Multi-angle projection method and device, equipment and storage medium

Abstract

According to the multi-angle projection method and device, the equipment and the storage medium provided by the invention, the three-dimensional coordinates of the plurality of key points of the target object are determined according to the at least one image of the target object, and the three-dimensional coordinates of the plurality of key points of the target object are calculated according to the target projection angle and the three-dimensional coordinates of the key points for any target projection angle in the plurality of projection angles. And determining projection coordinates of each key point on a projection surface under the target projection angle, generating an image of the target object under the target projection angle according to each projection coordinate, extracting three-dimensional coordinates of the plurality of key points of the target object, and projecting the plurality of key points to obtain the image of the target object under the target projection angle. And finally, according to the projection coordinates of the plurality of key points, an image under the projection angle is generated, and compared with generation of the three-dimensional model of the target object and projection transformation of the three-dimensional model, the method has the advantages that coordinate points to be processed are greatly reduced, so that the calculation amount is reduced.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional display technology, and in particular to a multi-angle projection method, device, equipment and storage medium. Background Art

[0002] With the continuous iterative development in the field of three-dimensional display technology, multi-angle projection of objects in images is a common problem in three-dimensional display. For example, when an image at a first angle is input, images at other angles can be output.

[0003] In the prior art, three-dimensional modeling of an object is usually performed first, and then the three-dimensional model is projected and rendered at multiple angles. When the three-dimensional model is projected and rendered at multiple angles, the three-dimensional model can be projected according to the projection matrix first, and then the projection result is rendered. Since there are many points in the model that need to be processed and there are many parameters involved in the projection matrix, the amount of calculation is large when performing projection and subsequent rendering. Therefore, the projection and rendering methods are usually deployed on the GPU.

[0004] In the process of multi-angle projection and rendering, how to reduce the amount of calculation without reducing the projection transformation effect is an urgent problem to be solved. Summary of the invention

[0005] The present invention provides a multi-angle projection method, device, equipment and storage medium, which are used to solve the problem of large calculation amount without reducing the projection transformation effect.

[0006] In a first aspect, the present invention provides a multi-angle projection method, the method comprising:

[0007] Determining three-dimensional coordinates of a plurality of key points of the target object based on at least one image of the target object;

[0008] For any target projection angle among the multiple projection angles, determine the projection coordinates of each key point on the projection surface at the target projection angle according to the target projection angle and the three-dimensional coordinates of each key point;

[0009] An image of the target object at the target projection angle is generated according to each of the projection coordinates.

[0010] Optionally, determining the three-dimensional coordinates of a plurality of key points of the target object according to at least one image of the target object includes:

[0011] The at least one image is input into a lightweight neural network model, and a plurality of three-dimensional coordinates output by the lightweight neural network model are determined as the three-dimensional coordinates of a plurality of key points of the target object.

[0012] Optionally, determining the projection coordinates of each key point on the projection surface at the target projection angle according to the target projection angle and the three-dimensional coordinates of each key point includes:

[0013] Determine the three-dimensional coordinates of the camera;

[0014] For any target key point among the multiple key points, determining a projection light according to the three-dimensional coordinates of the camera and the three-dimensional coordinates of the target key point;

[0015] The intersection point of the projection light and the projection surface is determined, and the projection coordinates of the target key point on the projection surface are determined according to the intersection point.

[0016] Optionally, the target key point coordinates and the camera coordinates are both coordinates in a world coordinate system; and determining the projection coordinates of the target key point on the projection surface according to the intersection point includes:

[0017] The shape parameters of the projection surface are acquired, and the coordinates of the intersection point in the world coordinate system are converted into projection coordinates in the projection surface coordinate system according to the shape parameters of the projection surface.

[0018] Optionally, generating an image of the target object at the target projection angle according to each of the projection coordinates includes:

[0019] Constructing a plurality of Delaunay triangles according to the projection coordinates;

[0020] Generate and store corresponding interpolation coefficients according to basic information corresponding to the at least one image; the basic information includes: the Delaunay triangle to which each pixel point belongs, the vertex coordinates of the Delaunay triangle to which each pixel point belongs, and the texture coordinate values ​​corresponding to the vertices of the Delaunay triangle;

[0021] For any target image to be processed of the target object, determine the texture coordinate value of each pixel point in the projected image according to the interpolation coefficients stored in the storage unit;

[0022] The pixel value of each pixel point is determined according to the texture coordinate value of each pixel point and the texture map corresponding to the target image, so as to generate an image of the target object in the target image at the target projection angle.

[0023] Optionally, the interpolation coefficient includes the basic information; and determining the texture coordinate value of each pixel point in the projected image according to the interpolation coefficient stored in the storage unit includes:

[0024] For any target pixel in the projected image, determine the weight information of the target pixel according to the position of the target pixel and the vertex coordinates of the Delaunay triangle to which the target pixel belongs in the interpolation coefficient;

[0025] The texture coordinate value of the target pixel point is determined according to the weight information and the texture coordinate values ​​corresponding to the vertices of the Delaunay triangle to which the target pixel point belongs.

[0026] Optionally, the interpolation coefficients include: a first array and a second array corresponding to each Delaunay triangle, and texture coordinate values ​​corresponding to each vertex of each Delaunay triangle, and the Delaunay triangle to which each pixel point belongs; determining the texture coordinate value of each pixel point in the projected image according to the interpolation coefficients stored in the storage unit includes:

[0027] For any target pixel point in the projected image, a first weight is determined according to the first array and the target pixel point coordinates, a second weight is determined according to the second array and the target pixel point coordinates, and a third weight is determined according to the first weight and the second weight; the first array and the second array are determined according to the vertex coordinates of the Delaunay triangle to which each pixel point belongs; the first weight and the second weight are functions of the target pixel point coordinates, respectively, and the first array and the second array are multiple constants in the corresponding functions;

[0028] The texture coordinate value of the target pixel point is determined according to the first weight, the second weight, the third weight, and the texture coordinate values ​​corresponding to each vertex of the Delaunay triangle to which the target pixel point belongs.

[0029] Optionally, constructing a plurality of Delaunay triangles according to the projection coordinates includes:

[0030] Determine remaining storage space information of the storage unit, and determine an upper limit on the number of constructed Delaunay triangles according to the remaining storage space information;

[0031] The Delaunay triangle is constructed according to the upper limit of quantity and each of the projection coordinates.

[0032] In a second aspect, the present invention provides a multi-angle projection device, the device comprising:

[0033] A first determination module, configured to determine three-dimensional coordinates of a plurality of key points of the target object based on at least one image of the target object;

[0034] A second determination module is used to determine, for any one target projection angle among the multiple projection angles, the projection coordinates of each key point on the projection surface at the target projection angle according to the target projection angle and the three-dimensional coordinates of each key point;

[0035] A generating module is used to generate an image of the target object at the target projection angle according to each of the projection coordinates.

[0036] In a third aspect, the present invention provides a multi-angle projection system, comprising: a computing unit and a storage unit;

[0037] The computing unit is used to execute the method according to any one of the first aspects;

[0038] The storage unit is used to store the intermediate results calculated by the calculation unit.

[0039] In a fourth aspect, the present invention provides an electronic device, comprising: at least one processor and a memory;

[0040] The memory stores computer-executable instructions;

[0041] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method as described in any one of the first aspects.

[0042] In a fifth aspect, the present invention provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, a method as described in any one of the first aspects is implemented.

[0043] In a sixth aspect, the present invention provides a computer program product, comprising a computer program, which implements any method described in the first aspect when executed by a processor.

[0044] The present invention provides a multi-angle projection method, device, equipment and storage medium. The method determines the three-dimensional coordinates of multiple key points of the target object based on at least one image of the target object. For any target projection angle among multiple projection angles, the projection coordinates of each key point on the projection surface at the target projection angle are determined based on the target projection angle and the three-dimensional coordinates of each key point. The image of the target object at the target projection angle is generated based on each projection coordinate. The method extracts the three-dimensional coordinates of multiple key points of the target object, projects the multiple key points, and finally generates the image at the projection angle based on the projection coordinates of the multiple key points. Compared with generating a three-dimensional model of the target object and performing a projection transformation on the three-dimensional model, the method greatly reduces the coordinate points to be processed, thereby reducing the amount of calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0046] Figure 1 An application scenario diagram of a multi-angle projection method provided by an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of a flow chart of a multi-angle projection method provided by an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of a flow chart of another multi-angle projection method provided by an embodiment of the present invention;

[0049] Figure 4 A schematic diagram of three-dimensional coordinates of multiple key points in a face provided by an embodiment of the present invention;

[0050] Figure 5 A schematic structural diagram of a multi-angle projection device provided by an embodiment of the present invention;

[0051] Figure 6 A schematic diagram of the architecture of a multi-angle projection system provided by an embodiment of the present invention;

[0052] Figure 7 A schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention.

[0053] The above drawings have shown clear embodiments of the present invention, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present invention in any way, but to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0054] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention.

[0055] Multi-angle projection refers to outputting an image at a projection angle according to the projection angle input by the user. For example, in some VR fields, multi-angle projection allows users to view and analyze target objects from different angles to enhance sensory effects. Figure 1 An application scenario diagram of a multi-angle projection method provided by an embodiment of the present invention, such as Figure 1 As shown, the projection surface can be a plane or a curved surface. The input image contains a target object. When the target object is viewed from three different projection angles, different projection images can be obtained at different positions of the projection surface.

[0056] In the prior art, when performing multi-angle projection, the target object in the image is usually modeled first to obtain a three-dimensional model of the target object, and then the projection image of the three-dimensional model is determined. When establishing a three-dimensional model, a fine modeling method (such as 3DMM, 3D Morphable Model) is usually used, and the fine modeling method has more model parameters and a large amount of calculation. When projecting the three-dimensional model, the projection image is obtained through the projection matrix. When the projection image is calculated using the projection matrix method, each point on a surface needs to be processed, and the amount of calculation is large. Moreover, when the projection surface is a curved surface, the amount of calculation will be even greater.

[0057] Based on the above problems, this application considers obtaining only the three-dimensional coordinates of multiple key points in the target object during three-dimensional modeling. When calculating the projection coordinates of the target object on the projection surface, it is only necessary to calculate the projection coordinates of multiple key points. Finally, all the pixel information of the entire image is generated based on the projection coordinates of the multiple key points to obtain the image of the target object at the target projection angle. In the three-dimensional modeling and projection transformation stages, the amount of calculation is greatly reduced by calculating multiple key points. Finally, by generating all the pixel information of the entire image based on the projection coordinates of multiple key points, it is possible to generate a projection image at any target projection angle without losing the accuracy of image transformation.

[0058] Figure 2 The present invention provides a flowchart of a multi-angle projection method according to an embodiment of the present invention. The method is applied to a multi-angle projection device, and the multi-angle projection device is arranged on an electronic device. The method includes steps S201 to S203:

[0059] Step S201: Determine the three-dimensional coordinates of multiple key points of the target object according to at least one image of the target object.

[0060] The electronic device can capture images or receive images transmitted by other devices, and there is no limit on the number of images captured or received. When it is necessary to display an image at a target projection angle to a user, the three-dimensional coordinates of multiple key points of the target object can be determined first.

[0061] Determining the three-dimensional coordinates of multiple key points of the target object refers to determining the three-dimensional coordinates of the multiple key points in the three-dimensional space according to the plane coordinates of the multiple key points in the two-dimensional image space. Here, the three-dimensional modeling of the target object is not performed, but the three-dimensional coordinates of the multiple key points of the target object are determined.

[0062] For example, when the target object is a human face, the three-dimensional coordinates of multiple key points in the face may be determined. For example, when 68 key points are used to represent the face, 6 key points may be used to represent the eyes, and 5 key points may be used to represent the eyebrows. In addition, other numbers of key points may be used to represent the face, such as 201 key points. The conventional method of establishing a three-dimensional model can determine the three-dimensional coordinates of each point in the face. Different from the conventional method of establishing a three-dimensional model, the number of three-dimensional coordinates obtained here is greatly reduced.

[0063] When determining the three-dimensional coordinates of multiple key points of the target object, the three-dimensional coordinates can be determined based on an image containing the target object, or based on multiple images containing the target object. Exemplarily, when the input is multiple images containing the target object, the three-dimensional coordinates of multiple key points of the target object can be determined based on the multiple images; or, when the input is a video stream containing the target object, one or more images containing the target object can be obtained from the video stream, and the three-dimensional coordinates of multiple key points of the target object can be determined.

[0064] Step S202: for any target projection angle among the multiple projection angles, determine the projection coordinates of each key point on the projection surface at the target projection angle according to the target projection angle and the three-dimensional coordinates of each key point.

[0065] After obtaining the three-dimensional coordinates of each key point in the target object, each key point can be mapped and transformed according to the needs of the projection transformation, that is, the projection coordinates corresponding to each key point on the projection surface are determined. When determining the projection coordinates, the target projection angle needs to be determined first.

[0066] like Figure 1 As shown, when the target projection angle is different, the projection coordinates of the target object on the projection surface are also different. For example, when the target projection angle is projection angle 1, the projection coordinates of each key point determined are located on the right side of the projection surface, and when the target projection angle is projection angle 2, the projection coordinates of each key point determined are located on the left side of the projection surface.

[0067] The target projection angle may be determined according to user needs. For example, when the user wishes to view the target object from projection angle 1, projection angle 1 is determined as the target projection angle.

[0068] After the target projection angle is determined, the projection coordinates of the three-dimensional coordinates of each key point on the projection surface under the target projection angle can be determined.

[0069] Since the three-dimensional coordinates of each key point are projected, the amount of calculation can be greatly reduced compared to processing the three-dimensional model through the projection matrix.

[0070] Step S203: Generate an image of the target object at the target projection angle according to each of the projection coordinates.

[0071] After obtaining the projection coordinates of each key point on the projection surface, it is also necessary to restore the pixel information of each point in the target object according to these projection coordinates to obtain the image of the target object at the target projection angle.

[0072] Optionally, rasterization can be used to achieve this. Rasterization is a basic method of graphics rendering. We use rasterization to obtain pixel information of each point in the target object.

[0073] The present invention provides a multi-angle projection method, which determines the three-dimensional coordinates of multiple key points of the target object according to at least one image of the target object, and for any target projection angle among multiple projection angles, determines the projection coordinates of each key point on the projection surface at the target projection angle according to the target projection angle and the three-dimensional coordinates of each key point, and generates the image of the target object at the target projection angle according to each projection coordinate, extracts the three-dimensional coordinates of multiple key points of the target object, projects the multiple key points, and finally generates the image at the projection angle according to the projection coordinates of the multiple key points. Compared with generating a three-dimensional model of the target object and performing projection transformation on the three-dimensional model, this method greatly reduces the coordinate points to be processed, thereby reducing the amount of calculation.

[0074] Figure 3 A schematic flow chart of another multi-angle projection method provided by an embodiment of the present invention, the method comprising steps S301 to S303:

[0075] Step S301: input the at least one image into a lightweight neural network model, and determine a plurality of three-dimensional coordinates output by the lightweight neural network model as the three-dimensional coordinates of a plurality of key points of the target object.

[0076] When determining the three-dimensional coordinates of multiple key points of the target object, a deep learning method can be used to achieve this. Specifically, the image can be input into a lightweight neural model, and the output of the lightweight neural network model is the three-dimensional coordinates of multiple key points corresponding to the target object. The lightweight neural network model here is a pre-trained network model that can determine the three-dimensional coordinates of multiple key points of the target object in the image.

[0077] Optionally, based on the type of the target object, an image containing an object of the same type as the target object can be pre-input into the lightweight neural network model, and the three-dimensional coordinates of multiple key points corresponding to the object can be used as the output of the lightweight neural network model to train the lightweight neural network model.

[0078] Exemplarily, when the target object is a human face, multiple images containing human faces can be pre-input into the lightweight neural network model, and the three-dimensional coordinates of multiple key points of each face can be obtained through a three-dimensional scanning device. The obtained three-dimensional coordinates of the multiple key points of each face are used as the output of the neural network model, thereby obtaining a trained lightweight neural network model.

[0079] Figure 4 A schematic diagram of three-dimensional coordinates of multiple key points in a face provided by an embodiment of the present invention, such as Figure 4 As shown, for the input face image, the three-dimensional coordinates of multiple key points are obtained.

[0080] When determining the three-dimensional coordinates of multiple key points of the target object, a single image containing the target object can be input into the lightweight neural network to improve the efficiency of determining the three-dimensional coordinates of the multiple key points of the target object.

[0081] The pre-trained lightweight neural network model can quickly map the two-dimensional key points on the image to the three-dimensional space.

[0082] Step S302: Determine the three-dimensional coordinates of the camera.

[0083] The camera indicates the location of the user, and the camera position indicates the location of the target object viewed by the user. After the target projection angle is determined, the camera position can be determined according to the positional relationship between the user and the electronic device. Optionally, the three-dimensional coordinates of the camera determined here can be three-dimensional coordinates in the world coordinate system.

[0084] Step S303: for any target key point among the multiple key points, determine the projection light according to the three-dimensional coordinates of the camera and the three-dimensional coordinates of the target key point.

[0085] For any target key point, the projection light can be determined according to the three-dimensional coordinates of the camera and the three-dimensional coordinates of the target key point, that is, the three-dimensional coordinates of the camera and the three-dimensional coordinates of the target key point are connected to obtain the projection light.

[0086] Here, the method can be described as determining the projection coordinates of each key point on the projection surface by ray tracing. Since the three-dimensional coordinates of each key point of the target object are obtained, the point-based method can be used to obtain the corresponding coordinates on the projection surface. Ray tracing refers to tracing the light from the target projection angle to determine the coordinates of the key point on the projection surface under the projection of the light.

[0087] Before determining the projection light, the target key points can be converted into coordinates in the world coordinate system. The coordinates of the target key points output by the lightweight neural network model may be based on a certain three-dimensional coordinate point in the target object as the coordinate origin. Since the three-dimensional coordinates of the camera are in the world coordinate system, each target key point also needs to be converted into coordinates in the world coordinate system.

[0088] Step S304: determine the intersection point of the projection light and the projection surface, and determine the projection coordinates of the target key point on the projection surface according to the intersection point.

[0089] After the projection light is determined, the intersection of the projection light and the projection surface can be calculated according to the intersection of the light, and the intersection coordinates here are also coordinates in the world coordinate system. The projection surface is the position of the screen receiving the projection in the electronic device, and the projection surface can be a plane or a curved surface.

[0090] By traversing each key point, the projection coordinates of each key point on the projection surface at the target projection angle can be calculated.

[0091] Since the intersection point of the projection light and the projection surface is calculated by the light intersection method, the projection coordinates of the key points on the projection surface can be quickly calculated regardless of whether the projection surface is a plane or a curved surface.

[0092] By calculating the projection light, the projection coordinates of the key points on the projection surface can be quickly calculated according to the intersection of the projection light and the projection surface. Compared with the existing projection matrix method, this method is also applicable to the scene where the projection surface is a curved surface, and will not increase the amount of calculation due to the curved projection surface.

[0093] Optionally, determining the projection coordinates of the target key point on the projection surface according to the intersection point includes:

[0094] The shape parameters of the projection surface are acquired, and the coordinates of the intersection point in the world coordinate system are converted into projection coordinates in the projection surface coordinate system according to the shape parameters of the projection surface.

[0095] Optionally, the target key point coordinates and the camera coordinates are both coordinates in the world coordinate system. In step S304, the determined projection coordinates are coordinates in the world coordinate system. Since the final acquired projection image needs to be displayed on the projection surface, it is necessary to determine the position of each projection coordinate relative to the projection surface, that is, to convert the projection coordinates from the world coordinate system to the projection surface's own coordinate system.

[0096] The projection surface can be a plane or a curved surface. When the projection surface is a plane, the projection coordinates in the world coordinate system can be directly converted to the projection coordinates in the projection surface coordinate system. When the projection surface is a curved surface, the coordinates of the projection coordinates in the projection surface coordinate system need to be determined according to the shape parameters of the projection surface, that is, the curved surface needs to be unfolded into a plane to determine the coordinates of the projection coordinates on the unfolded plane.

[0097] Optionally, when the projection surface is a curved surface, a curve integral can be first performed according to the shape parameters of the projection surface to obtain the line length of the curve. For example, when the curved surface is curved in the horizontal direction and not curved in the vertical direction, the line length corresponding to the curved surface in the horizontal direction can be calculated by curve integral.

[0098] For the coordinates of the intersection point in the world coordinate system, the coordinates of the coordinates relative to the surface can be calculated, and the coordinates of the coordinates on the plane after the curve is unfolded can be calculated through curve integration of the converted coordinates.

[0099] When the projection surface is a curved surface, it is unfolded into a plane and the projection coordinates relative to the projection plane are calculated, so as to facilitate the subsequent calculation of the pixel values ​​of other points on the target object on the projection plane and to facilitate the final display of the projected image.

[0100] Rendering through rasterization means that when an image is projected at multiple angles, a triangle can be constructed based on the projection coordinates of each key point corresponding to the image. For each pixel in the projected image, the texture coordinate value corresponding to the pixel can be calculated based on the vertex coordinates and texture coordinate values ​​corresponding to the Delaunay triangle to which the pixel belongs. Because the texture coordinate value of the pixel is related to the texture coordinate values ​​of each vertex of the Delaunay triangle, after calculating the texture coordinate value corresponding to the pixel, RGB information can be extracted from the input image based on the texture coordinate value to obtain the projected image.

[0101] However, in practice, we may project multiple consecutively input images that are related. The existence of a correlation means that the local part of the target object changes position, that is, the target object performs a relatively small movement. For example, when it is necessary to perform multi-angle projection on multiple images of a person speaking, there is no need to calculate the three-dimensional coordinates of multiple key points of each image, and the projection coordinates of these three-dimensional coordinates on the projection surface. For any target image, the image of the target image at the target projection angle can be directly calculated based on the multiple projection coordinates corresponding to one of the multiple images.

[0102] When it is necessary to calculate the image of at least one image at a target projection angle, in order to reduce the amount of calculation, interpolation coefficients can be stored, and the pixel value corresponding to each pixel point in the projected image corresponding to the target image can be quickly calculated based on the interpolation coefficients.

[0103] Step S305: construct multiple Delaunay triangles according to the projection coordinates.

[0104] In order to render the pixel information of each pixel in the projected image by using the rasterization method, multiple Delaunay triangles can be constructed according to the projection coordinates of each key point. The Bowyer-Watson algorithm can be used to construct the Delaunay triangle.

[0105] Optionally, constructing a plurality of Delaunay triangles according to the projection coordinates includes:

[0106] Determine the remaining storage space information of the storage unit, and determine the upper limit of the number of constructed Delaunay triangles according to the remaining storage space information; and construct the Delaunay triangle according to the upper limit and each of the projection coordinates.

[0107] Since the stored interpolation coefficients are related to the number of constructed Delaunay triangles, the more the number of constructed Delaunay triangles, the more interpolation coefficients need to be stored; on the contrary, the fewer the number of constructed Delaunay triangles, the fewer the interpolation coefficients need to be stored. The interpolation coefficients need to be stored in the storage unit, so the number of stored interpolation coefficients depends on the remaining storage space of the storage unit. When the remaining storage space is large, more Delaunay triangles can be constructed, and when the remaining storage space is small, fewer Delaunay triangles can be constructed.

[0108] Optionally, the remaining storage space information of the storage unit can be determined first, and an upper limit on the number of Delaunay triangles that can be constructed can be determined based on the remaining storage space information, so that when Delaunay triangles are constructed according to each projection coordinate, the number of constructed Delaunay triangles needs to be less than the upper limit.

[0109] By determining the upper limit of the number of Delaunay triangles, the interpolation coefficients corresponding to each Delaunay triangle can be successfully stored after the Delaunay triangle is constructed, avoiding the need to reconstruct the Delaunay triangle again when the interpolation coefficients of each Delaunay triangle cannot be stored smoothly, thereby improving processing efficiency.

[0110] Step S306: Generate and store corresponding interpolation coefficients according to the basic information corresponding to the at least one image.

[0111] The basic information includes: the Delaunay triangle to which each pixel point belongs, the vertex coordinates of the Delaunay triangle to which each pixel point belongs, and the texture coordinate values ​​corresponding to the vertices of the Delaunay triangle.

[0112] When storing the interpolation coefficient, the interpolation coefficient may be basic information corresponding to at least one image, and illustratively, may be basic information corresponding to the first image of the at least one input image.

[0113] Since the interpolation coefficient is related to the constructed Delaunay triangle, and the Delaunay triangle is related to the projection coordinates, and the projection coordinates are related to the target projection angle, in order to obtain the projection images under multiple projection angles, it is necessary to determine the projection coordinates under multiple projection angles, and calculate the interpolation coefficients based on the projection coordinates for storage. Exemplarily, when there are three projection angles, the interpolation coefficients corresponding to the three projection angles can be determined and stored.

[0114] For each pixel in the projected image, the texture coordinate value corresponding to the pixel can be calculated according to the vertex coordinates and texture coordinate values ​​corresponding to the Delaunay triangle to which the pixel belongs. In this process, the Delaunay triangle to which the pixel belongs can be determined. If the information can be stored after calculating the Delaunay triangle to which each pixel belongs, then when calculating the projected image for other input images later, it is not necessary to calculate the Delaunay triangle to which each pixel belongs again, which greatly reduces the amount of calculation, so that the calculation process can be implemented on the CPU (Central Processing Unit) without relying on the GPU (Graphics Processing Unit) to implement it.

[0115] For a pixel point, the texture coordinate value corresponding to the pixel point is related to the texture coordinate value of the Delaunay triangle to which the pixel point belongs. The Delaunay triangle has three vertices, and there is a corresponding weight for each vertex. The weight represents the degree of influence of the texture coordinate value of the vertex on the texture coordinate value of the pixel point. Therefore, the texture coordinate value of the pixel point can be determined based on the weight and the texture coordinate values ​​of the three vertices. The weight here is related to the position of the pixel point and each vertex. Therefore, it is also necessary to store the texture coordinate values ​​corresponding to the vertices of each Delaunay triangle and the information related to the vertex coordinates of each Delaunay triangle.

[0116] Step S307 : for any target image to be processed of the target object, determine the texture coordinate value of each pixel point in the projected image according to the interpolation coefficients stored in the storage unit.

[0117] Optionally, the target object may be any image of at least one input graphic, for example, the first image. In this case, since the interpolation coefficients have been stored in advance, the texture coordinate values ​​of each pixel point in the projected image corresponding to the first image may be determined based on the interpolation coefficients.

[0118] Optionally, the target object can also be any image after the first image. When processing the first image, the interpolation coefficient is determined according to the projection coordinates corresponding to the first image and stored. When processing any image after the first image, the texture coordinate value of each pixel point in the target image can be calculated according to the interpolation coefficient.

[0119] Step S308: determining the pixel value of each pixel point according to the texture coordinate value of each pixel point and the texture map corresponding to the target image, so as to generate an image of the target object in the target image at the target projection angle.

[0120] For the target image, after determining the texture coordinate value of each pixel in the projected image, the pixel value of each pixel can be obtained according to the target image. The pixel value of the pixel can be obtained by sampling the texture map corresponding to the target image according to the texture coordinate value of any pixel. The pixel value can be the RGB value of the pixel. The pixel can be displayed according to the pixel value, thereby obtaining the image of the target image at the target projection angle.

[0121] In addition, the texture coordinate values ​​of each pixel can be directly stored, but for an image, the number of pixels is large. If the texture coordinate value of each pixel is stored, a large storage space is required. Therefore, in order to balance the amount of calculation and the amount of storage occupied, the interpolation coefficient is stored here, which does not require a high storage space. That is, the most performant image transformation effect is achieved by balancing the relationship between the amount of calculation and the amount of storage in the image rendering process.

[0122] By pre-storing the interpolation coefficients, the projection image corresponding to the input target image can be quickly determined based on the interpolation coefficients, without having to perform steps such as determining the three-dimensional coordinates of multiple key points of the target image and determining the projection coordinates corresponding to the three-dimensional coordinates. In addition, by storing the Delaunay triangle to which each pixel point belongs, when processing the target image, there is no need to recalculate the Delaunay triangle to which each pixel point belongs, thereby reducing the amount of calculation.

[0123] When storing the interpolation coefficients in the above step S306, the interpolation coefficients can be in two forms. One is to directly store the basic information, and the other is to calculate the first array and the second array according to the vertex coordinates of each Delaunay triangle in the basic information. The weight information of the target pixel point can be directly calculated according to the first array and the second array and the coordinates of each pixel point.

[0124] The following describes in detail how to calculate the weight information of the target pixel point according to the above two interpolation coefficients, and then calculate the texture coordinate value of the target pixel point.

[0125] Optionally, the interpolation coefficient includes the basic information; and determining the texture coordinate value of each pixel point in the projected image according to the interpolation coefficient stored in the storage unit includes:

[0126] For any target pixel in the projected image, determine the weight information of the target pixel according to the position of the target pixel and the vertex coordinates of the Delaunay triangle to which the target pixel belongs in the interpolation coefficient;

[0127] The texture coordinate value of the target pixel point is determined according to the weight information and the texture coordinate values ​​corresponding to the vertices of the Delaunay triangle to which the target pixel point belongs.

[0128] When the stored interpolation coefficients are the basic information, the pixel points in the projected image can be traversed, and for the target pixel point (the target pixel point is a point in a certain Delaunay triangle), the target Delaunay triangle is obtained according to the stored basic information. The target Delaunay triangle is the Delaunay triangle to which the target pixel point belongs, and the vertex coordinates of the target Delaunay triangle are obtained, and the weight is determined according to the coordinates of the target pixel point and the three vertex coordinates of the target Delaunay triangle. Among them, the vertices of the Delaunay triangle include vertex A, vertex B and vertex C, and the weight can be three values, namely the first weight, the second weight and the third weight.

[0129] Exemplarily, the weights can be calculated according to the following interpolation calculation formula of the Delaunay triangle, where α is the first weight, β is the second weight, and γ is the third weight. a ,y a ), (x b ,y b ), (x c ,y c ) are the coordinates of the three vertices of the Delaunay triangle, and (x, y) are the coordinates of the target pixel.

[0130]

[0131]

[0132] γ=1-α-β Formula 3

[0133] When calculating the texture coordinate value of the target pixel, the first weight is multiplied by the texture coordinate value of vertex A, the second weight is multiplied by the texture coordinate value of vertex B, and the third weight is multiplied by the texture coordinate value of vertex C to obtain three multiplication results, and the texture coordinate value of the pixel is obtained by adding the three multiplication results. Among them, the texture coordinate value of the pixel contains two values, and the above formula can be used to calculate the two texture coordinate values ​​respectively.

[0134] By storing the basic information as the interpolation coefficient, the basic information can be directly stored in the storage unit without deforming it, which makes the operation simpler.

[0135] Optionally, the interpolation coefficients include: a first array and a second array corresponding to each Delaunay triangle, and texture coordinate values ​​corresponding to each vertex of each Delaunay triangle, and the Delaunay triangle to which each pixel point belongs; determining the texture coordinate value of each pixel point in the projected image according to the interpolation coefficients stored in the storage unit includes:

[0136] For any target pixel point in the projected image, a first weight is determined according to the first array and the target pixel point coordinates, a second weight is determined according to the second array and the target pixel point coordinates, and a third weight is determined according to the first weight and the second weight; the first array and the second array are determined according to the vertex coordinates of the Delaunay triangle to which each pixel point belongs; the first weight and the second weight are functions of the target pixel point coordinates, respectively, and the first array and the second array are multiple constants in the corresponding functions;

[0137] The texture coordinate value of the target pixel point is determined according to the first weight, the second weight, the third weight, and the texture coordinate values ​​corresponding to each vertex of the Delaunay triangle to which the target pixel point belongs.

[0138] When storing interpolation coefficients, the basic information obtained can also be transformed to obtain the first array and the second array, and the first array and the second array can be stored, which can further simplify the process of calculating weights, thereby further reducing the amount of calculation when calculating the first weights and the second weights in real time.

[0139] Among them, the first array and the second array are only related to the three vertex coordinates of the Delaunay triangle, and the first weight and the second weight are respectively two-variable linear functions about the target pixel point, and the first array and the second array are respectively constants in the two-variable linear function. Exemplarily, the above formula 1 can be deformed to obtain a two-variable linear function of α about x and y, and the constant in the two-variable linear function is the first array. By storing the first array and the second array, after determining the value of the target pixel point (x, y), the first weight and the second weight can be obtained by simple multiplication and addition operations.

[0140] Compared with the above formula 1 and formula 2, by storing the first array and the second array, the process of calculating the first weight and the second weight is simplified.

[0141] After determining the first weight and the second weight, the third weight may be calculated, and then the texture coordinate value of the target pixel may be determined according to the three weights and the texture coordinate values ​​corresponding to the vertices of the Delaunay triangle to which the target pixel belongs.

[0142] By storing the first array and the second array, the weight calculation process can be further simplified, saving the computing power burden of the real-time calculation process.

[0143] Figure 5 A schematic diagram of the structure of a multi-angle projection device provided by an embodiment of the present invention, the device 50 comprises:

[0144] A first determination module 501 is used to determine the three-dimensional coordinates of multiple key points of the target object according to at least one image of the target object;

[0145] A second determination module 502 is used to determine, for any one target projection angle among the multiple projection angles, the projection coordinates of each key point on the projection surface at the target projection angle according to the target projection angle and the three-dimensional coordinates of each key point;

[0146] The generating module 503 is used to generate an image of the target object at the target projection angle according to each of the projection coordinates.

[0147] Optionally, the first determining module 501 is specifically configured to:

[0148] The at least one image is input into a lightweight neural network model, and a plurality of three-dimensional coordinates output by the lightweight neural network model are determined as the three-dimensional coordinates of a plurality of key points of the target object.

[0149] Optionally, the second determining module 502 is specifically configured to:

[0150] Determine the three-dimensional coordinates of the camera;

[0151] For any target key point among the multiple key points, determining a projection light according to the three-dimensional coordinates of the camera and the three-dimensional coordinates of the target key point;

[0152] The intersection point of the projection light and the projection surface is determined, and the projection coordinates of the target key point on the projection surface are determined according to the intersection point.

[0153] Optionally, the target key point coordinates and the camera coordinates are both coordinates in a world coordinate system; when the second determination module 502 determines the projection coordinates of the target key point on the projection surface according to the intersection point, it is specifically used to:

[0154] The shape parameters of the projection surface are acquired, and the coordinates of the intersection point in the world coordinate system are converted into projection coordinates in the projection surface coordinate system according to the shape parameters of the projection surface.

[0155] Optionally, the generating module 503 is specifically used for:

[0156] Constructing a plurality of Delaunay triangles according to the projection coordinates;

[0157] Generate and store corresponding interpolation coefficients according to basic information corresponding to the at least one image; the basic information includes: the Delaunay triangle to which each pixel point belongs, the vertex coordinates of the Delaunay triangle to which each pixel point belongs, and the texture coordinate values ​​corresponding to the vertices of the Delaunay triangle;

[0158] For any target image to be processed of the target object, determine the texture coordinate value of each pixel point in the projected image according to the interpolation coefficients stored in the storage unit;

[0159] The pixel value of each pixel point is determined according to the texture coordinate value of each pixel point and the texture map corresponding to the target image, so as to generate an image of the target object in the target image at the target projection angle.

[0160] Optionally, the interpolation coefficient includes the basic information; when the generation module 503 determines the texture coordinate value of each pixel point in the projected image according to the interpolation coefficient stored in the storage unit, it is specifically used to:

[0161] For any target pixel in the projected image, determine the weight information of the target pixel according to the position of the target pixel and the vertex coordinates of the Delaunay triangle to which the target pixel belongs in the interpolation coefficient;

[0162] The texture coordinate value of the target pixel point is determined according to the weight information and the texture coordinate values ​​corresponding to the vertices of the Delaunay triangle to which the target pixel point belongs.

[0163] Optionally, the interpolation coefficients include: a first array and a second array respectively corresponding to each Delaunay triangle, and texture coordinate values ​​corresponding to each vertex of each Delaunay triangle, and the Delaunay triangle to which each pixel point belongs; when the generation module 503 determines the texture coordinate value of each pixel point in the projected image according to the interpolation coefficients stored in the storage unit, it is specifically used to:

[0164] For any target pixel point in the projected image, a first weight is determined according to the first array and the target pixel point coordinates, a second weight is determined according to the second array and the target pixel point coordinates, and a third weight is determined according to the first weight and the second weight; the first array and the second array are determined according to the vertex coordinates of the Delaunay triangle to which each pixel point belongs; the first weight and the second weight are functions of the target pixel point coordinates, respectively, and the first array and the second array are multiple constants in the corresponding functions;

[0165] The texture coordinate value of the target pixel point is determined according to the first weight, the second weight, the third weight, and the texture coordinate values ​​corresponding to each vertex of the Delaunay triangle to which the target pixel point belongs.

[0166] Optionally, when constructing a plurality of Delaunay triangles according to the projection coordinates, the generating module 503 is specifically configured to:

[0167] Determine remaining storage space information of the storage unit, and determine an upper limit on the number of constructed Delaunay triangles according to the remaining storage space information;

[0168] The Delaunay triangle is constructed according to the upper limit of quantity and each of the projection coordinates.

[0169] The multi-angle projection device provided by the embodiment of the present invention can realize the above-mentioned Figure 2 and Figure 3 The implementation principle and technical effect of the multi-angle projection method in the illustrated embodiment are similar and will not be described in detail here.

[0170] Figure 6 A schematic diagram of the architecture of a multi-angle projection system provided by an embodiment of the present invention is shown in FIG. Figure 6 As shown, the multi-angle projection system provided in this embodiment includes: a computing unit and a storage unit;

[0171] The calculation unit is used to execute the method in the above method embodiment; wherein the storage unit is used to store the intermediate results calculated by the calculation unit.

[0172] like Figure 6As shown, the multi-angle projection system includes a computing unit and a storage unit. The computing unit can be a CPU. For the input first image, the computing unit can be used to extract the three-dimensional coordinates of multiple key points through a lightweight neural network model, and then the camera position and the projection surface are used to perform ray tracing to obtain the projection coordinates of each key point under the target projection angle (corresponding to the camera position). After calculating the projection coordinates, the interpolation coefficients can be determined and stored in the storage unit. The interpolation coefficients are used to calculate the projection images corresponding to other images (such as the second image). Since the interpolation coefficients are stored, when calculating the projection image corresponding to the second image, the amount of calculation is small, and the CPU can be used for calculation.

[0173] The intermediate results calculated by the computing unit can be interpolation coefficients. By storing them, the amount of calculation can be reduced, and the requirements for computing resources can be reduced. The projection rendering of the target object in the image can be achieved by relying only on the CPU.

[0174] Figure 7 The hardware structure diagram of an electronic device provided by an embodiment of the present invention is shown in FIG. Figure 7 As shown, the electronic device provided by this embodiment includes: at least one processor 701 and a memory 702. The processor 701 and the memory 702 are connected via a bus 703.

[0175] In a specific implementation process, at least one processor 701 executes the computer execution instructions stored in the memory 702, so that at least one processor 701 executes the method in the above method embodiment.

[0176] The specific implementation process of the processor 701 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0177] In the above Figure 7 In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0178] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk storage.

[0179] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0180] An embodiment of the present invention further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method of the above method embodiment is implemented.

[0181] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method of the above method embodiment when executed by a processor.

[0182] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0183] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0184] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0185] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0186] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0187] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A multi-angle projection method, characterized in that: The method comprises: Determining three-dimensional coordinates of a plurality of key points of the target object based on at least one image of the target object; For any target projection angle among the multiple projection angles, determine the projection coordinates of each key point on the projection surface at the target projection angle according to the target projection angle and the three-dimensional coordinates of each key point; An image of the target object at the target projection angle is generated according to each of the projection coordinates.

2. The method according to claim 1, characterized in that: Determining three-dimensional coordinates of a plurality of key points of the target object according to at least one image of the target object includes: The at least one image is input into a lightweight neural network model, and a plurality of three-dimensional coordinates output by the lightweight neural network model are determined as the three-dimensional coordinates of a plurality of key points of the target object.

3. The method according to claim 1, characterized in that Determining the projection coordinates of each key point on the projection surface at the target projection angle according to the target projection angle and the three-dimensional coordinates of each key point includes: Determine the three-dimensional coordinates of the camera; For any target key point among the multiple key points, determining a projection light according to the three-dimensional coordinates of the camera and the three-dimensional coordinates of the target key point; The intersection point of the projection light and the projection surface is determined, and the projection coordinates of the target key point on the projection surface are determined according to the intersection point.

4. The method according to claim 3, characterized in that The target key point coordinates and the camera coordinates are both coordinates in the world coordinate system; Determining the projection coordinates of the target key point on the projection surface according to the intersection point includes: The shape parameters of the projection surface are acquired, and the coordinates of the intersection point in the world coordinate system are converted into projection coordinates in the projection surface coordinate system according to the shape parameters of the projection surface.

5. The method according to any one of claims 1 to 4, characterized in that: Generating an image of the target object at the target projection angle according to each of the projection coordinates includes: Constructing a plurality of Delaunay triangles according to the projection coordinates; Generate and store corresponding interpolation coefficients according to basic information corresponding to the at least one image; the basic information includes: the Delaunay triangle to which each pixel point belongs, the vertex coordinates of the Delaunay triangle to which each pixel point belongs, and the texture coordinate values ​​corresponding to the vertices of the Delaunay triangle; For any target image to be processed of the target object, determine the texture coordinate value of each pixel point in the projected image according to the interpolation coefficients stored in the storage unit; The pixel value of each pixel point is determined according to the texture coordinate value of each pixel point and the texture map corresponding to the target image, so as to generate an image of the target object in the target image at the target projection angle.

6. The method according to claim 5, characterized in that The interpolation coefficient includes the basic information; and determining the texture coordinate value of each pixel point in the projected image according to the interpolation coefficient stored in the storage unit includes: For any target pixel in the projected image, determine the weight information of the target pixel according to the position of the target pixel and the vertex coordinates of the Delaunay triangle to which the target pixel belongs in the interpolation coefficient; The texture coordinate value of the target pixel point is determined according to the weight information and the texture coordinate values ​​corresponding to the vertices of the Delaunay triangle to which the target pixel point belongs.

7. The method according to claim 5, characterized in that The interpolation coefficients include: the first array and the second array corresponding to each Delaunay triangle, as well as the texture coordinate values ​​corresponding to each vertex of each Delaunay triangle and the Delaunay triangle to which each pixel belongs; the texture coordinate value of each pixel in the projected image is determined according to the interpolation coefficients stored in the storage unit, including: For any target pixel point in the projected image, a first weight is determined according to the first array and the target pixel point coordinates, a second weight is determined according to the second array and the target pixel point coordinates, and a third weight is determined according to the first weight and the second weight; the first array and the second array are determined according to the vertex coordinates of the Delaunay triangle to which each pixel point belongs; the first weight and the second weight are functions of the target pixel point coordinates, respectively, and the first array and the second array are multiple constants in the corresponding functions; The texture coordinate value of the target pixel point is determined according to the first weight, the second weight, the third weight, and the texture coordinate values ​​corresponding to each vertex of the Delaunay triangle to which the target pixel point belongs.

8. The method according to claim 5, characterized in that Constructing a plurality of Delaunay triangles according to the projection coordinates, including: Determine remaining storage space information of the storage unit, and determine an upper limit on the number of constructed Delaunay triangles according to the remaining storage space information; The Delaunay triangle is constructed according to the upper limit of quantity and each of the projection coordinates.

9. A multi-angle projection device, characterized in that: The device comprises: A first determination module, configured to determine three-dimensional coordinates of a plurality of key points of the target object based on at least one image of the target object; A second determination module is used to determine, for any one target projection angle among the multiple projection angles, the projection coordinates of each key point on the projection surface at the target projection angle according to the target projection angle and the three-dimensional coordinates of each key point; A generating module is used to generate an image of the target object at the target projection angle according to each of the projection coordinates.

10. A multi-angle projection system, characterized in that: include: Computational unit and storage unit; The computing unit is used to execute the method according to any one of claims 1 to 8; The storage unit is used to store the intermediate results calculated by the calculation unit.

11. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 8 is implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images