Video processing method, device, equipment and storage medium

Abstract

The disclosed embodiments relate to a video processing method, device, equipment and storage medium. The method comprises: obtaining three-dimensional reconstruction data of a video image and a first texture image corresponding to the current moment, determining that the target pixel position on the first texture image corresponds to the target brightness at the current moment according to the mapping relationship between the pixel position and the initial brightness and the brightness change rate obtained in advance, adjusting the brightness of the target pixel position on the first texture image to the target brightness, obtaining a second texture image, and mapping the second texture image to the video image based on the three-dimensional reconstruction data to obtain a target video image. In this way, the same pixel position on the video image can present different textures and brightness at different moments, and different pixel positions on the video image can present different textures and brightness at the same moment, thereby achieving a flowing and flickering texture effect on the video image, thereby improving the interest of the video.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of video processing technology, and in particular to a video processing method, apparatus, device and storage medium. Background Art

[0002] The video applications provided by related technologies can shoot videos of real scenes and share them on video consumption platforms for consumers to watch. However, with the development of video applications, sharing of real scenes can no longer meet the growing needs of users. Therefore, how to add specific texture effects to video scenes to improve the fun of videos is a technical problem that needs to be solved urgently. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present disclosure provide a video processing method, apparatus, device and storage medium.

[0004] A first aspect of the present disclosure provides a video processing method, the method comprising:

[0005] Acquire three-dimensional reconstruction data of the video image and a first texture image corresponding to the current moment;

[0006] Determine, based on a pre-obtained mapping relationship between a pixel position and an initial brightness and a brightness change rate, that a target pixel position on the first texture image corresponds to a target brightness at a current moment;

[0007] Adjusting the brightness of the target pixel position on the first texture image to the target brightness to obtain a second texture image;

[0008] Based on the three-dimensional reconstruction data, the second texture image is mapped onto the video image to obtain a target video image.

[0009] A second aspect of the embodiments of the present disclosure provides a video processing device, the device comprising:

[0010] An acquisition module, used to acquire three-dimensional reconstruction data of a current video image and a first texture image corresponding to a current moment;

[0011] A determination module, used to determine that the target pixel position on the first texture image corresponds to the target brightness at the current moment according to the mapping relationship between the pixel position, the initial brightness and the brightness change rate obtained in advance;

[0012] A brightness adjustment module, used for adjusting the brightness of a target pixel position on the first texture image to a target brightness, so as to obtain a second texture image;

[0013] The texture mapping module is used to map the second texture image onto the video image based on the three-dimensional reconstruction data to obtain a target video image.

[0014] A third aspect of an embodiment of the present disclosure provides an electronic device, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the video processing method of the first aspect can be implemented.

[0015] A fourth aspect of an embodiment of the present disclosure provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the video processing method of the first aspect described above can be implemented.

[0016] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0017] The disclosed embodiment, after acquiring the three-dimensional reconstruction data of the video image and the first texture image corresponding to the current moment, can determine the target brightness corresponding to the target pixel position on the first texture image at the current moment based on the mapping relationship between the pixel position and the initial brightness and the brightness change rate obtained in advance, and adjust the brightness of the target pixel position on the first texture image to the target brightness to obtain the second texture image, and obtain the target video image by mapping the second texture image to the video image. The scheme provided by the disclosed embodiment can make the same pixel position on the video image present different textures and brightness at different moments, and make different pixel positions on the video image present different textures and brightness at the same moment, thereby realizing a flowing and flickering texture effect on the video image, and improving the interest of the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 is a flowchart of a video processing method provided by an embodiment of the present disclosure;

[0021] Figure 2a is a schematic diagram of a reference texture image provided by an embodiment of the present disclosure;

[0022] Figure 2bis a schematic diagram of a preset template provided in an embodiment of the present disclosure;

[0023] Figure 2c is a schematic diagram of a noise texture image provided by an embodiment of the present disclosure;

[0024] Figure 3 is a flowchart of another video processing method provided by an embodiment of the present disclosure;

[0025] Figure 4 is a structural schematic diagram of a video processing device provided by an embodiment of the present disclosure;

[0026] Figure 5 It is a structural schematic diagram of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0028] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0029] In related technologies, video applications can shoot videos of real scenes and share the videos of real scenes on video consumption platforms for consumers to watch.

[0030] With the development of video applications, the sharing of real scenes can no longer meet the growing needs of users. Users hope to add texture effects with specific effects to video scenes to increase the fun of the video, thereby meeting the growing needs of users.

[0031] In order to meet user needs, an embodiment of the present disclosure provides a video processing method, which can add a texture with a flowing flickering effect to a video image.

[0032] The video processing method provided by the embodiment of the present disclosure is described below in conjunction with exemplary embodiments.

[0033] Figure 1 A schematic flow chart of a video processing method provided by an embodiment of the present disclosure is shown.

[0034] In some embodiments of the present disclosure, Figure 1The video processing method shown can be executed by an electronic device, which may include a mobile phone, a tablet computer, a desktop computer, a laptop computer, a vehicle-mounted terminal, a wearable device, an all-in-one device, a smart home device, and other devices with video processing functions, and may also include a device simulated by a virtual machine or a simulator.

[0035] like Figure 1 As shown, the video processing method may include the following steps.

[0036] S110, acquiring three-dimensional reconstruction data of a video image and a first texture image corresponding to a current moment.

[0037] Specifically, before obtaining the three-dimensional reconstruction data and the first texture image of the video image, a shooting device can be used to obtain a scene to which a specific effect needs to be added from different angles to obtain a scene video of the scene, and the scene video is sent to an electronic device. After the electronic device obtains the scene video, the scene is reconstructed based on the video image in the scene video to obtain the three-dimensional reconstruction data of the video image in the scene video.

[0038] Optionally, the above-mentioned shooting equipment may include a three-dimensional scanner, a camera, a laser, a depth camera and other equipment, which is not limited here.

[0039] Optionally, the video images may include game video images, virtual reality images, audio and video images, etc., which are not limited here.

[0040] In the embodiment of the present disclosure, the first texture image may be an initial material image for adding a specific effect to a video image, and the first texture image defines the texture of each pixel position and the brightness of each pixel position. In the embodiment of the present disclosure, the first texture image corresponding to different moments may be different, and the first texture image in the embodiment of the present disclosure may be generated in real time or may be preset.

[0041] Optionally, the first texture image may be a text image, a landscape image, a building image or other types of images, which are not limited here.

[0042] S120: Determine, based on a pre-obtained mapping relationship between a pixel position and an initial brightness and a brightness change rate, that a target pixel position on the first texture image corresponds to a target brightness at a current moment.

[0043] The pixel position referred to in the embodiments of the present disclosure may be understood as a coordinate area or a coordinate point.

[0044] The embodiment of the present disclosure pre-sets the initial brightness and brightness change rate of each pixel position, wherein the initial brightness corresponding to different pixel positions may be the same or different, and the brightness change rates corresponding to different pixel positions may be the same or different.

[0045] The target pixel position referred to in the embodiment of the present disclosure can be understood as the pixel position whose brightness value needs to be changed. There can be one or more target pixel positions in the embodiment of the present disclosure.

[0046] The target brightness of the target pixel position may be understood as the brightness of the target pixel position, which changes from the initial brightness at the initial moment to the brightness value obtained at the current moment at a preset brightness change rate.

[0047] In an exemplary implementation of the embodiment of the present disclosure, S120 may include:

[0048] S1. Obtain the initial brightness and brightness change rate of the target pixel position according to the pre-obtained mapping relationship between the pixel position and the initial brightness and the brightness change rate.

[0049] S2. Determine the brightness change amount of the target pixel position corresponding to the current moment based on the time parameter at the current moment and the brightness change rate of the target pixel position.

[0050] S3. Determine the target brightness of the target pixel position corresponding to the current moment based on the brightness change of the target pixel position corresponding to the current moment and the initial brightness of the target pixel position.

[0051] In the embodiment of the present disclosure, the brightness of each pixel position may be configured to change periodically within a preset range (such as a range from 0 to 1, but not limited to a range from 0 to 1).

[0052] In one implementation of the embodiment of the present disclosure, the time parameter at the current moment can be understood as the time change amount at the current moment in the current change cycle.

[0053] Optionally, the target brightness can be calculated as:

[0054] Z=f xy (t×v+a)

[0055] Where Z is the target brightness of the target pixel position, x is the horizontal coordinate of the target pixel position, y is the vertical coordinate of the target pixel position, t is the time parameter at the current moment, v is the brightness change rate of the target pixel position, a is the initial brightness of the target pixel position, and f xy (*) is the calculation function of target brightness.

[0056] Wherein, a and v can be preset values.

[0057] Therefore, the initial brightness and brightness change rate of the target pixel position can be obtained based on the mapping relationship between the pre-obtained pixel position and the initial brightness and the brightness change rate, and the time parameter at the current moment and the brightness change rate of the target pixel position can be multiplied to obtain the brightness change amount of the target pixel position corresponding to the current moment. Further, based on the brightness change amount of the target pixel position corresponding to the current moment and the initial brightness of the target pixel position, the target brightness of the target pixel position corresponding to the current moment can be accurately determined.

[0058] S130: Adjust the brightness of the target pixel position on the first texture image to the target brightness to obtain a second texture image.

[0059] Specifically, after the electronic device determines that the target pixel position on the first texture image corresponds to the target brightness at the current moment, it can adjust the brightness of the target pixel position to the target brightness to change the brightness value of the pixel point at the target pixel position so that the target pixel position corresponds to different brightness values ​​at different moments.

[0060] S140 . Mapping the second texture image onto the video image based on the three-dimensional reconstruction data to obtain a target video image.

[0061] In the embodiment of the present disclosure, the three-dimensional reconstruction data includes data of a three-dimensional mesh constituting a three-dimensional model of the scene, and the data of the three-dimensional mesh includes vertex coordinates and normal directions.

[0062] Among them, the three-dimensional grid can be understood as the basic unit that constitutes the three-dimensional model of the scene.

[0063] In an exemplary implementation of the embodiment of the present disclosure, the vertex coordinates and normal direction of the three-dimensional mesh can be extracted from the three-dimensional reconstruction data of the scene through a vertex shader.

[0064] In the embodiments of the present disclosure, texture mapping may be understood as a process of mapping a texture on a texture image onto a three-dimensional model of a scene.

[0065] In an exemplary implementation of the embodiments of the present disclosure, the sampling coordinates can be first determined based on the vertex coordinates and normal directions in the data of the three-dimensional grid; then, the preset second texture image is sampled based on the sampling coordinates; and based on the association between the sampling coordinates and the vertices, the sampled second texture image is mapped to the video image to obtain the target video image.

[0066] Specifically, since the sampling coordinates are determined based on the three-dimensional coordinates and normal direction of the grid, after the texture image is sampled based on the sampling coordinates, the sampled second texture image can be mapped to the video image based on the association between the sampling coordinates and the grid, so that the captured texture image is mapped to the three-dimensional scene of the video.

[0067] In the embodiment of the present disclosure, after acquiring the three-dimensional reconstruction data of the video image and the first texture image corresponding to the current moment, it is possible to determine that the target pixel position on the first texture image corresponds to the target brightness at the current moment based on the mapping relationship between the pixel position and the initial brightness and the brightness change rate obtained in advance, and adjust the brightness of the target pixel position on the first texture image to the target brightness to obtain the second texture image, and obtain the target video image by mapping the second texture image to the video image. The scheme provided by the embodiment of the present disclosure can make the same pixel position on the video image present different textures and brightness at different moments, and make different pixel positions on the video image present different textures and brightness at the same moment, thereby achieving a flowing and flickering texture effect on the video image, and improving the interest of the video.

[0068] In another embodiment of the present disclosure, the texture on the pre-obtained reference texture image may be randomly sampled to at least part of the pixel positions of the preset template to obtain a first texture image corresponding to the current moment.

[0069] The pre-obtained reference texture image may be a texture image to be sampled.

[0070] The preset template may be a sampling template obtained by pre-dividing pixels on the screen of the electronic device into a plurality of uniform blocks.

[0071] The size of each pixel position in the preset template may be a fixed size, so that when the texture data in the reference texture image is sampled to the pixel position of the preset template, deformation of the texture data in the pixel position of the preset template is avoided.

[0072] In some embodiments, randomly sampling the texture of the pre-obtained reference texture image to at least part of the pixel positions of the preset template may include:

[0073] S10. For any pixel position of at least part of the pixel positions of the preset template, randomly select a pixel position from a pre-obtained noise texture image as a sampling position, wherein the noise texture image includes information of random coordinates corresponding to the sampling position.

[0074] S12. Extract random coordinates corresponding to the sampling positions from the noise texture image.

[0075] S14, sampling the texture at the position corresponding to the random coordinate in the reference texture image onto the preset template.

[0076] The random coordinate may be a randomly generated coordinate.

[0077] S14 may include: using the random coordinates as sampling coordinates to sample the texture at the position corresponding to the random coordinates in the reference texture image onto the preset template.

[0078] Specifically, after the electronic device obtains the reference texture image, the pre-obtained noise texture image and the preset template, for any pixel position among at least part of the pixel positions of the preset template, it randomly selects a pixel position from the pre-obtained noise texture image as a sampling position, determines the random coordinates of the sampling position, and uses the random coordinates as the sampling coordinates to sample the texture at the position corresponding to the random coordinates in the reference texture image onto the preset template, until all sampling is completed, and a first texture image corresponding to the current moment is obtained.

[0079] Figure 2a A schematic diagram of a reference texture image provided by an embodiment of the present disclosure is shown. Figure 2b A schematic diagram of a preset template provided by an embodiment of the present disclosure is shown. Figure 2c A schematic diagram of a noise texture image provided by an embodiment of the present disclosure is shown.

[0080] Figure 2a The reference texture image shown may be a text image. Figure 2b Each small square of the preset template shown is a pixel position. Figure 2c Each sampling position in the noise texture image shown corresponds to one or more random coordinates.

[0081] See also Figure 2a-2c , the electronic device targets pixel position 1 and pixel position 2 in at least part of the pixel positions of the preset template, at the moment before the current moment, the texture corresponding to pixel position 1 is A4, and the texture corresponding to pixel position 2 is A3. At the current moment, the electronic device randomly selects a pixel position from the noise texture image as a sampling position, determines the random coordinates of the sampling position, uses the random coordinates as the sampling coordinates, samples the texture A1 at the position corresponding to the random coordinates in the reference texture image to pixel position 1, so that the texture at pixel position 1 is adjusted from A4 to A1, and samples the texture A2 at the position corresponding to the random coordinates in the reference texture image to pixel position 2, so that the texture at pixel position 2 is adjusted from A3 to A2, and obtains a first texture image corresponding to the current moment.

[0082] Therefore, in the embodiment of the present disclosure, a noise texture image can be introduced, and random coordinates corresponding to the sampling position can be extracted from the noise texture image. Based on the random coordinates, the texture on the pre-obtained reference texture image is randomly sampled to at least part of the pixel position of the preset template to obtain a first texture image corresponding to the current moment. In a continuous time period, the texture information of the collected first texture image can be continuously changed to present an effect of dynamic texture change.

[0083] In some other embodiments, randomly sampling the texture of the pre-obtained reference texture image to at least part of the pixel positions of the preset template may include:

[0084] S20. For any pixel position in at least part of the pixel positions of the preset template, coordinate offset processing is performed on the coordinates of any pixel position on the preset template to obtain offset coordinates corresponding to any pixel position.

[0085] S22. Obtain random coordinates at corresponding positions from a preset noise texture image based on the offset coordinates.

[0086] S24, collecting the texture at the position corresponding to the random coordinate in the reference texture image to any pixel position.

[0087] Among them, the offset coordinates can be the coordinates obtained after the initial coordinates corresponding to the pixel position are subjected to coordinate offset processing, wherein the offset processing can, for example, be superimposing the initial coordinates with the time parameters corresponding to the current moment, but is not limited to superimposing the initial coordinates with the time parameters corresponding to the current moment.

[0088] Specifically, after the electronic device obtains the reference texture image, the pre-obtained noise texture image and the preset template, for any pixel position in at least part of the pixel positions of the preset template, the coordinate offset processing is performed on the coordinates of any pixel position on the preset template, and based on the initial coordinates of any pixel position, the offset coordinates corresponding to any pixel position are obtained, and then, the position corresponding to the offset coordinates is used as the sampling position in the preset noise texture image, and the random coordinates corresponding to the sampling position are determined, and further, the random coordinates are used as sampling coordinates to sample the texture at the position corresponding to the random coordinates in the reference texture image to any pixel position, so as to obtain the first texture image corresponding to the current moment.

[0089] Therefore, in the embodiment of the present disclosure, a time parameter can be introduced, and for any pixel position in at least part of the pixel positions of the preset template, a coordinate offset processing can be performed on the coordinates of any pixel position on the preset template to obtain the offset coordinates corresponding to any pixel position, and a noise texture image can be introduced. Based on the offset position and the noise texture image, a first texture image corresponding to the current moment is obtained. In a continuous time period, the texture information of the collected first texture image can be continuously changed to present an effect of dynamic texture change.

[0090] In some further embodiments, randomly sampling the texture of the pre-obtained reference texture image to at least part of the pixel positions of the preset template may include:

[0091] S30. For any pixel position in at least part of the pixel positions of the preset template, coordinate offset processing is performed on the coordinates of any pixel position on the preset template to obtain offset coordinates corresponding to any pixel position.

[0092] S32, collecting the texture at the position corresponding to the offset coordinate in the reference texture image to any pixel position.

[0093] Specifically, after the electronic device obtains the reference texture image and the preset template, for any pixel position among at least part of the pixel positions of the preset template, the coordinate offset processing is performed on the coordinates of any pixel position on the preset template, and based on the initial coordinates of any pixel position, the offset coordinates corresponding to any pixel position are obtained, and then, the offset coordinates are used as sampling coordinates to sample the texture at the position corresponding to the offset coordinates in the reference texture image to any pixel position, so as to obtain the first texture image corresponding to the current moment.

[0094] The disclosed embodiment obtains a second texture image by adjusting the brightness of the first texture image obtained in the above manner, and maps the second texture image onto a three-dimensional scene, so that the three-dimensional scene can present a dynamic change effect synthesized by a brightness change effect and a texture change effect, for example, an effect of generating character rain by a texture flickering effect and a flow effect can appear, thereby improving the interest of the video.

[0095] In another embodiment of the present disclosure, the vertex coordinate information and normal direction information of the three-dimensional mesh in the three-dimensional reconstruction data of the video image can be processed to obtain the three-dimensional coordinates and normal directions of the fragments in the three-dimensional mesh, and based on the three-dimensional coordinates and normal directions of the fragments, the second texture image is sampled, and the sampled texture is further mapped to the fragment to obtain the target video image.

[0096] Figure 3 A flow chart of another video processing method provided by an embodiment of the present disclosure is shown.

[0097] like Figure 1 As shown, the video processing method may include the following steps.

[0098] S310: Acquire three-dimensional reconstruction data of a video image and a first texture image corresponding to a current moment.

[0099] S320: Determine, based on a pre-obtained mapping relationship between a pixel position and an initial brightness and a brightness change rate, that a target pixel position on the first texture image corresponds to a target brightness at a current moment.

[0100] S330: Adjust the brightness of the target pixel position on the first texture image to the target brightness to obtain a second texture image.

[0101] Among them, S310 to S330 are similar to S110 to S130, and are not described in detail here.

[0102] S340 , performing differential processing on the three-dimensional grid in the three-dimensional reconstruction data to obtain segments in the three-dimensional grid and offset positions of the segments in the three-dimensional grid.

[0103] In the embodiment of the present disclosure, the differential processing may be based on the vertices of the three-dimensional mesh, and the three-dimensional mesh may be discretized according to a preset step size using an interpolation function, so that the three-dimensional mesh is discretized into one or more fragments.

[0104] In the embodiment of the present disclosure, a segment refers to the smallest unit obtained by dividing a three-dimensional grid into equal proportions.

[0105] The offset position of the fragment relative to the vertex of the three-dimensional mesh may be determined according to the step length of the fragment relative to the vertex of the three-dimensional mesh and the position of the vertex.

[0106] Specifically, after the electronic device extracts the three-dimensional coordinates of the vertices and the vertex normal directions of the three-dimensional mesh from the three-dimensional reconstruction data, the extracted three-dimensional coordinates of the vertices and the vertex normal directions of the three-dimensional mesh can be input into the fragment shader. The fragment shader can be based on the finite difference method, according to the vertices of the three-dimensional mesh, and use the interpolation function to discretize the three-dimensional mesh according to a preset step size, so that the three-dimensional mesh is discretized into one or more fragments, and the fragments in the three-dimensional mesh and the offset positions of the fragments in the three-dimensional mesh are obtained.

[0107] S350 : Determine the coordinates and normal direction of the fragment based on the offset position, and the vertex coordinates and normal direction of the three-dimensional mesh.

[0108] In an embodiment of the present disclosure, S350 may include: for each fragment, according to the three-dimensional coordinates of the vertices of the three-dimensional mesh and the offset position of each fragment in the three-dimensional mesh, calculating the coordinates of each fragment, according to the coordinates of the fragment, constructing the normal of the fragment, and using the normal direction of the three-dimensional mesh as the normal direction of each fragment.

[0109] Specifically, after the electronic device obtains the fragments in the three-dimensional grid and the offset positions of the fragments in the three-dimensional grid, it can use the fragment shader to use the coordinates corresponding to the offset positions of each fragment in the three-dimensional grid as the coordinates of the fragment, construct the normal of the fragment based on the coordinates of the fragment, and use the normal direction of the three-dimensional grid as the normal direction of each fragment.

[0110] Therefore, in the embodiment of the present disclosure, after obtaining the second texture image, the three-dimensional mesh is differentially processed using a fragment shader to obtain fragments in the three-dimensional mesh and the offset positions of the fragments in the three-dimensional mesh, and further based on the offset positions, as well as the vertex coordinates and normal directions of the three-dimensional mesh, the coordinates and normal directions of the fragments are accurately determined.

[0111] S360: Sample the second texture image based on the coordinates and normal direction of the fragment.

[0112] In the embodiment of the present disclosure, optionally, S360 may include:

[0113] S11. Based on the coordinates and normal direction of the fragment, determine the distance relationship between the normal of the fragment and three coordinate axes in a preset three-dimensional coordinate system.

[0114] S13. In response to the normal being closest to the first coordinate axis among the three coordinate axes, the sampling coordinates are composed based on the components of the three-dimensional coordinates on the second coordinate axis and the third coordinate axis among the three coordinate axes. Alternatively, in response to the normal being closest to the first coordinate axis and the second coordinate axis, the sampling coordinates can be composed based on the components of the three-dimensional coordinates on the second coordinate axis and the third coordinate axis, or the components of the three-dimensional coordinates on the first coordinate axis and the third coordinate axis.

[0115] The fact that the normal line is closest to the first coordinate axis and the second coordinate axis means that the distances between the normal line and the first coordinate axis and the second coordinate axis are the same and smaller than the distance between the normal line and the third coordinate axis.

[0116] S15. Sample the second texture image based on the sampling coordinates.

[0117] In some embodiments, S11 may include:

[0118] S111. Normalize the normal length of the fragment to obtain a target normal.

[0119] S113. Determine the components of the target normal on three coordinate axes based on the three-dimensional coordinates and normal direction of the fragment.

[0120] S115. Determine the distance relationship between the target normal and the three coordinate axes based on the components of the target normal on the three coordinate axes.

[0121] Among them, the distance between the target normal and the coordinate axis is inversely proportional to the component of the target normal on the coordinate axis.

[0122] In some other embodiments, S11 may include:

[0123] S211. Calculate the angle between the normal line of the fragment and the straight line where the three coordinate axes of the preset three-dimensional coordinate system are located according to the three-dimensional coordinates and the normal direction of the fragment.

[0124] S213. Use the coordinate axis corresponding to the minimum angle as the first coordinate axis closest to the normal line of the fragment, and use the coordinate axes corresponding to the maximum angle and the second largest angle as the second coordinate axis and the third coordinate axis that are not closest to the normal line of the fragment, wherein the angle can be any acute angle less than 90°.

[0125] The preset three-dimensional coordinate system may be a three-dimensional coordinate system where the three-dimensional grid model to which the fragment belongs is located.

[0126] The angle between the normal line of the fragment and the straight line where the three coordinate axes are located can be used to represent the proximity between the normal line and the three coordinate axes.

[0127] It can be understood that the larger the angle between the normal of the fragment and the straight line where the coordinate axis is located, the farther the normal of the fragment is from the coordinate axis, and therefore, the greater the distance between the normal of the fragment and the coordinate axis. Conversely, the smaller the angle between the normal of the fragment and the straight line where the coordinate axis is located, the closer the normal of the fragment is to the coordinate axis, and therefore, the smaller the distance between the normal of the fragment and the coordinate axis.

[0128] Therefore, in the embodiment of the present disclosure, the distance relationship between the target normal and the three coordinate axes can be determined based on the three-dimensional coordinates of the fragment and the components of the normalized target normal on the three coordinate axes, or the distance relationship between the target normal and the three coordinate axes can be determined based on the angle between the normal of the fragment and the coordinate axes to further determine the sampling coordinates based on the distance relationship.

[0129] Further, for S13, the components of the three-dimensional coordinates on the second and third axes of the three coordinate axes can be understood as the coordinates of the three-dimensional coordinates on the second and third axes of the three coordinate axes. Similarly, the components of the three-dimensional coordinates on the second and third axes can be understood as the coordinates of the three-dimensional coordinates on the second and third axes of the three coordinate axes, and the components of the three-dimensional coordinates on the first and third axes can be understood as the coordinates of the three-dimensional coordinates on the first and third axes of the three coordinate axes.

[0130] Further, with respect to S15, the electronic device may sample texture information of a pixel point at a position corresponding to the sampling coordinate on the texture image based on the sampling coordinate to obtain a corresponding texture.

[0131] S370: Map the sampled texture onto the fragment.

[0132] In the embodiments of the present disclosure, texture mapping may be understood as a process of mapping a texture on a texture image onto a three-dimensional model of a scene.

[0133] In the embodiment of the present disclosure, S370 may include:

[0134] S3701: In response to the normal line being closest to a first coordinate axis in a preset three-dimensional coordinate system, map the sampled texture onto the fragment along the direction of the first coordinate axis.

[0135] Specifically, since the sampling coordinates are determined based on the three-dimensional coordinates and normal direction of the fragment, after the texture image is sampled based on the sampling coordinates, the sampled texture image can be mapped to the fragment along the direction of the first coordinate axis that is closest to the normal according to the distance between the normal and the coordinate axis in the preset three-dimensional coordinate system, in response to the normal being closest to the first coordinate axis in the preset three-dimensional coordinate system, so that the captured texture image is mapped to the three-dimensional scene of the video.

[0136] Optionally, a tri-planar mapping method or a common mapping method may be used to map the texture information sampled from the second texture image onto the video image, so that the second texture image is mapped onto the three-dimensional model of the scene.

[0137] Therefore, in the embodiment of the present disclosure, since the three-dimensional coordinates and normal direction of the fragment are taken into account when determining the sampling coordinates, the texture image acquired based on the sampling coordinates can be mapped to the three-dimensional scene of the video and fit the three-dimensional scene of the video, and present a natural and realistic effect, thereby improving the interest of the video.

[0138] Figure 4is a schematic diagram of the structure of a video processing device provided by an embodiment of the present disclosure, and the processing device can be understood as the above-mentioned electronic device or a part of the functional modules in the above-mentioned electronic device. Figure 4 As shown, the video processing device 400 may include: an acquisition module 410 , a determination module 420 , a brightness adjustment module 430 and a texture mapping module 440 .

[0139] The acquisition module 410 is used to acquire the 3D reconstruction data of the current video image and the first texture image corresponding to the current moment;

[0140] A determination module 420, configured to determine that the target pixel position on the first texture image corresponds to the target brightness at the current moment according to a mapping relationship between the pixel position, the initial brightness, and the brightness change rate obtained in advance;

[0141] A brightness adjustment module 430, configured to adjust the brightness of a target pixel position on the first texture image to a target brightness, so as to obtain a second texture image;

[0142] The texture mapping module 440 is used to map the second texture image onto the video image based on the three-dimensional reconstruction data to obtain a target video image.

[0143] Optionally, the determination module 420 includes:

[0144] The acquisition submodule is used to obtain the initial brightness and brightness change rate of the target pixel position according to the mapping relationship between the pixel position, the initial brightness and the brightness change rate obtained in advance;

[0145] A first determination submodule, configured to determine a brightness change amount of a target pixel position corresponding to the current moment based on a time parameter at the current moment and a brightness change rate of the target pixel position;

[0146] The second determination submodule is used to determine the target brightness of the target pixel position corresponding to the current moment based on the brightness change amount of the target pixel position corresponding to the current moment and the initial brightness of the target pixel position.

[0147] Optionally, the acquisition module 410 includes:

[0148] The random sampling submodule is used to randomly sample the texture on the pre-obtained reference texture image to at least part of the pixel positions of the preset template at the current moment to obtain a first texture image corresponding to the current moment.

[0149] Optional, random sampling submodule, including:

[0150] A selection subunit is used to randomly select a pixel position from a pre-obtained noise texture image as a target pixel position for any pixel position among at least some of the pixel positions, wherein the noise texture image includes information of random coordinates corresponding to the target pixel position;

[0151] An extraction subunit, used for extracting random coordinates corresponding to target pixel positions from the noise texture image;

[0152] The first sampling subunit samples the texture corresponding to the random coordinates in the reference texture image to any pixel position.

[0153] Optional, random sampling submodule, including:

[0154] A processing subunit, configured to perform coordinate shift processing on the coordinates of any pixel position on a preset template for any pixel position among at least some of the pixel positions, so as to obtain shifted coordinates corresponding to any pixel position;

[0155] An acquisition subunit, used for acquiring random coordinates corresponding to a target pixel position from a preset noise texture image based on the offset coordinates, wherein the position coordinates of the target pixel position in the noise texture image match the offset coordinates;

[0156] The second sampling subunit is used to collect the texture corresponding to the random coordinates in the reference texture image to any pixel position.

[0157] Optionally, the three-dimensional reconstruction data includes vertex coordinate information and normal direction information of the three-dimensional mesh;

[0158] The texture mapping module 440 includes:

[0159] A differential processing module is used to perform differential processing on the three-dimensional grid to obtain fragments in the three-dimensional grid and offset positions of the fragments in the three-dimensional grid;

[0160] A third determination module, for determining the coordinates and normal direction of the fragment based on the offset position and the vertex coordinates and normal direction of the three-dimensional mesh;

[0161] An image sampling module, for sampling a second texture image based on the coordinates and normal direction of the fragment;

[0162] The texture mapping submodule is used to map the sampled texture onto the fragment.

[0163] Optionally, the texture mapping submodule is used to:

[0164] In response to the normal being closest to a first coordinate axis in a preset three-dimensional coordinate system, the sampled texture is mapped onto the fragment along the direction of the first coordinate axis.

[0165] The device provided in this embodiment can perform the above Figure 1-Figure 3 The method of any embodiment has similar execution mode and beneficial effects, which will not be described in detail here.

[0166] The present disclosure also provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the above-mentioned Figure 1 , Figure 3 ,as well as Figure 2a-2c The method of any one of the embodiments.

[0167] For example, Figure 5 This is a schematic diagram of the structure of a terminal device in an embodiment of the present disclosure. Figure 5 , which shows a schematic diagram of the structure of an electronic device 500 suitable for implementing the embodiment of the present disclosure. The electronic device 500 in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The terminal device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0168] like Figure 5 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0169] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0170] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0171] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0172] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0173] The computer-readable medium may be included in the terminal device, or may exist independently without being installed in the terminal device.

[0174] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains three-dimensional reconstruction data of the video image and a first texture image corresponding to the current moment;

[0175] Determine, based on a pre-obtained mapping relationship between a pixel position and an initial brightness and a brightness change rate, that a target pixel position on the first texture image corresponds to a target brightness at a current moment;

[0176] Adjusting the brightness of the target pixel position on the first texture image to the target brightness to obtain a second texture image;

[0177] Based on the three-dimensional reconstruction data, the second texture image is mapped onto the video image to obtain a target video image.

[0178] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0179] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0180] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0181] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0182] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0183] The present disclosure also provides a computer-readable storage medium in which a computer program is stored. When the computer program is executed by a processor, the above-mentioned Figure 1 , Figure 3 ,as well as Figure 2a-2c The method of any embodiment has similar execution mode and beneficial effects, which will not be described in detail here.

[0184] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0185] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video processing method, characterized in that: include: Acquire three-dimensional reconstruction data of the video image and a first texture image corresponding to the current moment; Determine, according to a pre-obtained mapping relationship between a pixel position and an initial brightness and a brightness change rate, that a target pixel position on the first texture image corresponds to a target brightness at a current moment, wherein the initial brightness and the brightness change rate of the pixel position are determined according to the pixel position; Adjusting the brightness of the target pixel position on the first texture image to the target brightness to obtain a second texture image; Based on the three-dimensional reconstruction data, the second texture image is mapped onto the video image to obtain a target video image, Among them, mapping the second texture image to the video image based on the three-dimensional reconstruction data includes: determining sampling coordinates based on the three-dimensional mesh data in the three-dimensional reconstruction data; sampling the second texture image based on the sampling coordinates; and mapping the sampled second texture image to the video image based on the sampling coordinates.

2. The method according to claim 1, characterized in that The step of determining, based on the pre-obtained mapping relationship between the pixel position and the initial brightness and the brightness change rate, that the target pixel position on the first texture image corresponds to the target brightness at the current moment comprises: Obtaining the initial brightness and brightness change rate of the target pixel position according to the mapping relationship between the pixel position, the initial brightness and the brightness change rate obtained in advance; Determine the brightness change amount of the target pixel position corresponding to the current moment based on the time parameter at the current moment and the brightness change rate of the target pixel position; Based on the brightness change amount of the target pixel position corresponding to the current moment and the initial brightness of the target pixel position, the target brightness of the target pixel position corresponding to the current moment is determined.

3. The method according to claim 1, characterized in that The obtaining of a first texture image corresponding to the current moment includes: For the current moment, the texture on the pre-obtained reference texture image is randomly sampled to at least part of the pixel positions of the preset template to obtain a first texture image corresponding to the current moment.

4. The method according to claim 3, characterized in that The step of randomly sampling the texture of the pre-obtained reference texture image to at least a portion of pixel positions of a preset template comprises: For any pixel position of at least part of the pixel positions of the preset template, randomly select a pixel position from a pre-obtained noise texture image as a sampling position, wherein the noise texture image includes information of random coordinates corresponding to the sampling position; Extracting random coordinates corresponding to the sampling positions from the noise texture image; The texture at the position corresponding to the random coordinate in the reference texture image is sampled to any pixel position.

5. The method according to claim 3, characterized in that: The step of randomly sampling the texture of the pre-obtained reference texture image to at least a portion of pixel positions of a preset template comprises: For any pixel position among at least part of the pixel positions of the preset template, coordinate offset processing is performed on the coordinates of the any pixel position on the preset template to obtain offset coordinates corresponding to the any pixel position; Acquire random coordinates at corresponding positions from a preset noise texture image based on the offset coordinates; The texture at the position corresponding to the random coordinate in the reference texture image is collected at any pixel position.

6. The method according to any one of claims 1 to 5, characterized in that The three-dimensional reconstruction data includes vertex coordinate information and normal direction information of the three-dimensional grid; The step of mapping the second texture image onto the video image based on the three-dimensional reconstruction data to obtain a target video image includes: Performing differential processing on the three-dimensional grid to obtain segments in the three-dimensional grid and offset positions of the segments in the three-dimensional grid; Determine the coordinates and normal direction of the fragment based on the offset position, and the vertex coordinates and normal direction of the three-dimensional mesh; Sampling the second texture image based on the coordinates and normal direction of the fragment; Map the sampled texture onto the fragment.

7. The method according to claim 6, characterized in that Mapping the sampled texture onto the fragment comprises: In response to the normal being closest to a first coordinate axis in a preset three-dimensional coordinate system, the sampled texture is mapped onto the fragment along the direction of the first coordinate axis.

8. A video processing device, characterized in that: include: An acquisition module, used to acquire three-dimensional reconstruction data of a current video image and a first texture image corresponding to a current moment; a determination module, configured to determine that a target pixel position on the first texture image corresponds to a target brightness at a current moment according to a pre-obtained mapping relationship between a pixel position and an initial brightness and a brightness change rate, wherein the initial brightness and the brightness change rate of the pixel position are determined according to the pixel position; A brightness adjustment module, used for adjusting the brightness of the target pixel position on the first texture image to the target brightness, so as to obtain a second texture image; A texture mapping module is used to map the second texture image onto the video image based on the three-dimensional reconstruction data to obtain a target video image, wherein mapping the second texture image onto the video image based on the three-dimensional reconstruction data includes: determining sampling coordinates based on three-dimensional grid data in the three-dimensional reconstruction data; sampling the second texture image based on the sampling coordinates; and mapping the sampled second texture image onto the video image based on the sampling coordinates.

9. The device according to claim 8, characterized in that The determining module comprises: An acquisition submodule, used to obtain the initial brightness and brightness change rate of the target pixel position according to the mapping relationship between the pixel position, the initial brightness and the brightness change rate obtained in advance; A first determination submodule, configured to determine a brightness change amount of the target pixel position corresponding to the current moment based on a time parameter at the current moment and a brightness change rate of the target pixel position; The second determination submodule is used to determine the target brightness of the target pixel position corresponding to the current moment based on the brightness change of the target pixel position corresponding to the current moment and the initial brightness of the target pixel position.

10. The device according to claim 8, characterized in that The acquisition module comprises: The random sampling submodule is used to randomly sample the texture on the pre-obtained reference texture image to at least part of the pixel positions of the preset template at the current moment to obtain a first texture image corresponding to the current moment.

11. The device according to claim 10, characterized in that The random sampling submodule includes: A selection subunit is used to randomly select a pixel position from a pre-obtained noise texture image as a target pixel position for any pixel position among the at least part of the pixel positions, wherein the noise texture image includes information of random coordinates corresponding to the target pixel position; An extraction subunit, used for extracting random coordinates corresponding to the target pixel position from the noise texture image; The first sampling subunit samples the texture corresponding to the random coordinates in the reference texture image to any pixel position.

12. The device according to claim 10, characterized in that The random sampling submodule comprises: a processing subunit, configured to perform coordinate offset processing on the coordinates of any pixel position in the at least part of the pixel positions on the preset template to obtain offset coordinates corresponding to the any pixel position; an acquisition subunit, configured to acquire random coordinates corresponding to a target pixel position from a preset noise texture image based on the offset coordinates, wherein the position coordinates of the target pixel position in the noise texture image match the offset coordinates; The second sampling subunit is used to collect the texture corresponding to the random coordinates in the reference texture image to any pixel position.

13. The device according to any one of claims 8 to 12, characterized in that The three-dimensional reconstruction data includes vertex coordinate information and normal direction information of the three-dimensional grid; The texture mapping module comprises: A differential processing module, used for performing differential processing on the three-dimensional grid to obtain segments in the three-dimensional grid and offset positions of the segments in the three-dimensional grid; A third determination module, configured to determine the coordinates and normal direction of the fragment based on the offset position and the vertex coordinates and normal direction of the three-dimensional mesh; An image sampling module, configured to sample the second texture image based on the coordinates and normal direction of the fragment; The texture mapping submodule is used to map the sampled texture onto the fragment.

14. The device according to claim 13, characterized in that The texture mapping submodule is used to: In response to the normal being closest to a first coordinate axis in a preset three-dimensional coordinate system, the sampled texture is mapped onto the fragment along the direction of the first coordinate axis.

15. An electronic device, characterized in that: include: A memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

16. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images