Video generation methods, apparatus, electronic devices and storage media

Abstract

This disclosure relates to a video generation method, apparatus, electronic device, and storage medium, belonging to the field of computer technology. The method includes: generating point cloud data based on the depth of pixels in a first image; offsetting the point cloud data according to a preset offset field; generating a second image based on the offset point cloud data; and generating a video containing the second image. This method can avoid having too many black pixels in the second image, improving the quality of the second image, and thus improving the quality of the generated video.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a video generation method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the popularization of multimedia technology, image-based video generation has become a common and popular method of video production.

[0003] In related technologies, taking the first image as an example, the offset field of each pixel in the first image is usually determined. Each pixel is then moved according to its offset field to obtain the second image. This second image is then used as a video frame to generate a video containing the second image. However, videos generated using this technique may contain a large number of black pixels, resulting in poor video quality. Summary of the Invention

[0004] This disclosure provides a video generation method, apparatus, electronic device, and storage medium, which can improve the quality of the generated video. The technical solution of this disclosure is as follows:

[0005] According to one aspect of the embodiments of this disclosure, a video generation method is provided, comprising:

[0006] Based on the depth of pixels in the first image, point cloud data is generated, and the point cloud data is used to represent the spatial location of pixels in the first image.

[0007] The point cloud data is offset according to a preset offset field, and a second image is generated based on the offset point cloud data.

[0008] Generate a video containing the second image.

[0009] In some embodiments, the step of offsetting the point cloud data according to a preset offset field and generating a second image based on the offset point cloud data includes:

[0010] The point cloud data is offset according to the offset field, and the second image is generated based on the offset point cloud data until the proportion of target pixels in the currently generated second image is greater than a preset proportion. Then, the step of generating a video containing the second image is executed, where the target pixels are pixels other than black pixels.

[0011] In some embodiments, the step of offsetting the point cloud data according to the offset field, generating the second image based on the offset point cloud data, and continuing until the proportion of target pixels in the currently generated second image is greater than a preset proportion, and then performing the step of generating a video containing the second image, includes:

[0012] The point cloud data is offset according to the offset field, and the second image is generated based on the offset point cloud data.

[0013] If the proportion of the target pixels in the second image is not greater than the preset proportion, the offset field is reduced.

[0014] The point cloud data is offset according to the reduced offset field, and a new second image is generated based on the offset point cloud data until the proportion of target pixels in the currently generated second image is greater than the preset proportion, and then the step of generating a video containing the second image is executed.

[0015] In some embodiments, the offset field includes at least one horizontal offset and at least one vertical offset. The step of offsetting the point cloud data according to the offset field, generating the second image based on the offset point cloud data, and continuing until the proportion of target pixels in the currently generated second image is greater than a preset proportion, and then generating a video containing the second image, includes:

[0016] The point cloud data is horizontally offset by at least one horizontal offset, and a second image is generated based on the point cloud data obtained after offset, until the proportion of target pixels in the currently generated second image is greater than the preset proportion.

[0017] While keeping the horizontal offset of the currently generated second image unchanged, the point cloud data corresponding to the currently generated second image is vertically offset by at least one vertical offset. A new second image is generated based on the point cloud data obtained after the offset, until the proportion of target pixels in the currently generated second image is greater than the preset proportion, and then the step of generating a video containing the second image is executed.

[0018] In some embodiments, the offset field includes at least one horizontal offset and at least one vertical offset. The step of offsetting the point cloud data according to the offset field, generating the second image based on the offset point cloud data, and continuing until the proportion of target pixels in the currently generated second image is greater than a preset proportion, and then generating a video containing the second image, includes:

[0019] The point cloud data is vertically offset by at least one vertical offset, and a second image is generated based on the point cloud data obtained after offset, until the proportion of target pixels in the currently generated second image is greater than the preset proportion.

[0020] While keeping the vertical offset of the currently generated second image unchanged, the point cloud data corresponding to the currently generated second image is horizontally offset by at least one horizontal offset. A new second image is generated based on the point cloud data obtained after the offset, until the proportion of target pixels in the currently generated second image is greater than the preset proportion, and then the step of generating a video containing the second image is executed.

[0021] In some embodiments, offsetting the point cloud data according to a preset offset field and generating a second image based on the offset point cloud data includes offsetting the point cloud data according to multiple offset fields and generating multiple second images based on the offset point cloud data respectively.

[0022] The generation of a video containing the second image includes generating a video containing the plurality of the second images.

[0023] In some embodiments, generating point cloud data based on the depth of pixels in the first image includes:

[0024] Obtain a depth map of the first image, wherein the depth map is used to represent the depth of pixels in the first image;

[0025] The point cloud data is generated based on the depth map.

[0026] In some embodiments, obtaining the depth map of the first image includes inputting the first image into a depth estimation model and obtaining the depth map output by the depth estimation model.

[0027] In some embodiments, the method further includes: filtering the depth map, wherein the filtering process includes at least one of the following: weighted mean filtering and joint bilateral filtering;

[0028] The step of generating the point cloud data based on the depth map includes: generating the point cloud data based on the filtered depth map.

[0029] In some embodiments, the method further includes filling black pixels in the second image with a color different from black.

[0030] According to another aspect of the present disclosure, a video generation apparatus is provided, the apparatus comprising:

[0031] A point cloud generation unit is configured to generate point cloud data based on the depth of pixels in a first image, the point cloud data being used to represent the spatial location of pixels in the first image.

[0032] The image generation unit is configured to offset the point cloud data according to a preset offset field, and generate a second image based on the offset point cloud data.

[0033] The video generation unit is configured to generate a video containing the second image.

[0034] In some embodiments, the image generation unit includes:

[0035] The image generation subunit is configured to offset the point cloud data according to the offset field, generate the second image based on the offset point cloud data, until the proportion of target pixels in the currently generated second image is greater than a preset proportion, triggering the video generation unit to execute the step of generating a video containing the second image, wherein the target pixels are pixels other than black pixels.

[0036] In some embodiments, the image generation subunit is configured as follows:

[0037] The point cloud data is offset according to the offset field, and the second image is generated based on the offset point cloud data.

[0038] If the proportion of the target pixels in the second image is not greater than the preset proportion, the offset field is reduced.

[0039] The point cloud data is offset according to the reduced offset field, and a new second image is generated based on the offset point cloud data until the proportion of target pixels in the currently generated second image is greater than the preset proportion, triggering the video generation unit to execute the step of generating a video containing the second image.

[0040] In some embodiments, the offset field includes at least one lateral offset and at least one vertical offset, and the image generation subunit is configured to:

[0041] The point cloud data is horizontally offset by at least one horizontal offset, and a second image is generated based on the point cloud data obtained after offset, until the proportion of target pixels in the currently generated second image is greater than the preset proportion.

[0042] While keeping the horizontal offset of the currently generated second image unchanged, the point cloud data corresponding to the currently generated second image is vertically offset by at least one vertical offset. A new second image is generated based on the point cloud data obtained after the offset, until the proportion of target pixels in the currently generated second image is greater than the preset proportion, triggering the video generation unit to execute the step of generating a video containing the second image.

[0043] In some embodiments, the offset field includes at least one lateral offset and at least one vertical offset, and the image generation subunit is configured to:

[0044] The point cloud data is vertically offset by at least one vertical offset, and a second image is generated based on the point cloud data obtained after offset, until the proportion of target pixels in the currently generated second image is greater than the preset proportion.

[0045] While keeping the vertical offset of the currently generated second image unchanged, the point cloud data corresponding to the currently generated second image is horizontally offset by at least one horizontal offset. A new second image is generated based on the point cloud data obtained after the offset, until the proportion of target pixels in the currently generated second image is greater than the preset proportion, triggering the video generation unit to execute the step of generating a video containing the second image.

[0046] In some embodiments, the image generation unit is configured to offset the point cloud data according to multiple offset fields, and generate multiple second images based on the offset point cloud data respectively.

[0047] The video generation unit is configured to generate a video containing the plurality of the second images.

[0048] In some embodiments, the point cloud generation unit includes:

[0049] The acquisition subunit is configured to acquire a depth map of the first image, the depth map being used to represent the depth of pixels in the first image;

[0050] The point cloud generation subunit is configured to generate the point cloud data based on the depth map.

[0051] In some embodiments, the acquisition subunit is configured to input the first image into a depth estimation model and acquire the depth map output by the depth estimation model.

[0052] In some embodiments, the apparatus further includes:

[0053] A filtering unit is configured to perform filtering processing on the depth map, the filtering processing including at least one of the following: weighted mean filtering, joint bilateral filtering;

[0054] The point cloud generation subunit is configured as follows:

[0055] The point cloud data is generated based on the filtered depth map.

[0056] In some embodiments, the apparatus further includes:

[0057] The fill unit is configured to fill black pixels in the second image with a color different from black.

[0058] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0059] processor;

[0060] Memory used to store the processor's executable instructions;

[0061] The processor is configured to execute the instructions to implement the video generation method as described above.

[0062] According to another aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the video generation method as described above.

[0063] According to another aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the video generation method as described above.

[0064] The solution provided in this embodiment first converts the pixels of the first image into point cloud data, and then offsets the point cloud data, instead of directly offsetting the pixels at the pixel level. This allows the generation of a second image based on the offset point cloud data, avoiding a large number of black pixels in the second image, improving the quality of the second image, and thus improving the quality of the generated video.

[0065] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0066] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0067] Figure 1 This is a flowchart illustrating a video generation method according to an exemplary embodiment;

[0068] Figure 2 This is a flowchart illustrating another video generation method according to an exemplary embodiment;

[0069] Figure 3 This is a flowchart illustrating another video generation method according to an exemplary embodiment;

[0070] Figure 4 This is a schematic diagram illustrating an adjustment of offset according to an exemplary embodiment;

[0071] Figure 5 This is a flowchart illustrating another video generation method according to an exemplary embodiment;

[0072] Figure 6 This is a schematic diagram illustrating an embodiment of obtaining a depth map;

[0073] Figure 7 This is a schematic diagram illustrating a filtering process for a depth map according to an exemplary embodiment;

[0074] Figure 8 This is a schematic diagram illustrating an original depth map and a filtered depth map according to an exemplary embodiment;

[0075] Figure 9 This is a schematic diagram illustrating a second image according to an exemplary embodiment;

[0076] Figure 10 This is a schematic diagram illustrating a second image obtained through interpolation processing according to an exemplary embodiment;

[0077] Figure 11 This is a schematic diagram illustrating an implementation environment according to an exemplary embodiment;

[0078] Figure 12 This is a structural block diagram of a video generation apparatus according to an exemplary embodiment;

[0079] Figure 13 This is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0080] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0081] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0082] The user information disclosed herein may be information authorized by the user or fully authorized by all parties.

[0083] Figure 1 This is a flowchart illustrating a video generation method according to an exemplary embodiment, such as... Figure 1 As shown, the method is performed by an electronic device, which may be a terminal or a server, or other devices. The method includes the following steps.

[0084] In step 101, the electronic device generates point cloud data based on the depth of the pixels in the first image.

[0085] The first image is either an image stored locally on the electronic device, an image currently captured by the electronic device, or an image downloaded by the electronic device. This embodiment of the disclosure does not limit the source of the first image.

[0086] In the first image, each pixel has not only color but also depth, which represents the distance between the pixel and the camera's imaging plane when the first image was captured. Different pixels in the first image may have the same or different depths.

[0087] In this embodiment, the position of each pixel in the first image is determined, which can be considered as the two-dimensional coordinates of each pixel in a two-dimensional space being determined. Furthermore, the first image also includes the depth of each pixel, which can represent the pixel's coordinates in a third dimension. Therefore, based on the depth of the pixels in the first image, the electronic device can convert each pixel into a point in three-dimensional space. Thus, multiple pixels in the first image, after conversion, can obtain point cloud data, which is used to represent the spatial position of the pixels in the first image.

[0088] In this point cloud data, the three-dimensional coordinates of each point are determined by the two-dimensional coordinates of the corresponding pixel in the first image and the depth of that pixel. For example, in the three-dimensional coordinates of each point, the first dimension is the x-coordinate of the corresponding pixel in the first image, the second dimension is the y-coordinate of the corresponding pixel in the first image, and the third dimension is the depth of the corresponding pixel.

[0089] In step 102, the electronic device offsets the point cloud data according to a preset offset field, and generates a second image based on the offset point cloud data.

[0090] The preset offset field includes the offset direction and offset amount of the point cloud data. This offset field can be a field set in the electronic device or an offset field input by the user. When the offset field is set in the electronic device, the electronic device sets the offset direction and offset amount of the point cloud data, and the offset field consists of this offset direction and offset amount. When the offset field is input by the user, the user can manually set the offset direction and offset amount of the point cloud data, and the offset field consists of this offset direction and offset amount.

[0091] In this embodiment of the present disclosure, the electronic device offsets the point cloud data according to the offset direction and offset amount indicated by the preset offset field, and obtains new point cloud data after offset. The new point cloud data indicates the spatial position of the offset pixel.

[0092] The point cloud data includes multiple points located at different spatial locations. The electronic device offsets the point cloud data according to the offset direction and offset amount indicated by the preset offset field, and the offset results in new point cloud data. This includes: the electronic device offsets each point in the point cloud data according to the offset direction and offset amount indicated by the preset offset field, and the offset multiple points constitute new point cloud data.

[0093] Subsequently, the electronic device generates a second image based on the offset point cloud data, that is, it maps the offset point cloud data into a two-dimensional space to obtain the second image.

[0094] In some embodiments, each point in the offset point cloud data is mapped to a two-dimensional space, and the multiple points mapped in the two-dimensional space can constitute a second image.

[0095] For example, each point in the point cloud data obtained after offset has three-dimensional coordinates. By retaining the coordinates in the first and second dimensions of each point and deleting the coordinates in the third dimension, each point in the point cloud data can be mapped to a two-dimensional space.

[0096] Furthermore, each pixel in the first image not only has a position and depth, but also a pixel value, which represents the color of that pixel. Therefore, by generating point cloud data based on the first image, offsetting the point cloud data, and then mapping the offset point cloud data to a two-dimensional space, the offset coordinates of each pixel can be obtained. Based on the pixel value before offset and the offset coordinates of each pixel, a second image can be generated. Thus, the second image can be seen as an image obtained by offsetting the content of the first image.

[0097] In this embodiment, the pixels of the first image are first converted into point cloud data, and then the point cloud data is offset, instead of directly offsetting the pixels at the pixel level. This is because offsetting pixels at the pixel level would produce many spaces, which are the black pixels in the image. Therefore, converting the pixels of the first image into point cloud data and then offsetting the point cloud data avoids having many black pixels in the generated second image, thus improving the quality of the second image.

[0098] In step 103, the electronic device generates a video containing the second image.

[0099] In this embodiment of the disclosure, after the electronic device generates the second image, it can use the second image as a video frame in the video to generate a video containing the second image.

[0100] Considering that directly offsetting pixels at the pixel level would produce many blank spaces, resulting in black pixels in the image, the method provided in this disclosure first converts the pixels of the first image into point cloud data, and then offsets the point cloud data, instead of directly offsetting the pixels at the pixel level. This allows the generation of a second image based on the offset point cloud data, avoiding a large number of black pixels in the second image, improving the quality of the second image, and thus improving the quality of the generated video.

[0101] Figure 2 This is a flowchart illustrating another video generation method according to an exemplary embodiment, such as... Figure 2 As shown, the method is performed by an electronic device and includes the following steps.

[0102] In step 201, the electronic device generates point cloud data based on the depth of the pixels in the first image.

[0103] In step 202, the electronic device offsets the point cloud data according to a preset offset field, and generates a second image based on the offset point cloud data.

[0104] Steps 201-202 are the same as steps 101-102 above, and will not be repeated here.

[0105] After performing step 202, the electronic device can measure the quality of the second image based on the proportion of target pixels in the second image in order to determine whether to generate a video containing the second image. For details, see steps 203 or 204-206 below.

[0106] In step 203, if the proportion of target pixels in the second image is greater than a preset proportion, the electronic device generates a video containing the second image.

[0107] In this context, target pixels refer to all pixels except black pixels. Black pixels are generated during the offsetting of point cloud data according to the offset field and during the generation of the second image based on the offset point cloud data. The proportion of target pixels refers to the percentage of pixels other than black pixels in the second image. A higher proportion of target pixels indicates fewer black pixels and higher image quality.

[0108] In this embodiment of the disclosure, a preset ratio is set based on the requirements for image quality. This preset ratio is used to measure the quality of the image. When the proportion of target pixels in the second image is greater than the preset ratio, it indicates that there are fewer black pixels in the second image, and the quality of the second image is higher. Therefore, a video containing the second image can be generated, thereby ensuring the quality of the generated video.

[0109] In step 204, the electronic device reduces the offset field if the proportion of the target pixel in the second image is not greater than a preset proportion.

[0110] If the proportion of target pixels in the second image is not greater than a preset proportion, it indicates that the second image has more black pixels and its quality is poor. In this case, directly generating a video containing the second image will result in poor video quality. Therefore, to improve video quality, the offset field can be reduced, and the image can be regenerated using the reduced offset field.

[0111] Here, "reduced offset field" refers to the offset included in the reduced offset field. The reduced offset can be achieved by multiplying the offset in the offset field by an adjustment factor less than 1, or by subtracting the step size from the offset. This step size can be a pre-set fixed value or a randomly generated value.

[0112] In step 205, the electronic device offsets the point cloud data according to the reduced offset field, and generates a new second image based on the offset point cloud data until the proportion of target pixels in the currently generated second image is greater than a preset proportion.

[0113] In this embodiment of the disclosure, the electronic device offsets the point cloud data according to the reduced offset field, and generates a new second image based on the point cloud data obtained after offset. Since the offset is reduced, the probability of generating black pixels is reduced, which can effectively reduce the number of black pixels in the generated second image, thereby improving the image quality as much as possible.

[0114] Therefore, in generating the new second image, the quality of the second image can be determined based on the proportion of target pixels in the currently generated second image. If the proportion of target pixels in the currently generated second image is greater than the preset proportion, it is determined that the offset field does not need to be further reduced. The current offset field is the offset field with the largest offset amount that can meet the image quality requirements when offsetting the point cloud data, and then step 206 is executed.

[0115] Alternatively, if the proportion of target pixels in the currently generated second image is not greater than the preset proportion, then the offset field is further reduced, and the point cloud data is further offset according to the reduced offset field until the proportion of target pixels in the currently generated second image is greater than the preset proportion, and step 206 is executed.

[0116] In this process, the adjustment coefficient or step size can be the same each time the offset is reduced, thus achieving a geometric or arithmetic reduction of the offset. In other words, fine-tuning is performed based on the original offset to find a more suitable offset.

[0117] In step 206, the electronic device generates a video containing the second image.

[0118] In some embodiments, the electronic device determines multiple offset fields, and performs steps 202 to 205 above for each offset field to obtain a second image. Thus, multiple second images corresponding to multiple offset fields can be obtained. Each second image can be regarded as an image obtained after a certain degree of offset from the first image. Then, a video containing the multiple second images is generated. The video can present the effect of the content of the first image gradually moving, forming a dynamic video picture and reflecting the offset trajectory of the first image.

[0119] In some embodiments, multiple offset fields are arranged in order, and the offsets in the multiple offset fields increase or decrease sequentially. After obtaining multiple second images corresponding to the multiple offset fields, the multiple second images are combined into a video according to the order of the multiple offset fields. The generated video can present the effect that the content of the first image gradually moves and the offset becomes larger or smaller, forming a dynamic video picture that reflects the offset trajectory of the first image.

[0120] In some embodiments, after generating the second image, the electronic device generates a video containing the first image and the second image. Since the second image is obtained by offsetting the first image, the first image and the second image are used as video frames, and the first image is located before the second image. The generated video will present the effect of the second image obtained by offsetting the first image, forming a dynamic video picture.

[0121] In other embodiments, multiple offset fields are arranged in sequence, and the offsets in the multiple offset fields increase sequentially. After obtaining multiple second images corresponding to the multiple offset fields, the first image is placed first, and the multiple second images are arranged according to the order of the multiple offset fields. Thus, the first image and the multiple second images constitute a video. The generated video can present the effect of starting with the content of the first image as the starting frame, and then the content of the first image gradually moves and the offset increases, forming a dynamic video picture that reflects the offset trajectory of the first image.

[0122] In some embodiments, after generating the second image, the electronic device can further fill the black pixels in the second image with a color different from black. These black pixels are generated during the process of offsetting the point cloud data according to the offset field and generating the second image based on the offset point cloud data. The electronic device can process the second image using traditional filling algorithms or other deep learning-based filling algorithms, such as inpainting networks. It can also perform mean filtering on the second image to fill in the black pixels, reducing the number of black pixels and thus improving the quality of the second image. The color-filled second image can then be used to generate video.

[0123] The method provided in this disclosure indicates that when the proportion of target pixels in the second image is greater than a preset proportion, it means that there are fewer black pixels in the second image, and the quality of the second image is higher. Therefore, a video containing the second image can be generated, thereby ensuring the quality of the generated video.

[0124] Furthermore, the method provided in this disclosure can reduce the offset field when the proportion of target pixels in the second image is not greater than a preset proportion, and offset the point cloud data according to the reduced offset field. A new second image is generated based on the offset point cloud data. The video containing the second image is only generated when the proportion of target pixels in the currently generated second image is greater than the preset proportion. This method of sequentially reducing the offset field can find the maximum offset field that meets the image quality requirements at the fastest speed, which not only improves the processing speed but also ensures the quality of the generated second image, thereby ensuring the quality of the generated video.

[0125] The method provided in this disclosure can adaptively generate video from images. Based on any input image, point cloud data is offset according to an offset field. A second image is generated based on the offset point cloud data. If the proportion of target pixels in the second image is not greater than a preset proportion, the offset field can be reduced. The point cloud data is offset according to the reduced offset field until the proportion of target pixels in the currently generated second image is greater than the preset proportion. Only then is a video containing the second image generated. Therefore, it can adaptively find an offset trajectory that meets the quality requirements of the generated second image. The generated video can present the effect of the image content gradually moving, forming a dynamic video picture and improving the viewing experience of the video.

[0126] Furthermore, the method provided in this disclosure can also generate multiple corresponding second images based on multiple offset fields, and then generate a video containing multiple second images. The multiple second images are arranged in the order of the offset fields. Therefore, the generated video can present the effect that the content of the first image gradually moves and the offset becomes larger or smaller, forming a dynamic video picture that reflects the offset trajectory of the first image.

[0127] Furthermore, in this embodiment, after generating a second image based on the offset point cloud data, the black pixels in the second image can be filled with a color different from black. This improves the quality of the second image, allowing for subsequent video generation using the color-filled second image, thereby enhancing the quality of the generated video.

[0128] Figure 3 This is a flowchart illustrating another video generation method according to an exemplary embodiment, such as... Figure 3 As shown, the method is performed by an electronic device and includes the following steps.

[0129] In step 301, the electronic device generates point cloud data based on the depth of the pixels in the first image.

[0130] Step 301 is the same as step 201 above, and will not be repeated here.

[0131] In step 302, the electronic device performs a lateral offset on the point cloud data by at least one lateral offset, and generates a second image based on the point cloud data obtained after the offset, until the proportion of target pixels in the currently generated second image is greater than a preset proportion.

[0132] The offset direction of the point cloud data includes horizontal and vertical directions. Accordingly, the offset field of the point cloud data includes at least one horizontal offset and at least one vertical offset, where the horizontal offset represents the length of the horizontal offset of the point cloud data and the vertical offset represents the length of the vertical offset of the point cloud data.

[0133] In this embodiment, the electronic device first performs a lateral offset on the point cloud data by at least one lateral offset, generates a second image based on the point cloud data obtained after the lateral offset, and determines whether the quality of the second image meets the requirements based on the proportion of target pixels in the currently generated second image. If the proportion of target pixels in the currently generated second image is greater than a preset proportion, it is determined that the lateral offset does not need to be adjusted, and the current lateral offset is the maximum offset that can meet the image quality requirements when performing a lateral offset on the point cloud data. Then, step 303 is executed to determine the vertical offset.

[0134] Alternatively, if the proportion of target pixels in the currently generated second image is not greater than the preset proportion, then the horizontal offset is further reduced, and the point cloud data is further offset according to the reduced horizontal offset until the proportion of target pixels in the currently generated second image is greater than the preset proportion. The horizontal offset adjustment is then determined to be complete. The current horizontal offset is the maximum offset that can meet the image quality requirements when the point cloud data is horizontally offset. Then, step 303 is executed to determine the vertical offset.

[0135] In this embodiment of the disclosure, this method of successively reducing the lateral offset can find the maximum lateral offset that meets the image quality requirements at the fastest speed, which not only improves the processing speed but also ensures the quality of the generated second image, thereby ensuring the quality of the generated video.

[0136] In step 303, while keeping the horizontal offset of the currently generated second image unchanged, the electronic device performs a vertical offset on the point cloud data corresponding to the currently generated second image according to at least one vertical offset, and generates a new second image based on the point cloud data obtained after the offset, until the proportion of target pixels in the currently generated second image is greater than a preset proportion.

[0137] In this embodiment, after determining the horizontal offset, the electronic device keeps the horizontal offset corresponding to the currently generated second image unchanged. The point cloud data corresponding to the currently generated second image is the point cloud data obtained by offsetting the point cloud data corresponding to the first image according to the horizontal offset. Then, the point cloud data corresponding to the currently generated second image is vertically offset according to at least one vertical offset. A new second image is generated based on the point cloud data obtained after the vertical offset. The quality of the second image is determined according to the proportion of target pixels in the new second image. If the proportion of target pixels in the new second image is greater than a preset proportion, it is determined that the vertical offset does not need to be adjusted, and the current vertical offset is the maximum offset that can meet the image quality requirements when vertically offsetting the point cloud data.

[0138] Alternatively, if the proportion of target pixels in the new second image is not greater than the preset proportion, the vertical offset is further reduced, and the point cloud data is further offset according to the reduced vertical offset until the proportion of target pixels in the currently generated second image is greater than the preset proportion. The vertical offset adjustment is then determined to be complete, and the current vertical offset is the maximum offset that can meet the image quality requirements when vertically offsetting the point cloud data.

[0139] In this embodiment of the disclosure, this method of successively reducing the vertical offset can find the maximum vertical offset that meets the image quality requirements at the fastest speed, which not only improves the processing speed but also ensures the quality of the generated second image, thereby ensuring the quality of the generated video.

[0140] For example, such as Figure 4 As shown, the electronic device acquires the depth of pixels in a first image, generates point cloud data based on the pixel depth, and then offsets the point cloud data by a preset offset (x, y), where x represents the horizontal offset and y represents the vertical offset. A second image is then generated based on the offset point cloud data. The electronic device performs a horizontal offset on the point cloud data by offset x, generates a second image based on the horizontally offset point cloud data, and determines whether the quality of the second image meets the requirements based on the proportion of the target pixels in the second image. If the proportion of the target pixels in the new second image is not greater than a preset proportion of 0.9, the offset x is reduced, and the horizontal offset on the point cloud data continues by the reduced offset x until the proportion of the target pixels in the currently generated second image is greater than the preset proportion of 0.9. At this point, the offset x is the maximum horizontal offset that meets the image quality requirements when horizontally offsetting the point cloud data. After determining this maximum horizontal offset x, the point cloud data is then vertically offset by offset y, and a second image is generated based on the vertically offset point cloud data. The quality of the second image is then determined based on the proportion of the target pixels in the second image. If the proportion of target pixels in the new second image is not greater than the preset proportion of 0.9, the offset y is reduced, and the point cloud data is continuously offset vertically according to the reduced offset y until the proportion of target pixels in the currently generated second image is greater than the preset proportion of 0.9. The offset y at this time is the maximum vertical offset that can meet the image quality requirements when offsetting the point cloud data vertically.

[0141] In step 304, the electronic device generates a video containing the second image.

[0142] Step 304 is the same as step 206 above, and will not be repeated here.

[0143] It should be noted that the above embodiment uses the example of performing a horizontal offset followed by a vertical offset to determine the most suitable horizontal and vertical offset amounts. In another embodiment, a vertical offset can be performed first, followed by a horizontal offset. That is, the electronic device first performs a vertical offset on the point cloud data by at least one vertical offset amount, generates a second image based on the point cloud data obtained after the vertical offset, and determines whether the quality of the second image meets the requirements based on the proportion of target pixels in the currently generated second image. If the proportion of target pixels in the currently generated second image is greater than a preset proportion, then it is determined that the vertical offset amount does not need to be adjusted, and the current vertical offset amount is the maximum vertical offset amount that can meet the image quality requirements when performing a vertical offset on the point cloud data. Alternatively, if the proportion of target pixels in the currently generated second image is not greater than a preset proportion, then the vertical offset amount is further reduced, and the point cloud data is continued to be offset according to the reduced vertical offset amount until the proportion of target pixels in the currently generated second image is greater than a preset proportion, at which point it is determined that the vertical offset adjustment is complete.

[0144] After determining the vertical offset, the electronic device keeps the vertical offset corresponding to the currently generated second image unchanged, and performs a horizontal offset on the point cloud data corresponding to the currently generated second image according to at least one horizontal offset. A new second image is generated based on the new point cloud data obtained after the horizontal offset, and the quality of the second image is determined according to the proportion of target pixels in the currently generated second image. If the proportion of target pixels in the currently generated second image is greater than a preset proportion, it is determined that the horizontal offset does not need to be adjusted, and the current horizontal offset is the maximum horizontal offset that can meet the image quality requirements when horizontally offsetting the point cloud data. Alternatively, if the proportion of target pixels in the currently generated second image is not greater than a preset proportion, the horizontal offset is further reduced, and the point cloud data is continued to be offset according to the reduced horizontal offset until the proportion of target pixels in the currently generated second image is greater than a preset proportion. At this point, the horizontal offset adjustment is determined to be complete, and step 304 above is executed.

[0145] The method provided in this disclosure offsets point cloud data according to both horizontal and vertical offsets. First, the point cloud data is offset by either the horizontal or vertical offset. A new second image is generated based on the offset point cloud data. This process continues until the proportion of target pixels in the currently generated second image is greater than a preset proportion. Once this is confirmed, the offset adjustment in that direction is complete. The offset in that direction is kept constant, and the offset in the other direction is adjusted until the proportion of target pixels in the currently generated second image is greater than a preset proportion. Only then is a video containing the second image generated. This method can find the maximum offset that meets the image quality requirements in the current direction at the fastest speed, without being affected by the offset in the other direction. Furthermore, it can fix the maximum offset in the current direction and quickly find the maximum offset that meets the image quality requirements in the other direction. This improves processing speed and ensures the quality of the generated second image, thereby ensuring the quality of the generated video.

[0146] Figure 5 This is a flowchart illustrating another video generation method according to an exemplary embodiment, such as... Figure 5 As shown, the method is performed by an electronic device and includes the following steps.

[0147] In step 501, the electronic device inputs the first image into the depth estimation model and obtains the depth map output by the depth estimation model.

[0148] The depth estimation model is used to obtain the depth of pixels in an image. Any image is input into the depth estimation model, and the model outputs a depth map. The depth map represents the depth of pixels in the image; it is an image where the depth of each pixel is used as its pixel value. In the depth map, brighter pixels have smaller depths, meaning they are closer to the camera's imaging plane, while darker pixels have larger depths, meaning they are farther from the camera's imaging plane.

[0149] In the embodiments disclosed herein, such as Figure 6 As shown, the electronic device inputs the first image into the depth estimation model, the depth estimation model obtains the depth of the pixels in the first image, and uses the depth of the pixels as the pixel value of the depth map to generate the depth map of the first image.

[0150] In some embodiments, the depth estimation model can be the depth estimation model proposed in Deep Lens (a shallow depth-of-field rendering algorithm), and the model can be accelerated using OpenVino (Open Visual Inference Neural Network Optimization, an open-source toolkit for model optimization and inference acceleration in deep neural networks), thereby improving the training and processing speed of the depth estimation model.

[0151] In some embodiments, after the electronic device acquires the depth map of the first image, the edges of the acquired depth map are inaccurate, containing discrete edge points. Therefore, the point cloud data generated based on these discrete edge points is also discrete, resulting in black pixels in the subsequently generated second image, affecting its quality. Furthermore, the edge boundaries of the depth map output by the depth estimation model are not clear, causing subsequent offsetting of the corresponding point cloud data to fail to completely shift the edges of the depth map, further impacting the quality of the generated second image. Therefore, filtering the depth map can improve its quality.

[0152] For example, after the electronic device acquires the depth map of the first image, it performs filtering processing on the depth map. The filtering processing includes at least one of the following: weighted mean filtering and joint bilateral filtering.

[0153] In some embodiments, after acquiring the depth map of the first image, the electronic device performs weighted mean filtering on the depth map. Weighted mean filtering is achieved by assigning different weights to the pixels in the weighted mean filtering template and then averaging the pixels in the depth map. Applying weighted mean filtering to the depth map optimizes its edges, making them sharper and more complete, and reducing the impact of discrete edge points in the depth map on subsequent point cloud data generation and the generation of the second image.

[0154] In some embodiments, after acquiring the depth map of the first image, the electronic device performs joint bilateral filtering on the depth map. Joint bilateral filtering is an edge-preserving filtering method that calculates a weighted average of pixels in the depth map with their surrounding pixels, achieving smoothing of the depth map's interior while preserving its edges.

[0155] In some embodiments, after acquiring the depth map of the first image, the electronic device first performs weighted mean filtering on the depth map, and then performs joint bilateral filtering, or first performs joint bilateral filtering, and then performs weighted mean filtering. This joint filtering method not only makes the edges of the depth map sharper and more complete, but also makes the interior of the depth map more continuous and smooth, improving the quality of the depth map. This is beneficial for subsequent generation of point cloud data and the generation of a second image based on the point cloud data, and also helps improve the quality of the second image.

[0156] For example, such as Figure 7 As shown, after the electronic device inputs the first image into the depth estimation model, it obtains the depth map of the first image generated by the depth estimation model. Then, it performs a weighted mean filter on the depth map of the first image to obtain a weighted mean filtered depth map. Finally, it performs a joint bilateral filter to obtain a joint bilateral filtered depth map. After these two filtering processes, the edges of the depth map are sharper and more complete, and the interior of the depth map is more continuous and smooth, thus improving its quality. For example, as... Figure 8 As shown, the left side is the original depth map of an image, and the right side is the depth map after the original depth map has been processed by weighted mean filtering and joint bilateral filtering. It can be seen that the edges of the depth map on the right are sharper and the interior is smoother.

[0157] Furthermore, because the edges of the depth map are sharper and more complete after the two filtering processes described above, the point cloud data corresponding to the edges of the depth map can be completely offset during subsequent offsetting of the point cloud data, thus improving the quality of the second image generated based on the offset point cloud data. For example, as... Figure 9 As shown, point cloud data is generated based on the original depth map, then the point cloud data is offset, and the point cloud data obtained after offset is used to generate the following... Figure 9 The second image shown on the left in the image is generated by filtering the depth map and then performing similar operations. Figure 9 The second image shown on the right shows that in the left image, the edge of the depth map is incomplete, resulting in edge discontinuity, while the right image does not show edge discontinuity.

[0158] In some embodiments, the resolution of the first image is reduced when obtaining the depth map of the first image through the depth estimation model to improve the speed of the depth map generation by the depth estimation model. Therefore, before filtering the depth map, interpolation algorithms such as bilinear interpolation and bicubic interpolation are used to interpolate the depth map to obtain a depth map with the original resolution. However, because the depth map contains discrete edge points, these discrete edge points become even more discrete after interpolation. The point cloud data generated based on these more discrete edge points is also discrete, and the second image generated based on the discrete point cloud data contains black pixels, affecting the quality of the generated second image. For example, after interpolating the depth map of an image, point cloud data is generated based on the depth map. The point cloud data is then offset according to an offset field, and a point cloud data is generated based on the offset point cloud data, such as... Figure 10 The second image shown contains a large number of black pixels, indicating poor image quality. Therefore, after interpolating the depth map, filtering can be applied to improve its quality, facilitating the subsequent generation of point cloud data based on the depth map and the generation of the second image from the point cloud data.

[0159] In step 502, the electronic device generates point cloud data based on the depth map.

[0160] In this embodiment of the disclosure, the electronic device acquires the pixel value of each pixel in the depth map. The pixel value represents the depth of the pixel at the same position in the first image. Therefore, based on the depth of each pixel in the first image, each pixel can be converted into a point in three-dimensional space. Thus, multiple pixels in the first image can be converted to obtain point cloud data.

[0161] In some embodiments, the electronic device generates point cloud data based on the filtered depth map. The filtered depth map has sharper and more complete edges, and its interior is more continuous and smoother, improving the quality of the depth map and facilitating the subsequent generation of point cloud data based on it.

[0162] In step 503, the electronic device offsets the point cloud data according to a preset offset field, and generates a second image based on the offset point cloud data.

[0163] In step 504, the electronic device generates a video containing the second image.

[0164] Steps 503-504 are the same as steps 102-103 above, and will not be repeated here.

[0165] The method provided in this disclosure generates point cloud data based on the depth map output by a depth estimation model. This allows for convenient extraction of pixel values ​​from the depth map to obtain the depth of pixels in the first image, improving the efficiency of point cloud data generation. Furthermore, instead of directly offsetting pixels at the pixel level, the point cloud data is offset, enabling the generation of a second image based on the offset point cloud data. This avoids a large number of black pixels in the second image, improving its quality and consequently enhancing the quality of the generated video.

[0166] Furthermore, in this embodiment, a depth map of the first image is obtained through a depth estimation model, and point cloud data is generated based on the depth map. Therefore, the depth map can intuitively represent the depth of pixels in the first image. When generating point cloud data based on the depth map, the depth of pixels in the first image can be easily obtained by extracting pixel values ​​from the depth map, facilitating the subsequent generation of point cloud data based on the depth of pixels in the first image.

[0167] In addition, the method provided in this disclosure can accelerate the depth estimation model using OpenVino, thereby improving the processing speed of the depth estimation model. After acceleration, the depth estimation model can achieve a processing speed of less than 20 milliseconds on a GPU (Graphics Processing Unit).

[0168] In addition, after obtaining the depth map of the first image, this embodiment of the present disclosure can also perform weighted mean filtering and joint bilateral filtering on the depth map. After the two filtering processes, the edges of the depth map are sharper and more complete, and the interior of the depth map is more continuous and smooth, which improves the quality of the depth map and facilitates the subsequent generation of point cloud data based on the depth map and the generation of the second image based on the point cloud data.

[0169] This application applies to electronic devices, which may be terminals or servers, or other electronic devices. Terminals may be laptops, mobile phones, tablets, etc. Servers may be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0170] This disclosure also provides an implementation environment, such as... Figure 11 As shown. The implementation environment includes electronic device 1101 and server 1102, which are connected via wired or wireless network.

[0171] After a user inputs a first image on electronic device 1101, electronic device 1101 sends the first image to server 1102. Server 1102 generates a video using the method provided in this embodiment and returns the video to electronic device 1101, which then plays the video. Alternatively, server 1102 publishes the video based on the account logged into electronic device 1101 for other users to watch.

[0172] In some embodiments, the electronic device 1101 is equipped with a target application, which is associated with and provided with services by the server 1102. The target application has video sharing or video playback functions, and also provides a one-click video generation function. When a user clicks the one-click video generation control in the target application and inputs a first image, the electronic device 1101 sends the first image to the server 1102. The server 1102 automatically generates the video using the method provided in this embodiment and returns the video to the target application for playback. Alternatively, the server 1102 can publish the video based on the account logged into the target application for other users to view.

[0173] Figure 12 This is a structural block diagram illustrating a video generation apparatus according to an exemplary embodiment. See also... Figure 12 The device includes:

[0174] The point cloud generation unit 1201 is configured to generate point cloud data based on the depth of pixels in the first image, and the point cloud data is used to represent the spatial position of pixels in the first image.

[0175] The image generation unit 1202 is configured to offset the point cloud data according to a preset offset field, and generate a second image based on the offset point cloud data.

[0176] The video generation unit 1203 is configured to generate a video containing a second image.

[0177] In some embodiments, the image generation unit 1202 includes:

[0178] The image generation subunit is configured to offset the point cloud data according to the offset field, generate a second image based on the offset point cloud data, and continue until the proportion of target pixels in the currently generated second image is greater than a preset proportion, triggering the video generation unit to execute the step of generating a video containing the second image. The target pixels are pixels other than black pixels.

[0179] In some embodiments, the image generation subunit is configured as follows:

[0180] The point cloud data is offset according to the offset field, and a second image is generated based on the offset point cloud data.

[0181] If the proportion of the target pixels in the second image is not greater than a preset proportion, the offset field is reduced.

[0182] The point cloud data is offset according to the reduced offset field, and a new second image is generated based on the offset point cloud data. This process continues until the proportion of target pixels in the currently generated second image is greater than a preset proportion, triggering the video generation unit to execute the step of generating a video containing the second image.

[0183] In some embodiments, the offset field includes at least one lateral offset and at least one longitudinal offset, and the image generation subunit is configured to:

[0184] The point cloud data is horizontally offset by at least one horizontal offset, and a second image is generated based on the point cloud data obtained after offset, until the proportion of target pixels in the currently generated second image is greater than a preset proportion.

[0185] While keeping the horizontal offset of the currently generated second image unchanged, the point cloud data corresponding to the currently generated second image is vertically offset by at least one vertical offset. A new second image is generated based on the point cloud data obtained after the offset, until the proportion of target pixels in the currently generated second image is greater than a preset proportion, triggering the video generation unit to execute the step of generating a video containing the second image.

[0186] In some embodiments, the offset field includes at least one lateral offset and at least one longitudinal offset, and the image generation subunit is configured to:

[0187] The point cloud data is vertically offset by at least one vertical offset, and a second image is generated based on the point cloud data obtained after offset, until the proportion of target pixels in the currently generated second image is greater than a preset proportion.

[0188] While keeping the vertical offset of the currently generated second image unchanged, the point cloud data corresponding to the currently generated second image is horizontally offset by at least one horizontal offset. A new second image is generated based on the point cloud data obtained after the offset, until the proportion of target pixels in the currently generated second image is greater than a preset proportion, triggering the video generation unit to execute the step of generating a video containing the second image.

[0189] In some embodiments, the image generation unit 1202 is configured to offset point cloud data according to multiple offset fields, and generate multiple second images based on the offset point cloud data respectively.

[0190] The video generation unit 1203 is configured to generate a video containing multiple second images.

[0191] In some embodiments, the point cloud generation unit 1201 includes:

[0192] The acquisition subunit is configured to acquire a depth map of a first image, the depth map being used to represent the depth of pixels in the first image;

[0193] The point cloud generation subunit is configured to generate point cloud data based on the depth map.

[0194] In some embodiments, the acquisition subunit is configured to input a first image into a depth estimation model and acquire a depth map output by the depth estimation model.

[0195] In some embodiments, the apparatus further includes: a filtering unit configured to perform filtering processing on the depth map, the filtering processing including at least one of the following: weighted mean filtering, joint bilateral filtering;

[0196] The point cloud generation subunit is configured to generate point cloud data based on the filtered depth map.

[0197] In some embodiments, the apparatus further includes:

[0198] The fill unit is configured to fill black pixels in the second image with a color different from black.

[0199] The apparatus provided in this embodiment first converts the pixels of the first image into point cloud data, and then offsets the point cloud data, instead of directly offsetting the pixels at the pixel level. This allows the generation of a second image based on the offset point cloud data, avoiding a large number of black pixels in the second image, improving the quality of the second image, and thus improving the quality of the generated video.

[0200] Figure 13 This is a structural block diagram of an electronic device according to an exemplary embodiment. In some embodiments, the electronic device 1300 includes a desktop computer, a laptop computer, a tablet computer, a smartphone, or other electronic devices. The electronic device 1300 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0201] Typically, electronic device 1300 includes a processor 1301 and a memory 1302.

[0202] In some embodiments, processor 1301 includes one or more processing cores, such as a quad-core processor, an octa-core processor, etc. In some embodiments, processor 1301 is implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). In some embodiments, processor 1301 also includes a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1301 integrates a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1301 also includes an AI (Artificial Intelligence) processor, which handles computational operations related to machine learning.

[0203] In some embodiments, memory 1302 includes one or more computer-readable storage media that are non-transitory. In some embodiments, memory 1302 also includes high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in memory 1302 is used to store executable instructions that are executed by processor 1301 to implement the video generation method provided in the method embodiments of this disclosure.

[0204] In some embodiments, the electronic device 1300 may optionally include a peripheral device interface 1303 and at least one peripheral device. In some embodiments, the processor 1301, memory 1302, and peripheral device interface 1303 are connected via a bus or signal line. In some embodiments, each peripheral device is connected to the peripheral device interface 1303 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 1304, a display screen 1305, a camera assembly 1306, an audio circuit 1307, a positioning assembly 1308, and a power supply 1309.

[0205] Peripheral interface 1303 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1301 and memory 1302. In some embodiments, processor 1301, memory 1302 and peripheral interface 1303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1301, memory 1302 and peripheral interface 1303 are implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0206] The radio frequency (RF) circuit 1304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1304 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1304 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. In some embodiments, the RF circuit 1304 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. In some embodiments, the RF circuit 1304 communicates with other electronic devices via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1304 also includes circuitry related to NFC (Near Field Communication), which is not limited herein.

[0207] Display screen 1305 is used to display a UI (User Interface). In some embodiments, the UI includes graphics, text, icons, videos, and any combination thereof. When display screen 1305 is a touch display screen, display screen 1305 also has the ability to collect touch signals on or above the surface of display screen 1305. In some embodiments, the touch signals are input as control signals to processor 1301 for processing. At this time, display screen 1305 is also used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, there is one display screen 1305, disposed on the front panel of electronic device 1300; in other embodiments, there are at least two display screens 1305, disposed on different surfaces of electronic device 1300 or in a folded design; in other embodiments, display screen 1305 is a flexible display screen, disposed on a curved surface or folded surface of electronic device 1300. Furthermore, display screen 1305 may also be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. In some embodiments, the display screen 1305 is made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0208] The camera assembly 1306 is used to acquire images or videos. In some embodiments, the camera assembly 1306 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the electronic device, and the rear-facing camera is located on the back of the electronic device. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1306 also includes a flash. In some embodiments, the flash is a single-color temperature flash, and in some embodiments, the flash is a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, used for light compensation at different color temperatures.

[0209] In some embodiments, the audio circuit 1307 includes a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input to the processor 1301 for processing, or input to the radio frequency circuit 1304 to achieve voice communication. For stereo acquisition or noise reduction purposes, in some embodiments, there are multiple microphones, each located at a different part of the electronic device 1300. In some embodiments, the microphone is an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert electrical signals from the processor 1301 or the radio frequency circuit 1304 into sound waves. In some embodiments, the speaker is a conventional diaphragm speaker; in some embodiments, the speaker is a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into sound waves that humans can hear, but also into sound waves that humans cannot hear for purposes such as distance measurement. In some embodiments, the audio circuit 1307 also includes a headphone jack.

[0210] The positioning component 1308 is used to locate the current geographical location of the electronic device 1300 to enable navigation or LBS (Location Based Service). In some embodiments, the positioning component 1308 is a positioning component based on the US GPS (Global Positioning System), China's BeiDou system, Russia's Granas positioning system, or the European Union's Galileo system.

[0211] Power supply 1309 is used to supply power to various components in electronic device 1300. In some embodiments, power supply 1309 is alternating current, direct current, a disposable battery, or a rechargeable battery. When power supply 1309 includes a rechargeable battery, the rechargeable battery is a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery is also used to support fast charging technology.

[0212] In some embodiments, the electronic device 1300 further includes one or more sensors 1310. The one or more sensors 1310 include, but are not limited to, an acceleration sensor 1311, a gyroscope sensor 1312, a pressure sensor 1313, an optical sensor 1314, and a proximity sensor 1315.

[0213] In some embodiments, the accelerometer 1311 detects the magnitude of acceleration along the three coordinate axes of a coordinate system established by the electronic device 1300. For example, the accelerometer 1311 is used to detect the components of gravitational acceleration along the three coordinate axes. In some embodiments, the processor 1301 controls the display screen 1305 to display the user interface in a landscape or portrait view based on the gravitational acceleration signal acquired by the accelerometer 1311. In some embodiments, the accelerometer 1311 is also used for acquiring game or user motion data.

[0214] In some embodiments, the gyroscope sensor 1312 detects the orientation and rotation angle of the electronic device 1300. The gyroscope sensor 1312 and the accelerometer sensor 1311 work together to acquire the user's 3D movements on the electronic device 1300. Based on the data acquired by the gyroscope sensor 1312, the processor 1301 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0215] In some embodiments, the pressure sensor 1313 is disposed on the side bezel of the electronic device 1300 and / or on the lower layer of the display screen 1305. When the pressure sensor 1313 is disposed on the side bezel of the electronic device 1300, it can detect the user's grip signal on the electronic device 1300, and the processor 1301 performs left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1313. When the pressure sensor 1313 is disposed on the lower layer of the display screen 1305, the processor 1301 controls the operable controls on the UI interface based on the user's pressure operation on the display screen 1305. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0216] Optical sensor 1314 is used to collect ambient light intensity. In one embodiment, processor 1301 controls the display brightness of display screen 1305 based on the ambient light intensity collected by optical sensor 1314. Specifically, when the ambient light intensity is high, the display brightness of display screen 1305 is increased; when the ambient light intensity is low, the display brightness of display screen 1305 is decreased. In another embodiment, processor 1301 also dynamically adjusts the shooting parameters of camera assembly 1306 based on the ambient light intensity collected by optical sensor 1314.

[0217] The proximity sensor 1315, also known as a distance sensor, is typically located on the front panel of the electronic device 1300. The proximity sensor 1315 is used to detect the distance between the user and the front of the electronic device 1300. In one embodiment, when the proximity sensor 1315 detects that the distance between the user and the front of the electronic device 1300 is gradually decreasing, the processor 1301 controls the display screen 1305 to switch from a screen-on state to a screen-off state; when the proximity sensor 1315 detects that the distance between the user and the front of the electronic device 1300 is gradually increasing, the processor 1301 controls the display screen 1305 to switch from a screen-off state to a screen-on state.

[0218] Those skilled in the art will understand that Figure 13 The structure shown does not constitute a limitation on the electronic device 1300, which may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0219] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, which can be executed by a processor of an electronic device to perform the virtual image display method in the above method embodiments. In some embodiments, the computer-readable storage medium may be ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage device, etc.

[0220] In an exemplary embodiment, a computer program product is also provided, which includes a computer program that, when executed by a processor, implements the video generation method described in the above method embodiments.

[0221] In some embodiments, the computer program involved in this disclosure may be deployed on an electronic device for execution, or executed on multiple electronic devices located in one location, or executed on multiple electronic devices distributed in multiple locations and interconnected through a communication network. Multiple electronic devices distributed in multiple locations and interconnected through a communication network may constitute a blockchain system.

[0222] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the foregoing claims.

[0223] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method of video generation, the method comprising: The method comprises: generating point cloud data based on the depth of the pixel points in the first image, the point cloud data being used to represent the spatial positions of the pixel points in the first image; offsetting the point cloud data according to a preset offset field, and generating a second image based on the point cloud data obtained after the offsetting; in a case where the proportion of target pixel points in the second image is not greater than a preset proportion, reducing the offset field, the target pixel points being pixel points other than black pixel points; offsetting the point cloud data according to the offset field after the reduction, and generating a new second image based on the point cloud data obtained after the offsetting, until the proportion of the target pixel points in the current generated second image is greater than the preset proportion, and a step of generating a video containing the second image is performed.

2. The method of claim 1, wherein, The offset field comprises at least one lateral offset and at least one longitudinal offset, and the method further comprises: laterally offsetting the point cloud data according to the at least one lateral offset, generating a second image based on the point cloud data obtained after the lateral offsetting, until the proportion of target pixel points in the current generated second image is greater than the preset proportion; in a case where the lateral offset corresponding to the current generated second image remains unchanged, longitudinally offsetting the point cloud data corresponding to the current generated second image according to the at least one longitudinal offset, generating a new second image based on the point cloud data obtained after the longitudinal offsetting, until the proportion of the target pixel points in the current generated second image is greater than the preset proportion, and the step of generating the video containing the second image is performed.

3. The method of claim 1, wherein, The offset field comprises at least one lateral offset and at least one longitudinal offset, and the method further comprises: longitudinally offsetting the point cloud data according to the at least one longitudinal offset, generating a second image based on the point cloud data obtained after the longitudinal offsetting, until the proportion of target pixel points in the current generated second image is greater than the preset proportion; in a case where the longitudinal offset corresponding to the current generated second image remains unchanged, laterally offsetting the point cloud data corresponding to the current generated second image according to the at least one lateral offset, generating a new second image based on the point cloud data obtained after the lateral offsetting, until the proportion of the target pixel points in the current generated second image is greater than the preset proportion, and the step of generating the video containing the second image is performed.

4. The method of claim 1, wherein, The offsetting the point cloud data according to the preset offset field and generating a second image based on the point cloud data obtained after the offsetting comprises: offsetting the point cloud data according to a plurality of offset fields respectively, and generating a plurality of second images based on the point cloud data obtained after the offsetting respectively; The generating a video containing the second image comprises: generating a video containing the plurality of second images.

5. The method of claim 1, wherein, The generating point cloud data based on the depth of the pixel points in the first image comprises: obtaining a depth map of the first image, the depth map being used to represent the depth of the pixel points in the first image; generating the point cloud data based on the depth map.

6. The method of claim 5, wherein, The obtaining the depth map of the first image comprises: inputting the first image into a depth estimation model, and obtaining the depth map output by the depth estimation model.

7. The method of claim 5, wherein, After the depth map of the first image is acquired, the method further includes: filtering the depth map, the filtering including at least one of weighted average filtering and joint bilateral filtering; generating the point cloud data based on the depth map includes: generating the point cloud data based on the filtered depth map.

8. The method according to any one of claims 1 to 7, characterized in that, After the point cloud data is offset according to the preset offset field and the second image is generated based on the point cloud data obtained after the offset, the method further includes: filling a color for a black pixel point in the second image, the filled color being different from black.

9. A video generating apparatus characterized by comprising: including: a point cloud generation unit configured to generate point cloud data based on depths of pixel points in a first image, the point cloud data being used to represent spatial positions of the pixel points in the first image; an image generation unit configured to offset the point cloud data according to a preset offset field, and generate a second image based on the point cloud data obtained after the offset; in a case where a proportion of target pixel points in the second image is not greater than a preset proportion, the target pixel points being pixel points other than black pixel points, reducing the offset field, offsetting the point cloud data according to the reduced offset field, and generating a new second image based on the point cloud data obtained after the offset, until the proportion of the target pixel points in the currently generated second image is greater than the preset proportion, triggering a video generation unit to perform a step of generating a video containing the second image.

10. An electronic device, comprising: including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the video generation method according to any one of claims 1 to 8.

11. A computer readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the video generation method according to any one of claims 1 to 8.

Images