A video generation method, device, equipment and storage medium
By comparing the average depth values on the depth image, determining the accuracy of the depth information and rendering the video, the problem of inaccurate depth information under occlusion or complex geometric structures of the monocular depth estimation algorithm is solved, and the visual effect of the video is significantly improved.
Patent Information
- Application Number
- CN202210032871.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-12
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-01-12
AI Technical Summary
Monocular depth estimation algorithm is difficult to accurately identify when occluded or complex geometric structures exist in images, resulting in inaccurate depth information and affecting the visual effect of the video.
By comparing the average depth values of the first set of pixels and the second set of pixels on the depth image, it is determined whether the object is a target object, thereby determining the accuracy of the depth information, and rendering the video based on this to improve the visual effect.
Ensure the accuracy of depth information, avoid the sense of separation between the target object and the background, and significantly improve the visual effect of the video.
Smart Images

Figure CN114387326B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of cloud computing technology, and in particular to a method, apparatus, device and storage medium for generating a video. Background Art
[0002] With the maturity of electronic technology, more digital products have been produced. The popularity and popularization of digital products such as digital cameras, tablet computers, and smart phones have brought great convenience to the general public. In daily life, people use digital products to take photos or record videos, which can be used to record life trajectories and share daily life. Video production is widely used in television, movies, and communication fields such as exhibitions, surveys, medical treatment, and games.
[0003] Video production adds the output of depth information on the basis of images, which can give people a unique sense of three-dimensionality and depth of field, and bring people a strong visual impact. The increase of depth information is usually achieved through a monocular depth estimation algorithm.
[0004] However, due to occlusions or complex geometric structures that are easy to appear in images, monocular depth estimation has difficulty in identifying these situations, which may cause the monocular depth estimation algorithm to have an erroneous understanding of the image information and incorrect object recognition, resulting in inaccurate depth information of the acquired monocular depth image, thereby resulting in poor visual effects of the video generated based on the monocular depth image with inaccurate depth information. Summary of the invention
[0005] The embodiments of the present application provide a video generation method, apparatus, device and storage medium, which are used to determine whether an object formed by a first set of pixel points on a depth image is a target object by comparing an average depth value of a first set of pixel points on a depth image with an average depth value of a second set of pixel points, thereby determining whether the depth information of the depth image is accurate, so as to render a video with accurate content information, thereby improving the visual effect of the video.
[0006] The present application provides a method for generating a video, including:
[0007] Obtain an original image and a depth image corresponding to the original image, wherein each pixel in the depth image has a corresponding depth value;
[0008] Acquire a first bounding box and an expanded search range corresponding to the first bounding box from the original image, wherein the first bounding box includes each pixel point of the target object;
[0009] Searching for a micro-motion area from the expanded search range according to a pixel condition, wherein a first bounding box is placed in the micro-motion area;
[0010] Generate a second bounding box according to the position of the slightly-moved area on the original image, wherein the slightly-moved area is placed in the second bounding box;
[0011] Determine a first average depth value of the slightly-moved area and a second average depth value of the area selected by the second bounding box according to the depth value of each pixel in the depth image;
[0012] If the first average depth value is less than the second average depth value, rendering is performed based on the original image, the micro-motion area, and the depth image to obtain a video.
[0013] On the other hand, the present application provides a video generation device, including:
[0014] An acquisition unit, used to acquire an original image and a depth image corresponding to the original image, wherein each pixel in the depth image has a corresponding depth value;
[0015] An acquisition unit, configured to acquire a first bounding box and an expanded search range corresponding to the first bounding box from an original image, wherein the first bounding box includes each pixel point of the target object;
[0016] The processing unit is further used to search for a micro-motion area from the expanded search range according to the pixel condition, wherein the first bounding box is placed in the micro-motion area;
[0017] The processing unit is further used to generate a second bounding box according to the position of the slightly-moved area on the original image, wherein the slightly-moved area is placed in the second bounding box;
[0018] A determining unit, configured to determine a first average depth value of the micro-motion area and a second average depth value of the area selected by the second bounding box according to the depth value of each pixel in the depth image;
[0019] The processing unit is further configured to render a video based on the original image, the micro-motion area, and the depth image if the first average depth value is less than the second average depth value.
[0020] In a possible design, in an implementation of another aspect of the embodiment of the present application, the processing unit may be specifically used for:
[0021] Perform image segmentation processing on the original image to obtain the target object;
[0022] According to the position of the target object on the original image, a border regression process is performed on the target object to obtain a first bounding box;
[0023] An expansion search range is determined according to the position of the first bounding box on the original image and a first preset expansion size.
[0024] In a possible design, in an implementation of another aspect of the embodiment of the present application, the processing unit may be specifically used for:
[0025] Perform a color score on each pixel on the original image to obtain a pixel score for each pixel, where the pixel score is positively correlated with the brightness value of each pixel;
[0026] Pixel points whose pixel scores meet a preset score are searched from the expanded search range to generate a micro-motion area.
[0027] In a possible design, in an implementation of another aspect of the embodiment of the present application, the processing unit may be specifically used for:
[0028] Convert the original image into a grayscale image;
[0029] The grayscale value of each pixel on the grayscale image is processed by average precision mean to obtain the pixel score of each pixel.
[0030] In a possible design, in an implementation of another aspect of the embodiment of the present application, the processing unit may be specifically used for:
[0031] Perform image filtering on the original image to obtain the filtering value of each pixel;
[0032] The filter value of each pixel is normalized respectively to obtain the pixel score of each pixel.
[0033] In a possible design, in an implementation of another aspect of the embodiment of the present application, the determining unit may be specifically used to:
[0034] Acquire a first pixel point set corresponding to the micro-motion area and a second pixel point set corresponding to the second bounding box from the depth image;
[0035] A first average depth value is calculated according to the depth values of the pixels in the first pixel set, and a second average depth value is calculated according to the depth values of the pixels in the second pixel set.
[0036] In a possible design, in an implementation of another aspect of the embodiment of the present application, the processing unit may be specifically used for:
[0037] Construct a target three-dimensional grid image based on the original image, the micro-motion area and the depth image;
[0038] Generate a camera motion trajectory according to each frame template image in the preset video template, wherein the camera motion trajectory is used to indicate the camera position and orientation information corresponding to each frame template image taken by the camera;
[0039] The target 3D grid image and the camera motion trajectory are rendered to obtain a video.
[0040] In a possible design, in an implementation of another aspect of the embodiment of the present application, the processing unit may be specifically used for:
[0041] Downsampling the original image and the depth image respectively to generate a downsampled image of the original image and a downsampled image of the depth image;
[0042] Generate an initial three-dimensional grid image by performing spatial projection according to the position of each pixel point on the downsampled image of the depth image;
[0043] Performing upsampling processing on the initial three-dimensional grid image to generate an intermediate three-dimensional grid image;
[0044] The intermediate three-dimensional grid image is color-complemented according to the color information of the original image to obtain a target three-dimensional grid image.
[0045] In a possible design, in an implementation of another aspect of the embodiment of the present application, the processing unit may be specifically used for:
[0046] According to the position of each pixel point on the downsampled image of the depth image, each pixel point is spatially projected to generate a spatial vertex corresponding to each pixel point;
[0047] Connecting the spatial vertices to a basic three-dimensional mesh image according to the depth value of each pixel in the downsampled image of the depth image;
[0048] Extracting first color information of each pixel from the downsampled image of the original image;
[0049] According to the mapping relationship between the original image and the depth image, the basic three-dimensional grid image is color-complemented according to the first color information of each pixel to obtain an initial three-dimensional grid image.
[0050] In a possible design, in an implementation of another aspect of the embodiment of the present application, the processing unit may be specifically used for:
[0051] Extracting second color information of each pixel from the original image;
[0052] The color of each spatial vertex in the intermediate three-dimensional grid image is supplemented according to the second color information of each pixel point to obtain a target three-dimensional grid image.
[0053] In a possible design, in an implementation of another aspect of the embodiment of the present application, the processing unit may be specifically used for:
[0054] According to the playback time of the preset video template, the camera motion rate corresponding to each frame template image in the preset video template is obtained;
[0055] The camera motion velocity is interpolated to obtain the camera motion trajectory.
[0056] In a possible design, in an implementation of another aspect of the embodiment of the present application,
[0057] The processing unit is further used to perform depth estimation processing on the original image to obtain an initial depth image and depth information of the initial depth image;
[0058] The processing unit is further used to perform inverse function processing on the initial depth image and the depth information of the initial depth image to obtain a depth image.
[0059] On the other hand, the present application provides a computer device, including: a memory, a transceiver, a processor, and a bus system;
[0060] Wherein, the memory is used to store programs;
[0061] The processor is used to implement the above-mentioned methods when executing the program in the memory;
[0062] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.
[0063] Another aspect of the present application provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer is enabled to execute the above-mentioned methods.
[0064] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:
[0065] By acquiring an original image and a depth image corresponding to the original image, acquiring a first bounding box and an expanded search range corresponding to the first bounding box from the original image, and searching for a micro-motion area from the expanded search range according to a pixel condition, a second bounding box can be generated according to the position of the micro-motion area on the original image, and then, according to the depth value of each pixel in the depth image, a first average depth value of the micro-motion area and a second average depth value of the area selected by the second bounding box can be determined, and when the first average depth value is less than the second average depth value, a video can be rendered based on the original image and the depth image to obtain a video. In the above manner, it is possible to intelligently search for a micro-motion area suitable for the original image, and by comparing the first average depth value of the micro-motion area with the second average depth value of the second bounding box, it is possible to determine whether the object contained in the micro-motion area on the depth image is the target object, thereby determining whether the depth information of the depth image is accurate, and then, based on the original image and the depth image with accurate depth information and the micro-motion area suitable for micro-motion, rendering can be performed, not only a video with accurate content information can be obtained, but also a clear sense of separation between the target object and the background in the video can be avoided by rendering the micro-motion area, thereby improving the visual effect of the video. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] Figure 1 This is a schematic diagram of the architecture of the image data control system in the embodiment of the present application;
[0067] Figure 2 is a flow chart of an embodiment of a method for generating a video in an embodiment of the present application;
[0068] Figure 3 is another embodiment flow chart of the method for generating a video in an embodiment of the present application;
[0069] Figure 4 is another embodiment flow chart of the method for generating a video in an embodiment of the present application;
[0070] Figure 5 is another embodiment flow chart of the method for generating a video in an embodiment of the present application;
[0071] Figure 6 is another embodiment flow chart of the method for generating a video in an embodiment of the present application;
[0072] Figure 7 is another embodiment flow chart of the method for generating a video in an embodiment of the present application;
[0073] Figure 8 is another embodiment flow chart of the method for generating a video in an embodiment of the present application;
[0074] Fig. 9is another embodiment flow chart of the method for generating a video in an embodiment of the present application;
[0075] Fig.10 It is a schematic diagram of a video generation principle flow chart of a method for generating a video in an embodiment of the present application;
[0076] Fig.11 It is another schematic flow chart of the principle of video generation in the method for generating a video in an embodiment of the present application;
[0077] Fig.12 It is a schematic diagram of an interface for generating a video in a method for generating a video in an embodiment of the present application;
[0078] Fig.13 It is another schematic diagram of an interface for generating a video in the method for generating a video in an embodiment of the present application;
[0079] Fig.14 It is a schematic diagram of image segmentation generation of a method for generating a video in an embodiment of the present application;
[0080] Fig.15 is a schematic diagram of depth image generation of a method for generating a video in an embodiment of the present application;
[0081] Fig.16 is another schematic diagram of depth image generation in the method for generating a video in an embodiment of the present application;
[0082] Fig.17 is a schematic diagram of generating an initial three-dimensional grid image in a method for generating a video in an embodiment of the present application;
[0083] Fig.18 This is a schematic diagram of the principle of obtaining pixel scores in the method for generating a video in an embodiment of the present application;
[0084] Fig.19 This is a schematic diagram of the principle of obtaining a target three-dimensional grid image in a method for generating a video in an embodiment of the present application;
[0085] Fig. 20 It is a schematic diagram of a target three-dimensional grid image effect of the method for generating a video in an embodiment of the present application;
[0086] Fig.21 This is a schematic diagram of the principle of obtaining the camera motion trajectory in the method for generating a video in an embodiment of the present application;
[0087] Fig. 22 is a schematic diagram of an embodiment of a device for generating a video in an embodiment of the present application;
[0088] Fig.23 It is a schematic diagram of an embodiment of a computer device in an embodiment of the present application. DETAILED DESCRIPTION
[0089] The embodiments of the present application provide a video generation method, apparatus, device and storage medium, which are used to determine whether an object formed by a first set of pixel points on a depth image is a target object by comparing an average depth value of a first set of pixel points on a depth image with an average depth value of a second set of pixel points, thereby determining whether the depth information of the depth image is accurate, so as to render a video with accurate content information, thereby improving the visual effect of the video.
[0090] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0091] With the rapid development of information, cloud technology has gradually entered all aspects of people's lives. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the rapid development and application of the Internet industry, each item may have its own identification mark in the future, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing.
[0092] Among them, cloud security refers to the general term for security software, hardware, users, organizations, and security cloud platforms based on cloud computing business model applications. Cloud security integrates emerging technologies and concepts such as parallel processing, grid computing, and unknown virus behavior judgment. Through a large number of meshed clients, it monitors the abnormal behavior of software in the network, obtains the latest information of Trojans and malicious programs on the Internet, and sends it to the server for automatic analysis and processing, and then distributes virus and Trojan solutions to each client. The video generation method provided in the embodiment of the present application can be implemented through cloud computing technology and cloud security technology.
[0093] It should be understood that the video generation method provided in the present application can be applied to the scenario of generating videos through images, for example, generating advertising videos through advertising plane images. As another example, generating product introduction videos through product images. As yet another example, generating news information videos through news information images.
[0094] It is understandable that the present application proposes a method for generating a video, which is applied to Figure 1 Please refer to the image data control system shown in Figure 1 , Figure 1 FIG. 1 is a schematic diagram of an architecture of an image data control system in an embodiment of the present application. Figure 1 As shown, the server obtains the original image provided by the terminal device and the depth image corresponding to the original image, extracts the first bounding box and the second bounding box from the original image, and obtains the first pixel set corresponding to the first bounding box and the second pixel set corresponding to the second bounding box from the depth image. Then, the first average depth value can be determined according to the first pixel set, and the second average depth value can be determined according to the second pixel set. When the first average depth value is less than the second average depth value, the video can be rendered based on the original image and the depth image. In the above manner, it is possible to determine whether the object formed by the first pixel set on the depth image is the target object by comparing the average depth value of the first pixel set on the depth image with the average depth value of the second pixel set, so as to determine whether the depth information of the depth image is accurate. Then, rendering can be performed based on the original image and the depth image with accurate depth information to obtain a video with accurate content information, thereby improving the visual effect of the video.
[0095] Understandably, Figure 1 Only one type of terminal device is shown in the figure. In actual scenarios, more types of terminal devices may participate in the data processing process. Terminal devices include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, car terminals, etc. The specific number and type depend on the actual scenario and are not limited here. Figure 1 One server is shown in the figure, but in actual scenarios, multiple servers may also be involved, especially in the scenario of multi-model training interaction. The number of servers depends on the actual scenario and is not limited here.
[0096] It should be noted that in this embodiment, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal device and the server can be directly or indirectly connected through wired or wireless communication, and the terminal device and the server can be connected to form a blockchain network, which is not limited in this application.
[0097] In order to solve the above problems, the present application proposes a method for generating a video, which is generally executed by a server or a terminal device. Accordingly, a generating device applied to the video is generally arranged in the server or the terminal device.
[0098] It is understandable that, in the video generation method, apparatus, device and storage medium disclosed in this application, multiple servers or terminal devices can form a blockchain, and the server or terminal device is a node on the blockchain. In practical applications, data sharing between nodes may be required in the blockchain, and each node may store image data and video data, etc.
[0099] The following is an introduction to the method of generating the video in this application. Figure 2 In one embodiment of the video generation method of the present application, the method includes:
[0100] In step S101, an original image and a depth image corresponding to the original image are obtained, wherein each pixel in the depth image has a corresponding depth value;
[0101] In this embodiment, if Fig.10 As shown, when the image needs to be converted into a video, the server can obtain the original image according to the conversion requirements, and can perform depth estimation on the obtained original image to obtain a depth image with a corresponding depth value for each pixel, so that the corresponding video can be subsequently rendered based on the obtained original image and the depth image corresponding to the original image.
[0102] Among them, the original image can be specifically expressed as an actual product image, food image, advertising graphic image or character image, etc., or it can be an artificially designed anime character image, cartoon character or animal image, product sticker, clothing design image or advertising design image, etc., or other images, which are not specifically limited here. The depth image is an image obtained by performing depth estimation on the original image. There is a one-to-one correspondence between the pixels in the original image and the pixels in the depth image. Depth estimation is to form a depth image by obtaining the distance between the target object and the shooting point, also known as an optical flow map. The depth image records the parallax of the same target object between different images, and then the distance between the target object and the shooting point can be converted through the camera parameters and the position information between the two shooting points. The depth image can be used to display the brightness proportional to the distance between the target object and the shooting point. It can be understood that the surface of the target object closer to the shooting point is brighter, and other surfaces are lighter. The target object can be specifically expressed as a person or object in the original image.
[0103] Among them, the parallax is used to obtain the depth information of the target object, that is, the distance between the target object and the shooting point. The parallax is also called optical flow. For ease of understanding, this embodiment can be explained by the changes of the target object outside the window seen by the passenger in the moving vehicle. For example, when the passenger is in the vehicle and looks out of the window when the vehicle moves, the passenger can see that the target object in the distance changes slowly, while the target object in the near changes very quickly. That is, as the vehicle moves, it is assumed that the passengers in the car take a picture at two consecutive moments. If there are some identical target objects in the two images, then the differences in the pixel coordinates of these target objects in the two images are usually different. If the pixel coordinate difference of a target object in the two pictures is very large, then it can be judged that the target object is relatively close to the shooting point. If the pixel coordinate difference of a target object in the two images is small, then it can be inferred that the target object is relatively close to the shooting point. That is, the parallax generated by the closer target object is larger, and the parallax generated by the farther target object is smaller. Therefore, the difference in pixel coordinates of the same target object in different images can be understood as parallax.
[0104] Specifically, Fig.11As shown, when it is necessary to convert an image into a video, for example, when it is necessary to generate a corresponding advertising video from an advertising design drawing or a promotional drawing, the server can receive the original image uploaded by the user or designer through the terminal device, or receive a video generation request sent by the terminal device, and obtain the original image from the local database according to the image address or image identifier carried in the video generation request. The original image can also be obtained by other means, such as crawling the original image from the network, etc., which are not specifically limited here. Then, the depth of the obtained original image can be estimated to obtain a depth image, so that the three-dimensional scene can be better and more accurately restored based on the original image and the depth image that can truly reflect the distance between the target object and the shooting point, so that the video can be rendered based on the restored three-dimensional scene.
[0105] For example, suppose there is an original image of an idle wooden bed at home, and the depth estimation of the original image can be obtained as follows Fig.15 Alternatively, assuming that an original image is an advertisement image of a mobile phone, the depth estimation of the original image can be obtained as follows: Fig.15 The illustrated image can display a target object with obvious brightness contrast changes, and can reflect a depth image with different depth information between the target object and the background information.
[0106] In step S102, a first bounding box and an expanded search range corresponding to the first bounding box are obtained from the original image, wherein the first bounding box includes each pixel point of the target object;
[0107] In this embodiment, if Fig.10 As shown, after acquiring the original image, the target object in the original image can be identified, and a first bounding box can be calculated for the target object identified in the original image. Then, since the edge of the bounding box may be cut on a non-pure color area with drastic color changes when extracting the first bounding box from the original image, a crack may be generated between the edge of the first bounding box and the background information. In this embodiment, an expanded search range corresponding to the first bounding box can be obtained to find the area where the crack may be generated and perform corresponding processing to avoid an obvious sense of separation between the edge of the target object and the background information in the subsequently rendered video.
[0108] The first bounding box can be specifically expressed as a rectangular frame that frames the target object on the original image, and the rectangular frame includes various pixel points of the target object, which can be used to mark the position of the target object on the original image (such as coordinate values) and the attributes of the target object (such as object name). The expanded search range is the area expanded on the basis of the first bounding box, that is, the first bounding box is placed within the expanded search range, and the expanded search range is used to find areas where cracks may occur and need to be slightly moved.
[0109] Specifically, since the acquired original image is not necessarily a real-life image, such as a product map, which does not have a corresponding real-world counterpart, it is easy to cause inaccurate depth information of the depth image. Therefore, in order to better and more accurately determine whether the depth information of the acquired depth image is accurate, after acquiring the original image, the target object in the original image can be identified to obtain the target object and background information.
[0110] Furthermore, in order to acquire the target object more completely and accurately so that the target object can be rendered better later, a sliding window method can be used to frame a rectangular box that completely contains all the pixel points of the target object from the original image, that is, the first bounding box. Specifically, the sliding window of different window sizes can be slid from left to right and from top to bottom on the original image, and a classifier is executed on the current window during each sliding (the classifier is pre-trained, such as an SVM classifier). If the current window can obtain a classification probability greater than a preset probability threshold, it can be considered that the target object is detected. The preset probability threshold is set according to actual application requirements and is not specifically limited here. Then, after detecting each sliding window of different window sizes, the target object labels detected by different windows can be obtained. Since these window sizes may have a high degree of repetition, non-maximum suppression can be used. The window screening is performed by using the method of neural network suppression (NMS), and the finally screened window can be used as the first bounding box, and the object in the window can be used as the target object. The first bounding box can also be extracted by other methods, such as the selective search (Selective Search, SS) candidate area algorithm, the edge box (Edge Boxes, EB) search candidate area algorithm or the region generation network (Region Proposal Network, RPN) algorithm, etc., which are not specifically limited here. Furthermore, after obtaining the first bounding box, the first bounding box can be proportionally expanded according to a preset magnification factor to form an expanded search range, and other acquisition methods can also be used to obtain the expanded search range, such as pixel search or pixel expansion, etc., which are not specifically limited here.
[0111] In step S103, a micro-motion area is searched from the expanded search range according to the pixel condition, wherein the first bounding box is placed in the micro-motion area;
[0112] In this embodiment, when the first bounding box is extracted from the original image, cracks may occur between the edge of the first bounding box and the background information because the edge of the bounding box cuts into a non-pure color area with drastic color changes. In this embodiment, these areas where cracks may occur may require micro-motion rendering to avoid an obvious sense of separation between the edge of the target object and the background information in the subsequently rendered video. Therefore, in this embodiment, the area where cracks may occur, i.e., the micro-motion area, may be searched between the expanded search range and the area selected by the first bounding box according to pixel conditions. That is, the first bounding box is placed in the micro-motion area so that the micro-motion area can be rendered subsequently, so that the target object in the video can move slightly in the micro-motion area to reduce the obvious sense of separation between the edge of the target object and the background information, and to form a better stereoscopic visual effect.
[0113] The pixel condition may be specifically expressed as a pixel value less than or equal to a pixel threshold, or a pixel color score less than or equal to a score threshold, or a pixel brightness value less than or equal to a brightness threshold, or other pixel conditions, which are not specifically limited here. The micro-motion area may be specifically expressed as an area where the color change of the region boundary is small, and there is no clear sense of separation between the region boundary and the background information. The micro-motion area can present the micro-motion of the target object in the rendered video to form a more three-dimensional visual difference, thereby improving the visual effect of the video to a certain extent.
[0114] Specifically, after obtaining the first bounding box and the expanded search range corresponding to the first bounding box, pixel points that meet the pixel conditions can be searched between the expanded search range and the area selected by the first bounding box according to pixel conditions such as the pixel value being less than or equal to the pixel threshold. It can be understood that these pixel points are all suitable for micro-motion. Then, the area enclosed by the searched pixel points that meet the pixel conditions can be used as the micro-motion area. The micro-motion area can form a transition area between the target object in the first bounding box and the background outside the first bounding box, so as to reduce the sense of separation between the target object and the background in the generated video, thereby improving the quality of the video to a certain extent and improving the visual effect of the video.
[0115] In step S104, a second bounding box is generated according to the position of the slightly-moved area on the original image, wherein the slightly-moved area is placed in the second bounding box;
[0116] In this embodiment, since the acquired original image is not necessarily a real-life image, such as a product sticker, which does not have a corresponding real-world counterpart, this may easily lead to inaccurate depth information of the depth image. Therefore, in order to be able to better and more accurately determine whether the depth information of the acquired depth image is accurate, after acquiring the micro-motion area, a second bounding box can be generated based on the position of the micro-motion area on the original image, so that the depth information of the depth image can be subsequently determined based on the micro-motion area and the second bounding box. This can maintain the accuracy of the video content to a certain extent and improve the visual effect of the video.
[0117] The second bounding box may be specifically expressed as a rectangular frame including a number of pixels of part of the background information and each pixel of the target object, that is, the micro-motion area is placed in the second bounding box.
[0118] Specifically, after acquiring the micro-motion area, the micro-motion area can be expanded according to the preset expansion size based on the position of the micro-motion area on the original image to obtain a second bounding box. For example, by expanding 10 pixels around the position of the micro-motion area, a second bounding box containing the micro-motion area can be obtained, wherein the preset expansion size is set according to actual application requirements and can usually be set to 10 pixels, and no specific limitation is made here.
[0119] In step S105 , a first average depth value of the slightly-moved area and a second average depth value of the area selected by the second bounding box are determined according to the depth value of each pixel in the depth image.
[0120] In this embodiment, due to the one-to-one correspondence between the pixels on the original image and the depth image, a set of pixel points corresponding to the micro-motion area and a set of pixel points corresponding to the area selected by the second bounding box can be found in the depth image. Then, based on the depth value of each pixel point in the depth image, the first average depth value of the micro-motion area and the second average depth value of the area selected by the second bounding box can be calculated.
[0121] Furthermore, after obtaining the micro-motion area and the second bounding box on the original image, a set of pixel points corresponding to the micro-motion area can be obtained from the depth image according to the correspondence between the original image and the depth image, or the micro-motion area can be directly mapped to the depth image to obtain the area selected by the micro-motion area on the depth image. Since each pixel in the depth image has a depth value, the depth value of each pixel in the set of pixel points corresponding to the micro-motion area can be obtained on the depth image to calculate the first average depth value of the micro-motion area. Similarly, the set of pixel points corresponding to the second bounding box can be obtained from the depth image, or the second bounding box can be directly mapped to the depth image to obtain the area selected by the second bounding box on the depth image, and then the depth value of each pixel in the set of pixel points corresponding to the second bounding box can be obtained on the depth image to calculate the second average depth value of the second bounding box.
[0122] In step S106 , if the first average depth value is less than the second average depth value, rendering is performed based on the original image, the micro-motion area, and the depth image to obtain a video.
[0123] In this embodiment, after obtaining the first average depth value and the second average depth value, the first average depth value can be compared with the second average depth value. If the first average depth value is less than the second average depth value, it can be determined that the depth estimation is accurate. Rendering based on the original image and the depth image with accurate depth estimation can obtain a video with accurate content information, thereby maintaining the video quality to a certain extent and improving the visual effect of the video.
[0124] Specifically, Fig.10 As shown, when the first average depth value and the second average depth value are obtained, since the depth value can be used to reflect the real distance between the target object and the shooting point, it can be understood that, relative to the background information, the target object can be understood as the foreground information of the original image. In the real world, relative to the distance between the background information and the shooting point, the foreground information should be closer to the shooting point, that is, the depth information of the target object should be smaller than the depth information of the background information. Only then can the depth information of the depth image truly reflect the original image, and then the depth estimation is accurate. Therefore, the first average depth value that can reflect the depth information of the target object can be compared with the second average depth value that can reflect the depth information of the background information around the target object. If the first average depth value is smaller than the second average depth value, that is, the depth information of the target object is smaller than the depth information of the background information around the target object, it can be determined that the depth information in the depth image of the acquired original image is accurate, so the video can be rendered based on the original image, the micro-motion area and the depth image. For example, Fig.12 The original image shown is a food image of fried chicken, which can be rendered into a fried chicken video, or, as shown in Fig.13 The original image shown is a takeaway information image of a takeaway platform, which can be rendered to obtain a takeaway information video.
[0125] Furthermore, if the first average depth value is greater than the second average depth value, that is, the depth information of the target object is greater than the depth information of the background information around the target object, it can be understood that there is an obvious depth difference of "small outside and large inside" near the boundary of the target object, which can easily lead to a crack between the target object and the background information, or the depth estimation identifies the target object as background information, and there is a recognition error. Therefore, the depth image can be filtered so that the original image can be processed accordingly and the depth estimation can be re-performed to obtain a depth image with relatively accurate depth information.
[0126] In an embodiment of the present application, a method for generating a video is provided. Through the above-mentioned method, by comparing the average depth value of the first pixel point set on the depth image and the average depth value of the second pixel point set on the depth image, it is possible to determine whether the object formed by the first pixel point set on the depth image is the target object, thereby determining whether the depth information of the depth image is accurate. Then, rendering can be performed based on the original image and the depth image with accurate depth information to obtain a video with accurate content information, thereby improving the visual effect of the video.
[0127] Optionally, in the above Figure 2 Based on the corresponding embodiment, in another optional embodiment of the method for generating a video provided in the embodiment of the present application, as Figure 3 As shown, extracting a first bounding box and an expanded search range corresponding to the first bounding box from the original image includes:
[0128] In step S301, the original image is segmented to obtain the target object;
[0129] In step S302, according to the position of the target object on the original image, a frame regression process is performed on the target object to obtain a first bounding box;
[0130] In step S303, an expansion search range is determined according to the position of the first bounding box on the original image and a first preset expansion size.
[0131] In this embodiment, when the original image is acquired, foreground information and background information in the original image can be acquired, wherein the foreground information includes the target object and text information (such as a title, price, or logo, etc.). In order to accurately acquire the target object, the original image can be subjected to image segmentation processing to accurately segment the target object and the background information. After acquiring the target object, the target object can be accurately marked on the original image by using borders or windows of different sizes, and then border regression can be performed on borders or windows of different sizes to screen out a border that is closest to the actual size of the target object on the original image, i.e., a first bounding box. Then, starting from the area selected by the first bounding box, an expanded search range can be determined according to a first preset expansion size, so that the search range can be expanded subsequently to better acquire the micro-motion area, and then the micro-motion area can be rendered, so that the target object in the video can move slightly in the micro-motion area, so as to reduce the obvious sense of separation between the edge of the target object and the background information, and to form a better stereoscopic visual effect, thereby improving the visual effect of the video to a certain extent.
[0132] Among them, the image segmentation algorithm can be specifically expressed as a semantic segmentation algorithm, an instance segmentation algorithm or a panoramic segmentation algorithm, etc., and can also be other segmentation algorithms, which are not specifically limited here. Among them, semantic segmentation is to label all pixels with categories, such as sky, car, cup, etc., but does not distinguish individuals of the same category, that is, all pixels belonging to "cup" are labeled with the same label. Instance segmentation requires not only the labeling of pixels, but also the distinction between different instances of the same category, such as "cup A" and "cup B". The first preset expansion size is an expandable pixel set according to actual application requirements, such as expanding by at least 10 to 30 pixels, etc., which is not specifically limited here. The second preset expansion size is also an expandable pixel set according to actual application requirements, such as expanding by 10 pixels, etc., which is not specifically limited here.
[0133] Specifically, when the original image is obtained, the obtained original image can be input into a pre-trained fully convolutional network FCN for semantic segmentation, and a low-resolution semantic feature map can be generated by converting the fully connected layer to the fully convolutional layer, and the obtained low-resolution semantic feature map is upsampled using a transposed convolution (such as a bilinear interpolation filter), and a skip connection layer is introduced after each convolution block to abstract the semantic features obtained by the upsampling to output the semantic segmentation result and obtain the target object. For example, Fig.14 As shown, assuming that the original image obtained is a black sesame paste advertisement image, the original image is subjected to image segmentation processing, and the areas where the target object, text information and background information in the original image are located can be marked respectively, that is, a segmented image can be obtained in which the circular object is the target object, the strip object is the text information, and the rest is the background information.
[0134] Furthermore, a selective search algorithm can be used to select several candidate boxes from the original image, the image in each candidate box can be scaled to a specified size and placed in a convolutional neural network CNN to extract target object features, and the extracted target object features can be classified using a classifier SVM, and a bounding box containing the target object can be output. Then, the bounding box containing the target object can be regressed, specifically by performing a translation transformation and scale scaling on the bounding box through the neural network R-CNN to obtain a first bounding box containing the target object.
[0135] Furthermore, after obtaining the first bounding box, the first bounding box can be expanded according to the position of the first bounding box on the original image and the area selected by the first bounding box according to a first preset expansion size. For example, by expanding 10 to 30 pixels in all directions from the position of the first bounding box, an expanded search range that includes the area selected by the first bounding box can be obtained.
[0136] Optionally, in the above Figure 3 Based on the corresponding embodiment, in another optional embodiment of the method for generating a video provided in the embodiment of the present application, as Figure 4 As shown, the micro-motion area is searched from the expanded search range according to the pixel conditions, including:
[0137] In step S401, color scoring is performed on each pixel on the original image to obtain a pixel score for each pixel, wherein the pixel score is positively correlated with the brightness value of each pixel;
[0138] In step S402, pixel points whose pixel scores satisfy a preset score are searched in the expanded search range to generate a micro-motion area.
[0139] In this embodiment, after obtaining the first bounding box, since the original image and the pixels on the depth image have a one-to-one correspondence, and the depth value of the pixel on the depth image is positively correlated with the bright spot value of the pixel, each pixel on the original image can be color scored to obtain the pixel score of each pixel, and then the brightness value of each pixel can be reflected by the pixel score of each pixel. Then, the area whose pixel score meets the score threshold can be searched in the expanded search range to obtain the micro-motion area, so that the micro-motion area can be rendered later, so that the target object in the video can move slightly in the micro-motion area to reduce the obvious sense of separation between the edge of the target object and the background information, and to form a better stereoscopic visual effect, thereby improving the visual effect of the video to a certain extent.
[0140] Specifically, color scoring is performed on each pixel on the original image, which can be an edge detection canny algorithm, an image filtering algorithm or a deep neural network, etc., or other algorithms can be used, which are not specifically limited here, and the pixel score of each pixel can be obtained. Among them, color scoring can be understood as edge detection of the original image, so that the color information of the original image can be reflected through color scoring, that is, the brightness change of each pixel in the original image can be reflected through the pixel score, and then the pixel points with obvious brightness changes, that is, edge points, can be obtained, so that the target object can be better determined based on the edge points.
[0141] Furthermore, after obtaining the pixel score of each pixel point, the search range can be expanded, starting from the area selected by the first bounding box, to search for pixel points whose pixel scores meet a preset score, such as pixel points whose pixel scores are less than a preset score threshold, and the boundary enclosed by the pixel points that meet the preset score can be used as the micro-motion area.
[0142] Optionally, in the above Figure 4 Based on the corresponding embodiment, in another optional embodiment of the method for generating a video provided in the embodiment of the present application, as Figure 5 As shown, the color score of each pixel on the original image is performed to obtain the pixel score of each pixel, including:
[0143] In step S501, the original image is converted into a grayscale image;
[0144] In step S502, average precision mean processing is performed on the grayscale value of each pixel on the grayscale image to obtain a pixel score for each pixel.
[0145] In this embodiment, after obtaining the first bounding box and expanding the search range, each pixel on the original image can be color scored. This can be done by converting the original image into a grayscale image and performing average precision mean processing on the grayscale value of each pixel on the grayscale image to obtain the pixel score of each pixel. The pixel score of each pixel can then be used to reflect the brightness value of each pixel, so that the subsequent search for an area outside the area selected by the first bounding box where the pixel score meets the score threshold can be performed to more accurately obtain the micro-motion area, thereby determining whether the depth information of the depth image is accurate based on the micro-motion area and the second bounding box, thereby being able to maintain the accuracy of the video content to a certain extent and improve the visual effect of the video.
[0146] Specifically, after obtaining the first bounding box micro-motion area, converting the original image into a grayscale image refers to the process of converting a color image into a grayscale image, which is called image grayscale. Specifically, the original image can be grayscaled using a component method, a maximum method, a mean method, or a weighted average method to obtain a grayscale image.
[0147] Among them, the pixel value in the color image is determined by the three components of RGB (R: Red, G: Green, B: Blue), namely the three primary colors of red, green and blue. The three components of RGB can be used to represent true color. Each component has 0-255 (256 types) to choose from, so the pixel value of a pixel point can have 16 million possibilities (256*256*256). Grayscale image refers to an image that only contains brightness information but not color information. Black and white photos are also grayscale images. The brightness of grayscale images changes from dark to bright, and the brightness changes continuously. The description of grayscale images, like color images, still reflects the distribution and characteristics of the overall and local chromaticity and brightness levels of the entire image. Therefore, the pixel value of a grayscale image can be understood as a special color image with the same RGB three component values, which has only 256 possibilities. Therefore, by graying the original image into a grayscale image, subsequent image processing can be facilitated and the amount of calculation can be reduced, thereby improving the efficiency of rendering video to a certain extent.
[0148] Among them, the component method is to use the brightness of the three RGB components in the color image as the grayscale value of the three grayscale images. The maximum method is to use the maximum value of the brightness of the three RGB components in the color image as the grayscale value of the grayscale image. The mean method is to obtain the grayscale value of the grayscale image by averaging the brightness of the three RGB components in the color image. The weighted average method is to weighted average the three RGB components with different weights according to importance and other indicators. Since the human eye is most sensitive to green and least sensitive to blue, a more reasonable grayscale image can be obtained by weighted averaging the three RGB components according to the following formula: f(i,j)=0.30R(i,j)+0.59G(i,j)+0.11B(i,j)).
[0149] Furthermore, after obtaining the grayscale image, Fig.18 As shown in the figure, the grayscale value of each pixel on the grayscale image can be processed by average precision. The mean average precision (MAP) matrix can be used to compare the grayscale value of each pixel in each row with the grayscale value of each pixel in the grayscale image. Fig.18 The preset first MAP matrix shown in FIG. is multiplied by a matrix to obtain a row score MAP. At the same time, the grayscale value of each pixel in each column is multiplied by the grayscale value of each pixel in each column. Fig.18The preset second MAP matrix shown is subjected to matrix multiplication to obtain a column score MAP, and then a weighted calculation is performed based on the obtained row score MAP and column score MAP to obtain a basic pixel score for each pixel, wherein a pure color area on the grayscale image indicates a lower pixel score, and an area with complex colors indicates a higher pixel score. Then, since the obtained basic pixel score may contain decimals, the basic pixel score can be normalized to obtain a pixel score belonging to [0, 1], so that the pixel points whose pixel scores meet the preset score can be subsequently searched from the expanded search range to generate a micro-motion area. Specifically, the boundary formed by searching for pixels with a pixel score of 0 in the expanded search range can be used as an intermediate bounding box.
[0150] Optionally, in the above Figure 4 Based on the corresponding embodiment, in another optional embodiment of the method for generating a video provided in the embodiment of the present application, as Figure 5 As shown, the color score of each pixel on the original image is performed to obtain the pixel score of each pixel, including:
[0151] In step S601, the original image is subjected to image filtering to obtain a filtering value for each pixel;
[0152] In step S602, the filter value of each pixel is normalized to obtain a pixel score of each pixel.
[0153] In this embodiment, after obtaining the first bounding box and expanding the search range, each pixel on the original image can be color scored. This can be done by performing image filtering on the original image to obtain a filter value for each pixel and normalizing the filter value to obtain a pixel score for each pixel. The pixel score of each pixel can then be used to reflect the brightness value of each pixel, so that the subsequent search for an area outside the area selected by the first bounding box where the pixel score meets the score threshold can be performed to more accurately obtain the micro-motion area, thereby determining whether the depth information of the depth image is accurate based on the micro-motion area and the second bounding box, thereby maintaining the accuracy of the video content to a certain extent and improving the visual effect of the video.
[0154] Specifically, after obtaining the first bounding box and expanding the search range, the original image is subjected to image filtering to obtain the filtering value of each pixel point. Specifically, the original image can be filtered by using image filtering algorithms such as mean filtering, median filtering or Gaussian filtering, and the filtering mean, filtering median or filtering weighted average value can be obtained. Other filtering algorithms can also be used, which are not specifically limited here. Among them, mean filtering is to use the moving average or box filter to take the average value of the pixel value in the K*K window as the output. Median filtering is to replace the pixel value of a pixel point in the original image with the median value of the pixel values of each pixel point in an area where the pixel point is located. Gaussian filtering is a process of weighted averaging the pixel values of all pixels in the original image, and the pixel value of each pixel point is obtained by weighted averaging itself and other pixel values in the neighborhood. Further, the filtering value of each pixel point is normalized respectively, and specifically, the obtained filtering value can be weighted averaged according to the preset weight value to calculate the pixel score of each pixel point.
[0155] Optionally, in the above Figure 2 Based on the corresponding embodiment, in another optional embodiment of the method for generating a video provided in the embodiment of the present application, as Figure 6 As shown, determining a first average depth value of the slightly-moved area and a second average depth value of the area selected by the second bounding box according to the depth value of each pixel in the depth image includes:
[0156] In step S701, a first pixel point set corresponding to the micro-motion area and a second pixel point set corresponding to the second bounding box are obtained from the depth image;
[0157] In step S702, a first average depth value is calculated according to the depth values of the pixels in the first pixel set, and a second average depth value is calculated according to the depth values of the pixels in the second pixel set.
[0158] In this embodiment, after obtaining the micro-motion area and the second bounding box on the original image, the micro-motion area and the second bounding box can be mapped to the depth image according to the mapping relationship between the original image and the depth image, and then the first pixel point set corresponding to the micro-motion area and the second pixel point set corresponding to the second bounding box can be obtained. Then, the first average depth value can be calculated according to the depth value of each first pixel point in the first pixel point set. Similarly, the second average depth value can be calculated according to the depth value of each second pixel point in the second pixel point set, so that the depth information between the target object and the background information in the depth image can be judged according to the obtained first average depth value and the second average depth value, thereby determining whether the depth information of the depth image is accurate.
[0159] Specifically, after obtaining the micro-motion area and the second bounding box on the original image, based on the one-to-one correspondence between the pixels on the original image and the depth image, a number of pixel points that correspond one-to-one to each pixel point of the target object in the micro-motion area in the original image can be indexed on the depth image, that is, the first pixel point set. Similarly, a number of pixel points that correspond one-to-one to each pixel point in the second bounding box in the original image can be obtained on the depth image, that is, the second pixel point set.
[0160] Furthermore, when a first set of pixels and a second set of pixels in a depth image are obtained, since each pixel on the depth image has a corresponding depth value, the depth values corresponding to the first pixels in the first set of pixels can be added together and the average of the summed values can be taken, i.e., the first average depth value. Similarly, the depth values corresponding to the second pixels in the second set of pixels can be added together and the average of the summed values can be taken, i.e., the second average depth value.
[0161] Optionally, in the above Figure 2 Based on the corresponding embodiment, in another optional embodiment of the method for generating a video provided in the embodiment of the present application, as Figure 7 As shown, a video is rendered based on the original image, the micro-motion area, and the depth image, including:
[0162] In step S801, a target three-dimensional grid image is constructed based on the original image, the micro-motion area and the depth image;
[0163] In step S802, a camera motion trajectory is generated according to each frame of the template image in the preset video template, wherein the camera motion trajectory is used to indicate the camera position and orientation information corresponding to each frame of the template image taken by the camera;
[0164] In step S803, the target three-dimensional grid image and the camera motion trajectory are rendered to obtain a video.
[0165] In this embodiment, when the first average depth value obtained is less than the second average depth value, the three-dimensional scene can be reconstructed based on the original image and the depth image to construct a target three-dimensional grid image. At the same time, a preset video template can be obtained, and a camera motion trajectory that can reflect the camera position and orientation information corresponding to each frame of the template image taken by the camera can be generated according to each frame of the template image in the preset video template. Then, a video can be rendered based on the target three-dimensional grid image and the camera motion trajectory. A high-precision target grid image corresponding to the original image and the depth image can be obtained through three-dimensional scene reconstruction. A high-precision video can be obtained by rendering based on the high-precision target grid image and the accurate camera motion trajectory, thereby improving the video quality and the visual effect of the video to a certain extent.
[0166] The target 3D mesh image is a high-precision 3D mesh obtained by a 3D reconstruction algorithm based on the original image and the depth image. The camera motion trajectory includes the camera position and orientation information corresponding to each frame of the template image taken by the camera, as well as the camera position such as camera rotation, and may also include other camera information, which is not specifically limited here. The preset video template is a video uploaded by the user or designer, and the video includes the video playback time, template image, and video playback style.
[0167] Specifically, Fig.19 As shown, when the first average depth value obtained is less than the second average depth value, the three-dimensional scene can be reconstructed for the original image and the depth image. Specifically, the pixel points in the depth image can be spatially projected to generate spatial vertices. Then, the spatial vertices can be subdivided to improve the image resolution, and connected to obtain a high-precision three-dimensional mesh image. Then, the three-dimensional mesh can be re-colored by extracting color information from the original image to obtain a high-precision target three-dimensional mesh image.
[0168] Furthermore, a preset video template corresponding to the original image can be obtained from a database, and a camera motion trajectory can be generated according to each frame template image in the preset video template. Specifically, an image interpolation algorithm can be used to simulate the movement rate of the camera and draw a curve to obtain a camera motion trajectory with a motion rhythm. Then, rendering can be performed based on a high-precision target grid image and the camera motion trajectory with a motion rhythm to obtain a high-precision video with good visual effects.
[0169] Optionally, in the above Figure 7 Based on the corresponding embodiment, in another optional embodiment of the method for generating a video provided in the embodiment of the present application, as Figure 8 As shown, a target three-dimensional grid image is constructed based on the original image, the slightly-moved area, and the depth image, including:
[0170] In step S901, downsampling is performed on the original image and the depth image respectively to generate a downsampled image of the original image and a downsampled image of the depth image;
[0171] In step S902, spatial projection is performed according to the position of each pixel point on the downsampled image of the depth image to generate an initial three-dimensional grid image;
[0172] In step S903, upsampling is performed on the initial three-dimensional grid image to generate an intermediate three-dimensional grid image;
[0173] In step S904, the intermediate three-dimensional grid image is color-complemented according to the color information of the original image to obtain a target three-dimensional grid image.
[0174] In this embodiment, when the first average depth value obtained is less than the second average depth value, the original image and the depth image can be downsampled respectively to obtain a downsampled image of the original image and a downsampled image of the depth image, and the initial three-dimensional grid image can be generated by spatially projecting the position of each pixel point on the downsampled image of the depth image, so that the resolution of the image can be reduced by downsampling without changing the rendering accuracy, and the amount of image calculation can be reduced, thereby improving the rendering efficiency of the video to a certain extent. Furthermore, the initial three-dimensional grid image can be upsampled to obtain an intermediate three-dimensional grid image, and the resolution of the intermediate three-dimensional grid image can be improved by upsampling, and the accuracy of the intermediate three-dimensional grid image can be improved. Then, the intermediate three-dimensional grid image is color-complemented according to the color information of the original image, and a high-precision target three-dimensional grid image can be obtained.
[0175] Specifically, Fig.19 As shown, when the first average depth value obtained is less than the second average depth value, the original image and the depth image can be downsampled respectively to obtain a downsampled image of the original image and a downsampled image of the depth image. Specifically, the original image I has a size of M*N, and it is downsampled s times to obtain a low-resolution image of (M / s)*(N / s) size, where s should be a common divisor of M and N, and s, M and N are all integers greater than 1.
[0176] For example, assuming there is an original image with a resolution of 1600*1600, by downsampling the original image by 2 times, a downsampled image of the original image with a low resolution of 800*800 can be obtained. Similarly, the depth image with a resolution of 1600*1600 can be downsampled by 2 times to obtain a downsampled image of the depth image with a low resolution of 800*800. Since the original image and the depth image are in a one-to-one correspondence, the downsampled image of the original image and the downsampled image of the depth image obtained by downsampling with the same multiple also have a one-to-one correspondence.
[0177] Furthermore, if Fig.17 As shown, after obtaining the downsampled image of the original image and the downsampled image of the depth image, spatial projection can be performed according to the position of each pixel point in the downsampled image of the depth image, that is, the coordinate value (x, y), and the spatial vertex corresponding to each pixel point can be obtained. The set of these spatial vertices can be called a point cloud, and then the vertices can be connected according to the depth value to obtain a low-resolution initial three-dimensional mesh image.
[0178] Further, after obtaining the low-resolution initial three-dimensional grid image, the initial three-dimensional grid image can be upsampled to obtain an intermediate three-dimensional grid image. Specifically, an interpolation algorithm can be used, that is, a new vertex is inserted between two spatial vertices using a suitable interpolation algorithm based on the pixels of the intermediate three-dimensional grid image, wherein the interpolation algorithm can be specifically expressed as a nearest neighbor method, a bilinear interpolation method, or a cubic interpolation method, etc., and other interpolation algorithms can also be used, which are not specifically limited here. For example, an 800*800 low-resolution initial three-dimensional grid image is upsampled, that is, a new spatial vertex is inserted between two spatial vertices, and the spatial vertices are connected to obtain an intermediate three-dimensional grid image with a resolution of 1600*1600. Then, the color information of the original image can be extracted, and each spatial vertex in the intermediate three-dimensional grid image can be recolored, that is, color supplemented, according to the corresponding relationship between each pixel point and the spatial vertex, so that a high-precision target three-dimensional grid image can be obtained.
[0179] Optionally, in the above Figure 8 Based on the corresponding embodiment, in another optional embodiment of the method for generating a video provided in the embodiment of the present application, as Fig. 9 As shown, the initial three-dimensional grid image is generated by spatial projection according to the position of each pixel point on it, including:
[0180] In step S1001, spatial projection is performed on each pixel point according to the position of each pixel point on the downsampled image of the depth image to generate a spatial vertex corresponding to each pixel point;
[0181] In step S1002, spatial vertices are connected to form a basic three-dimensional mesh image according to the depth value of each pixel in the downsampled image of the depth image;
[0182] In step S1003, first color information of each pixel is extracted from the downsampled image of the original image;
[0183] In step S1004, according to the mapping relationship between the original image and the depth image, the basic three-dimensional grid image is color-complemented according to the first color information of each pixel to obtain an initial three-dimensional grid image.
[0184] In this embodiment, after obtaining the downsampled image of the depth image, each pixel point can be spatially projected according to the position of each pixel point on the downsampled image of the depth image, and the spatial vertex corresponding to each pixel point can be obtained. According to the one-to-one correspondence between the pixel point and the spatial vertex, the depth value of each pixel point is the depth value of each spatial vertex, and then the spatial vertices can be connected according to the depth values of the spatial vertices, and the following can be obtained. Fig.19A low-resolution basic 3D grid image composed of spatial vertices and triangular faces is shown. Then, the first color information of each pixel can be extracted from the downsampled image of the original image. Then, according to the mapping relationship between the original image and the depth image, there can be a mapping relationship between the downsampled image of the original image and the downsampled image of the depth image, that is, the pixels between the downsampled image of the original image and the downsampled image of the depth image are one-to-one corresponding. Therefore, the first color information of each pixel can be extracted from the downsampled image of the original image to color the corresponding spatial vertices in the basic 3D grid image, that is, color supplement, to obtain an initial 3D grid image. The low-resolution basic 3D grid image is obtained by spatial projection, thereby obtaining the initial 3D grid image, which can reduce the resolution of the image, reduce the amount of image calculation, and thus improve the video rendering efficiency to a certain extent.
[0185] Specifically, Fig.17 As shown, after obtaining the downsampled image of the depth image, the adjacent pixels can be connected by edges, and a triangular face formed by the pixels and the edges can be obtained. According to the depth value of each pixel on the downsampled image of the depth image, when searching for discontinuous depth values between adjacent pixels or the difference between adjacent depth values is greater than a preset depth threshold, the edges between the adjacent pixels can be disconnected, that is, it can be understood as disconnecting the edges between the pixels of the target object and the background information, so that the pixels of the target object can be connected, and the pixels of the background information can be connected. Then, according to the one-to-one correspondence between the pixels and the spatial vertices, each pixel can be spatially projected according to the position of each pixel on the downsampled image of the depth image to obtain the corresponding spatial vertex, and then, according to the one-to-one correspondence between the pixels and the spatial vertices, the spatial vertices with continuous depth values can be connected according to the edges between the pixels to form a triangular face, and a low-resolution basic three-dimensional grid image composed of spatial vertices and triangular faces can be obtained.
[0186] Furthermore, a general color grabbing tool can be used to extract the color of each pixel, that is, the first color information, from the downsampled image of the original image. Then, based on the one-to-one correspondence between the pixels of the downsampled image of the original image and the downsampled image of the depth image, and the one-to-one correspondence between the downsampled image of the depth image and the spatial vertices in the basic three-dimensional mesh image, the corresponding spatial vertices in the basic three-dimensional mesh image can be colored according to the first color information of each pixel extracted from the downsampled image of the original image to obtain an initial three-dimensional mesh image.
[0187] It can be understood that the initial three-dimensional grid image is generated by spatial projection according to the position of each pixel point on the image. It can also be that the layered depth image (Layered Depth Image, LDI) with explicit pixel connectivity is used to represent the downsampled image of the depth image, and then the color and depth of the occluded area in the downsampled image of the input depth image are iteratively repaired. Specifically, the downsampled image of the original image and the downsampled image of the depth image are used as input, and are initialized as a single-layer LDI respectively. By detecting pixel points with discontinuous depth values, they are defined as depth edges, and the pixel points of the single-layer LDI with discontinuous depth value edges are disconnected. Then, based on the depth edges, the single-layer LDI can be divided into different areas, and the edges, depth values and color values of the occluded areas in the single-layer LDI are repaired by using a deep learning repair network. It is repeated until no new edges are generated. Then, the synthetic pixels of the single-layer LDI after the repair are merged into the target LDI, and the target LDI is converted into a 3D grid representation to obtain the initial three-dimensional grid image.
[0188] Optionally, in the above Figure 8 Based on the corresponding embodiment, in another optional embodiment of the method for generating a video provided in the embodiment of the present application, as Fig. 9 As shown, the intermediate three-dimensional grid image is color-complemented according to the color information of the original image to obtain a target three-dimensional grid image, including:
[0189] In step S1101, second color information of each pixel is extracted from the original image;
[0190] In step S1102, the color of each spatial vertex in the intermediate three-dimensional grid image is supplemented according to the second color information of each pixel point to obtain a target three-dimensional grid image.
[0191] In this embodiment, after the intermediate three-dimensional grid image is obtained, the second color information of each pixel can be extracted from the original image, and the corresponding spatial vertices in the intermediate three-dimensional grid image can be colored according to the second color information of each pixel in the original image to obtain the target three-dimensional grid image. By improving the resolution of the initial three-dimensional grid image, a high-precision intermediate three-dimensional grid image can be obtained, and then the high-precision intermediate three-dimensional grid image is supplemented with color according to the second color information of the original image, so that a high-precision target three-dimensional grid image can be obtained, thereby improving the quality of the video to a certain extent, and improving the visual effect of the video.
[0192] Specifically, Fig.19As shown, after acquiring the intermediate 3D grid image, a general color capture tool can be used to extract the color of each pixel, i.e., the second color information, from the original image. Then, based on the one-to-one correspondence between the pixels of the original image and the depth image, and the one-to-one correspondence between the depth image and the spatial vertices in the intermediate 3D grid image, the corresponding spatial vertices in the intermediate 3D grid image can be recolored according to the second color information of each pixel extracted from the original image. It can be understood that, based on the second color information of each pixel extracted, the color of the previously colored spatial vertices in the intermediate 3D grid image is overwritten, and the correspondence between the pixels and the newly inserted spatial vertices is supplemented with the corresponding colors to obtain the target 3D grid image. For example, Fig. 20 As shown, assuming that an original image is a milk powder advertisement image, a high-precision target three-dimensional image can be obtained through three-dimensional scene reconstruction.
[0193] Optionally, in the above Figure 7 Based on the corresponding embodiment, in another optional embodiment of the method for generating a video provided in the embodiment of the present application, as Figure 8 As shown, the camera motion trajectory is generated according to each frame template image in the preset video template, including:
[0194] In step 1201, according to the playback time of the preset video template, the camera motion rate corresponding to each template image frame in the preset video template is obtained;
[0195] In step 1202, the camera motion rate is interpolated to obtain the camera motion trajectory.
[0196] In this embodiment, before performing video rendering based on the original image and the depth image, the playback time of the preset video template can be obtained, and the camera motion rate corresponding to each frame template image in the preset video template can be obtained based on the obtained playback time of the preset video template, and the camera motion rate can be interpolated. This can better simulate the camera's motion rate, thereby accurately obtaining the camera motion trajectory, so that subsequent rendering based on a high-precision target grid image and an accurate camera motion trajectory can obtain a high-precision video, thereby improving the video quality and the visual effect of the video to a certain extent.
[0197] Specifically, Fig. 20 As shown in FIG, before performing video rendering based on the original image and the depth image, the playback time of the preset video template can be obtained, and according to the obtained playback time of the preset video template, the camera motion rate corresponding to each frame template image in the preset video template can be accurately obtained through the calculation formula of time and speed or the deep neural network, such as Fig.11As shown, assuming that the playback time of a preset video template is 0:06, that is, 6 seconds, if there are 8 frames of template images in 1 second, the camera speed can be represented by the frame rate, that is, the camera movement rate is 8fps, which is used to indicate that the camera can capture a maximum of 8 frames of images in 1 second.
[0198] Furthermore, after obtaining the camera motion rate, the camera motion rate can be interpolated to obtain the camera motion trajectory. Specifically, the camera motion rate can be simulated by using a Bezier curve, or other interpolation algorithms can be used, such as B-spline or cubic spline curve interpolation, etc., which are not specifically limited here. Then, the simulated adjacent points can be connected by smooth curves to obtain a smooth curve, namely the camera motion trajectory, wherein the camera motion trajectory specifies the camera position, orientation information and rotation angle of the camera corresponding to each frame image. It can be understood that each frame image is a scene photo "taken" by the camera. Then, each frame image can be combined with the camera motion trajectory to obtain a video.
[0199] Furthermore, if Fig.21 As shown, since the camera motion trajectory includes the rotation angle of the camera, in order to ensure that the camera only captures the space where the original image has the corresponding 3D scene, the camera is corrected once each time each frame of the original image and depth image is rendered, so that the line connecting the camera and the center of the scene is perpendicular to the imaging plane of the camera, and the imaging range is less than or equal to the maximum range of the scene, and a video with a normal visual angle can be rendered, thereby improving the visual effect of the video to a certain extent. Specifically, the camera correction can be performed by detecting the current camera orientation information and rotation angle when rendering each frame of the image, and the camera angle can be adjusted so that the line connecting the camera and the center of the scene is perpendicular to the imaging plane of the camera, and the three are on a horizontal line.
[0200] Optionally, in the above Figure 2 Based on the corresponding embodiment, in another optional embodiment of the method for generating a video provided in the embodiment of the present application, as Figure 3 As shown, before obtaining the original image and the depth image corresponding to the original image, the method further includes:
[0201] In step 1301, depth estimation is performed on the original image to obtain an initial depth image and depth information of the initial depth image;
[0202] In step 1302, an inverse function is performed on the initial depth image and the depth information of the initial depth image to obtain a depth image.
[0203] In this embodiment, before obtaining the depth image of the original image, the original image can be subjected to depth estimation processing to obtain an initial depth image and depth information D of the initial depth image. Then, since in actual application scenarios, the original image is not necessarily a real shot image, directly obtaining the depth image will lead to inaccurate depth estimation. Therefore, an inverse log function is used to slow down the changing trend of the depth value of the depth image, thereby reducing the error of the depth information of the obtained depth image, so that the subsequent rendering can be based on the depth image with less error to obtain a video with accurate content information, thereby improving the video quality and the visual effect of the video.
[0204] Specifically, before obtaining the depth image of the original image, the original image can be subjected to depth estimation processing. Specifically, a monocular depth estimation algorithm can be used to fit the original image to obtain an initial depth image and the depth information of the initial depth image. Other depth estimation algorithms, such as a binocular depth estimation algorithm, can also be used. No specific restrictions are made here. The monocular estimation based on deep learning reflects the depth relationship through the relationship between pixel values. Specifically, the original image I is mapped into the initial depth image D=f(I) by fitting a function f.
[0205] Furthermore, after obtaining the initial depth image and the depth information of the initial depth image, an inverse log function calculation may be performed according to D=f(I), which may be obtained by the inverse log function formula: The depth image and the depth information C of the depth image are calculated, where ∈ represents a constant small quantity, which can usually be taken as (1e-8).
[0206] The following is a detailed description of the video generation device in this application. Fig. 22 , Fig. 22 This is a schematic diagram of an embodiment of a video generation device in an embodiment of the present application. The video generation device 20 includes:
[0207] An acquisition unit 201 is used to acquire an original image and a depth image corresponding to the original image, wherein each pixel in the depth image has a corresponding depth value;
[0208] An acquisition unit 201 is used to acquire a first bounding box and an expanded search range corresponding to the first bounding box from an original image, wherein the first bounding box includes each pixel point of the target object;
[0209] The processing unit 202 is further configured to search for a slightly motion region from the expanded search range according to a pixel condition, wherein the first bounding box is placed in the slightly motion region;
[0210] The processing unit 202 is further configured to generate a second bounding box according to the position of the slightly-moved area on the original image, wherein the slightly-moved area is placed within the second bounding box;
[0211] A determining unit 203 is used to determine a first average depth value of the micro-motion area and a second average depth value of the area selected by the second bounding box according to the depth value of each pixel in the depth image;
[0212] The processing unit 202 is further configured to render a video based on the original image and the depth image if the first average depth value is less than the second average depth value.
[0213] Optionally, in the above Fig. 22 On the basis of the corresponding embodiment, in another embodiment of the video generation device provided by the embodiment of the present application, the processing unit 202 can be specifically used for:
[0214] Perform image segmentation processing on the original image to obtain the target object;
[0215] According to the position of the target object on the original image, a border regression process is performed on the target object to obtain a first bounding box;
[0216] An expansion search range is determined according to the position of the first bounding box on the original image and a first preset expansion size.
[0217] Optionally, in the above Fig. 22 On the basis of the corresponding embodiment, in another embodiment of the video generation device provided by the embodiment of the present application, the processing unit 202 can be specifically used for:
[0218] Perform a color score on each pixel on the original image to obtain a pixel score for each pixel, where the pixel score is positively correlated with the brightness value of each pixel;
[0219] Pixel points whose pixel scores meet a preset score are searched from the expanded search range to generate a micro-motion area.
[0220] Optionally, in the above Fig. 22 On the basis of the corresponding embodiment, in another embodiment of the video generation device provided by the embodiment of the present application, the processing unit 202 can be specifically used for:
[0221] Convert the original image into a grayscale image;
[0222] The grayscale value of each pixel on the grayscale image is processed by average precision mean to obtain the pixel score of each pixel.
[0223] Optionally, in the above Fig. 22On the basis of the corresponding embodiment, in another embodiment of the video generation device provided by the embodiment of the present application, the processing unit 202 can be specifically used for:
[0224] Perform image filtering on the original image to obtain the filtering value of each pixel;
[0225] The filter value of each pixel is normalized respectively to obtain the pixel score of each pixel.
[0226] Optionally, in the above Fig. 22 On the basis of the corresponding embodiment, in another embodiment of the video generation device provided by the embodiment of the present application, the determination unit 203 can be specifically used to:
[0227] Acquire a first pixel point set corresponding to the micro-motion area and a second pixel point set corresponding to the second bounding box from the depth image;
[0228] A first average depth value is calculated according to the depth values of the pixels in the first pixel set, and a second average depth value is calculated according to the depth values of the pixels in the second pixel set.
[0229] Optionally, in the above Fig. 22 On the basis of the corresponding embodiment, in another embodiment of the video generation device provided by the embodiment of the present application, the processing unit 202 can be specifically used for:
[0230] Construct a target three-dimensional grid image based on the original image, the micro-motion area and the depth image;
[0231] Generate a camera motion trajectory according to each frame template image in the preset video template, wherein the camera motion trajectory is used to indicate the camera position and orientation information corresponding to each frame template image taken by the camera;
[0232] The target 3D grid image and the camera motion trajectory are rendered to obtain a video.
[0233] Optionally, in the above Fig. 22 On the basis of the corresponding embodiment, in another embodiment of the video generation device provided by the embodiment of the present application, the processing unit 202 can be specifically used for:
[0234] Downsampling the original image and the depth image respectively to generate a downsampled image of the original image and a downsampled image of the depth image;
[0235] Generate an initial three-dimensional grid image by performing spatial projection according to the position of each pixel point on the downsampled image of the depth image;
[0236] Performing upsampling processing on the initial three-dimensional grid image to generate an intermediate three-dimensional grid image;
[0237] The intermediate three-dimensional grid image is color-complemented according to the color information of the original image to obtain a target three-dimensional grid image.
[0238] Optionally, in the above Fig. 22 On the basis of the corresponding embodiment, in another embodiment of the video generation device provided by the embodiment of the present application, the processing unit 202 can be specifically used for:
[0239] According to the position of each pixel point on the downsampled image of the depth image, each pixel point is spatially projected to generate a spatial vertex corresponding to each pixel point;
[0240] Connecting the spatial vertices to a basic three-dimensional mesh image according to the depth value of each pixel in the downsampled image of the depth image;
[0241] Extracting first color information of each pixel from the downsampled image of the original image;
[0242] According to the mapping relationship between the original image and the depth image, the basic three-dimensional grid image is color-complemented according to the first color information of each pixel to obtain an initial three-dimensional grid image.
[0243] Optionally, in the above Fig. 22 On the basis of the corresponding embodiment, in another embodiment of the video generation device provided by the embodiment of the present application, the processing unit 202 can be specifically used for:
[0244] Extracting second color information of each pixel from the original image;
[0245] The color of each spatial vertex in the intermediate three-dimensional grid image is supplemented according to the second color information of each pixel point to obtain a target three-dimensional grid image.
[0246] Optionally, in the above Fig. 22 On the basis of the corresponding embodiment, in another embodiment of the video generation device provided by the embodiment of the present application, the processing unit 202 can be specifically used for:
[0247] According to the playback time of the preset video template, the camera motion rate corresponding to each frame template image in the preset video template is obtained;
[0248] The camera motion velocity is interpolated to obtain the camera motion trajectory.
[0249] Optionally, in the above Fig. 22 On the basis of the corresponding embodiment, in another embodiment of the video generation device provided in the embodiment of the present application,
[0250] The processing unit 202 is further used to perform depth estimation processing on the original image to obtain an initial depth image and depth information of the initial depth image;
[0251] The processing unit 202 is further configured to perform inverse function processing on the initial depth image and the depth information of the initial depth image to obtain a depth image.
[0252] On the other hand, the present application provides another schematic diagram of a computer device, such as Fig.23 As shown, Fig.23 3 is a schematic diagram of a computer device structure provided in an embodiment of the present application. The computer device 300 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 310 (for example, one or more processors) and a memory 320, and one or more storage media 330 (for example, one or more mass storage devices) storing application programs 331 or data 332. Among them, the memory 320 and the storage medium 330 may be temporary storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the computer device 300. Furthermore, the central processing unit 310 may be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the computer device 300.
[0253] The computer device 300 may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input and output interfaces 360, and / or one or more operating systems 333, such as Windows Server 2000, Windows Server 2003, Windows Server 2003E, Windows Server 2003R ... TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.
[0254] The computer device 300 is also used to perform the following steps: Figures 2 to 9 The steps in the corresponding embodiments.
[0255] Another aspect of the present application provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, and when the instructions are executed on a computer, the computer executes the following Figures 2 to 9 The illustrated embodiments describe steps in a method.
[0256] Another aspect of the present application provides a computer program product comprising instructions, which when executed on a computer or a processor, causes the computer or the processor to execute the following Figures 2 to 9 The illustrated embodiments describe steps in a method.
[0257] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0258] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0259] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0260] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0261] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.
Claims
1. A method for generating a video, characterized in that: include: Acquire an original image and a depth image corresponding to the original image, wherein each pixel in the depth image has a corresponding depth value; Acquire a first bounding box and an expanded search range corresponding to the first bounding box from the original image, wherein the first bounding box includes each pixel point of the target object; Performing a color score on each pixel on the original image to obtain a pixel score for each pixel, wherein the pixel score is positively correlated with the brightness value of each pixel; Searching for pixel points whose pixel scores satisfy a preset score in the expanded search range to generate a micro-motion area, wherein the first bounding box is placed in the micro-motion area; the micro-motion area is an area where the color change of the area boundary is small and there is no obvious sense of separation between the area boundary and the background information; generating a second bounding box according to the position of the slightly-moved area on the original image, wherein the slightly-moved area is placed within the second bounding box; Determine a first average depth value of the micro-motion area and a second average depth value of the area selected by the second bounding box according to the depth value of each pixel in the depth image; If the first average depth value is less than the second average depth value, rendering is performed based on the original image, the micro-motion area, and the depth image to obtain a video.
2. The method according to claim 1, characterized in that The acquiring a first bounding box and an expanded search range corresponding to the first bounding box from the original image includes: Performing image segmentation processing on the original image to obtain the target object; According to the position of the target object on the original image, performing border regression processing on the target object to obtain the first bounding box; The expansion search range is determined according to the position of the first bounding box on the original image and a first preset expansion size.
3. The method according to claim 1, characterized in that: The step of performing a color score on each pixel on the original image to obtain a pixel score for each pixel includes: Converting the original image into a grayscale image; The grayscale value of each pixel on the grayscale image is processed by average precision mean to obtain a pixel score of each pixel.
4. The method according to claim 1, characterized in that: Performing a color score on each pixel on the original image to obtain a pixel score for each pixel includes: Performing image filtering on the original image to obtain a filtering value for each pixel; The filter value of each pixel is normalized respectively to obtain a pixel score of each pixel.
5. The method according to claim 1, characterized in that The determining, according to the depth value of each pixel in the depth image, a first average depth value of the micro-motion area and a second average depth value of the area selected by the second bounding box comprises: Acquire a first pixel point set corresponding to the micro-motion area and a second pixel point set corresponding to the second bounding box from the depth image; The first average depth value is calculated according to the depth values of the pixels in the first pixel set, and the second average depth value is calculated according to the depth values of the pixels in the second pixel set.
6. The method according to claim 1, characterized in that The rendering based on the original image, the micro-motion area and the depth image to obtain a video includes: constructing a target three-dimensional grid image based on the original image, the micro-motion area and the depth image; Generate a camera motion trajectory according to each frame template image in a preset video template, wherein the camera motion trajectory is used to indicate the camera position and orientation information corresponding to each frame template image captured by the camera; The target three-dimensional grid image and the camera motion trajectory are rendered to obtain the video.
7. The method according to claim 6, characterized in that The constructing a target three-dimensional grid image based on the original image, the micro-motion area and the depth image includes: Downsampling the original image and the depth image respectively to generate a downsampled image of the original image and a downsampled image of the depth image; Generate an initial three-dimensional grid image by spatially projecting the position of each pixel point on the downsampled image of the depth image; Performing upsampling processing on the initial three-dimensional grid image to generate an intermediate three-dimensional grid image; The intermediate three-dimensional grid image is color-complemented according to the color information of the original image to obtain the target three-dimensional grid image.
8. The method according to claim 7, characterized in that The generating an initial three-dimensional grid image by spatially projecting the position of each pixel point on the downsampled image of the depth image comprises: According to the position of each pixel point on the downsampled image of the depth image, perform spatial projection on each pixel point to generate a spatial vertex corresponding to each pixel point; Connecting the spatial vertices into a basic three-dimensional mesh image according to the depth value of each pixel in the downsampled image of the depth image; Extracting first color information of each pixel from the downsampled image of the original image; According to the mapping relationship between the original image and the depth image, the basic three-dimensional grid image is color-complemented according to the first color information of each pixel to obtain the initial three-dimensional grid image.
9. The method according to claim 7, characterized in that: The color supplementing of the intermediate three-dimensional grid image according to the color information of the original image to obtain the target three-dimensional grid image includes: Extracting second color information of each pixel from the original image; The color of each spatial vertex in the intermediate three-dimensional grid image is supplemented according to the second color information of each pixel point to obtain the target three-dimensional grid image.
10. The method according to claim 6, characterized in that The step of generating a camera motion trajectory according to each frame template image in a preset video template includes: According to the playback time of the preset video template, obtaining the camera motion rate corresponding to each frame template image in the preset video template; The camera motion rate is interpolated to obtain the camera motion trajectory.
11. The method according to claim 1, characterized in that: Before acquiring the original image, the micro-motion area, and the depth image corresponding to the original image, the method further includes: Performing depth estimation processing on the original image to obtain an initial depth image and depth information of the initial depth image; An inverse function is performed on the initial depth image and the depth information of the initial depth image to obtain the depth image.
12. A video generation device, characterized in that: include: An acquisition unit, configured to acquire an original image and a depth image corresponding to the original image, wherein each pixel in the depth image has a corresponding depth value; The acquisition unit is configured to acquire a first bounding box and an expanded search range corresponding to the first bounding box from the original image, wherein the first bounding box includes each pixel point of the target object; A processing unit is used to perform color scoring on each pixel point on the original image to obtain a pixel score of each pixel point, wherein the pixel score is positively correlated with the brightness value of each pixel point; search for pixel points whose pixel scores meet a preset score from the expanded search range to generate a micro-motion area, wherein the first bounding box is placed in the micro-motion area; the micro-motion area is an area where the color change of the area boundary is small and there is no obvious sense of separation between the area boundary and the background information; The processing unit is further configured to generate a second bounding box according to the position of the slightly-moved area on the original image, wherein the slightly-moved area is placed within the second bounding box; a determining unit, configured to determine a first average depth value of the micro-motion area and a second average depth value of the area selected by the second bounding box according to the depth value of each pixel in the depth image; The processing unit is further configured to render a video based on the original image and the depth image if the first average depth value is less than the second average depth value.
13. The device according to claim 12, characterized in that The processing unit is specifically used for: Performing image segmentation processing on the original image to obtain the target object; According to the position of the target object on the original image, performing border regression processing on the target object to obtain the first bounding box; The expansion search range is determined according to the position of the first bounding box on the original image and a first preset expansion size.
14. The device according to claim 12, characterized in that The processing unit is specifically used for: Converting the original image into a grayscale image; The grayscale value of each pixel on the grayscale image is processed by average precision mean to obtain a pixel score of each pixel.
15. The device according to claim 12, characterized in that The processing unit is specifically used for: Performing image filtering on the original image to obtain a filtering value for each pixel; The filter value of each pixel is normalized respectively to obtain a pixel score of each pixel.
16. The device according to claim 12, characterized in that The determining unit is specifically configured to: Acquire a first pixel point set corresponding to the micro-motion area and a second pixel point set corresponding to the second bounding box from the depth image; The first average depth value is calculated according to the depth values of the pixels in the first pixel set, and the second average depth value is calculated according to the depth values of the pixels in the second pixel set.
17. The device according to claim 12, characterized in that The processing unit is specifically used for: constructing a target three-dimensional grid image based on the original image, the micro-motion area and the depth image; Generate a camera motion trajectory according to each frame template image in a preset video template, wherein the camera motion trajectory is used to indicate the camera position and orientation information corresponding to each frame template image captured by the camera; The target three-dimensional grid image and the camera motion trajectory are rendered to obtain the video.
18. The device according to claim 17, characterized in that The processing unit is specifically used for: Downsampling the original image and the depth image respectively to generate a downsampled image of the original image and a downsampled image of the depth image; Generate an initial three-dimensional grid image by spatially projecting the position of each pixel point on the downsampled image of the depth image; Performing upsampling processing on the initial three-dimensional grid image to generate an intermediate three-dimensional grid image; The intermediate three-dimensional grid image is color-complemented according to the color information of the original image to obtain the target three-dimensional grid image.
19. The device according to claim 18, characterized in that The processing unit is specifically used for: According to the position of each pixel point on the downsampled image of the depth image, perform spatial projection on each pixel point to generate a spatial vertex corresponding to each pixel point; Connecting the spatial vertices into a basic three-dimensional mesh image according to the depth value of each pixel in the downsampled image of the depth image; Extracting first color information of each pixel from the downsampled image of the original image; According to the mapping relationship between the original image and the depth image, the basic three-dimensional grid image is color-complemented according to the first color information of each pixel to obtain the initial three-dimensional grid image.
20. The device according to claim 18, characterized in that The processing unit is specifically used for: Extracting second color information of each pixel from the original image; The color of each spatial vertex in the intermediate three-dimensional grid image is supplemented according to the second color information of each pixel point to obtain the target three-dimensional grid image.
21. The device according to claim 17, characterized in that The processing unit is specifically used for: According to the playback time of the preset video template, obtaining the camera motion rate corresponding to each frame template image in the preset video template; The camera motion rate is interpolated to obtain the camera motion trajectory.
22. The device according to claim 12, characterized in that The processing unit is further used for: Performing depth estimation processing on the original image to obtain an initial depth image and depth information of the initial depth image; An inverse function is performed on the initial depth image and the depth information of the initial depth image to obtain the depth image.
23. A computer device, characterized in that: include: memories, transceivers, processors, and bus systems; Wherein, the memory is used to store programs; The processor is used to implement the method according to any one of claims 1 to 11 when executing the program in the memory; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.
24. A computer-readable storage medium comprising instructions, which, when executed on a computer, causes the computer to perform the method according to any one of claims 1 to 11.
25. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.
