Video data acquisition method, video creation method and related products

Abstract

The embodiment of the present application provides a video data acquisition method, a video creation method and related products, which includes: acquiring image data and posture data of at least one frame of an original video; calculating target augmented reality AR data according to the image data and the posture data; the target AR data includes at least one of plane data, anchor point data and grid data; the target AR data is used for video augmented reality creation. The embodiment of the present application can improve the processing effect of video augmented reality.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video data acquisition method, a video creation method and related products. Background Art

[0002] In recent years, with the widespread use of mobile Internet, people's entertainment needs have been increasing, from the initial simple photo taking, beauty photo taking, to short videos, live broadcasts and other two-dimensional forms, new elements have been gradually incorporated. Currently, the video editing or video creation tools on the market are still at the two-dimensional processing level, and the sense of reality of augmented reality is poor. Summary of the invention

[0003] The embodiments of the present application provide a video data acquisition method, a video creation method and related products to improve the processing effect of video augmented reality.

[0004] A first aspect of an embodiment of the present application provides a video data acquisition method, including:

[0005] Collecting image data and posture data of at least one frame of an original video;

[0006] Target augmented reality (AR) data is calculated based on the image data and the posture data; the target AR data includes at least one of plane data, anchor point data, and grid data; and the target AR data is used for video augmented reality creation.

[0007] When collecting video data, the embodiment of the present application calculates target AR data for video augmented reality creation based on image data and posture data of at least one frame of the collected original video. Since the target AR data includes plane data, anchor point data, and grid data, the plane of addable materials for each frame of image can be determined based on the plane data, the three-dimensional anchor point coordinates and material orientation of the addable materials in each frame of image can be determined based on the anchor point data, and the three-dimensional grid of each frame of image can be determined based on the grid data, so that the combination of the added material and each frame of image in the video is more realistic, thereby improving the processing effect of video augmented reality.

[0008] Optionally, calculating target augmented reality (AR) data according to the image data and the posture data includes:

[0009] Calculate initial AR data according to the image data and the posture data;

[0010] The initial AR data are filtered according to the quality scores of the initial AR data, and the target AR data having a quality score greater than a set threshold is retained.

[0011] The embodiment of the present application can filter the initial AR data by the quality score of the initial AR data, so that the reliability of the filtered target AR data is greatly enhanced, and video creation is performed on the reliable target AR data, thereby reducing the probability of distortion of the video creation effect.

[0012] Optionally, after calculating augmented reality target AR data according to the image data and the posture data, the method further includes:

[0013] The target AR data is used to add three-dimensional material to the original video to obtain an augmented reality video.

[0014] Among them, after collecting image data and posture data, the target AR data can be used to add three-dimensional materials to the original video to obtain an augmented reality video, so that the combination of the added three-dimensional material and each frame of the video is more realistic, thereby improving the processing effect of video augmented reality.

[0015] Optionally, the method further includes:

[0016] Determining video classification data, wherein the video classification data includes any one of video voice keywords, video text data, and video scene data;

[0017] After calculating the target augmented reality (AR) data according to the image data and the posture data, the method further includes:

[0018] Encoding the image data, the target AR data, and the depth data of the at least one frame of image, or encoding the image data, the target AR data, the depth data of the at least one frame of image, and the video classification data to obtain an encoded video file or video stream data;

[0019] The encoded video file or video stream data is uploaded to the server.

[0020] The video classification data can be used as a reference for adding 3D materials later, to avoid the theme of the added 3D materials being inconsistent with the video content, and to improve the video fusion effect. The encoded video stream data is uploaded to the server, and the server can push the video stream data to the video playback client for video editing and video playback, which is convenient for subsequent video editing and video playback.

[0021] Optionally, the depth data is calculated based on the image data and the posture data; or, the depth data is calculated based on initial depth information and the image data collected by a depth camera, or the depth data is calculated based on a binocular image collected by a binocular camera.

[0022] A second aspect of the embodiments of the present application provides a video creation method, including:

[0023] Get video files or video stream data;

[0024] Decoding the video file or video stream data to obtain image data of each frame image, target augmented reality AR data and depth data;

[0025] The target AR data and the depth data are used to add three-dimensional material to each frame of the image, and the three-dimensional material is fused with the image data to obtain an augmented reality video.

[0026] When creating a video, since the target AR data includes plane data, anchor point data, and grid data, the plane to which materials can be added to each frame of the image can be determined based on the plane data, the three-dimensional anchor point coordinates and material orientation of the materials that can be added to each frame of the image can be determined based on the anchor point data, and the three-dimensional grid of each frame of the image can be determined based on the grid data. This makes the combination of the added materials with each frame of the video more realistic, thereby improving the processing effect of video augmented reality.

[0027] Optionally, the target AR data includes plane data and anchor point data, and adding a three-dimensional material to each frame image by using the target AR data and the depth data includes:

[0028] Determine the three-dimensional space coordinates of each frame of image according to the depth data, determine the plane to which the material can be added in each frame of image according to the plane data, and determine the three-dimensional anchor point coordinates and material orientation of the material to be added in each frame of image according to the anchor point data and the three-dimensional space coordinates;

[0029] Determine the size of the addable material according to the size of the plane of the addable material and the three-dimensional anchor point coordinates, and select the three-dimensional material according to the size of the addable material;

[0030] The three-dimensional material is added to the three-dimensional anchor point coordinates according to the material orientation.

[0031] Optionally, the target AR data further includes grid data, and before fusing the three-dimensional material with the image data, the method further includes:

[0032] Determine a three-dimensional grid of each frame of image according to the grid data and the three-dimensional space coordinates, perform collision detection on the three-dimensional material and the three-dimensional grid, and obtain a collision detection result;

[0033] The fusing process of the three-dimensional material with the image data includes:

[0034] The three-dimensional material and the image data are fused and processed according to the collision detection result.

[0035] The electronic device can perform collision detection based on the three-dimensional material and the three-dimensional grid to obtain the collision detection result. The collision detection result can be used in the subsequent fusion processing of the three-dimensional material and the image data, so as to avoid the overlap of the three-dimensional material and the three-dimensional grid, and avoid the visual perception of the three-dimensional material embedded in the three-dimensional grid, thereby improving the display effect of the augmented reality video. Among them, when the three-dimensional material is in motion, when the three-dimensional coordinates of the three-dimensional material and the coordinates of the three-dimensional grid overlap, it can be judged that the three-dimensional material is in contact with the three-dimensional grid, and the collision between the two is detected. After the collision, the trajectory of the three-dimensional material after the collision can be simulated according to the movement direction of the three-dimensional material. Performing collision detection based on the three-dimensional material and the three-dimensional grid can avoid the overlap of the three-dimensional material and the three-dimensional grid, and avoid the visual perception of the three-dimensional material embedded in the three-dimensional grid, thereby improving the display effect of the augmented reality video.

[0036] Optionally, after decoding the video file or video stream data to obtain image data, target AR data and depth data of each frame image, before adding a three-dimensional material to each frame image using the target AR data and the depth data and fusing the three-dimensional material with the image data to obtain an augmented reality video, the method further includes:

[0037] Get video classification data;

[0038] The three-dimensional material corresponding to the video classification data is selected from a material library.

[0039] Video classification data can be used as a reference when adding three-dimensional materials to avoid the theme of the added three-dimensional materials being incompatible with the video content, thereby improving the video fusion effect.

[0040] Optionally, after adding a three-dimensional material to each frame of the image using the target AR data and the depth data, and fusing the three-dimensional material with the image data to obtain an augmented reality video, the method further includes:

[0041] Determining a virtual scene rendering effect of the augmented reality video according to a user portrait of the player;

[0042] The augmented reality video is rendered using the virtual scene rendering effect.

[0043] Different user portraits can be matched with corresponding rendering effects to achieve a video experience that is different for each person.

[0044] Optionally, the 3D material includes a 3D video or a 3D picture.

[0045] A third aspect of an embodiment of the present application provides a video data acquisition device, including:

[0046] A collection unit, used to collect image data and posture data of at least one frame of an original video;

[0047] A computing unit is used to compute target augmented reality (AR) data based on the image data and the posture data; the target AR data includes at least one of plane data, anchor point data, and grid data; the target AR data is used for video augmented reality creation.

[0048] A fourth aspect of an embodiment of the present application provides a video creation device, including:

[0049] An acquisition unit, used for acquiring video files or video stream data;

[0050] A decoding unit, used to decode the video file or video stream data to obtain image data of each frame image, target augmented reality AR data and depth data;

[0051] The video processing unit is used to add three-dimensional materials to each frame of the image using the target AR data and the depth data, and fuse the three-dimensional materials with the image data to obtain an augmented reality video.

[0052] The fifth aspect of the embodiments of the present application provides an electronic device, including a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the step instructions in the first aspect or the second aspect of the embodiments of the present application.

[0053] The sixth aspect of the embodiments of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program enables the computer to execute part or all of the steps described in the first aspect or the second aspect of the embodiments of the present application.

[0054] A seventh aspect of the embodiments of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in the first aspect or the second aspect of the embodiments of the present application. The computer program product may be a software installation package.

[0055] In an embodiment of the present application, image data and posture data of at least one frame of an original video are collected; target augmented reality AR data is calculated based on the image data and the posture data; the target AR data includes at least one of plane data, anchor point data, and grid data; the target AR data is used for video augmented reality creation. In an embodiment of the present application, when collecting video data, target AR data for video augmented reality creation is calculated based on the image data and posture data of at least one frame of an original video collected. Since the target AR data includes plane data, anchor point data, and grid data, the plane of the addable material of each frame of the image can be determined based on the plane data, the three-dimensional anchor point coordinates and material orientation of the addable material in each frame of the image can be determined based on the anchor point data, and the three-dimensional grid of each frame of the image can be determined based on the grid data, so that the combination of the added material and each frame of the video is more realistic, thereby improving the processing effect of video augmented reality. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0057] Figure 1 It is a flowchart of a video data acquisition method provided in an embodiment of the present application;

[0058] Figure 2 is a schematic diagram of the structure of a face grid provided in an embodiment of the present application;

[0059] Figure 3 is a schematic diagram of a grid structure provided in an embodiment of the present application;

[0060] Figure 4 It is a flowchart of another video data acquisition method provided in an embodiment of the present application;

[0061] Figure 5 It is a schematic diagram of another process of video creation provided by an embodiment of the present application;

[0062] Figure 6 A schematic diagram of the structure of a video data acquisition device provided in an embodiment of the present application;

[0063] Figure 7 A schematic diagram of the structure of a video creation device provided in an embodiment of the present application;

[0064] Figure 8 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0066] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0067] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0068] The electronic devices involved in the embodiments of the present application may include devices with a camera and image and video processing capabilities, such as mobile phones, tablet computers, etc.

[0069] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a video data acquisition method provided in an embodiment of the present application. Figure 1 As shown, the method includes the following steps.

[0070] 101. An electronic device collects image data and posture data of at least one frame of an original video.

[0071] In an embodiment of the present application, the electronic device can collect image data of each frame of video image through a camera, and collect the posture data corresponding to each frame of video image through an inertial measurement unit (IMU). Each frame of video image can be referred to as each frame of image. Each frame of image can include image data and posture data. Image data is the image content of each frame of image, such as the RGB value of the pixel of the image. The posture data can be the displacement and rotation information of three dimensions collected by the IMU. The IMU can measure the three-axis attitude angle, three-axis acceleration and three-axis displacement of the electronic device. In each frame of image, the IMU can measure the posture data corresponding to the frame of video image.

[0072] 102. The electronic device calculates target augmented reality (AR) data based on the image data and the posture data. The target AR data is used for video augmented reality creation.

[0073] The target AR data includes at least one of plane data, anchor point data, and grid data.

[0074] The electronic device can also determine the depth data of each frame of the image based on the image data and the posture data. The depth data can be calculated based on the image data and the posture data. The depth data can be measured by a rotatable camera. According to the rotation of the camera, the camera is equivalent to taking two pictures, that is, a parallax of the two pictures, and an effective depth is finally calculated based on the rotation of the camera that took the picture.

[0075] The depth data in the implementation of the present application may be: depth information of pixels in each frame of the video.

[0076] Optionally, after executing step 102, the following steps may also be performed:

[0077] The electronic device uses the target AR data to add three-dimensional materials to the original video to obtain an augmented reality video.

[0078] In the embodiment of the present application, after the video shooter shoots the original video, the three-dimensional material can be automatically added to the original video according to the target AR data to obtain an augmented reality video. Since the target AR data includes plane data, anchor point data, and grid data, the plane to which the material can be added in each frame of the image can be determined according to the plane data, the three-dimensional anchor point coordinates and material orientation of the material to be added in each frame of the image can be determined according to the anchor point data, and the three-dimensional grid of each frame of the image can be determined according to the grid data, so that the combination of the added material and each frame of the video is more realistic, thereby improving the processing effect of video augmented reality.

[0079] After the electronic device uses the target AR data to add three-dimensional materials to the original video to obtain an augmented reality video, the electronic device can also play the augmented reality video.

[0080] Optionally, the depth data can also be calculated using the initial depth information collected by the depth camera and the image data. The depth camera can obtain the depth data of each pixel on each frame of the image. The depth data can accurately describe the depth of the pixel and convert the depth of each pixel into an accurate coordinate in the three-dimensional space. A three-dimensional object can be placed on any pixel in the three-dimensional space.

[0081] Optionally, the depth data can also be calculated by binocular images collected by binocular cameras. Specifically, two cameras simultaneously take a frame of image, calculate the parallax of the images collected by the two cameras, and the distance between the two cameras, and finally calculate an effective depth.

[0082] Optionally, when calculating the depth data of each pixel on each frame of the image, the depth information of only some of the points can be calculated. For example, the depth information of a point is found every three pixels or five pixels. The depth information of the pixels in the middle can be calculated in combination with the RGB image data of each frame of the image.

[0083] After determining the depth data of each frame of image, the depth data of each pixel point of each frame of image can be converted into accurate coordinates in three-dimensional space, and the three-dimensional coordinates of each pixel point of each frame of image in three-dimensional space can be obtained. Each frame of image can be mapped to three-dimensional space through the depth data of each pixel point of each frame of image. With the three-dimensional coordinates of each pixel point of each frame of image in three-dimensional space, it is convenient to calculate plane data, anchor point data, grid data, etc.

[0084] Plane data refers to the data of planar objects in each frame of image. For example, planar objects may include tables, walls, floors, ceilings, etc. in the image. Planar object data may include the size of the planar object, its coordinate information in the image, and its three-dimensional coordinate information in three-dimensional space.

[0085] Anchor point data can be the position where the 3D material is placed on each frame of the image. The anchor point can be set on a plane, or it can be set above or below the plane. For example, after detecting the plane, you can set an anchor point at a certain position on the plane. This anchor point is a three-dimensional point. When you place the 3D material in the future, you can place the 3D material on this anchor point. Setting the anchor point on the plane will give the 3D material a sense of fit with the plane, so that the inserted 3D material will not be too abrupt, the 3D material will blend better with the plane, and the 3D effect will be more realistic. Setting the anchor point on the plane will create a sense of drift.

[0086] The anchor point data includes the 3D position data and 3D normal vector of the anchor point. The 3D position data of the anchor point is used to indicate the coordinates of the anchor point in the 3D space. The 3D normal vector of the anchor point is used to indicate the direction of the anchor point. The normal vector can be understood as the orientation of the 3D material placed at the anchor point. For example, if you put a point on a wall plane in 2D space, the point is right here and has no 3D information. In 3D space, if you put a point on the wall plane, such as on the west wall, the normal vector is facing east. This anchor point has a direction information, which reflects the 3D sense, so that the 3D materials added later have a richer 3D effect.

[0087] The normal vector is related to the visual effect of the 3D material to be added. For example, there is a wall in the picture, and the video creator wants to put a 3D object on the wall. If the video creator wants the 3D object to be completely attached to the wall, the normal vector should be set perpendicular to the wall. If the video creator wants the wall to crack, the normal vector should be set slightly offset in the vertical direction. The 3D position data of the anchor point determines where the added 3D object is, and the normal vector determines how the 3D object rotates.

[0088] Mesh data, also called mesh data, is similar to mesh information. Figure 2 , Figure 2 Schematic diagram of the structure of a face grid provided in an embodiment of the present application. Figure 2 There are many facial feature points in the picture. If these facial feature points are connected into triangles according to a certain rule, the face is equivalent to a piece of information, which is equivalent to a mesh. This piece of information is equivalent to putting a mask on the face. In three-dimensional space, if texture information is not considered (RGB information of the image is not considered), the entire person or object can be described by mesh. For example, if the video creator wants a small ball to run down from an irregular object, he will do a collision detection with the small ball along the mesh in the image. In this way, it feels like the small ball is rolling on the irregular object, which is equivalent to moving forward along the surface mesh of the irregular object.

[0089] Figure 3The plane in the video can be equivalent to a mesh with many points on it. For example, when a ball rolls on a plane, in fact, the final result is a collision detection between the ball and the mesh layer below. When the ball rolls in three-dimensional space, because it is a three-dimensional space, the physical properties of the display space are not displayed in the three-dimensional space (in the real space, if the ball is on the table, it must be on the table. This is because the table gives the ball a supporting force, so the ball will not fall off the table). In the computer, a plane is detected, which is a mesh with a three-dimensional ball. Although this mesh is a plane, it can also be understood as a three-dimensional object. When the three-dimensional ball is added to the video, a collision detection is performed between the mesh and the three-dimensional ball. The collision detection means that when the mesh contacts the three-dimensional ball, it is considered that no further movement can be made. At this time, you can see the ball moving on the mesh. If there is no mesh, the video creator or video editing software has no idea which object the ball is doing collision detection with. In three-dimensional space, mesh is equivalent to Figure 3 This table, with the ball placed on it, uses its characteristics to detect that the ball is placed on the table, and the ball will not fall off the table, but will move forward along the table. Some information in three-dimensional space can be used to describe some things that correspond to real physics. Because there is no concept of supporting force in computers, video creators or video editing software must know whether the inserted three-dimensional material overlaps with certain elements in the image in three-dimensional space to avoid losing realism. Whether the inserted three-dimensional material overlaps with certain elements in the image can be determined by whether the inserted three-dimensional material overlaps with the grid in the image. If so, it indicates overlap.

[0090] The mesh can be calculated. There are several ways, such as Figure 2 For this kind of mesh of a face, we first detect key points, and detect some two-dimensional key point information. According to the key point information and the face information, we map it into three-dimensional space to get the mesh data of the face. For another example, we can also use parallax. When there is motion (movement of the camera of an electronic device or movement of the photographed object), the position data measured by IMU will have depth because of parallax formed by motion. With depth, we can know how far the photographed object (for example, a wall) is from us, and we can use the depth data to calculate the mesh data of the photographed object.

[0091] Mesh data can be divided into: object mesh data, human body mesh data, and face mesh data.

[0092] like Figure 3 , it will detect that there is an upper plane and a lower plane in the image, Figure 3The little robot in the video was put in later. The original video is two-dimensional. Because the little robot is placed in a three-dimensional space, this three-dimensional space is the information of this plane when shooting. So when the little robot is put in like this, it feels like the little robot is standing on the table or the wall.

[0093] Optionally, the target AR data may also include at least one of object segmentation data, human expression data, and human skeleton data.

[0094] Object segmentation data is the data of objects that can be segmented in an image. The segmented objects can be human bodies, human faces, human heads, etc. Segmentation data is mainly used for occlusion. For example, when we put three-dimensional advertising materials, there is a wall behind the three-dimensional advertising materials. When someone passes in front of the wall, the put-in three-dimensional advertising materials will not block the human body (for example, the human head and face).

[0095] Facial expression data and human skeleton data can be calculated through artificial intelligence algorithms. When a certain frame of a video contains a face, the facial expression data in that frame can be calculated; when a certain frame of a video contains a human body, the human skeleton data in that frame can be calculated.

[0096] Video editing software or video creators can determine the plane where materials can be added to each frame of the image based on the plane data, determine the three-dimensional anchor point coordinates and material orientation of the materials that can be added to each frame of the image based on the anchor point data, and determine the three-dimensional grid of each frame of the image based on the grid data, so that the combination of the added materials and each frame of the video is more realistic, thereby improving the processing effect of video augmented reality.

[0097] When creating a video, the above-mentioned target AR data can be displayed in the video to assist the video creator in the creation. For example, when a video creator wants to put a three-dimensional object into a video, the video creator can put the three-dimensional object on a plane. For example, if there is a table in the shot video, the video creator or video editing software will put a three-dimensional toy on the table after recognizing the table. When the video creator or video editing software puts the toy in the three-dimensional space, the position of the plane in the three-dimensional space can be known based on the plane data. So when the video creator or video editing software puts the toy in, the toy and the video are integrated together. This effect is completely different from that of a two-dimensional one, and it is obviously more realistic and more integrated than a two-dimensional one.

[0098] Optionally, step 102 may include the following steps:

[0099] (11) the electronic device calculates initial AR data based on the image data and the posture data;

[0100] (12) The electronic device filters the initial AR data according to the quality score of the initial AR data, and retains the target AR data whose quality score is greater than a set threshold.

[0101] In the embodiment of the present application, the electronic device can calculate the initial AR data according to the image data and the posture data through the algorithm framework. The algorithm framework may include a general algorithm framework such as ARcore, ARkit, Slam, etc. It can be used to detect the target AR data in the video. The target AR data may include at least one of plane data, anchor point data, and grid data.

[0102] The initial AR data calculated by the general algorithm framework may have certain errors, and the calculated plane data, anchor point data, and grid data may not meet the needs of subsequent video creation. If the plane calculated by the algorithm framework is not necessarily a plane, errors will occur in the subsequent video creation, resulting in distorted video creation effects.

[0103] The embodiment of the present application can filter the initial AR data by the quality score of the initial AR data, so that the reliability of the filtered target AR data is greatly enhanced, and video creation is performed on the reliable target AR data, thereby reducing the probability of distortion of the video creation effect.

[0104] For example, for plane data, after the general algorithm framework calculates the N planes, it can calculate the degree of discreteness between the pixels on the N planes and the calculated planes, and calculate the quality score of the plane based on the degree of discreteness. The degree of discreteness can be calculated by the three-dimensional coordinates of the pixels on the first plane, and the mean of the vertical distance between the pixels on the first plane and the calculated first plane. If the mean is larger, the degree of discreteness is larger and the quality score is lower; if the mean is smaller, the degree of discreteness is smaller and the quality score is higher. For example, based on the quality score of the plane, we can filter out planes with a quality score less than 0.5.

[0105] See also Figure 4 , Figure 4 It is a flowchart of another video data acquisition method provided in an embodiment of the present application. Figure 4 is Figure 1 Based on the further optimization, Figure 4 As shown, the method includes the following steps.

[0106] 401, the electronic device collects image data and posture data of at least one frame of an original video.

[0107] 402, the electronic device calculates target augmented reality (AR) data according to the image data and the posture data, and the target AR data is used for video augmented reality creation.

[0108] The specific implementation of steps 401 to 402 can be found in Figure 1 Steps 101 to 102 shown are not repeated here.

[0109] 403 , the electronic device determines video classification data, where the video classification data includes any one of video voice keywords, video text data, and video scene data.

[0110] In the embodiment of the present application, in the process of collecting the original video, the video voice can be collected and the video voice keywords in the video voice can be extracted. The video text data in the original video can be extracted by the OCR algorithm. The video scene data scene can be an overview of the video content, and the video scene data can exist in the form of tags. For example, the video content can be tagged with scene tags such as wedding, pet, scenery, etc.

[0111] The video classification data can be used as a reference when adding three-dimensional materials later, so as to avoid the theme of the added three-dimensional materials being incompatible with the video content and improve the video fusion effect.

[0112] Specifically, the electronic device can collect the video voice in the original video through the microphone of the electronic device, convert the video voice into voice text through the voice recognition algorithm, and extract the video voice keywords from the voice text. The electronic device can extract the video text data in the original video through the OCR algorithm. The electronic device can determine the video scene data by identifying the objects in the video content. If the objects in the video content are identified to include scenery, the video content can be marked with a scene tag of "scenery", if the objects in the video content are identified to include pets, the video content can be marked with a scene tag of "pet", and if the objects in the video content are identified to include a wedding stage, the video content can be marked with a scene tag of "wedding".

[0113] Video scene data can be used for subsequent 3D advertising insertion. After identifying the scene tag, the corresponding 3D advertisement can be automatically inserted into the video according to the scene tag. Improve the relevance of the inserted 3D advertisement to the video content, improve the effectiveness of advertising, and improve the order conversion rate of advertising traffic. For example, if the scene tag is "scenery", 3D advertisements for sports equipment can be inserted into the video. If the scene tag is "pet", 3D advertisements for dog food and dog clothes can be inserted into the video. If the scene tag is "wedding", 3D advertisements related to diamond rings, wedding dresses, and wedding decorations can be inserted into the video. 3D advertisements can exist in the form of videos or pictures.

[0114] Optionally, after executing step 403, the following steps may also be executed:

[0115] 404, the electronic device encodes the image data, the target AR data, and the depth data of at least one frame of image, or encodes the image data, the target AR data, the depth data of at least one frame of image, and the video classification data to obtain an encoded video file or video stream data;

[0116] 405, the electronic device uploads the encoded video file or video stream data to the server.

[0117] In an embodiment of the present application, a digital video compression format such as H.264, H.265, H.266, etc. may be used to encode image data, target AR data, and depth data of at least one frame of image, or to encode image data, target AR data, depth data of at least one frame of image, and video classification data.

[0118] The target AR data, the depth data of the at least one frame of image, and the video classification data may be stored in a set area of ​​the video, or saved through an additional file, and only the corresponding frames are required.

[0119] Video files can be saved locally or uploaded to a server. Video stream data can be uploaded to a server, supporting simultaneous downloading and playback on the consumer side.

[0120] The encoded video stream data is uploaded to the server, and the server can push the video stream data to the video playback client for video editing and video playback, which is convenient for subsequent video editing and video playback.

[0121] On the video player client, you can add 3D elements to the video before it plays, such as 3D advertisements, 3D characters, 3D videos, 3D pictures, etc.

[0122] See also Figure 5 , Figure 5 FIG. 1 is a flow chart of a video creation method provided in an embodiment of the present application. Figure 5 As shown, the method includes the following steps.

[0123] 501, the electronic device obtains a video file or video stream data.

[0124] In the embodiment of the present application, the electronic device can download video files from a server, or download video streaming data online.

[0125] 502, the electronic device decodes the video file or the video stream data to obtain image data of each frame image, target augmented reality AR data and depth data.

[0126] The target AR data may include at least one of plane data, anchor point data, and grid data. Figures 1 to 4The relevant description in will not be repeated here.

[0127] The decoding step of the embodiment of the present application corresponds to the encoding step of step 404 and will not be repeated here.

[0128] 503, the electronic device uses the target AR data and the depth data to add a three-dimensional material to each frame of the image, and fuses the three-dimensional material with the image data to obtain an augmented reality video.

[0129] After determining the depth data of each frame of image, the depth data of each pixel of each frame of image can be converted into accurate coordinates in three-dimensional space, and the three-dimensional coordinates of each pixel of each frame of image in three-dimensional space can be obtained. Each frame of image can be mapped to three-dimensional space through the depth data of each pixel of each frame of image. With the three-dimensional coordinates of each pixel of each frame of image in three-dimensional space, it is easy to calculate plane data, anchor point data, grid data, etc.

[0130] In the embodiment of the present application, when creating a video, since the target AR data includes plane data, anchor point data, and grid data, the plane to which the material can be added in each frame of the image can be determined based on the plane data, the three-dimensional anchor point coordinates and material orientation of the material that can be added in each frame of the image can be determined based on the anchor point data, and the three-dimensional grid of each frame of the image can be determined based on the grid data, so that the combination of the added material and each frame of the video is more realistic, thereby improving the processing effect of video augmented reality. The three-dimensional material and image data are fused and processed, and the three-dimensional image and image data are more integrated, so that the inconsistency between the material added in each frame and the image can be avoided.

[0131] In the embodiment of the present application, the description of the plane data, anchor point data and grid data can be found in the above Figures 1 to 4 The relevant description in will not be repeated here.

[0132] Figure 5 Electronic equipment and Figures 1 to 4 The electronic devices in the device may be different or the same.

[0133] Optionally, the target AR data includes plane data and anchor point data. In step 503, the electronic device may add a three-dimensional material to each frame of the image using the target AR data and the depth data, which may include the following steps:

[0134] (21) The electronic device determines the three-dimensional spatial coordinates of each frame of the image based on the depth data, determines the plane of the addable material for each frame of the image based on the plane data, and determines the three-dimensional anchor point coordinates and material orientation of the addable material in each frame of the image based on the anchor point data and the three-dimensional spatial coordinates.

[0135] (22) The electronic device determines the size of the addable material according to the size of the plane of the addable material and the three-dimensional anchor point coordinates, and selects the three-dimensional material according to the size of the addable material.

[0136] (23) The electronic device adds the three-dimensional material to the three-dimensional anchor point coordinates according to the orientation of the material.

[0137] In the embodiment of the present application, the electronic device can determine at least one plane of each frame image to which materials can be added based on the plane data. The planes in each frame image are generally suitable for adding materials, but in some cases, the plane is not suitable for adding certain materials. For example, the plane of the ceiling is not suitable for adding a three-dimensional figure standing on the ceiling, and the plane of the wall is not suitable for adding a cup of water.

[0138] The anchor point data may include the three-dimensional position data and the three-dimensional normal vector of the anchor point. The three-dimensional position data of the anchor point is used to represent the coordinates of the anchor point in the three-dimensional space. The three-dimensional normal vector of the anchor point is used to represent the direction of the anchor point.

[0139] The electronic device can determine the 3D anchor point coordinates of the addable material in each frame image based on the 3D position data of the anchor point and the 3D spatial coordinates of each frame image, and can determine the material orientation of the addable material in each frame image based on the 3D normal vector of the anchor point.

[0140] The embodiment of the present application can determine the size of the addable material based on the size of the plane of the addable material and the three-dimensional anchor point coordinates. Specifically, it can be determined based on the projection size of the addable material on the two-dimensional plane (for example, the plane where the screen is located). If the projection size of the addable material in two dimensions is within a suitable range, it indicates that the size of the addable material is suitable; if the projection size of the addable material in two dimensions is less than the minimum size of the suitable range, it indicates that the size of the addable material is too small; if the projection size of the addable material in two dimensions is less than the maximum size of the suitable range, it indicates that the size of the addable material is too large. Generally speaking, the larger the size of the plane, the larger the size of the addable material. If the three-dimensional anchor point is closer to the screen side, the smaller the size of the addable material, if the coordinates of the three-dimensional anchor point are farther from the screen side, the larger the size of the addable material.

[0141] When inserting a 3D object in a large space, you need to make the 3D object close to nature. If the 3D object is large and close to the screen, the whole picture will be out of harmony. If the 3D object is small and far from the screen, the 3D object will not be clearly visible. In addition, when inserting an object, it is better to insert the 3D object in an appropriate scale and direction. If it is far from the screen, choose a large 3D object; if it is close to the screen, choose a small 3D object. For a water cup, if it is far from the screen, you can put a large water cup, and the size of the water cup projected to the two-dimensional is moderate; if it is close to the screen, you can put a small water cup, and the size of the water cup projected to the two-dimensional is also moderate. This can avoid the picture from being out of harmony and too abrupt.

[0142] Optionally, the target AR data further includes grid data, and before the electronic device fuses the three-dimensional material with the image data, the following steps may be performed:

[0143] (31) The electronic device determines a three-dimensional grid of each frame image according to the grid data and the three-dimensional space coordinates;

[0144] (32) the electronic device performs collision detection on the three-dimensional material and the three-dimensional grid to obtain a collision detection result;

[0145] The electronic device fuses the three-dimensional material with the image data, which may specifically include the following steps:

[0146] The electronic device fuses the three-dimensional material with the image data according to the collision detection result.

[0147] Any physical object in the image can be represented by mesh data, which is three-dimensional data. There can be multiple three-dimensional meshes in the image. The electronic device can determine the three-dimensional mesh of each frame of the image based on the mesh data. Any three-dimensional object in the image can be represented by mesh data. For example, the wall, table, cup, face, etc. in the image. The three-dimensional mesh can be composed of multiple points connected on the surface of the three-dimensional object in the image. If the added three-dimensional material overlaps with the three-dimensional mesh, the visual feeling of the three-dimensional material embedded in the three-dimensional object will appear, resulting in a significant reduction in the effect of the augmented reality video. For example, the inserted three-dimensional material is a ping-pong ball, and the mesh data is the three-dimensional mesh of the table. When the inserted three-dimensional material overlaps with the three-dimensional mesh of the table, the visual feeling of the ping-pong ball embedded in the table will appear, and the visual effect of the ping-pong ball rolling or bouncing on the table cannot be presented, resulting in a significant reduction in the visual effect.

[0148] The electronic device can perform collision detection based on the three-dimensional material and the three-dimensional grid to obtain a collision detection result. The collision detection result can include the possible motion trajectory of the three-dimensional material in the three-dimensional space of the continuous multi-frame image. The electronic device can determine the motion trajectory of the three-dimensional material in the video based on the above collision detection result, which can avoid the overlap of the three-dimensional material and the three-dimensional grid in the augmented reality video after fusion processing, and avoid the visual experience of the three-dimensional material embedded in the three-dimensional grid in the augmented reality video, thereby improving the display effect of the augmented reality video. Among them, when the three-dimensional material is in motion, when the three-dimensional coordinates of the three-dimensional material coincide with the coordinates of the three-dimensional grid, it can be judged that the three-dimensional material is in contact with the three-dimensional grid, and the collision between the two is detected. After the collision, the trajectory of the three-dimensional material after the collision can be simulated according to the movement direction of the three-dimensional material. Performing collision detection based on the three-dimensional material and the three-dimensional grid can avoid the overlap of the three-dimensional material and the three-dimensional grid, and avoid the visual experience of the three-dimensional material embedded in the three-dimensional grid, thereby improving the display effect of the augmented reality video.

[0149] Optionally, after executing step 502 and before executing step 503, the following steps may also be performed:

[0150] (41) The electronic device obtains video classification data;

[0151] (42) The electronic device selects three-dimensional material corresponding to the video classification data from the material library.

[0152] In the embodiment of the present application, the video classification data includes any one of video voice keywords, video text data, and video scene data. The video classification data can be a label of the video content, and the video content can have one or more labels. The video classification data can be used as a reference when adding three-dimensional materials to avoid the theme of the added three-dimensional materials being incompatible with the video content, thereby improving the video fusion effect.

[0153] For example, if the video classification data includes "scenery", you can insert 3D advertising materials for sports goods into the video. If the video classification data includes "pets", you can insert 3D advertising materials for dog food and dog clothes into the video. If the video classification data includes "wedding", you can insert 3D advertising materials related to diamond rings, wedding dresses, and wedding decorations into the video. 3D advertising materials can exist in the form of videos or pictures.

[0154] Optionally, after executing step 503, the following steps may also be executed:

[0155] (51) The electronic device determines a virtual scene rendering effect of the augmented reality video according to a user portrait of the player;

[0156] (52) The electronic device uses the virtual scene rendering effect to render the augmented reality video.

[0157] In the embodiment of the present application, the user portrait may include the user's age, gender, skin color, interests, hobbies, etc. Different user portraits can match corresponding virtual scene rendering effects. The virtual scene rendering inside the video can be triggered in combination with the player's preferences, browsing data, etc., to achieve a video experience with thousands of faces for thousands of people.

[0158] Among them, after the electronic device determines the virtual scene rendering effect of the augmented reality video according to the user portrait of the player, the electronic device can also use the virtual scene rendering effect to render the augmented reality video, and after obtaining the rendered augmented reality video, play the rendered augmented reality video.

[0159] Optionally, the electronic device determines the virtual scene rendering effect of the augmented reality video according to the player's location based services (LBS) data. The LBS data may include GPS data.

[0160] The 3D material includes a 3D video or a 3D picture. When the 3D material is a 3D video, a picture-in-picture effect can be achieved. The 3D video can be presented in the form of a television or a movie screen.

[0161] The above mainly introduces the scheme of the embodiment of the present application from the perspective of the execution process on the method side. It is understandable that, in order to realize the above functions, the electronic device includes a hardware structure and / or software module corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiments provided herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0162] The embodiment of the present application can divide the electronic device into functional units according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0163] See also Figure 6 , Figure 6The schematic diagram of the structure of a video data acquisition device provided in an embodiment of the present application is shown in FIG. 60. The video data acquisition device 600 is applied to an electronic device. The video data acquisition device 600 may include an acquisition unit 601 and a calculation unit 602, wherein:

[0164] The acquisition unit 601 is used to acquire image data and posture data of at least one frame of an original video;

[0165] The calculation unit 602 is used to calculate target augmented reality (AR) data based on the image data and the posture data; the target AR data includes at least one of plane data, anchor point data, and grid data; the target AR data is used for video augmented reality creation.

[0166] Optionally, the calculation unit 602 calculates target augmented reality AR data based on the image data and the posture data, including: calculating initial AR data based on the image data and the posture data; filtering the initial AR data according to a quality score of the initial AR data, and retaining the target AR data having a quality score greater than a set threshold.

[0167] Optionally, the video data acquisition device 600 may further include an adding unit 603;

[0168] The adding unit 603 is used to add three-dimensional materials to the original video using the target AR data to obtain the augmented reality video after the calculating unit 602 calculates the augmented reality target AR data according to the image data and the posture data.

[0169] Optionally, the video data acquisition device 600 may further include a first determination unit 604, an encoding unit 605 and an uploading unit 606;

[0170] The first determining unit 604 is configured to determine video classification data, where the video classification data includes any one of video voice keywords, video text data, and video scene data.

[0171] The encoding unit 605 is used to encode the image data, the target AR data and the depth data of the at least one frame of image, or to encode the image data, the target AR data, the depth data of the at least one frame of image and the video classification data to obtain an encoded video file or video stream data;

[0172] The uploading unit 606 is used to upload the encoded video file or video stream data to the server.

[0173] Optionally, the depth data is calculated based on the image data and the posture data; or, the depth data is calculated based on initial depth information and the image data collected by a depth camera, or the depth data is calculated based on a binocular image collected by a binocular camera.

[0174] The acquisition unit 601 in the embodiment of the present application is a camera module and an IMU in an electronic device. The calculation unit 602, the adding unit 603, the first determination unit 604 and the testing unit 603 can be a processor of an electronic device, and the uploading unit 606 can also be a communication module in the electronic device.

[0175] In an embodiment of the present application, when video data is collected, target AR data for video augmented reality creation is calculated based on image data and posture data of at least one frame of the collected original video. Since the target AR data includes plane data, anchor point data, and grid data, the plane of addable materials for each frame of image can be determined based on the plane data, the three-dimensional anchor point coordinates and material orientation of the addable materials in each frame of image can be determined based on the anchor point data, and the three-dimensional grid of each frame of image can be determined based on the grid data. This makes the combination of the added material with each frame of image in the video more realistic, thereby improving the processing effect of video augmented reality.

[0176] See also Figure 7 , Figure 7 A schematic diagram of the structure of a video creation device provided in an embodiment of the present application, wherein the video creation device 700 is applied to an electronic device, and the video creation device 700 may include an acquisition unit 701, a decoding unit 702, and a video processing unit 703, wherein:

[0177] The acquisition unit 701 is used to acquire video files or video stream data;

[0178] A decoding unit 702 is used to decode the video file or video stream data to obtain image data of each frame image, target augmented reality AR data and depth data;

[0179] The video processing unit 703 is used to add three-dimensional materials to each frame of the image using the target AR data and the depth data, and fuse the three-dimensional materials with the image data to obtain an augmented reality video.

[0180] Optionally, the target AR data includes plane data and anchor point data, and the video processing unit 703 uses the target AR data and the depth data to add three-dimensional material to each frame image, including: determining the three-dimensional space coordinates of each frame image according to the depth data, determining the plane of the addable material for each frame image according to the plane data, determining the three-dimensional anchor point coordinates and material orientation of the addable material in each frame image according to the anchor point data and the three-dimensional space coordinates; determining the addable size according to the size of the plane of the addable material and the three-dimensional anchor point coordinates, and selecting the three-dimensional material according to the size of the addable material; and adding the three-dimensional material to the three-dimensional anchor point coordinates according to the material orientation.

[0181] Optionally, the target AR data further includes grid data, and the video processing unit 703 is further configured to determine a three-dimensional grid of each frame of the image according to the grid data and the three-dimensional space coordinates before fusing the three-dimensional material with the image data, and perform collision detection between the three-dimensional material and the three-dimensional grid to obtain a collision detection result;

[0182] The video processing unit 703 performs fusion processing on the 3D material and the image data, including:

[0183] The video processing unit 703 fuses the three-dimensional material with the image data according to the collision detection result.

[0184] Optionally, the video creation device 700 may also include and selection unit 704.

[0185] The acquisition unit 701 is further configured to acquire video classification data after the decoding unit 702 decodes the video file or video stream data to obtain image data, target AR data and depth data of each frame of image;

[0186] The selection unit 704 is configured to select the three-dimensional material corresponding to the video classification data from a material library.

[0187] Optionally, the video creation device 700 may further include a second determination unit 705 .

[0188] The second determination unit 705 is used to add three-dimensional materials to each frame image by using the target AR data and the depth data in the video processing unit 703, fuse the three-dimensional materials with the image data, and obtain an augmented reality video. Then, the virtual scene rendering effect of the augmented reality video is determined according to the user portrait of the player; and the augmented reality video is rendered using the virtual scene rendering effect.

[0189] Optionally, the 3D material includes a 3D video or a 3D picture.

[0190] Among them, the decoding unit 702, the video processing unit 703, the selection unit 704, and the second determination unit 705 in the embodiment of the present application can be a processor of an electronic device, and the acquisition unit 701 can also be a communication module in the electronic device.

[0191] When creating a video, the embodiment of the present application can determine the plane of the addable material for each frame image based on the plane data, determine the three-dimensional anchor point coordinates and material orientation of the addable material in each frame image based on the anchor point data, and determine the three-dimensional grid of each frame image based on the grid data, so that the combination of the added material and each frame image in the video is more realistic, thereby improving the processing effect of video augmented reality.

[0192] See also Figure 8 , Figure 8 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 8 As shown, the electronic device 800 includes a processor 801 and a memory 802, and the processor 801 and the memory 802 can be connected to each other via a communication bus 803. The communication bus 803 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus 803 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The memory 802 is used to store computer programs. The computer programs include program instructions. The processor 801 is configured to call the program instructions. The program includes instructions for executing Figure 1 , Figure 4 or Figure 5 The method shown.

[0193] The processor 801 may be a general purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the above program.

[0194] The memory 802 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processor via a bus. The memory may also be integrated with the processor.

[0195] The electronic device 800 may further include a camera module 804, which may include at least one camera, a camera sensor, an image processing module, etc. The camera sensor may include an IMU. The electronic device 800 may further include a communication module.

[0196] The electronic device 800 may also include devices such as a display screen and a speaker, and may also include devices such as a radio frequency circuit and an antenna.

[0197] In an embodiment of the present application, when video data is collected, target AR data for video augmented reality creation is calculated based on image data and posture data of at least one frame of the collected original video; when creating the video, since the target AR data includes plane data, anchor point data, and grid data, the plane of addable materials for each frame of the image can be determined based on the plane data, the three-dimensional anchor point coordinates and material orientation of the addable materials in each frame of the image can be determined based on the anchor point data, and the three-dimensional grid of each frame of the image can be determined based on the grid data, so that the combination of the added material and each frame of the video is more realistic, thereby improving the processing effect of video augmented reality.

[0198] An embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute part or all of the steps of any one of the methods recorded in the above method embodiments.

[0199] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0200] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0201] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be electrical or other forms.

[0202] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0203] In addition, the functional units in the various embodiments of the application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software program modules.

[0204] If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program codes.

[0205] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, which can include: a flash drive, a read-only memory, a random access memory, a magnetic disk or an optical disk, etc.

[0206] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video data acquisition method, characterized in that: include: Collecting image data and posture data of at least one frame of an original video; Calculate target AR data according to the image data and the posture data; The target AR data includes at least one of plane data, anchor point data, and grid data; the target AR data is used for video augmented reality creation; Determining video classification data; Encoding the image data, the target AR data, and the depth data of the at least one frame of image, or encoding the image data, the target AR data, the depth data of the at least one frame of image, and the video classification data to obtain an encoded video file or video stream data; Decoding the video file or video stream data to obtain image data, target AR data, and depth data of each frame of image; Adding three-dimensional material to each frame of the image using the target AR data and the depth data, and fusing the three-dimensional material with the image data to obtain an augmented reality video; The target AR data includes plane data and anchor point data, and adding a three-dimensional material to each frame image by using the target AR data and the depth data includes: Determine the three-dimensional space coordinates of each frame of image according to the depth data, determine the plane to which the material can be added in each frame of image according to the plane data, and determine the three-dimensional anchor point coordinates and material orientation of the material to be added in each frame of image according to the anchor point data and the three-dimensional space coordinates; Determine the size of the addable material according to the size of the plane of the addable material and the three-dimensional anchor point coordinates, and select the three-dimensional material according to the size of the addable material; The three-dimensional material is added to the three-dimensional anchor point coordinates according to the material orientation.

2. The method according to claim 1, characterized in that The calculating target AR data according to the image data and the posture data comprises: Calculate initial AR data according to the image data and the posture data; The initial AR data are filtered according to the quality scores of the initial AR data, and the target AR data having a quality score greater than a set threshold is retained.

3. The method according to claim 1, characterized in that After calculating the target AR data according to the image data and the posture data, the method further includes: The target AR data is used to add three-dimensional material to the original video to obtain an augmented reality video.

4. The method according to claim 1, characterized in that The video classification data includes any one of video voice keywords, video text data and video scene data; The method further includes: uploading the encoded video file or video stream data to a server.

5. The method according to claim 1, characterized in that: The depth data is calculated based on the image data and the posture data; or, the depth data is calculated using initial depth information collected by a depth camera and the image data, or the depth data is calculated using a binocular image collected by a binocular camera.

6. A video creation method, characterized in that: include: Get video files or video stream data; Decoding the video file or video stream data to obtain image data, target AR data, and depth data of each frame of image; Adding three-dimensional material to each frame of the image using the target AR data and the depth data, and fusing the three-dimensional material with the image data to obtain an augmented reality video; The target AR data includes plane data and anchor point data, and adding a three-dimensional material to each frame image by using the target AR data and the depth data includes: Determine the three-dimensional space coordinates of each frame of image according to the depth data, determine the plane to which the material can be added in each frame of image according to the plane data, and determine the three-dimensional anchor point coordinates and material orientation of the material to be added in each frame of image according to the anchor point data and the three-dimensional space coordinates; Determine the size of the addable material according to the size of the plane of the addable material and the three-dimensional anchor point coordinates, and select the three-dimensional material according to the size of the addable material; The three-dimensional material is added to the three-dimensional anchor point coordinates according to the material orientation.

7. The method according to claim 6, characterized in that The target AR data further includes grid data. Before fusing the three-dimensional material with the image data, the method further includes: Determine a three-dimensional grid of each frame of image according to the grid data and the three-dimensional space coordinates, perform collision detection on the three-dimensional material and the three-dimensional grid, and obtain a collision detection result; The fusing process of the three-dimensional material with the image data includes: The three-dimensional material and the image data are fused and processed according to the collision detection result.

8. The method according to claim 6 or 7, characterized in that: After decoding the video file or video stream data to obtain image data, target AR data and depth data of each frame image, adding three-dimensional material to each frame image by using the target AR data and the depth data, fusing the three-dimensional material with the image data, and obtaining an augmented reality video, the method further includes: Get video classification data; The three-dimensional material corresponding to the video classification data is selected from a material library.

9. The method according to claim 6 or 7, characterized in that: After adding a three-dimensional material to each frame of the image by using the target AR data and the depth data, and fusing the three-dimensional material with the image data to obtain an augmented reality video, the method further includes: Determining a virtual scene rendering effect of the augmented reality video according to a user portrait of the player; The augmented reality video is rendered using the virtual scene rendering effect.

10. The method according to claim 6 or 7, characterized in that: The 3D material includes a 3D video or a 3D picture.

11. A video data acquisition device, characterized in that: include: A collection unit, used to collect image data and posture data of at least one frame of an original video; A calculation unit, configured to calculate target AR data according to the image data and the posture data; the target AR data includes at least one of plane data, anchor point data, and grid data; the target AR data is used for video augmented reality creation; A first determining unit, configured to determine video classification data; an encoding unit, configured to encode the image data, the target AR data, and the depth data of the at least one frame of image, or to encode the image data, the target AR data, the depth data of the at least one frame of image, and the video classification data, to obtain an encoded video file or video stream data; A decoding unit, used to decode the video file or video stream data to obtain image data, target AR data and depth data of each frame of image; A video processing unit, configured to add a three-dimensional material to each frame of the image using the target AR data and the depth data, and fuse the three-dimensional material with the image data to obtain an augmented reality video; The target AR data includes plane data and anchor point data, and the video processing unit uses the target AR data and the depth data to add three-dimensional materials to each frame image, including: determining the three-dimensional space coordinates of each frame image according to the depth data, determining the plane of the addable material for each frame image according to the plane data, determining the three-dimensional anchor point coordinates and material orientation of the addable material in each frame image according to the anchor point data and the three-dimensional space coordinates; determining the size of the addable material according to the size of the plane of the addable material and the three-dimensional anchor point coordinates, and selecting the three-dimensional material according to the size of the addable material; and adding the three-dimensional material to the three-dimensional anchor point coordinates according to the material orientation.

12. A video creation device, characterized in that: include: An acquisition unit, used for acquiring video files or video stream data; A decoding unit, used to decode the video file or video stream data to obtain image data, target AR data and depth data of each frame of image; A video processing unit, configured to add a three-dimensional material to each frame of the image using the target AR data and the depth data, and fuse the three-dimensional material with the image data to obtain an augmented reality video; The target AR data includes plane data and anchor point data, and the video processing unit uses the target AR data and the depth data to add three-dimensional materials in each frame image, including: determining the three-dimensional space coordinates of each frame image according to the depth data, determining the plane of the addable material for each frame image according to the plane data, determining the three-dimensional anchor point coordinates and material orientation of the addable material in each frame image according to the anchor point data and the three-dimensional space coordinates; determining the addable size according to the size of the plane of the addable material and the three-dimensional anchor point coordinates, and selecting three-dimensional materials according to the size of the addable material; and adding the three-dimensional material to the three-dimensional anchor point coordinates according to the material orientation.

13. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method according to any one of claims 1 to 10.

Images