Video restoration method, device, equipment, storage medium and program product

Abstract

This application provides a video restoration method, apparatus, device, storage medium, and program product, applied in the field of audio and video technology. The method includes acquiring multiple original images captured from multiple preset locations at a target position; analyzing and identifying the main object in the multiple original images; constructing a target model based on the image information of the multiple original images, the main object, and multiple original models in a historical database; and using a virtual camera to supplement the target model from the multiple preset locations to restore a first surround video of the target position at those locations. This method reduces the deployment of cameras and data transmission in the physical environment, using only a few original images from the physical environment as a basis to restore the video of the target position using a virtual camera in digital space, thus reducing the difficulty of the video restoration process.

Description

Technical Field

[0001] This application relates to the field of audio and video technology, and in particular to a video restoration method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the development of photography technology, volumetric photography is increasingly being used in fields such as music video shooting, live sports broadcasts, and 3D medical simulations. Unlike panoramic photography, this technology utilizes a large number of cameras to capture target tasks and scenes in a studio from multiple angles, generating and outputting dynamic 3D images and models. However, most existing volumetric video reconstruction methods use tolerance stitching. The synthesized video data is then processed by a streaming server and video codec before being transmitted to the broadcast end for full decoding and presentation to the user. This method involves complex front-end physical deployment, large data transmission volumes, and high costs for physical acquisition equipment, making video reconstruction a challenging process. Summary of the Invention

[0003] This application provides a video restoration method, apparatus, device, storage medium, and program product to solve the problem of high difficulty in the video restoration process in existing video restoration methods.

[0004] To solve the above-mentioned technical problems, this application is implemented as follows:

[0005] In a first aspect, embodiments of this application provide a video restoration method, which includes:

[0006] Acquire multiple raw images of the target location taken from multiple preset orientations, wherein at least one raw image is taken from each preset orientation;

[0007] The multiple original images are analyzed to identify the main object in the multiple original images;

[0008] Based on the image information of the multiple original images, the main object, and multiple original models in the historical database, a target model is constructed;

[0009] A virtual camera is used to supplement the image of the target model from the multiple preset positions to restore the first surround video of the target position in the multiple preset positions.

[0010] Optionally, the step of identifying and analyzing the multiple original images to determine the main object in the multiple original images includes:

[0011] Extract edge information, color histogram, and gradient histogram from each of the multiple original images;

[0012] Based on the edge information, color histogram, and gradient histogram of each original image, the feature vector of each original image is obtained;

[0013] Based on the feature vector of each original image, the edge contour information of each original image is obtained;

[0014] Based on the edge contour information of each original image, calculate the maximum closed contour area of ​​each original image;

[0015] The objects corresponding to the maximum closed contour area of ​​each original image are matched with each other. If the objects corresponding to the maximum closed contour area of ​​N original images are the same, the objects corresponding to the maximum closed contour area of ​​N original images are determined as the main object, where N is an integer greater than 1.

[0016] Optionally, the plurality of original models include a scene model and a plurality of subject models. The step of constructing a target model based on the image information of the plurality of original images, the subject objects, and the plurality of original models in the historical database includes:

[0017] From the plurality of subject models, determine the target subject model that matches the subject object;

[0018] Based on the image information of the multiple original images, the target subject model and the scene model are vector-aligned, and then the target subject model is superimposed on the scene model to obtain the target model.

[0019] Optionally, determining the target subject model matching the subject object from the plurality of subject models includes:

[0020] Calculate multiple first Euclidean distances between the feature vector of the main object and the feature vectors of the multiple main models, wherein each first Euclidean distance is the Euclidean distance between the feature vector of the main object and the feature vector of each main model;

[0021] Based on the plurality of first Euclidean distances, a target subject model matching the subject object is determined from the plurality of subject models, wherein the first Euclidean distance between the feature vector of the target subject model and the feature vector of the subject object is the minimum Euclidean distance among the plurality of first Euclidean distances.

[0022] Optionally, when the main object is an object, the step of calculating multiple first Euclidean distances between the feature vector of the main object and the feature vectors of the multiple main models includes:

[0023] Multiple second Euclidean distances are calculated based on multiple sets of first parameter pairs. Each first parameter pair includes a first parameter and a second parameter corresponding to the first parameter. The first parameter is the feature vector of the object in a preset position, and the second parameter is the feature vector of the first main body model in the corresponding preset position.

[0024] Calculate the sum of the multiple second Euclidean distances to obtain the first Euclidean distance between the feature vector of the subject object and the feature vector of the first subject model, where the first subject model is any one of the multiple subject models.

[0025] Optionally, when the main object is a person, the step of calculating multiple first Euclidean distances between the feature vector of the main object and the feature vectors of the multiple main object models includes:

[0026] Multiple third Euclidean distances are calculated based on multiple sets of second parameter pairs, wherein each set of second parameter pairs includes a third parameter and a fourth parameter corresponding to the third parameter. The third parameter is used to characterize the skeletal joints of the character, and the fourth parameter is used to characterize the corresponding skeletal joints of the first main body model.

[0027] Calculate the sum of the multiple third Euclidean distances to obtain the first Euclidean distance between the feature vector of the subject object and the feature vector of the first subject model, where the first subject model is any one of the multiple subject models.

[0028] Optionally, the step of using a virtual camera to supplement the target model from the plurality of preset locations to restore the first surround video of the target location at the plurality of preset locations includes:

[0029] Obtain the actual shooting angle of the target location in the multiple preset directions;

[0030] Based on the actual shooting angles of the target location in multiple preset directions, the virtual camera is used to supplement the image of the target model from the multiple preset directions to obtain the initial surround video of the target location in the multiple preset directions.

[0031] Extract key information from the multiple original images;

[0032] Using the key information, the initial surround video of the target location in multiple preset directions is rendered to obtain the first surround video of the target location in multiple preset directions.

[0033] Optionally, the method of using a virtual camera to supplement the target model from the plurality of preset positions and restore the target position after the first surround video from the plurality of preset positions further includes:

[0034] After aligning the player's playback angle with the actual shooting angle, the first surround video of the target location at the multiple preset positions is sent to the player.

[0035] Optionally, the method of using a virtual camera to supplement the target model from the plurality of preset positions and restore the target position after the first surround video from the plurality of preset positions further includes:

[0036] Receive correction information sent by the player;

[0037] Based on the correction information and the first surround video of the target location in the multiple preset directions, a second surround video is generated;

[0038] After aligning the playback angle of the player with the actual shooting angle, a second surround video of the target location at the multiple preset positions is sent to the player.

[0039] Secondly, embodiments of this application also provide a video restoration apparatus, which includes:

[0040] The first acquisition module is used to acquire multiple original images of the target location taken from multiple preset directions, wherein at least one original image is taken from each preset direction;

[0041] The first determining module is used to identify and analyze the multiple original images to determine the main object in the multiple original images;

[0042] The first construction module is used to construct a target model based on the image information of the multiple original images, the main object, and multiple original models in the historical database;

[0043] The first restoration module is used to supplement the target model with images from the multiple preset positions using a virtual camera, so as to restore the first surround video of the target position in the multiple preset positions.

[0044] Optionally, the first determining module includes:

[0045] The first extraction unit is used to extract edge information, color histogram, and gradient histogram of each of the multiple original images;

[0046] The first processing unit is used to obtain the feature vector of each original image based on the edge information, color histogram and gradient histogram of each original image;

[0047] The second processing unit is used to obtain the edge contour information of each original image based on the feature vector of each original image;

[0048] The first calculation unit is used to calculate the maximum closed contour area of ​​each original image based on the edge contour information of each original image.

[0049] The first determining unit is used to perform feature matching on the objects corresponding to the maximum closed contour area of ​​each original image, and when the objects corresponding to the maximum closed contour area of ​​N original images are the same, determine the objects corresponding to the maximum closed contour area of ​​N original images as the main object, where N is an integer greater than 1.

[0050] Optionally, the plurality of original models include a scene model and a plurality of subject models, and the first construction module includes:

[0051] The second determining unit is used to determine, from the plurality of subject models, a target subject model that matches the subject object;

[0052] The third processing unit is used to perform vector alignment between the target subject model and the scene model based on the image information of the multiple original images, and then superimpose the target subject model onto the scene model to obtain the target model.

[0053] Optionally, the second determining unit includes:

[0054] The first calculation subunit is used to calculate multiple first Euclidean distances between the feature vector of the subject object and the feature vectors of the multiple subject models, wherein each first Euclidean distance is the Euclidean distance between the feature vector of the subject object and the feature vector of each subject model.

[0055] The first determining subunit is used to determine a target subject model that matches the subject object from the plurality of subject models based on the plurality of first Euclidean distances, wherein the first Euclidean distance between the feature vector of the target subject model and the feature vector of the subject object is the minimum Euclidean distance among the plurality of first Euclidean distances.

[0056] Optionally, when the main object is an object, the first calculation subunit is specifically used for:

[0057] Multiple second Euclidean distances are calculated based on multiple sets of first parameter pairs. Each first parameter pair includes a first parameter and a second parameter corresponding to the first parameter. The first parameter is the feature vector of the object in a preset position, and the second parameter is the feature vector of the first main body model in the corresponding preset position.

[0058] Calculate the sum of the multiple second Euclidean distances to obtain the first Euclidean distance between the feature vector of the subject object and the feature vector of the first subject model, where the first subject model is any one of the multiple subject models.

[0059] Optionally, when the main object is a person, the first calculation subunit is specifically used for:

[0060] Multiple third Euclidean distances are calculated based on multiple sets of second parameter pairs, wherein each set of second parameter pairs includes a third parameter and a fourth parameter corresponding to the third parameter. The third parameter is used to characterize the skeletal joints of the character, and the fourth parameter is used to characterize the corresponding skeletal joints of the first main body model.

[0061] Calculate the sum of the multiple third Euclidean distances to obtain the first Euclidean distance between the feature vector of the subject object and the feature vector of the first subject model, where the first subject model is any one of the multiple subject models.

[0062] Optionally, the first restoration module includes:

[0063] The first acquisition unit is used to acquire the actual shooting angle of the target position in the multiple preset directions;

[0064] The fourth processing unit is used to supplement the target model with images from the multiple preset directions based on the actual shooting angle of the target position in the multiple preset directions using the virtual camera, so as to obtain the initial surround video of the target position in the multiple preset directions.

[0065] The first extraction unit is used to extract key information from the multiple original images;

[0066] The fifth processing unit is used to render the initial surround video of the target location in the multiple preset directions using the key information, so as to obtain the first surround video of the target location in the multiple preset directions.

[0067] Optionally, the device further includes:

[0068] The first sending module is used to align the playback view of the player with the actual shooting view, and then send the first surround video of the target position at the multiple preset positions to the player.

[0069] Optionally, the device further includes:

[0070] The first receiving module is used to receive correction information sent by the player;

[0071] The first generation module is used to generate a second surround video based on the correction information and the first surround video of the target position in the multiple preset directions;

[0072] The second sending module is used to align the playback angle of the player with the actual shooting angle, and then send the second surround video of the target position at the multiple preset positions to the player.

[0073] Thirdly, embodiments of this application provide an electronic device, which includes a transceiver and a processor, wherein the transceiver is used for:

[0074] Acquire multiple raw images of the target location taken from multiple preset orientations, wherein at least one raw image is taken from each preset orientation;

[0075] The processor is used for:

[0076] The multiple original images are analyzed to identify the main object in the multiple original images;

[0077] Based on the image information of the multiple original images, the main object, and multiple original models in the historical database, a target model is constructed;

[0078] A virtual camera is used to supplement the image of the target model from the multiple preset positions to restore the first surround video of the target position in the multiple preset positions.

[0079] Optionally, the processor is specifically used for:

[0080] Extract edge information, color histogram, and gradient histogram from each of the multiple original images;

[0081] Based on the edge information, color histogram, and gradient histogram of each original image, the feature vector of each original image is obtained;

[0082] Based on the feature vector of each original image, the edge contour information of each original image is obtained;

[0083] Based on the edge contour information of each original image, calculate the maximum closed contour area of ​​each original image;

[0084] The objects corresponding to the maximum closed contour area of ​​each original image are matched with each other. If the objects corresponding to the maximum closed contour area of ​​N original images are the same, the objects corresponding to the maximum closed contour area of ​​N original images are determined as the main object, where N is an integer greater than 1.

[0085] Optionally, the processor is specifically used for:

[0086] From the plurality of subject models, determine the target subject model that matches the subject object;

[0087] Based on the image information of the multiple original images, the target subject model and the scene model are vector-aligned, and then the target subject model is superimposed on the scene model to obtain the target model.

[0088] Optionally, the processor is specifically used for:

[0089] Calculate multiple first Euclidean distances between the feature vector of the main object and the feature vectors of the multiple main models, wherein each first Euclidean distance is the Euclidean distance between the feature vector of the main object and the feature vector of each main model;

[0090] Based on the plurality of first Euclidean distances, a target subject model matching the subject object is determined from the plurality of subject models, wherein the first Euclidean distance between the feature vector of the target subject model and the feature vector of the subject object is the minimum Euclidean distance among the plurality of first Euclidean distances.

[0091] Optionally, when the main object is an object, the processor is specifically used to:

[0092] Multiple second Euclidean distances are calculated based on multiple sets of first parameter pairs. Each first parameter pair includes a first parameter and a second parameter corresponding to the first parameter. The first parameter is the feature vector of the object in a preset position, and the second parameter is the feature vector of the first main body model in the corresponding preset position.

[0093] Calculate the sum of the multiple second Euclidean distances to obtain the first Euclidean distance between the feature vector of the subject object and the feature vector of the first subject model, where the first subject model is any one of the multiple subject models.

[0094] Optionally, when the main object is a person, the processor is specifically used for:

[0095] Multiple third Euclidean distances are calculated based on multiple sets of second parameter pairs, wherein each set of second parameter pairs includes a third parameter and a fourth parameter corresponding to the third parameter. The third parameter is used to characterize the skeletal joints of the character, and the fourth parameter is used to characterize the corresponding skeletal joints of the first main body model.

[0096] Calculate the sum of the multiple third Euclidean distances to obtain the first Euclidean distance between the feature vector of the subject object and the feature vector of the first subject model, where the first subject model is any one of the multiple subject models.

[0097] Optionally, the processor is specifically used for:

[0098] Obtain the actual shooting angle of the target location in the multiple preset directions;

[0099] Based on the actual shooting angles of the target location in multiple preset directions, the virtual camera is used to supplement the image of the target model from the multiple preset directions to obtain the initial surround video of the target location in the multiple preset directions.

[0100] Extract key information from the multiple original images;

[0101] Using the key information, the initial surround video of the target location in multiple preset directions is rendered to obtain the first surround video of the target location in multiple preset directions.

[0102] Optionally, the transceiver is further used for:

[0103] After aligning the player's playback angle with the actual shooting angle, the first surround video of the target location at the multiple preset positions is sent to the player.

[0104] Optionally, the transceiver is further used for:

[0105] Receive correction information sent by the player;

[0106] Based on the correction information and the first surround video of the target location in the multiple preset directions, a second surround video is generated;

[0107] After aligning the playback angle of the player with the actual shooting angle, a second surround video of the target location at the multiple preset positions is sent to the player.

[0108] Fourthly, embodiments of this application also provide an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the video restoration method described above.

[0109] Fifthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the video restoration method described above.

[0110] In a sixth aspect, embodiments of this application provide a computer program product including computer instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0111] The video restoration method of this application includes acquiring multiple original images of a target location captured from multiple preset locations, wherein at least one original image is captured from each preset location; performing identification and analysis on the multiple original images to determine the main object in the multiple original images; constructing a target model based on the image information of the multiple original images, the main object, and multiple original models in a historical database; and using a virtual camera to supplement the target model from the multiple preset locations to restore a first surround video of the target location at the multiple preset locations. This method reduces the deployment of cameras and the amount of data transmission in the front-end physical environment, using only a few original images from the physical environment as a basis to restore the video of the target location using a virtual camera in digital space, thus reducing the difficulty of the video restoration process. Attached Figure Description

[0112] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0113] Figure 1 This is a flowchart of the video restoration method provided in the embodiments of this application;

[0114] Figure 2 This is a schematic diagram of the video restoration method provided in an embodiment of this application;

[0115] Figure 3 This is a schematic diagram of the extraction of the edge contour information of a person provided in an embodiment of this application;

[0116] Figure 4 This is a schematic diagram illustrating the process of adding images to an initial surround video, as provided in an embodiment of this application.

[0117] Figure 5 This is a structural diagram of a video restoration device provided in an embodiment of this application;

[0118] Figure 6 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0119] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0120] This application provides a video restoration method. See also... Figure 1 , Figure 1 This is a flowchart of the video restoration method provided in the embodiments of this application, such as... Figure 1 As shown, the steps include the following. It should be noted that the following steps can be performed by the server:

[0121] Step 101: Acquire multiple original images of the target location taken from multiple preset directions, wherein at least one original image is taken from each preset direction;

[0122] In this step, for example, three cameras can be set up in the actual physical environment to capture original images of the target location from several preset positions, namely the top (Z-axis), front (X-axis), and side (Y-axis).

[0123] Step 102: Perform identification and analysis on the multiple original images to determine the main object in the multiple original images;

[0124] In this step, the server does not stitch together the multiple original images obtained from multiple preset locations, but performs image recognition analysis. Based on the proportion of each object in the frame in each original image, the main object of each original image is determined. If the main object is the same in multiple original images, the subsequent analysis is based on that main object.

[0125] Step 103: Based on the image information of the multiple original images, the main object, and multiple original models in the historical database, construct the target model;

[0126] In practical applications, such as large-scale performances, the venue layout and performers are information already known before the live broadcast. This information is stored in a historical database, forming initial models. For example, different subject models can be formed based on the characteristic information of different performers, and scene models can be formed based on the venue layout information.

[0127] Based on the main object identified from the aforementioned multiple original images, a target main object model matching the main object is determined from multiple main object models stored in the historical database. Then, the target main object model obtained from the image information of the multiple original images is bound to the scene model to obtain the target model.

[0128] Step 104: Use a virtual camera to supplement the image of the target model from the multiple preset positions to restore the first surround video of the target position in the multiple preset positions.

[0129] In this step, the aforementioned target model is constructed in digital space, and its actual image information is limited. A virtual camera can be obtained using 3D rendering, and then the virtual camera can be used to supplement the target model with images from multiple preset directions to reconstruct the first surround video of the target position in multiple preset directions. For example, multiple images can be captured in 360 degrees in the X direction, and then a set of first surround videos based on the X direction can be generated using a stitching and synthesis algorithm. The first surround videos in other directions can be generated in the same way.

[0130] Existing volumetric photography technology primarily involves physically deploying multiple cameras to capture raw images (e.g., 24-120 cameras are needed for a free-viewing perspective). All raw images are then stitched together using algorithms, processed by a server, and finally displayed to the user by selecting their preferred viewing angle. In one implementation, a smaller number of cameras are deployed in the physical space to capture multiple raw images of the target location from multiple preset locations. These raw images are not stitched together initially but are analyzed to identify the main subject. Then, a target model related to the main subject is obtained using a historical database. Based on this target model, the video footage of the target location is reconstructed in digital space. Figure 2 For example, a camera captures three original images of a cup from three preset positions: the front, side, and top. These images are then analyzed to identify the cup as the main subject. Using a cup model matched to the cup in a historical database and clues extracted during the image analysis (such as fluorescent lights, wood grain tables, etc.), the cup image is reconstructed in digital space based on this model. The reconstructed images are then stitched together using an algorithm. This implementation reduces the deployment of cameras and the amount of data transmitted in the physical environment. It uses only a few original images from the physical environment as a basis to reconstruct the video of the target location using a virtual camera in digital space, thus reducing the difficulty of the video reconstruction process.

[0131] Optionally, the step of identifying and analyzing the multiple original images to determine the main object in the multiple original images includes:

[0132] Extract edge information, color histogram, and gradient histogram from each of the multiple original images;

[0133] Based on the edge information, color histogram, and gradient histogram of each original image, the feature vector of each original image is obtained;

[0134] Based on the feature vector of each original image, the edge contour information of each original image is obtained;

[0135] Based on the edge contour information of each original image, calculate the maximum closed contour area of ​​each original image;

[0136] The objects corresponding to the maximum closed contour area of ​​each original image are matched with each other. If the objects corresponding to the maximum closed contour area of ​​N original images are the same, the objects corresponding to the maximum closed contour area of ​​N original images are determined as the main object, where N is an integer greater than 1.

[0137] In one implementation, edge information, color histogram, and gradient histogram are extracted from each original image, and these three are combined to form a complete feature vector. The specific process for extracting the feature vector can be found in the following formula:

[0138] F i =σ j (I i )

[0139] Where F represents the feature vector, σ represents the feature extraction method, I represents the input original image, j represents the j-th feature extraction method, and i represents the i-th original image.

[0140] Then, the edge contour information of each original image is obtained based on its feature vector, and the maximum closed contour area of ​​each original image is calculated using Green's formula. In the specific implementation, to reduce the edge contour information extraction time, see [link to relevant documentation]. Figure 3 When the object is identified as a person, the OpenPose algorithm can be used to quickly extract the skeletal pose of the person in the image. When the object is identified as an object, the ContourDetection algorithm can be used to quickly extract the object's contour lines.

[0141] Then, a feature matching method is used to determine whether the object corresponding to the largest closed contour area is the same in each original image. If the object corresponding to the largest closed contour area is the same in multiple original images, then the object is determined to be the main object. In this implementation, by identifying and analyzing multiple original images, the main object in multiple original images can be obtained, which is beneficial for subsequent reconstruction of the video at the target location based on the main object.

[0142] Optionally, the plurality of original models include a scene model and a plurality of subject models. The step of constructing a target model based on the image information of the plurality of original images, the subject objects, and the plurality of original models in the historical database includes:

[0143] From the plurality of subject models, determine the target subject model that matches the subject object;

[0144] Based on the image information of the multiple original images, the target subject model and the scene model are vector-aligned, and then the target subject model is superimposed on the scene model to obtain the target model.

[0145] In practical applications, such as large-scale performances, the venue layout and performers are information already known before the live broadcast. This information is stored in a historical database, forming initial models. For example, different subject models can be formed based on the characteristic information of different performers, and scene models can be formed based on the venue layout information.

[0146] In one implementation, the identified subject object is matched with multiple subject models in the database. Specifically, the Euclidean distance between the feature vector of the subject object and the feature vector of each subject model can be calculated. The subject model corresponding to the smallest calculated Euclidean distance is determined as the subject model that best matches the subject object, i.e., the target subject model. The determined target subject model is then bound to a scene model. This binding process relies on image information from multiple original images. These images come from different preset orientations, such as the X-axis, Y-axis, and Z-axis. Therefore, the target subject model and scene model need to be vector-aligned in each orientation by referring to the real image information from each orientation. This ensures that the target model obtained by superimposing the target subject model and scene model can more realistically reproduce the actual situation of the target location.

[0147] Optionally, determining the target subject model matching the subject object from the plurality of subject models includes:

[0148] Calculate multiple first Euclidean distances between the feature vector of the main object and the feature vectors of the multiple main models, wherein each first Euclidean distance is the Euclidean distance between the feature vector of the main object and the feature vector of each main model;

[0149] Based on the plurality of first Euclidean distances, a target subject model matching the subject object is determined from the plurality of subject models, wherein the first Euclidean distance between the feature vector of the target subject model and the feature vector of the subject object is the minimum Euclidean distance among the plurality of first Euclidean distances.

[0150] In one implementation, a target subject model can be determined from multiple subject models by calculating the first Euclidean distance between the feature vector of the subject object and the feature vector of each subject model. The calculation of the first Euclidean distance can be found in the following formula:

[0151]

[0152] Where d represents the first Euclidean distance, a1 represents the feature vector of the main object, and a2 represents the feature vector of the main model.

[0153] The first Euclidean distance between the feature vector of the target subject model and the subject object is the smallest among the calculated first Euclidean distances, indicating that the target subject model has the greatest correlation with the subject object and is the subject model that best matches the subject object among multiple subject models.

[0154] In this implementation, by calculating the first Euclidean distance between the feature vector of the subject object and the feature vector of each of the multiple subject models, it is beneficial to quickly determine the target subject model that matches the subject object from the multiple subject models.

[0155] Optionally, when the main object is an object, the step of calculating multiple first Euclidean distances between the feature vector of the main object and the feature vectors of the multiple main models includes:

[0156] Multiple second Euclidean distances are calculated based on multiple sets of first parameter pairs. Each first parameter pair includes a first parameter and a second parameter corresponding to the first parameter. The first parameter is the feature vector of the object in a preset position, and the second parameter is the feature vector of the first main body model in the corresponding preset position.

[0157] Calculate the sum of the multiple second Euclidean distances to obtain the first Euclidean distance between the feature vector of the subject object and the feature vector of the first subject model, where the first subject model is any one of the multiple subject models.

[0158] In one embodiment, when the aforementioned main object is an object, the second Euclidean distance between the feature vector of the object at each preset orientation and the feature vector of each main model at the corresponding preset orientation can be calculated first. Then, the second Euclidean distances calculated for each preset orientation are added together to obtain the first Euclidean distance. For example, when the multiple preset orientations are X-axis orientation, Y-axis orientation, and Z-axis orientation, the calculation of the first Euclidean distance in this embodiment can refer to the following formula:

[0159] d=sqrt(x1-x2)^2+(y1-y2)^2+(z1-z2)^2);

[0160] Where d represents the first Euclidean distance, x1 represents the feature vector of the object in the x-axis direction, x2 represents the feature vector of the main model in the x-axis direction, y1 represents the feature vector of the object in the y-axis direction, y2 represents the feature vector of the main model in the y-axis direction, z1 represents the feature vector of the object in the z-axis direction, and z2 represents the feature vector of the main model in the z-axis direction.

[0161] In this implementation, when the main object is an object, by first calculating the second Euclidean distance between the feature vectors of the object at each preset position and the feature vectors of the main model at each preset position, and then obtaining the first Euclidean distance based on the second Euclidean distance, it is beneficial to improve the accuracy of the calculated first Euclidean distance according to the type of the main object.

[0162] Optionally, when the main object is a person, the step of calculating multiple first Euclidean distances between the feature vector of the main object and the feature vectors of the multiple main object models includes:

[0163] Multiple third Euclidean distances are calculated based on multiple sets of second parameter pairs, wherein each set of second parameter pairs includes a third parameter and a fourth parameter corresponding to the third parameter. The third parameter is used to characterize the skeletal joints of the character, and the fourth parameter is used to characterize the corresponding skeletal joints of the first main body model.

[0164] Calculate the sum of the multiple third Euclidean distances to obtain the first Euclidean distance between the feature vector of the subject object and the feature vector of the first subject model, where the first subject model is any one of the multiple subject models.

[0165] In one implementation, when the subject is a human figure, the first Euclidean distance can be calculated based on multiple skeletal joints along the figure's skeleton. Specifically, the third Euclidean distance between each skeletal joint of the human figure and the corresponding skeletal joint of the main model can be calculated first. Multiple third Euclidean distances can be calculated for multiple pairs of skeletal joints, and then these multiple third Euclidean distances are added together to obtain the first Euclidean distance. The calculation of the first Euclidean distance in this implementation can be found in the following formula:

[0166] d=sqrt[∑((a i -b i )^2)](i=12…n);

[0167] Where d represents the first Euclidean distance, a represents the skeletal joint of the character, b represents the skeletal joint of the main model corresponding to the skeletal joint of the character, and i represents the i-th skeletal joint.

[0168] In this implementation, when the main object is a human figure, the third Euclidean distance between each bone joint on the human figure's skeleton line and the corresponding bone joint on the main model is calculated first, and then the first Euclidean distance is obtained based on the third Euclidean distance. This helps to improve the accuracy of the calculated first Euclidean distance according to the type of the main object.

[0169] Optionally, the step of using a virtual camera to supplement the target model from the plurality of preset locations to restore the first surround video of the target location at the plurality of preset locations includes:

[0170] Obtain the actual shooting angle of the target location in the multiple preset directions;

[0171] Based on the actual shooting angles of the target location in multiple preset directions, the virtual camera is used to supplement the image of the target model from the multiple preset directions to obtain the initial surround video of the target location in the multiple preset directions.

[0172] Extract key information from the multiple original images;

[0173] Using the key information, the initial surround video of the target location in multiple preset directions is rendered to obtain the first surround video of the target location in multiple preset directions.

[0174] In one embodiment, for example, the preset orientation includes X-axis orientation, Y-axis orientation, and Z-axis orientation. Let θ be the actual shooting angle of the target position in the X-axis orientation. In digital space, a virtual camera is constructed around θ using 3D rendering, and a set of images with F(x) = θ (0°-360°) are captured using the virtual camera. Then, the multiple sets of images captured around θ are stitched together using a stitching and compositing algorithm to generate a set of initial 360° surround videos based on the X-axis orientation. Similarly, initial 360° surround videos in the Y-axis orientation and initial 360° surround videos in the Z-axis orientation are obtained.

[0175] See Figure 4 Since the initial surround video contains limited substantive image information, key information can be extracted from the original image to form a "key information controller." This controller can then be used to render the initial surround video using a Control Net (C-Net) algorithm. For example, the key information may include the color depth, color accuracy, color gamut, lighting, hue, and image style of the original image, thus obtaining the first surround video after rendering the initial video.

[0176] In this implementation, based on the actual shooting perspective of the target location, a virtual camera is used to supplement the image of the target model, and then the original image is used to perform secondary rendering of the obtained image, which helps to improve the fidelity of the video of the target location.

[0177] Optionally, the method of using a virtual camera to supplement the target model from the plurality of preset positions and restore the target position after the first surround video from the plurality of preset positions further includes:

[0178] After aligning the player's playback angle with the actual shooting angle, the first surround video of the target location at the multiple preset positions is sent to the player.

[0179] Referring to the example of the aforementioned implementation method, let θ be the actual shooting angle of the target position in the X-axis direction. To align the playback angle of the player with the actual shooting angle, let θ be the playback angle of the player in the X-axis direction. 0 Then, after the player receives the first surround video of the target position on the X-axis sent by the server, the user can watch the first surround video by sliding the dial on the player. The angle adjustment and video restoration of other preset positions are the same.

[0180] In this implementation, aligning the playback viewpoint of the player with the actual shooting viewpoint helps users synchronize the video playback with the actual scene at the target location, thereby improving the fidelity of the video at the target location.

[0181] Optionally, the method of using a virtual camera to supplement the target model from the plurality of preset positions and restore the target position after the first surround video from the plurality of preset positions further includes:

[0182] Receive correction information sent by the player;

[0183] Based on the correction information and the first surround video of the target location in the multiple preset directions, a second surround video is generated;

[0184] After aligning the playback angle of the player with the actual shooting angle, a second surround video of the target location at the multiple preset positions is sent to the player.

[0185] In one implementation, the user can further modify the first surround video according to their needs and preferences. For example, if the user wants the dancers in the performance to look like rabbits and the performance style to be forest-themed, the modification information can include the variables "rabbit" and "forest style." Then, the first surround video is modified based on this modification information to obtain a second surround video that meets the user's needs and preferences. This implementation method helps increase the diversity of videos that recreate the target location, enhancing the user's enjoyment of watching videos on the player.

[0186] See Figure 5 , Figure 5 This is a structural diagram of a video restoration device provided in an embodiment of this application, as shown below. Figure 6 As shown, the video restoration device 500 includes:

[0187] The first acquisition module 501 is used to acquire multiple original images of the target position taken from multiple preset positions, wherein at least one original image is taken from each preset position.

[0188] The first determining module 502 is used to identify and analyze the multiple original images to determine the main object in the multiple original images;

[0189] The first construction module 503 is used to construct a target model based on the image information of the multiple original images, the main object, and multiple original models in the historical database;

[0190] The first restoration module 504 is used to supplement the target model with images from the multiple preset positions using a virtual camera, so as to restore the first surround video of the target position in the multiple preset positions.

[0191] Optionally, the first determining module includes:

[0192] The first extraction unit is used to extract edge information, color histogram, and gradient histogram of each of the multiple original images;

[0193] The first processing unit is used to obtain the feature vector of each original image based on the edge information, color histogram and gradient histogram of each original image;

[0194] The second processing unit is used to obtain the edge contour information of each original image based on the feature vector of each original image;

[0195] The first calculation unit is used to calculate the maximum closed contour area of ​​each original image based on the edge contour information of each original image.

[0196] The first determining unit is used to perform feature matching on the objects corresponding to the maximum closed contour area of ​​each original image, and when the objects corresponding to the maximum closed contour area of ​​N original images are the same, determine the objects corresponding to the maximum closed contour area of ​​N original images as the main object, where N is an integer greater than 1.

[0197] Optionally, the plurality of original models include a scene model and a plurality of subject models, and the first construction module includes:

[0198] The second determining unit is used to determine, from the plurality of subject models, a target subject model that matches the subject object;

[0199] The third processing unit is used to perform vector alignment between the target subject model and the scene model based on the image information of the multiple original images, and then superimpose the target subject model onto the scene model to obtain the target model.

[0200] Optionally, the second determining unit includes:

[0201] The first calculation subunit is used to calculate multiple first Euclidean distances between the feature vector of the subject object and the feature vectors of the multiple subject models, wherein each first Euclidean distance is the Euclidean distance between the feature vector of the subject object and the feature vector of each subject model.

[0202] The first determining subunit is used to determine a target subject model that matches the subject object from the plurality of subject models based on the plurality of first Euclidean distances, wherein the first Euclidean distance between the feature vector of the target subject model and the feature vector of the subject object is the minimum Euclidean distance among the plurality of first Euclidean distances.

[0203] Optionally, when the main object is an object, the first calculation subunit is specifically used for:

[0204] Multiple second Euclidean distances are calculated based on multiple sets of first parameter pairs. Each first parameter pair includes a first parameter and a second parameter corresponding to the first parameter. The first parameter is the feature vector of the object in a preset position, and the second parameter is the feature vector of the first main body model in the corresponding preset position.

[0205] Calculate the sum of the multiple second Euclidean distances to obtain the first Euclidean distance between the feature vector of the subject object and the feature vector of the first subject model, where the first subject model is any one of the multiple subject models.

[0206] Optionally, when the main object is a person, the first calculation subunit is specifically used for:

[0207] Multiple third Euclidean distances are calculated based on multiple sets of second parameter pairs, wherein each set of second parameter pairs includes a third parameter and a fourth parameter corresponding to the third parameter. The third parameter is used to characterize the skeletal joints of the character, and the fourth parameter is used to characterize the corresponding skeletal joints of the first main body model.

[0208] Calculate the sum of the multiple third Euclidean distances to obtain the first Euclidean distance between the feature vector of the subject object and the feature vector of the first subject model, where the first subject model is any one of the multiple subject models.

[0209] Optionally, the first restoration module includes:

[0210] The first acquisition unit is used to acquire the actual shooting angle of the target position in the multiple preset directions;

[0211] The fourth processing unit is used to supplement the target model with images from the multiple preset directions based on the actual shooting angle of the target position in the multiple preset directions using the virtual camera, so as to obtain the initial surround video of the target position in the multiple preset directions.

[0212] The first extraction unit is used to extract key information from the multiple original images;

[0213] The fifth processing unit is used to render the initial surround video of the target location in the multiple preset directions using the key information, so as to obtain the first surround video of the target location in the multiple preset directions.

[0214] Optionally, the device further includes:

[0215] The first sending module is used to align the playback view of the player with the actual shooting view, and then send the first surround video of the target position at the multiple preset positions to the player.

[0216] Optionally, the device further includes:

[0217] The first receiving module is used to receive correction information sent by the player;

[0218] The first generation module is used to generate a second surround video based on the correction information and the first surround video of the target position in the multiple preset directions;

[0219] The second sending module is used to align the playback angle of the player with the actual shooting angle, and then send the second surround video of the target position at the multiple preset positions to the player.

[0220] The video restoration device 500 can achieve Figure 1 The various processes implemented in the illustrated method embodiments can achieve the same beneficial effects, and will not be described again here to avoid repetition.

[0221] This application also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the various processes of the above-described video restoration method embodiments applied to electronic devices and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0222] For details, see Figure 6 As shown, this embodiment of the invention also provides an electronic device, including a bus 601, a transceiver 602, an antenna 603, a bus interface 604, a processor 605, and a memory 606.

[0223] The transceiver 602 is used for:

[0224] Acquire multiple raw images of the target location taken from multiple preset orientations, wherein at least one raw image is taken from each preset orientation;

[0225] The processor 605 is used for:

[0226] The multiple original images are analyzed to identify the main object in the multiple original images;

[0227] Based on the image information of the multiple original images, the main object, and multiple original models in the historical database, a target model is constructed;

[0228] A virtual camera is used to supplement the image of the target model from the multiple preset positions to restore the first surround video of the target position in the multiple preset positions.

[0229] Optionally, the processor 605 is specifically used for:

[0230] Extract edge information, color histogram, and gradient histogram from each of the multiple original images;

[0231] Based on the edge information, color histogram, and gradient histogram of each original image, the feature vector of each original image is obtained;

[0232] Based on the feature vector of each original image, the edge contour information of each original image is obtained;

[0233] Based on the edge contour information of each original image, calculate the maximum closed contour area of ​​each original image;

[0234] The objects corresponding to the maximum closed contour area of ​​each original image are matched with each other. If the objects corresponding to the maximum closed contour area of ​​N original images are the same, the objects corresponding to the maximum closed contour area of ​​N original images are determined as the main object, where N is an integer greater than 1.

[0235] Optionally, the processor 605 is specifically used for:

[0236] From the plurality of subject models, determine the target subject model that matches the subject object;

[0237] Based on the image information of the multiple original images, the target subject model and the scene model are vector-aligned, and then the target subject model is superimposed on the scene model to obtain the target model.

[0238] Optionally, the processor 605 is specifically used for:

[0239] Calculate multiple first Euclidean distances between the feature vector of the main object and the feature vectors of the multiple main models, wherein each first Euclidean distance is the Euclidean distance between the feature vector of the main object and the feature vector of each main model;

[0240] Based on the plurality of first Euclidean distances, a target subject model matching the subject object is determined from the plurality of subject models, wherein the first Euclidean distance between the feature vector of the target subject model and the feature vector of the subject object is the minimum Euclidean distance among the plurality of first Euclidean distances.

[0241] Optionally, when the main object is an object, the processor 605 is specifically used for:

[0242] Multiple second Euclidean distances are calculated based on multiple sets of first parameter pairs. Each first parameter pair includes a first parameter and a second parameter corresponding to the first parameter. The first parameter is the feature vector of the object in a preset position, and the second parameter is the feature vector of the main model in the corresponding preset position.

[0243] Calculate the sum of the multiple second Euclidean distances to obtain the first Euclidean distance between the feature vector of the subject object and the feature vector of the first subject model, where the first subject model is any one of the multiple subject models.

[0244] Optionally, when the main object is a person, the processor 605 is specifically used for:

[0245] Multiple third Euclidean distances are calculated based on multiple sets of second parameter pairs, wherein each set of second parameter pairs includes a third parameter and a fourth parameter corresponding to the third parameter. The third parameter is used to characterize the skeletal joints of the character, and the fourth parameter is used to characterize the corresponding skeletal joints of the first main body model.

[0246] Calculate the sum of the multiple third Euclidean distances to obtain the first Euclidean distance between the feature vector of the subject object and the feature vector of the first subject model, where the first subject model is any one of the multiple subject models.

[0247] Optionally, the processor 605 is specifically used for:

[0248] Obtain the actual shooting angle of the target location in the multiple preset directions;

[0249] Based on the actual shooting angles of the target location in multiple preset directions, the virtual camera is used to supplement the image of the target model from the multiple preset directions to obtain the initial surround video of the target location in the multiple preset directions.

[0250] Extract key information from the multiple original images;

[0251] Using the key information, the initial surround video of the target location in multiple preset directions is rendered to obtain the first surround video of the target location in multiple preset directions.

[0252] Optionally, the transceiver 602 is further configured to:

[0253] After aligning the player's playback angle with the actual shooting angle, the first surround video of the target location at the multiple preset positions is sent to the player.

[0254] Optionally, the transceiver 602 is further configured to:

[0255] Receive correction information sent by the player;

[0256] Based on the correction information and the first surround video of the target location in the multiple preset directions, a second surround video is generated;

[0257] After aligning the playback angle of the player with the actual shooting angle, a second surround video of the target location at the multiple preset positions is sent to the player.

[0258] exist Figure 6 In this document, a bus architecture (represented by bus 601) is used. Bus 601 can include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 605 and memory represented by memory 606. Bus 601 can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 604 provides an interface between bus 601 and transceiver 602. Transceiver 602 can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 605 is transmitted over a wireless medium via antenna 603, which further receives data and transmits data to processor 605.

[0259] Processor 605 manages bus 601 and general processing, and also provides various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Memory 606 can be used to store data used by processor 605 during operation.

[0260] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described video restoration method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0261] This application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above-described... Figure 1 The various processes of the method embodiments shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0262] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0263] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0264] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A video restoration method, characterized in that, The method includes: Acquire multiple raw images of the target location taken from multiple preset orientations, wherein at least one raw image is taken from each preset orientation; The multiple original images are analyzed to identify the main object in the multiple original images; Based on the image information of the multiple original images, the main object, and multiple original models in the historical database, a target model is constructed; Using a virtual camera, the target model is supplemented with images from the multiple preset locations to restore the first surround video of the target location from the multiple preset locations; The multiple original models include a scene model and multiple subject models. The construction of the target model based on the image information of the multiple original images, the subject objects, and the multiple original models in the historical database includes: From the plurality of subject models, determine the target subject model that matches the subject object; Based on the image information of the multiple original images, the target subject model and the scene model are vector-aligned, and then the target subject model is superimposed on the scene model to obtain the target model.

2. The video restoration method according to claim 1, characterized in that, The step of determining the target subject model that matches the subject object from the plurality of subject models includes: Calculate multiple first Euclidean distances between the feature vector of the main object and the feature vectors of the multiple main models, wherein each first Euclidean distance is the Euclidean distance between the feature vector of the main object and the feature vector of each main model; Based on the plurality of first Euclidean distances, a target subject model matching the subject object is determined from the plurality of subject models, wherein the first Euclidean distance between the feature vector of the target subject model and the feature vector of the subject object is the minimum Euclidean distance among the plurality of first Euclidean distances.

3. The video restoration method according to claim 2, characterized in that, When the main object is an object, the step of calculating multiple first Euclidean distances between the feature vector of the main object and the feature vectors of the multiple main models includes: Multiple second Euclidean distances are calculated based on multiple sets of first parameter pairs. Each first parameter pair includes a first parameter and a second parameter corresponding to the first parameter. The first parameter is the feature vector of the object in a preset position, and the second parameter is the feature vector of the first main body model in the corresponding preset position. Calculate the sum of the plurality of second Euclidean distances to obtain the first Euclidean distance between the feature vector of the subject object and the feature vector of the first subject model, wherein the first subject model is any one of the plurality of subject models.

4. The video restoration method according to claim 2, characterized in that, When the main object is a person, the step of calculating multiple first Euclidean distances between the feature vector of the main object and the feature vectors of the multiple main object models includes: Multiple third Euclidean distances are calculated based on multiple sets of second parameter pairs, wherein each set of second parameter pairs includes a third parameter and a fourth parameter corresponding to the third parameter. The third parameter is used to characterize the skeletal joints of the character, and the fourth parameter is used to characterize the corresponding skeletal joints of the first main body model. Calculate the sum of the multiple third Euclidean distances to obtain the first Euclidean distance between the feature vector of the subject object and the feature vector of the first subject model, where the first subject model is any one of the multiple subject models.

5. The video restoration method according to claim 1, characterized in that, The step of using a virtual camera to supplement the target model from the multiple preset locations to restore the first surround video of the target position in the multiple preset locations includes: Obtain the actual shooting angle of the target location in the multiple preset directions; Based on the actual shooting angles of the target location in multiple preset directions, the virtual camera is used to supplement the image of the target model from the multiple preset directions to obtain the initial surround video of the target location in the multiple preset directions. Extract key information from the multiple original images; Using the key information, the initial surround video of the target location in multiple preset directions is rendered to obtain the first surround video of the target location in multiple preset directions.

6. The video restoration method according to claim 5, characterized in that, The method of using a virtual camera to supplement the target model from the multiple preset positions and restore the target position after the first surround video from the multiple preset positions further includes: After aligning the playback angle of the player with the actual shooting angle, the first surround video of the target location in the multiple preset directions is sent to the player. or, Receive correction information sent by the player; Based on the correction information and the first surround video of the target location in the multiple preset directions, a second surround video is generated; After aligning the playback angle of the player with the actual shooting angle, a second surround video of the target location at the multiple preset positions is sent to the player.

7. An electronic device, characterized in that, It includes a transceiver, a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the video restoration method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the video restoration method as described in any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that, The computer program is executed by at least one processor to implement the steps of the video restoration method as described in any one of claims 1-6.

Images