Video generation method and apparatus, electronic device, and medium

Abstract

Embodiments of the application disclose a video generation method and device, electronic equipment and medium, and relate to the technical field of artificial intelligence. A specific implementation of the method includes: determining a motion direction of a target object in a target scene, the target scene being previously arranged with a shooting array formed by at least one camera; determining a directional camera matched with the motion direction in the shooting array according to the motion direction; determining a main camera group and a secondary camera group from the shooting array based on the directional camera; and generating a free-viewpoint video used for constructing a three-dimensional virtual space based on image sequences collected by the directional camera, the main camera group and the secondary camera group. The method can dynamically match and activate corresponding camera groups according to the motion direction of a user to perform targeted collection, thereby reducing the amount of data collection and processing overhead while ensuring the required detail accuracy of the generated free-viewpoint video.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a video generation method, apparatus, electronic device, and medium. Background Technology

[0002] When constructing video content in a 3D virtual space, it is necessary to collect sufficient video or image materials in the real world and generate high-fidelity digital images through 3D modeling and rendering. How to efficiently collect materials while balancing performance overhead and reconstruction granularity has become a key issue in this field.

[0003] Most related technologies use three cameras (left, center, and right) to collect data simultaneously, or use continuous video streams to collect data continuously. The collected materials are then used for 3D modeling and rendering to obtain video content for constructing a three-dimensional virtual space.

[0004] However, the above three-channel acquisition or continuous acquisition methods have obvious shortcomings: three-channel acquisition is prone to insufficient acquisition granularity and loss of details due to the limited number of cameras; while continuous acquisition, although it has a large amount of data, will generate a huge amount of data and processing overhead. Neither of them can achieve a good balance between performance overhead and reconstruction granularity. Summary of the Invention

[0005] This application provides a video generation method, apparatus, electronic device, and medium.

[0006] In a first aspect, embodiments of this application provide a video generation method, the method comprising: determining the motion direction of a target object in a target scene, wherein a shooting array formed by at least one camera is pre-arranged in the target scene;

[0007] Based on the direction of motion, determine the direction of the shooting array and the directional camera that matches the direction of motion; Based on the directional camera, the main camera group and the secondary camera group are determined from the shooting array; Based on image sequences acquired by the directional camera, the main camera group, and the secondary camera group, free-viewpoint videos are generated to construct a three-dimensional virtual space.

[0008] In some embodiments, determining a directional camera whose shooting array matches the direction of motion, based on the direction of motion, includes: Determine the angle between the direction vector of each camera in the shooting array and the motion direction vector of the motion direction, and identify the camera corresponding to the target angle as the directional camera.

[0009] In some embodiments, the main camera group includes a main center camera, a first main camera, and a second main camera. Determining the main camera group and the secondary camera group from the shooting array based on the directional camera includes: Set the directional camera as the main center camera; Based on the main central camera and the preset jump selection step size, cameras located on both sides of the directional camera are selected in the shooting array and used as the first main camera and the second main camera, respectively. Based on the main central camera, the secondary camera group is determined from the shooting array.

[0010] In some embodiments, the secondary camera group includes a secondary central camera, a first secondary camera, and a second secondary camera. Determining the secondary camera group from the shooting array based on the main central camera includes: Based on the direction of motion and the direction of the camera, determine the secondary center camera in the shooting array that is symmetrical to the main center camera; Based on the sub-center camera and the preset jump selection step size, cameras located on both sides of the sub-center camera are selected in the shooting array, serving as the first and second sub-cameras respectively.

[0011] In some embodiments, generating a free-viewpoint video for constructing a three-dimensional virtual space based on image sequences acquired by a directional camera, a main camera group, and a secondary camera group includes: Using a directional camera, acquire frontal motion sequence images of the target object when the gait landing event is triggered; Use the main center camera in the main camera group to obtain the background environment information of the target object; Using the first and second main cameras in the main camera group, the frontal physical texture information of the target object's body is obtained from the opposite side. Use the secondary center camera in the secondary camera group to obtain the forward scene environment information of the target object; The first and second secondary cameras in the secondary camera group are used to obtain the physical texture information of the back of the target object's body on the opposite side of the back. The frontal motion sequence images, the back scene environment information, the frontal physical texture information, the frontal scene environment information, and the back physical texture information are combined into a multi-view image set; A collection of images from multiple perspectives is input into a 3D reconstruction model to generate a free-viewpoint video.

[0012] In some embodiments, the method further includes: During the process of acquiring image sequences using a directional camera, a main camera group, and a secondary camera group, the updated motion direction, effective movement distance, and movement speed of the target object are determined. Based on the updated direction of motion, effective distance traveled, and speed of motion, determine the motion bias of the target object; Based on motion bias, the update camera is determined from the shooting array to update at least one camera in the main camera group and / or the secondary camera group according to the update camera.

[0013] In some embodiments, after updating at least one camera in the main camera group and / or the secondary camera group according to the updated camera, the method further includes: Determine the coverage area of ​​the triangular region formed by the main center camera, the first main camera, and the second main camera in the updated main camera group; Based on the coverage area, update at least one camera in the main camera group and / or the secondary camera group.

[0014] Secondly, embodiments of this application provide a video generation apparatus, the apparatus comprising: The detection unit is used to determine the direction of motion of the target object in the target scene, where an array of cameras is pre-arranged. The first determining unit is used to determine the direction camera that matches the direction of motion of the shooting array based on the direction of motion. The second determining unit is used to determine the main camera group and the secondary camera group from the shooting array based on the directional camera; The generation unit is used to generate free-viewpoint videos for constructing a three-dimensional virtual space based on image sequences acquired by the directional camera, the main camera group, and the secondary camera group.

[0015] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor, wherein, when the processor runs the computer program, it performs the method described in any embodiment of the first aspect.

[0016] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described in any embodiment of the first aspect.

[0017] Fifthly, embodiments of this application provide a computer program product including a computer program that, when executed by a processor, performs the method described in any embodiment of the first aspect.

[0018] This application provides a video generation method that determines the movement direction of a target object in a target scene, where a shooting array consisting of at least one camera is pre-arranged. Based on the movement direction, a directional camera matching the movement direction is determined within the shooting array. Based on the directional camera, a main camera group and a secondary camera group are determined from the shooting array. Based on the image sequences acquired by the directional camera, the main camera group, and the secondary camera groups, a free-viewpoint video for constructing a three-dimensional virtual space is generated. This method dynamically matches and activates the corresponding camera group based on the user's movement direction for targeted acquisition, reducing data acquisition volume and processing overhead while ensuring the detail accuracy required for generating free-viewpoint video.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are merely embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort, and this application can be applied to other similar scenarios based on the provided drawings.

[0021] Figure 1 A flowchart illustrating a video generation method provided in this application embodiment; Figure 2 A flowchart illustrating the second video generation method provided in this application embodiment; Figure 3 A schematic diagram of a directional camera provided in an embodiment of this application; Figure 4 A schematic diagram of a main camera assembly provided in an embodiment of this application; Figure 5 A schematic diagram of a secondary camera group provided in an embodiment of this application; Figure 6 A schematic diagram illustrating the adjustment of the left secondary camera, provided as an embodiment of this application; Figure 7 A schematic diagram illustrating a specific coverage area provided for an embodiment of this application; Figure 8 A schematic diagram of a specific main camera group and a secondary camera group provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of a video generation device 900 provided in an embodiment of this application; Figure 10This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. The described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0023] It should be noted that the terms "system," "device," "unit," and / or "module" used in this application are methods of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they can be replaced by other expressions.

[0024] Hereinafter, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include one or more of that feature.

[0025] To achieve the best digital space display effect, it is usually necessary to collect sufficient video or image materials in the real world, and then adjust, process, and 3D model and render them to finally construct the image content in the digital space.

[0026] Currently, the relevant technologies mainly employ the acquisition of three video streams (left, center, and right) or the acquisition of real-world images through continuous acquisition, and then perform 3D modeling and rendering on the obtained video streams.

[0027] However, neither conventional three-channel acquisition nor continuous video stream acquisition can simultaneously meet the dual requirements of performance overhead and acquisition granularity.

[0028] To address the aforementioned issues, this application adaptively matches directional cameras from a fixed shooting array by real-time detection of the target object's motion direction, and determines the main and secondary camera groups based on the directional cameras in layers. This allows the acquired image sequences to closely follow the user's motion posture and perspective changes, thereby reducing redundant data acquisition and processing overhead while ensuring the integrity of the multi-angle, high-precision 3D information required to generate free-viewpoint videos, achieving a balance between acquisition efficiency and reconstruction quality.

[0029] The following section provides a detailed description of a video generation method provided in this application, with reference to the accompanying drawings.

[0030] Figure 1 A flowchart of a video generation method provided in an embodiment of this application is shown. Figure 1 As shown, the video generation method includes steps 101-104.

[0031] Step 101: Determine the direction of motion of the target object in the target scene, where a shooting array formed by at least one camera is pre-arranged.

[0032] In the embodiments of this application, the motion direction of the target object can be determined in a variety of ways: for example, by setting an inertial sensor on the target object for detection; or by using a camera that is pre-arranged and turned on in the target scene (e.g., the center camera of the array located at the center of the shooting array, or the global camera in the target scene), combined with computer vision algorithms (such as optical flow method, target tracking) to detect the displacement direction of the target object (usually a human body) in three-dimensional space in real time, thereby obtaining its motion direction vector.

[0033] The target object is the main subject of the video footage to be captured, usually a moving human body or a movable object.

[0034] The target scene is a physical space where shooting equipment is set up in advance, such as a circular studio or a motion capture area.

[0035] A camera array is a group of acquisition devices consisting of multiple cameras arranged in a specific geometric layout (such as a ring) in a target scene. Each camera has a fixed spatial position and shooting optical axis direction, and is used to synchronously or selectively acquire image sequences of the target object from different perspectives.

[0036] The direction of motion is the instantaneous velocity direction of the target object in three-dimensional space, represented by a unit vector (e.g., due north, due east, etc.).

[0037] Step 102: Based on the direction of motion, determine the direction camera that matches the direction of motion for the shooting array.

[0038] In embodiments of this application, in a circularly arranged camera array, each camera has a fixed optical axis direction. This application can calculate the angle between the direction vector of each camera and the motion direction vector of the target object (e.g., using cosine similarity), and determine the directional camera based on this angle. This directional camera faces the frontal motion direction of the target object, making it most suitable for capturing motion sequences.

[0039] A directional camera is the camera that best matches the target's direction of movement at any given moment, and is used to continuously record the target's frontal movements.

[0040] Step 103: Based on the directional camera, determine the main camera group and the secondary camera group from the shooting array.

[0041] In the embodiments of this application, a main center camera is determined based on a directional camera. Then, cameras with fixed step distances are selected on either side of the main center camera according to a preset selection step size (calculated based on the target's size and field of view overlap rate). These cameras serve as the left and right main cameras in the main camera group. Simultaneously, a secondary center camera, mirror-symmetrical to the main center camera, is found based on the extension line of the movement direction. The left and right secondary cameras are then selected using the secondary center camera as a reference. The main camera group primarily captures the body texture from the front and sides of the target; the secondary camera group captures the body texture from the back and environmental information.

[0042] The main camera group is a collection of cameras responsible for capturing the physical attributes of the front and side of the target object. It typically includes the main center camera (i.e., the directional camera), the first main camera (left main camera), and the second main camera (right main camera).

[0043] The secondary camera group is a collection of cameras responsible for capturing the physical properties of the target object from behind and the surrounding space. It includes the secondary center camera, the first secondary camera (left secondary), and the second secondary camera (right secondary).

[0044] It is understood that the main central camera described in this application is the directional camera determined based on the direction of movement of the target object, and it is located at the center of the field of view relative to the current direction of movement of the target object. Furthermore, the imaging array also includes a pre-set array central camera located at the geometric center of the array. This array central camera can be the same camera as the directional camera, or it can be a different camera. When the two are different cameras, this application disables the array central camera after determining the directional camera.

[0045] Step 104: Based on the image sequences acquired by the directional camera, the main camera group, and the secondary camera group, generate a free-viewpoint video for constructing a three-dimensional virtual space.

[0046] In some embodiments, all selected cameras synchronously acquire images, each capturing specific content: a directional camera captures a frontal action sequence, a main center camera captures a back view of the scene environment, left and right main cameras capture frontal body textures on opposite sides, a secondary center camera captures a back view action sequence and a frontal scene environment, and left and right secondary cameras capture backal body textures on opposite sides. These multi-view images are time-synchronized, depth-estimated, and texture-mapped before being input into a 3D reconstruction model (such as a neural radiation field or a mesh-based reconstruction algorithm), outputting a free-viewpoint video that can be played from any perspective.

[0047] An image sequence is a sequence of frames captured continuously over time by each camera, including action sequence frames and environmental information frames.

[0048] Free-viewpoint video is a video format that allows users to freely choose their viewing angle (e.g., 6 degrees of freedom) during playback. Viewers can move the viewpoint forward, backward, left, right, up, and down, or circle around to observe the target object.

[0049] In summary, the video generation method proposed in this application dynamically matches the most suitable directional camera by sensing the motion direction of the target object in real time. Using this camera as a reference, the main and secondary camera groups are activated in layers to collect frontal motion, front and back body textures, and front and back environmental information, respectively. Finally, these are fused to generate a free-viewpoint video. This method avoids data redundancy and high overhead caused by all cameras working simultaneously, achieves on-demand allocation of acquisition resources, and significantly reduces the size of the source material and processing load while ensuring the detail accuracy of 3D reconstruction.

[0050] Figure 2 This is a flowchart illustrating another data processing method provided in an embodiment of the present disclosure. Figure 2 based on Figure 1 The illustrated embodiment further defines step 102. Figure 2 In the illustrated embodiment, step 102 includes step 202. For example... Figure 2 As shown, the method may include: Step 201: Determine the direction of motion of the target object in the target scene, where a shooting array formed by at least one camera is pre-arranged.

[0051] In the embodiments of this application, several shooting devices are arranged in a ring in the target scene to form a ring-shaped shooting matrix. This shooting matrix is ​​used to capture video images corresponding to free-angle video or digital spatial video streams. In this shooting matrix, the selected cameras are in working mode, while the unselected cameras are in standby mode and do not actually shoot, thereby saving resources and reducing the size of the captured material.

[0052] Based on this, this application can first use sensors or pre-activated cameras to detect the movement trajectory of the target object in the target scene, and determine the movement direction of the target object based on the velocity vector (i.e., the motion direction vector) in the movement direction of the target object.

[0053] Step 202: Determine the angle between the direction vector of each camera in the shooting array and the motion direction vector of the motion direction, and determine the camera corresponding to the target angle as the directional camera.

[0054] In the embodiments of this application, the camera's direction vector This refers to the fixed shooting optical axis direction (i.e., the direction the lens is facing) that each camera has, represented by a three-dimensional unit vector. For example, the direction vector of the north camera points south (towards the center of the circle), the east camera points west, and so on.

[0055] Motion direction vector It represents the instantaneous direction of motion of the target object at the current moment, and is also a three-dimensional unit vector. For example, if the user is moving in the northeast direction, then... Pointing northeast.

[0056] The included angle is and The spatial angle between the two. The smaller the angle, the closer the camera's optical axis is to the user's actual direction of movement. The target angle refers to the smallest possible angle.

[0057] Specifically, this application can determine the directional camera in the shooting matrix based on the user's current direction of movement (i.e., the motion direction vector). That is, in the target scene, this application detects the user's movement trajectory and, based on the velocity vector in the user's movement direction (i.e., the motion direction vector), determines the camera with the smallest angle between the motion direction and the camera direction; this camera is the directional camera.

[0058] Alternatively, this application can compute the orientation vector of each camera. The camera whose cosine value is closest to 1 (i.e., the smallest angle) is selected from the angles between the camera and the user's motion direction vector v.

[0059]

[0060] Wherein, the user location is set as The direction vector of motion is The position of each camera in the matrix is The wide-angle direction vector of the camera is And calculate the similarity dot product between the user and the camera. ,choose The smallest camera is used as the directional camera. This is achieved by calculating the angle between the direction each camera points and the user's movement direction, and selecting the camera with the smallest angle as the directional camera to ensure that this camera can track the user's movement most directly.

[0061] like Figure 3 The diagram shown is a schematic of a directional camera provided in this application. The directional camera in this application is used to record a frontal motion sequence image P1 of the target object's dynamic movements after the latest action occurs (i.e., triggering a gait landing event, such as when a user's single foot moves). It is a sequence because after each walking action, images are captured at a preset frequency between the beginning of the action and the next action; the temporal order of the images and the actions forms the motion sequence.

[0062] Step 203: Based on the directional camera, determine the main camera group and the secondary camera group from the shooting array.

[0063] In some embodiments, the main camera group includes a main center camera, a first main camera, and a second main camera. Specifically, this application uses the directional camera determined in step 202 as the main center camera; then, based on the main center camera and a preset selection step size, cameras located on both sides of the main center camera are selected in the shooting array, respectively, as the first main camera and the second main camera. Based on this, this application further determines a secondary camera group based on the main center camera. The secondary camera group includes a secondary center camera, a first secondary camera, and a second secondary camera. First, according to the movement direction of the target object and the main center camera, cameras that are mirror-symmetrical to the main center camera in the shooting array are determined as secondary center cameras; then, based on the secondary center camera and a preset selection step size, cameras located on both sides of the secondary center camera are selected in the shooting array, respectively, as the first secondary camera and the second secondary camera.

[0064] Specifically, this application acquires the motion direction of the target object in real time at time intervals t, and redetermines the directional camera in the shooting matrix based on time interval t. Simultaneously, it determines whether this directional camera is the same device as the center camera of the array. Specifically, when the motion direction vector of the target object changes frequently, after a direction change at time t, the newly determined directional camera is usually not the same camera as the center camera of the array; conversely, if the direction is stable, they are the same.

[0065] After obtaining the directional camera, this application determines the directional camera as the main center camera and performs a jump selection based on the left and right order of the camera to obtain the left main camera (first main camera) and the right main camera (second main camera) in the main direction. The left and right main cameras respectively capture the physical texture information of the front side of the moving user's body, and the main center camera captures the background scene information of the target object, which together obtain P2.

[0066] For skip selection, it is a camera selection strategy based on preset topology rules. If the positive direction of movement is taken as c of the center camera in the camera matrix, then the index of the left main camera can be c. The right main camera index can be c+step, where step is the step size for selection, which can be adjusted according to the required shooting angle baseline and the target's body shape.

[0067] For the step function, the selection method is as follows:

[0068] in, To ensure good target coverage, the ideal baseline length between the left and right main cameras can be estimated as follows: , For the target body width, For the expected overlap rate, This refers to the field of view of a single camera. This represents the spacing between the camera matrix.

[0069] like Figure 4 The diagram shown is a schematic representation of a main camera assembly provided in this application. (Refer to...) Figure 4 , Figure 4 The central camera coincides with the center camera of the array. The left and right main cameras obtained in this application capture the frontal physical texture information of the target object's body from opposite sides. This physical texture information refers to the skin's reaction under different lighting conditions using a spherical light stage or camera array from multiple angles, in order to create hyper-realistic digital human skin texture. Based on this data, a PBR (Physical Background Repair) map set, including diffuse maps, normal maps, and roughness maps, can be calculated, thus comprehensively defining the physical properties of the character.

[0070] After obtaining the main camera group through the above operations, this application can determine the secondary camera group based on the main center camera in the main camera group.

[0071] Taking the positive direction of user movement as an example, find the secondary center camera that is mirror-symmetrical to the main center camera on the extension line of this positive direction, so as to capture the positive scene environment information P3 (i.e., the image sequence of the positive scene environment information of the target object) that is consistent with the main center camera. Optionally, such as Figure 5 As shown, this application provides a schematic diagram of a secondary camera assembly. The positive direction of user movement is... An extension line is constructed in this direction, and the corresponding secondary center camera is found based on this extension line (the secondary center camera is symmetrical to the primary center camera; since the primary center camera is the directional camera, the secondary center camera is symmetrical to the directional camera). The frontal scene environment information P3 is obtained through the secondary center camera, and based on the above-mentioned jump-selection of the same camera strategy, the left secondary center camera (first secondary center camera) and the right secondary center camera (second secondary center) are found, and the left and right secondary cameras are respectively used to capture the physical texture of the target object's back side body and the scene environment information P4 (i.e., the back physical texture information P4).

[0072] Step 204: Based on the image sequences acquired by the directional camera, the main camera group, and the secondary camera group, generate a free-viewpoint video for constructing a three-dimensional virtual space.

[0073] In some embodiments, a directional camera is used to acquire a frontal motion sequence image of the target object when the gait landing event is triggered; a main center camera in the main camera group is used to acquire the back scene environment information of the target object; a first main camera and a second main camera in the main camera group are used to acquire the frontal physical texture information of the target object's opposite side of the body; a secondary center camera in the secondary camera group is used to acquire the frontal scene environment information of the target object; a first secondary camera and a second secondary camera in the secondary camera group are used to acquire the back physical texture information of the target object's opposite side of the body; the frontal motion sequence image, the back scene environment information, the frontal physical texture information, the frontal scene environment information, and the back physical texture information are combined into a multi-view image set (P={P1; P2; P3; P4}); the multi-view image set is input into a 3D reconstruction model (e.g., Vid3D) to generate a free viewpoint video (i.e., a 3D virtual space video stream).

[0074] Furthermore, this application can continuously detect the user's positive motion direction vector (i.e., the updated motion direction vector that updates the motion direction), effective movement distance, and movement speed while simultaneously capturing images using a predetermined main center camera, left and right main cameras, secondary center camera, and left and right secondary cameras. The purpose of obtaining the user's positive motion direction vector is to obtain the user's movement trajectory in real time and adjust the camera positions and relative distances based on the movement process, thereby acquiring image resources of consistent quality with fewer acquisition resources. In other words, this application can determine the updated motion direction, effective movement distance, and movement speed of the target object during the process of acquiring image sequences using directional cameras, the main camera group, and the secondary camera group; determine the motion bias of the target object based on the updated motion direction, effective movement distance, and movement speed; and determine the updated camera from the shooting array based on the motion bias, so as to update at least one camera in the main camera group and / or the secondary camera group accordingly.

[0075] Meanwhile, this application can also continuously obtain the ratio of effective moving distance to moving speed. If the ratio is lower than the preset ratio threshold, it is determined that the data overhead of the device has increased. At this time, the camera should be further adjusted based on the increased overhead.

[0076] Specifically, when the target object (i.e., the mobile user) is relatively stationary in the positive direction of movement, the dynamic positions of the left and right secondary cameras are adjusted according to the user's moving speed V and the effective moving distance. These positions are adjusted and selected by quantifying the distance between the two devices.

[0077] Optionally, such as Figure 6 The diagram shown is a schematic representation of adjusting the left secondary camera according to this application. (Refer to...) Figure 6 When the positive bias is relatively stationary, calculate the user's longitudinal movement bias. (i.e., the skewness of motion), that is .

[0078] Here, the directional offset θ means that the user is moving longitudinally. In order to maintain tracking, the camera needs to translate laterally to compensate. The amount of displacement is proportional to the size of the offset angle, the user's moving speed, and the prediction time. The effective travel distance represents the actual distance a user travels within the actual travel time R. The lateral component of the travel is calculated using sin(θ). When θ is small, sin(θ) ≈ θ, which is used for gain calculation. (Position adjustment gain) is used to fine-tune the tracking sensitivity, Δ >0 indicates moving forward, Δ <0 indicates moving backward.

[0079] Another scenario is that the user's positive bias is not static; in this case, Δ It will become and The square root of the squares of the two components is used to represent the actual changes in the user's movement.

[0080] This application can then be based on the user's vertical movement bias. Calculate the new baseline distance In this application, dynamically adjusting the baseline distance can optimize the depth perception effect of stereo vision at different speeds. Generally, higher speeds require a larger baseline distance to improve the resolution of distant objects. This baseline distance is used to determine the distance between the left and right secondary cameras.

[0081] Among them, the new baseline distance The calculation formula is: This represents the current distance between the left and right secondary cameras. This is a correction factor for the size of the camera itself.

[0082] Finally, this application can be based on the new baseline distance. This determines the final updated positions of the left and right secondary cameras within the matrix. Specifically, it determines the new midpoint coordinates based on the previously determined midpoint coordinates of the left and right secondary cameras. ,in, This provides the old midpoint coordinates of either the left or right secondary camera. Then, the updated position of the left secondary camera is calculated based on these midpoint coordinates. ,in, Let x be the coordinates of the new midpoint. The y-axis component represents the old midpoint coordinates. Finally, the updated position of the left secondary camera is calculated based on the midpoint coordinates: Based on the updated location, the system determines which secondary camera to update and replaces the original secondary camera with the updated one.

[0083] It's important to note that when the updated positions of the left secondary camera and / or the right main secondary camera, calculated based on user movement, are more than half the radius / half-width of the field center (axis), the position is considered too close to the edge of the field and may even be outside the effective shooting range. In this case, for safety reasons (to prevent the cameras from capturing meaningless edge areas or to avoid mechanical / logical boundary violations), position updates are not performed; that is, the left and right secondary cameras remain in their original positions.

[0084] Specifically, when the user's movement speed is relatively constant, the updated position of the main center camera is determined based on the degree of oscillation of the component in the positive direction of movement. Simply put, although the user's movement speed is constant, it's not in a straight line, but rather in an arc or curve. Therefore, when the user is not moving in a straight line, the position of the center camera needs to be readjusted to better capture the most forward-facing view.

[0085] At this point, it is necessary to calculate the lateral displacement. (i.e., the partial motion): .in, For user movement speed, The time it takes for the positive direction component to travel. The oscillation is the component in the positive direction. This is the oscillation sensitivity coefficient.

[0086] And on Apply threshold range restrictions, i.e., those that conform to the range. Values ​​are retained, while those that do not meet the requirements are discarded, thus effectively filtering out unnecessary minor vibrations and preventing frequent fine-tuning of the camera.

[0087] Ultimately, at the original location of the main central camera... Based on this, adjustments and corrections are made to obtain the updated position of the main central camera. . .in, The original midpoint coordinates of the main center camera. The calculated lateral displacement is given by θ, which represents the directional offset of the moving user. For user movement speed, This represents the time during which the positive directional component of the movement occurs. This logic indicates that the camera is moving at a constant speed (V) while following the user. Simultaneously with Δt, the lateral displacement is adjusted according to the direction of the swing. .

[0088] Furthermore, after updating at least one camera in the main camera group and / or the secondary camera group according to the updated camera, this application can also determine the coverage area of ​​the triangular region formed by the main center camera, the first main camera, and the second main camera in the updated main camera group; based on the coverage area, at least one camera in the main camera group and / or the secondary camera group is updated. The coverage area can be initially set based on the main cameras, or it can be based on the directional cameras when the upstream system changes.

[0089] Specifically, such as Figure 7 The diagram shown illustrates a specific coverage area provided in this application. (Refer to...) Figure 7 This application can determine whether it is necessary to increase or decrease the number of main cameras based on the latest camera location data, the location of the main central camera, and the coverage area of ​​the triangular topology formed by the central camera and the left and right main cameras.

[0090] Optionally, this application can calculate the position of the central camera and the area of ​​the triangular topology formed by the central camera and the left and right main cameras (i.e., the coverage area of ​​the triangular region). When the area is sufficient to cover 1 / 3 but less than 1 / 2 of the target scene area, no adjustment is required. If it is greater than 1 / 2, the position of the main central camera is reversed, and the original main central camera is used as the main central camera. If it is less than 1 / 3, based on the user's positive directional bias, the secondary camera on the side with the greater bias is replaced as the right main camera.

[0091] After the above processing, such as Figure 8 The diagram shown illustrates a specific main camera assembly and a secondary camera assembly provided in this application. (Refer to...) Figure 8 This application ultimately obtains a set of multi-degree-of-freedom video stream materials by selecting the main center camera, left and right main cameras, sub-center camera, and left and right sub-sub cameras, and finally inputs the set of multi-degree-of-freedom video stream materials into Vid3D to generate a 3D virtual space video stream.

[0092] In summary, this application achieves a shift from passive data acquisition to active optimization by dynamically adjusting the number, arrangement, and task allocation of cameras in a circular camera array based on the user's real-time movement direction. This reduces the number of devices and data redundancy while ensuring the focus and accuracy of the acquisition.

[0093] In order to implement the above embodiments, this application also provides a video generation apparatus. Figure 9 This is a schematic diagram of the structure of a video generation device 900 provided in an embodiment of this application. Figure 9 As shown, the device includes: The detection unit 910 is used to determine the direction of movement of the target object in the target scene, where a shooting array formed by at least one camera is pre-arranged. The first determining unit 920 is used to determine the direction camera that matches the direction of motion of the shooting array based on the direction of motion. The second determining unit 930 is used to determine the main camera group and the secondary camera group from the shooting array based on the directional camera; The generation unit 940 is used to generate free-viewpoint videos for constructing a three-dimensional virtual space based on image sequences acquired by the directional camera, the main camera group, and the secondary camera group.

[0094] In some embodiments, the first determining unit 920 is configured to: Determine the angle between the direction vector of each camera in the shooting array and the motion direction vector of the motion direction, and identify the camera corresponding to the target angle as the directional camera.

[0095] In some embodiments, the main camera group includes a main center camera, a first main camera, and a second main camera; the second determining unit 930 is configured to: Set the directional camera as the main center camera; Based on the main central camera and the preset jump selection step size, cameras located on both sides of the directional camera are selected in the shooting array and used as the first main camera and the second main camera, respectively. Based on the main central camera, the secondary camera group is determined from the shooting array.

[0096] In some embodiments, the secondary camera group includes a secondary central camera, a first secondary camera, and a second secondary camera; the second determining unit 930 is configured to: Based on the direction of motion and the direction of the camera, determine the secondary center camera in the shooting array that is symmetrical to the main center camera; Based on the sub-center camera and the preset jump selection step size, cameras located on both sides of the sub-center camera are selected in the shooting array, serving as the first and second sub-cameras respectively.

[0097] In some embodiments, the generating unit 940 is configured to: Using a directional camera, acquire frontal motion sequence images of the target object when the gait landing event is triggered; Use the main center camera in the main camera group to obtain the background environment information of the target object; Using the first and second main cameras in the main camera group, the frontal physical texture information of the target object's body is obtained from the opposite side. Use the secondary center camera in the secondary camera group to obtain the forward scene environment information of the target object; The first and second secondary cameras in the secondary camera group are used to obtain the physical texture information of the back of the target object's body on the opposite side of the back. The frontal motion sequence images, the back scene environment information, the frontal physical texture information, the frontal scene environment information, and the back physical texture information are combined into a multi-view image set; A collection of images from multiple perspectives is input into a 3D reconstruction model to generate a free-viewpoint video.

[0098] In some embodiments, the second determining unit 930 is configured to: During the process of acquiring image sequences using a directional camera, a main camera group, and a secondary camera group, the updated motion direction, effective movement distance, and movement speed of the target object are determined. Based on the updated direction of motion, effective distance traveled, and speed of motion, determine the motion bias of the target object; Based on motion bias, the update camera is determined from the shooting array to update at least one camera in the main camera group and / or the secondary camera group according to the update camera.

[0099] In some embodiments, the second determining unit 930 is configured to: after updating at least one camera in the main camera group and / or the secondary camera group according to the updated camera, determine the coverage area of ​​the triangular region formed by the main center camera, the first main camera and the second main camera in the updated main camera group. Based on the coverage area, update at least one camera in the main camera group and / or the secondary camera group.

[0100] The methods and apparatus provided in the embodiments of this application have been described above. To implement the functions of the methods provided in the embodiments of this application, the electronic device may include a hardware structure and software modules, and may implement the above functions in the form of a hardware structure, software modules, or a hardware structure plus software modules. One of the above functions may be executed in the form of a hardware structure, software modules, or a hardware structure plus software modules.

[0101] Figure 10 This is a block diagram illustrating an electronic device 1000 for implementing the above-described video generation method, according to an exemplary embodiment. For example, the electronic device 1000 may be a mobile phone, computer, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0102] Reference Figure 10The electronic device 1000 may include a communication interface 1001, capable of interacting with other devices; a processor 1002, connected to the communication interface 1001 to interact with other devices, used to execute the methods provided by one or more of the above-described technical solutions when running a computer program; and a memory 1003, on which the computer program is stored. Specifically, the specific processing procedure of the processor 1002 can refer to the video generation method described in the above embodiments of this disclosure.

[0103] Of course, in practical applications, the various components in electronic device 1000 are coupled together through bus system 1004. It can be understood that bus system 1004 is used to realize the connection and communication between these components. In addition to a data bus, bus system 1004 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in... Figure 10 The general labeled all buses as Bus System 1004.

[0104] The memory 1003 in this embodiment is used to store various types of data to support the operation of the electronic device 1000. Examples of such data include any computer program used to operate on the electronic device 1000.

[0105] The methods disclosed in the embodiments of this application can be applied to processor 1002, or implemented by processor 1002. Processor 1002 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 1002 or by instructions in the form of software. The processor 1002 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 1002 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 1003. Processor 1002 reads the information in memory 1003 and completes the steps of the aforementioned method in combination with its hardware.

[0106] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0107] Embodiments of this disclosure also propose a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the video generation method described in the above embodiments of this disclosure.

[0108] Embodiments of this disclosure also provide a computer program product, including a computer program that is executed by a processor using the video generation method described in the above embodiments of this disclosure.

[0109] Embodiments of this disclosure also propose a chip including one or more interface circuits and one or more processors; the interface circuits are used to receive signals from the memory of an electronic device and send signals to the processors, the signals including computer instructions stored in the memory, which, when executed by the processor, cause the electronic device to perform the video generation method described in the above embodiments of this disclosure.

[0110] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0111] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0112] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0113] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processing module, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (control method), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0114] It should be understood that various parts of the embodiments of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0115] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0116] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc.

[0117] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A video generation method, characterized in that, include: Determine the direction of movement of the target object in the target scene, wherein the target scene is pre-arranged with a shooting array formed by at least one camera; Based on the direction of motion, determine the directional camera in the shooting array that matches the direction of motion; Based on the directional camera, the main camera group and the secondary camera group are determined from the shooting array; Based on the image sequences acquired by the directional camera, the main camera group, and the secondary camera group, a free-viewpoint video for constructing a three-dimensional virtual space is generated.

2. The method according to claim 1, characterized in that, The step of determining the direction camera that matches the shooting array with the direction of motion based on the direction of motion includes: Determine the angle between the direction vector of each camera in the shooting array and the motion direction vector of the motion direction, and identify the camera corresponding to the target angle as the directional camera.

3. The method according to claim 1, characterized in that, The main camera group includes a main center camera, a first main camera, and a second main camera. Determining the main camera group and the secondary camera group from the shooting array based on the directional cameras includes: The directional camera is identified as the main center camera; Based on the main central camera and the preset jump selection step size, cameras located on both sides of the directional camera are selected in the shooting array, and are respectively used as the first main camera and the second main camera; Based on the main central camera, the secondary camera group is determined from the shooting array.

4. The method according to claim 1, characterized in that, The secondary camera group includes a secondary center camera, a first secondary camera, and a second secondary camera. Determining the secondary camera group from the shooting array based on the main center camera includes: Based on the direction of motion and the direction camera, determine the secondary center camera in the shooting array that is symmetrical to the main center camera; Based on the sub-center camera and the preset jump selection step size, cameras located on both sides of the sub-center camera are selected in the shooting array, and are respectively used as the first sub-camera and the second sub-camera.

5. The method according to claim 1, characterized in that, The process of generating free-viewpoint video for constructing a three-dimensional virtual space based on image sequences acquired by the directional camera, the main camera group, and the secondary camera group includes: Using the directional camera, acquire a frontal motion sequence image of the target object when the gait landing event is triggered; Using the main center camera in the main camera group, obtain the background scene environment information of the target object; Using the first and second main cameras in the main camera group, the frontal physical texture information of the target object's body on the opposite side is obtained; The sub-center camera in the sub-camera group is used to acquire the forward scene environment information of the target object; The physical texture information of the back of the target object's body is obtained using the first and second secondary cameras in the secondary camera group. The frontal action sequence image, the back scene environment information, the frontal physical texture information, the frontal scene environment information, and the back physical texture information are combined into a multi-view image set; The multi-view image set is input into the 3D reconstruction model to generate the free viewpoint video.

6. The method according to claim 1, characterized in that, The method further includes: During the process of acquiring image sequences using the directional camera, the main camera group, and the secondary camera group, the updated motion direction, effective movement distance, and movement speed of the target object are determined; The motion bias of the target object is determined based on the updated motion direction, the effective movement distance, and the movement speed. Based on the motion bias, an update camera is determined from the shooting array to update at least one camera in the main camera group and / or the secondary camera group according to the update camera.

7. The method according to claim 6, characterized in that, After updating at least one camera in the main camera group and / or the secondary camera group according to the updated camera, the method further includes: Determine the coverage area of ​​the triangular region formed by the main center camera, the first main camera, and the second main camera in the updated main camera group; Based on the coverage area, update at least one camera in the main camera group and / or the secondary camera group.

8. A video generation apparatus, characterized in that, The device includes: A detection unit is used to determine the direction of movement of a target object in a target scene, wherein a shooting array formed by at least one camera is pre-arranged in the target scene; The first determining unit is configured to determine, based on the direction of motion, the directional camera of the shooting array that matches the direction of motion; The second determining unit is used to determine the main camera group and the secondary camera group from the shooting array based on the directional camera; The generation unit is used to generate free-viewpoint videos for constructing a three-dimensional virtual space based on the image sequences acquired by the directional camera, the main camera group, and the secondary camera group.

9. An electronic device, characterized in that, include: The processor and the memory used to store computer programs that can run on the processor. When the processor is used to run the computer program, it performs the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 7.

Images