A display control method, apparatus, electronic device, and readable storage medium

Abstract

This application discloses a display control method, apparatus, and electronic device. The method includes: in response to a shooting command for a first virtual character in a virtual scene, acquiring a reference video, wherein the reference video is obtained by shooting the first character; performing segmentation processing on the reference video to obtain multiple segmented clips; extracting target image features from the first character in video frames of the segmented clips; determining target lens parameters corresponding to the video frames from a pre-configured lens parameter template library based on the target image features extracted from the first character in the video frames; and shooting the first virtual character based on the target lens parameters. This method allows users to shoot virtual objects without requiring complex camera movement knowledge, resulting in videos with camera movement effects.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically to a display control method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] In virtual reality technology, virtual cameras are typically used to film the virtual scene.

[0003] Currently, virtual objects and at least one corresponding camera movement control control are typically displayed in the virtual scene's interface. Users then control the virtual camera using this control, such as adjusting the lens by sliding a slider. However, this method demands a high level of camera movement skill from the user, requiring some photography knowledge to capture videos with dynamic camera movements that align with the narrative's rhythm. Therefore, there is an urgent need for a shooting method that allows users to capture videos with dynamic camera movements without requiring complex camera movement knowledge. Summary of the Invention

[0004] In view of this, this application provides a display control method, apparatus, electronic device, and computer-readable storage medium that enables users to shoot videos with camera movement effects without requiring them to have complex camera movement knowledge.

[0005] In a first aspect, embodiments of this application provide a display control method, the method comprising:

[0006] In response to a shooting instruction for a first virtual character in a virtual scene, a reference video is acquired, wherein the reference video is obtained by shooting the first character as the main subject;

[0007] The reference video is processed into multiple storyboard segments.

[0008] Extract target image features from the first person in the video frames of the storyboard segment;

[0009] Based on the target image features extracted from the first person in the video frame, the target shot parameters corresponding to the video frame are determined from a pre-configured shot parameter template library. The shot parameter template library includes a video frame template, a template image feature corresponding to the second person in the video frame template, and a correspondence between the shot parameters corresponding to the template image feature.

[0010] The first virtual character is photographed based on the target camera parameters.

[0011] Secondly, embodiments of this application provide a display control device, the device comprising:

[0012] The acquisition module is used to acquire a reference video in response to a shooting instruction for a first virtual character in a virtual scene, wherein the reference video is obtained by shooting the main body of the first character;

[0013] The storyboard module is used to perform storyboard processing on the reference video to obtain multiple storyboard segments.

[0014] The extraction module is used to extract target image features from the first person subject in the video frames of the storyboard segment;

[0015] The determining module is used to extract the target image features for the first person in the video frame and determine the target shot parameters corresponding to the video frame from a pre-configured shot parameter template library. The shot parameter template library includes a video frame template, a template image feature corresponding to the second person in the video frame template, and a correspondence between the shot parameters corresponding to the template image feature.

[0016] The shooting module is used to shoot the first virtual character based on the target camera parameters.

[0017] Thirdly, embodiments of this application provide an electronic device, including:

[0018] Processor; and

[0019] The memory is used to store data processing programs, which, when the electronic device is powered on and run by the processor, execute the method as described in the first aspect.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium storing a data processing program that is executed by a processor to perform the method as described in the first aspect.

[0021] This application provides a display control method, which includes: in response to a shooting command for a first virtual character in a virtual scene, acquiring a reference video, wherein the reference video is obtained by shooting the first character; performing segmentation processing on the reference video to obtain multiple segmented clips; extracting target image features from the first character in video frames of the segmented clips; determining target lens parameters corresponding to the video frames from a pre-configured lens parameter template library based on the target image features extracted from the first character in the video frames; and shooting the first virtual character based on the target lens parameters. This method allows users to shoot virtual objects without requiring complex camera movement knowledge, resulting in videos with camera movement effects. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A first flowchart of the display control method provided in this application embodiment;

[0024] Figure 2 This is a second schematic diagram of the display control method provided in the embodiments of this application;

[0025] Figure 3 A schematic diagram of the display control device provided in this application embodiment;

[0026] Figure 4 This application provides a block diagram of an electronic device for implementing a display control method. Detailed Implementation

[0027] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0028] It should be noted that the terms "first," "second," "third," etc., in the claims, specification, and drawings of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. Such data are interchangeable where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown or described herein. Furthermore, the terms "comprising," "having," and their variations are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0029] It should be understood that in the embodiments of this application, "at least one" means one or more, and "more than one" means two or more. "And / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "Contains A, B and / or C" means containing any one, two, or three of A, B, and C.

[0030] It should be understood that in the embodiments of this application, "B corresponding to A", "B corresponding to A", "A corresponds to B" or "B corresponds to A" means that B is associated with A, and B can be determined based on A. Determining B based on A does not mean that B is determined solely based on A; B can also be determined based on A and / or other information.

[0031] In view of the problems existing in the above-mentioned related technologies, the present application provides a display control method, device, electronic device and computer-readable storage medium.

[0032] The display control method provided in this application can be executed by an electronic device, which can be a terminal or a server. The terminal can be a smartphone, tablet, laptop, or other similar device. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. It is understood that this application does not specifically limit the executing entity of the display control method.

[0033] In one alternative embodiment, when the display control method runs on a terminal device, the terminal device stores an application program that implements the display control method. The terminal device interacts with the user through a graphical user interface (GUI). The terminal device can provide the GUI to the user in various ways, such as rendering it on the terminal device's display screen or presenting the GUI via holographic projection.

[0034] In an optional embodiment, when the display control method runs on a server, it can be implemented and executed based on a cloud service system. A cloud service system refers to a service model based on cloud computing. A cloud service system includes servers and client devices. The application running the display control method and the display presentation are separate; the storage and execution of the display control method are completed on the server. The video presentation is completed on the client, which is mainly used for receiving and sending video data and presenting the video. For example, the client can be a display device with data transmission capabilities located close to the user, such as a mobile terminal, television, computer, PDA, personal digital assistant, head-mounted display device (HMD), etc. However, the electronic device implementing the display control method is a server in the cloud. For example, the user operates the client to send instructions to the server; the server controls and runs the display control method according to the instructions, encodes and compresses data such as video images, returns it to the client via the network, and finally, the client decodes and outputs the video image.

[0035] It should be noted that, in this embodiment, the execution entity of the display control method can be a terminal device or a server. The terminal device can be a local terminal device or a client device in the aforementioned cloud service system. This embodiment does not limit the type of execution entity.

[0036] The technical solution of this application will be described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments described below are used to explain the technical solution of this application and are not intended to limit actual use.

[0037] Before detailing the implementation methods of this application, the relevant concepts of this application will be further introduced first.

[0038] Image features refer to specific attributes or patterns in an image or video that can be recognized and processed by computer vision algorithms. These features describe the visual characteristics of objects, scenes, people, etc., and are the foundation for computers to understand and analyze visual data. Different image features are suitable for different types of tasks, such as object detection, classification, tracking, and matching. Visual features can include color features, texture features, shape features, local features, global features, and motion features, etc.

[0039] Transition_x is used to represent the translation parameter in the x-direction of the lens;

[0040] Transition_y is used to represent the translation parameter of the lens in the y-direction;

[0041] Transition_z is used to represent the translation parameter in the z-direction of the lens;

[0042] Pitch controls the vertical angle of the camera, determining how much the camera looks up or down.

[0043] Yaw is used to control the horizontal angle of the lens, determining the degree to which the lens looks to the left or right.

[0044] Roll is used to control the rotation of the lens, which determines whether the image remains level.

[0045] FOV (Field of View) is used to indicate the field of view of a lens, and is usually expressed in degrees.

[0046] Before detailing the implementation methods of this application, the relevant technologies of this application will be further introduced first.

[0047] In related technologies, virtual reality technology typically utilizes virtual cameras set up within a virtual scene to capture that scene.

[0048] Currently, the virtual scene display interface typically shows virtual objects and at least one corresponding camera movement control control. Users then control the virtual camera using this control. For example, users adjust the camera by sliding a slider. In other words, the user performs the camera movement to film the virtual scene. However, this requires a high level of camera movement skill and a certain level of photography knowledge to capture videos with camera movements that match the rhythm of the storyline.

[0049] To address the problems existing in related technologies, embodiments of this application provide a display control method, such as... Figure 1 As shown, Figure 1 This is an example flowchart of a display control method provided in an embodiment of this application. It should be noted that the steps shown may be executed in a logical order different from that shown in the flowchart. The method may include the following steps S101 to S105.

[0050] Step S101: In response to the shooting instruction for the first virtual character in the virtual scene, obtain a reference video, which is obtained by shooting the first character.

[0051] Virtual scenes refer to three-dimensional environments created using computer graphics technology, which users can interact with through various devices. These scenes are widely used in video games, VR (Virtual Reality), AR (Augmented Reality), simulation training, architectural design visualization, and film special effects production, among other fields.

[0052] In one optional embodiment, the virtual scene can be a virtual scene provided in a game, and the first virtual character is a virtual character in the game. The reference video can be a segment from a movie or TV series, or a segment from a game video, etc. This application uses a game scene as an example for illustration.

[0053] In one alternative embodiment, in response to a shooting instruction for a first virtual character in a virtual scene, the user may trigger an import control in the game interface to import a reference video into an application implementing the display control method, so that the first virtual character can be shot based on the lens parameters corresponding to the reference video.

[0054] It should be noted that the primary character can be a virtual character in a game, a real-life character in a film or television drama, or a virtual character in anime; this application does not impose any restrictions on this.

[0055] In an optional embodiment, when the reference video includes multiple first candidate first person subjects, it is necessary to determine the first person subject.

[0056] In one optional embodiment, target detection is performed on the reference video to obtain multiple first candidate first person subjects in the reference video;

[0057] Based on the area and position of each first candidate first person subject in each video frame, a second candidate first person subject is determined for each video frame. This second candidate first person subject can be the first candidate first person subject with the largest area in the video frame and / or the one closest to the center of the video frame. The second candidate first person subject with the largest proportion in the reference video is determined as the first person subject. Of course, other methods can also be used to determine the first person subject, and this application does not limit this.

[0058] In one optional embodiment, target detection is performed on the reference video to obtain multiple first candidate first person subjects in the reference video, including: performing face detection on the reference video to obtain candidate faces; performing human body detection on the reference video to obtain candidate bodies; and matching the candidate faces and candidate bodies using a nearest neighbor matching method to obtain the first candidate first person subjects.

[0059] Considering that video frames often contain multiple objects, extracting features from only the main subject within a frame improves processing efficiency and accuracy, maximizing the use of limited computing resources while maintaining performance. By focusing solely on the first person, the amount of data to be processed is significantly reduced, thus lowering computational complexity. Unnecessary background information processing is also reduced, allowing the algorithm to complete feature extraction and subsequent analysis tasks more quickly. Focusing on the first person means the algorithm can more accurately capture object features, avoiding interference from irrelevant backgrounds. In complex and changing environments, such as crowded scenes or multi-object interactions, feature extraction from only the first person makes the algorithm more stable and less susceptible to changes in the surrounding environment. When the first person is partially occluded, if the target area is pre-determined, useful features can still be extracted from the visible portion without needing to re-search the entire frame. Only feature information related to the first person needs to be stored, thus greatly reducing the required storage space.

[0060] It should be noted that the primary subject is the person bound to the virtual camera via the target number parameter. In other words, this reference video was obtained by filming the primary subject using this virtual camera.

[0061] Step S102: Perform scene segmentation on the reference video to obtain multiple scene clips.

[0062] Considering that the lens parameters vary significantly across different scenarios, this application embodiment performs segmentation processing on the reference video to ensure the accuracy of the subsequently acquired target lens parameters.

[0063] It should be noted that segmenting a reference video typically refers to dividing the video content into several independent shots or scenes. This process can be done manually in video editing software or with the assistance of automated tools.

[0064] In one alternative embodiment, a video scene detection tool such as PySceneDetect is used to perform scene segmentation on the reference video.

[0065] It's important to explain the principles behind PySceneDetect's video scene detection tool: flicker detection and threshold detection. Flicker detection compares the histogram differences between adjacent video frames; when the difference exceeds a set threshold, a scene change is considered to have occurred. Threshold detection is based on pixel value changes between adjacent frames; when the difference exceeds a set threshold, a scene change is also considered to have occurred. This tool combines these two detection methods to segment scenes.

[0066] In one optional embodiment, the reference video is split into segments to obtain segment 1, segment 2, ..., segment m, each segment having an independent shot or scene.

[0067] In this step, segmenting the input video into multiple clips is a fundamental step in achieving high-quality video processing. This process provides the necessary material for subsequent video processing, allowing for more precise control over camera movements. Segmentation breaks the video down into smaller, independently meaningful segments or scenes, enabling more accurate control of the story's rhythm during later processing. This allows for the selection of the most appropriate editing points based on the content of each clip, ensuring a smooth and rhythmic narrative.

[0068] Step S103: Extract target image features from the first person in the video frame of the storyboard segment.

[0069] In one optional embodiment, for a storyboard segment, a target video frame is determined every N video frames, where N is a positive integer greater than 1; target image features are extracted for the first person in the target video frame.

[0070] In one optional example, for storyboard segment A, which has 30 video frames, the target video frames are determined every 3 frames. Therefore, the target video frames are video frames 1, 5, 9, 13, 17, 21, 25, and 29. Feature extraction is performed on video frames 1, 5, 9, 13, 17, 21, 25, and 29 to obtain the target image features.

[0071] In one optional embodiment, the reference video is segmented to obtain multiple segmented clips. Subsequently, target image features can be extracted from the first person in each video frame of each segmented clip to obtain the corresponding target shot parameters. This avoids situations where the accuracy of the determined target shot parameters is low due to significant changes in shot parameters across scenes.

[0072] In one alternative example, the reference video includes video frames 1, 2, 3, 4, 5, 6, 7, and 8. Storyboard segment 1 includes video frames 1, 2, 3, and 4; storyboard segment 2 includes video frames 5, 6, and 7; and storyboard segment 3 includes video frames 8 and 9.

[0073] If no scene segmentation is performed beforehand, the target video frame is determined every 3 video frames. The target image features are then extracted for the first person in the target video frame. Therefore, the target image features need to be extracted for the first person in video frames 1, 5, and 9.

[0074] When determining the target shot parameters for non-target video frames during interpolation, the target shot parameters for video frames 2, 3, and 4 are determined based on the target shot parameters of video frames 1 and 5. Clearly, video frames 1 and 5 belong to different scenes, leading to lower accuracy in the determined target shot parameters. Similarly, video frames 5 and 9, also belonging to different scenes, will also result in lower accuracy in the determined target shot parameters. Pre-segmenting the reference video into segments can avoid these problems.

[0075] Feature extraction every N frames can significantly reduce computational complexity and required hardware resources; it reduces the amount of data to be processed, thereby speeding up the entire process; in many cases, the changes between consecutive frames are very small and almost repetitive, and feature extraction every N frames can avoid processing too much of this similar information and focus on capturing more meaningful changes.

[0076] In one optional embodiment, extracting target image features for a first person subject in a video frame includes one or more of the following steps:

[0077] Extract facial position features of the first person in the video frame;

[0078] Extract the head pose features of the first person in the video frame;

[0079] Extract the human body position features of the first person in the video frame;

[0080] Extract depth features of the first person in the video frame.

[0081] In one alternative example, the facial position features of the first person subject can be represented as (x1, y1, x2, y2), the head pose features of the first person subject can be represented as H = (r1, r2, r3), the body position features of the first person subject can be represented as (a1, b1, a2, b2), and the depth features of the first person subject can be represented as R = (3, H, W).

[0082] Among them, x1, y1, x2, and y2 in the facial position features are the facial position coordinates;

[0083] The first component of head posture features, r1, represents the angle of vertical tilt, also known as pitch. A positive value indicates that the head is tilted upwards, while a negative value indicates that the head is tilted downwards.

[0084] The second component of the head pose feature, r2, represents the angle of left and right rotation, also known as the yaw angle. A positive value indicates that the head is turning to the right, and a negative value indicates that the head is turning to the left.

[0085] The third component of head posture features, r3, is the angle of left and right tilt, also known as the roll angle. A positive value indicates that the head tilts to the right, and a negative value indicates that the head tilts to the left.

[0086] In the human body position features, a1, b1, a2, and b2 are the human body position coordinates.

[0087] The first component 3 in the depth feature represents the three color channels (RGB);

[0088] The second component H in the depth feature represents the height (number of rows) of the virtual character in the video frame;

[0089] The third component W in the depth feature represents the width (number of columns) of the virtual character in the video frame.

[0090] In one optional embodiment, features of the first person subject are extracted using pre-configured face detection models such as Retinaface and ArcFace to obtain the facial position features and head pose features of the first person subject.

[0091] In one optional embodiment, the first person subject is subjected to feature extraction using a pre-configured human detection model such as YOLOv to obtain the human body position features of the first person subject.

[0092] In one alternative embodiment, the first person subject is subjected to feature extraction using a depth detection model such as DepthAnything to obtain the depth features of the first person subject.

[0093] In one optional embodiment, extracting target image features from the first person subject in a video frame of a storyboard segment includes:

[0094] Using a pre-configured deep learning model, target image features are extracted from the first person in the video frame of the storyboard segment;

[0095] The method includes accelerating the feature extraction process of deep learning models through a pre-configured acceleration model.

[0096] In one alternative example, the TensorRT model is used to accelerate the models described above. This acceleration is achieved through optimization, quantization, and efficient kernel execution, thus speeding up the inference process of the deep learning model. After acceleration, the inference speed can often be increased by 10 to 20 times, depending on the complexity of the model structure, significantly saving computational resources and time.

[0097] Step S104: Based on the target image features extracted from the first person subject in the video frame, determine the target shot parameters corresponding to the video frame from the pre-configured shot parameter template library. The shot parameter template library includes video frame templates, template image features corresponding to the second person subject in the video frame template, and the correspondence between the shot parameters corresponding to the template image features.

[0098] Based on the description of step S103 above, the image features in the embodiments of this application may include one or more of the following features: facial position features of the subject, head posture features, human body position features, and depth features.

[0099] In one optional embodiment, based on the target image features extracted from the first person subject in the video frame, the target shot parameters corresponding to the video frame are determined from a pre-configured shot parameter template library, including:

[0100] Determine the difference data between the target image features of the first person in the video frame and the template image features of the second person in the video frame template in the lens parameter template library; determine the target lens parameters corresponding to the video frame from the lens parameter template library based on the difference data.

[0101] Different image features yield different difference data, and the target lens parameters determined based on different difference data are also different.

[0102] In one optional embodiment, taking facial position features including the face area of ​​the main body, head posture features including head posture data including pitch angle, yaw angle and roll angle, human body position features including human body position coordinates, and depth features including depth data as an example, the method of determining the target lens parameters corresponding to the video frame from a pre-configured lens parameter template library based on the target image features extracted from the first main body in the video frame is described in detail.

[0103] In one optional embodiment, determining the target lens parameters corresponding to the video frame from the lens parameter template library based on the difference data includes: determining the target lens rotation parameters corresponding to the video frame from the lens parameter template library based on the difference data; and / or, determining the target lens translation parameters corresponding to the video frame from the lens parameter template library based on the difference data.

[0104] The target lens translation parameters may include one or more of the following parameters: Transition_x, Transition_y, Transition_z, and FOV; the target lens rotation parameters may include one or more of the following parameters: Pitch, Yaw, and Roll.

[0105] In one optional embodiment, determining the difference data between the target image features of the first person in the video frame and the template image features of the second person in the video frame template in the lens parameter template library includes: determining the difference in posture data between the head posture data of the first person in the video frame and the head posture data of the second person in the video frame template in the lens parameter template library.

[0106] Determine the target lens rotation parameters corresponding to the video frame from the lens parameter template library based on the difference data, including: taking the rotation parameters corresponding to the video frame template with the smallest attitude data difference in the lens parameter template library as the target lens rotation parameters corresponding to the video frame.

[0107] In one optional example, determining the difference between the head pose data of a first person in a video frame and the head pose data of a second person in a video frame template in a lens parameter template library includes:

[0108] The following formula is used to determine the difference between the head pose data of the first subject in the video frame and the head pose data of the second subject in the video frame template in the lens parameter template library:

[0109] Headpose_delta=MSE(H, H[m]);

[0110] Wherein, Headpose_delta is used to represent the difference between the head pose data of the first person in the video frame and the head pose data of the second person in the video frame template in the lens parameter template library, H is used to represent the head pose data of the first person in the video frame, H[m] is used to represent the head pose data of the second person in the m-th video frame template in the lens parameter template library, and MSE represents the mean square error.

[0111] In one optional embodiment, determining the difference data between the target image features of a first person in a video frame and the template image features of a second person in a video frame template in a shot parameter template library includes:

[0112] Determine the difference between the head pose data of the first person in the video frame and the head pose data of the second person in the video frame template in the lens parameter template library.

[0113] Determine the difference between the face area of ​​the first person in the video frame and the face area of ​​the second person in the video frame template in the lens parameter template library;

[0114] Determine the coordinate difference between the human body position coordinates of the first person in the video frame and the human body position coordinates of the second person in the video frame template in the lens parameter template library.

[0115] Determine the depth data difference between the depth data of the first subject in the video frame and the depth data of the second subject in the video frame template in the lens parameter template library.

[0116] It should be noted that if, in the embodiments of this application, it is necessary to determine both the target lens translation parameters and the target lens rotation parameters, and the difference between the head posture data of the first subject in the video frame and the head posture data of the second subject in the video frame template in the lens parameter template library is required when determining the target lens translation parameters and the target lens rotation parameters, then only the step of "determining the difference between the head posture data of the first subject in the video frame and the head posture data of the second subject in the video frame template in the lens parameter template library" needs to be performed once, and there is no need to repeat the execution.

[0117] In one optional example, determining the area difference between the face area of ​​a first person in a video frame and the face area of ​​a second person in a video frame template from a lens parameter template library includes:

[0118] Use the following formula to determine the face area of ​​the first person in the video frame and the face area of ​​the second person in the video frame template in the lens parameter template library;

[0119] area = (y2 - y1) * (x2 - x1);

[0120] Where area represents the face area of ​​the first person in the video frame and the face area of ​​the second person in the video frame template in the lens parameter template library, and (x1, y1) and (x2, y2) represent the position coordinates of the face.

[0121] The following formula is used to determine the difference between the face area of ​​the first person in a video frame and the face area of ​​the second person in a video frame template from the lens parameter template library:

[0122] area_delta=abs(area-area[m]);

[0123] Where area_delta represents the difference between the face area of ​​the first person in the video frame and the face area of ​​the second person in the video frame template in the lens parameter template library, area[m] represents the face area of ​​the second person in the m-th video frame template in the lens parameter template library, area represents the face area of ​​the first person in the video frame, and abs() represents the absolute value.

[0124] In one optional example, determining the coordinate difference between the body position coordinates of a first person in a video frame and the body position coordinates of a second person in a video frame template in a camera parameter template library includes:

[0125] The following formula is used to determine the difference between the coordinates of the first person in the video frame and the coordinates of the second person in the video frame template in the camera parameter template library:

[0126] position_delta=abs((a1-a1[m])+(b1-b1[m])+(a2-a2[m])+(b2-b2[m]));

[0127] Wherein, position_delta is used to represent the coordinate difference between the human body position coordinates of the first person in the video frame and the human body position coordinates of the second person in the video frame template in the lens parameter template library, m is used to represent the m-th video frame template in the lens parameter template library, abs() represents the absolute value, and a1, a2, b2, b2, a1[m], a2[m], a2, b1[m] and b2[m] are used to represent the human body position coordinates.

[0128] In one optional example, determining the depth data difference between the depth data of a first subject in a video frame and the depth data of a second subject in a video frame template in a lens parameter template library includes:

[0129] The following formula is used to determine the depth data difference between the depth data of the first person in the video frame and the depth data of the second person in the video frame template in the lens parameter template library:

[0130] depth_delta=MSE(R, R[m]);

[0131] Where depth_delta represents the difference between the depth data of the first subject in the video frame and the depth data of the second subject in the video frame template in the lens parameter template library, R represents the depth data of the first subject in the video frame, R[m] represents the depth data of the second subject in the video frame template in the lens parameter template library, and MSE represents the mean square error.

[0132] In one optional embodiment, determining the target lens translation parameters corresponding to a video frame from a lens parameter template library based on difference data includes:

[0133] The target video frame template is determined from the lens parameter template library based on at least one of the differences in pose data, area, coordinates, and depth data.

[0134] The translation parameters corresponding to the target video frame template are determined as the target lens translation parameters corresponding to the video frame.

[0135] In one alternative example, when determining the target video frame template from the lens parameter template library based on area difference, coordinate difference, and depth data difference, the target video frame template is determined from the lens parameter template library using the following formula:

[0136] L=alpha1*area_delta+alpha2*position_delta+alpha3*depth_delta;

[0137] Where L represents the sum of the differences corresponding to the video frame templates in the lens parameter template library, alpha1, alpha2, and alpha3 are hyperparameters, area_delta is used to represent the area difference between the face area of ​​the first subject in the video frame and the face area of ​​the second subject in the video frame template in the lens parameter template library, position_delta is used to represent the coordinate difference between the body position coordinates of the first subject in the video frame and the body position coordinates of the second subject in the video frame template in the lens parameter template library, and depth_delta is used to represent the depth data difference between the depth data of the first subject in the video frame and the depth data of the second subject in the video frame template in the lens parameter template library.

[0138] Next, the video frame template with the lowest sum of differences in the lens parameter template library is determined as the target video frame template, and the translation parameters corresponding to the target video frame template are determined as the target lens translation parameters corresponding to the video frame.

[0139] It should be noted that the formula L = alpha1 * area_delta + alpha2 * position_delta + alpha3 *

[0140] depth_delta can be adaptively varied based on different difference data, and this application does not limit this. When determining the target video frame template from the lens parameter template library based on only one type of difference data, the target video frame template can be directly determined from the lens parameter template library based on the difference value corresponding to the difference data, without needing to use the above formula for calculation.

[0141] In an optional embodiment, the method further includes the following steps: for an intermediate video frame located between the first target video frame and the second target video frame, interpolating the target shot parameters corresponding to the first target video frame and the target shot parameters of the second target video frame to obtain the target shot parameters corresponding to the intermediate video frame; the first target video frame and the second target video frame are any two adjacent target video frames in the storyboard segment.

[0142] In one optional example, for any intermediate video frame between two adjacent target video frames, such as video frames 2, 3, and 4 between video frame 1 and video frame 5, or video frames 6, 7, and 8 between video frame 5 and video frame 9, the target shot parameters of any two adjacent target video frames are interpolated to obtain the target shot parameters corresponding to the intermediate video frame.

[0143] In an optional embodiment, the camera parameters matched by the previous video frame are: before_cam_params = [Transition_x1, Transition_y1, Transition_z1, Pitch1, Yaw1, Roll1, fov1];

[0144] The camera parameters matched to the next video frame are:

[0145] after_cam_params=[Transition_x2, Transition_y2, Transition_z2, Pitch2, Yaw2, Roll2, fov2];

[0146] For a video frame with N frames in between, the camera parameters for each video frame can be:

[0147] Cam(i)=ratio*before_cam_params+(1-ratio)*after_cam_params, i=(1,...,N-1);

[0148] Where i represents the i-th video frame, and ratio = 1 - (i / N) represents the scaling factor.

[0149] By processing only a subset of frames, computational resource consumption can be significantly reduced. Since detailed analysis and processing of every single frame is unnecessary, processing speed is increased and hardware requirements are reduced. Interpolation algorithms help ensure smoother transitions between non-critical frames, maintaining consistency and fluency throughout the sequence. Because not all frames are fully saved or transmitted, but rather intermediate frames are generated through interpolation, the amount of data is reduced, thus saving storage space and bandwidth.

[0150] In one alternative embodiment, the pre-configured lens parameter template library includes the following steps:

[0151] Step 1: Set multiple sets of camera parameters, including one or more of the following parameters: Transition_x, Transition_y, Transition_z, FOV, Pitch, Yaw, and Roll.

[0152] FOV (Field of View) is a measure of the angular range that an optical system can observe. The FOV value varies across different applications, such as photography, filmmaking, virtual reality, augmented reality, surveillance cameras, and gaming. In photography and videography, if the lens is a standard lens, the FOV value can be set to 40-55 degrees to simulate the natural field of view of the human eye. If the lens is a wide-angle lens, the FOV value can be set to greater than 60 degrees to capture a wider scene. If the lens is a telephoto lens, the FOV value can be set to less than 30 degrees, suitable for shooting at long distances.

[0153] When setting multiple sets of shot parameters, it's necessary to consider whether the second subject can be included in the frame under those shot parameters. Only video frame templates where the second subject is included in the frame can be stored in the shot parameter template library.

[0154] It should be noted that different FOVs correspond to different ranges for Transition_x, Transition_y, and Transition_z. For example, when the FOV is 30, the range of Transition_x, Transition_y, and Transition_z is (0, 10); when the FOV is 35, the range is (0, 8); and when the FOV is 38, the range is (0, 7). That is, when selecting the FOV, the corresponding Transition_x, Transition_y, and Transition_z must be selected within the above ranges to avoid capturing video frame templates that cannot be stored in the lens parameter template library.

[0155] It should be noted that the numerical values ​​given in the embodiments of this application are exemplary.

[0156] In one alternative embodiment, as shown in Table 1, six sets of lens parameters are listed as examples:

[0157] FOV 30 30 30 30 40 60 Transition_x 0 0 1 4 1 0 Transition_y 0 0 2 5 2 0 Transition_z 2 2 3 6 3 2 Pitch 30 31 40 37 67 45 Yaw 30 31 45 28 68 78 Roll 60 31 49 91 79 89

[0158] Table 1

[0159] Step 2: Set up the lens based on the determined multiple sets of lens parameters, use the lens to shoot, and randomly sample the captured video footage to obtain multiple video frame templates.

[0160] For each set of lens parameters, the camera is used to shoot, and the video footage is randomly sampled to obtain the video frame template corresponding to that set of lens parameters.

[0161] Step 3: Extract template image features from the video frame template.

[0162] The process of extracting template image features from video frame templates in this step is similar to the process of extracting target image features from the first person subject in the video frame of the storyboard segment in step S103, and will not be described in detail here.

[0163] Step 4: Record the correspondence between the video frame template, the template image features corresponding to the video frame template, and the lens parameters corresponding to the template image features, and store this correspondence in the lens parameter template library.

[0164] In an optional embodiment, the lens parameter template library may further store a video frame template, a depth map of the second subject in the video frame template, template image features corresponding to the video frame template, and a correspondence between the lens parameters corresponding to the template image features. The depth map of the second subject in the video frame template is used to obtain the depth data of the second subject in the video frame template.

[0165] Step S105: Take a picture of the first virtual character based on the target camera parameters.

[0166] In one alternative embodiment, filming the first virtual character based on target camera parameters includes:

[0167] Determine the facial expressions and / or actions of the first subject in the reference video; control the first virtual character to present virtual facial expressions based on the facial expressions of the first subject, and / or control the first virtual character to present virtual actions based on the actions of the first subject; while the first virtual character is presenting virtual facial expressions and / or virtual actions, film the first virtual character using a virtual camera based on the target camera parameters.

[0168] It should be noted that facial expressions include at least one or more of the following: sadness, happiness, crying, anger, etc., and actions include one or more of the following: running, jumping, shooting, etc.

[0169] In this embodiment, the method involves acquiring a reference video, extracting target image features from the reference video, and matching suitable lens parameters from a lens parameter template library based on the extracted target image features.

[0170] The first virtual character is filmed based on the target camera parameters. This method allows users to film virtual objects without needing complex camera movement knowledge, resulting in videos with effective camera movement.

[0171] This application also provides a task logic diagram of a display control method, which includes the following steps:

[0172] Step 1: Obtain video frame templates and store them in the lens parameter template library;

[0173] Step 2: Obtain the template image features of the video frame template from the lens parameter template library;

[0174] Step 3: Obtain reference videos;

[0175] Step 4: Perform storyboarding on the reference video to obtain storyboard clips;

[0176] Step 5: Accelerate the model;

[0177] Step 6: Extract target image features from the target video frame every N frames in the storyboard segment using models such as face detection, human detection, and depth estimation;

[0178] Step 7: Match the target image features of the target video frame with the template image features of the video frame templates in the lens parameter template library to obtain the target lens parameters corresponding to the target video frame;

[0179] Step 8: For non-target video frames, perform lens parameter interpolation based on the target lens parameters corresponding to the target video frame to obtain the target lens parameters of the non-target video frames;

[0180] Step 9: Output the target shot parameters of the reference video;

[0181] Step 10: Use the target camera parameters of the reference video to shoot the virtual object and obtain video frames with camera movement effects.

[0182] Corresponding to the display control method provided in the embodiments of this application, the embodiments of this application also provide a display control device 300, such as... Figure 3As shown, the device includes:

[0183] The acquisition module 301 is used to acquire a reference video in response to a shooting instruction for a first virtual character in a virtual scene, wherein the reference video is obtained by shooting the first character.

[0184] The storyboard module 302 is used to perform storyboard processing on the reference video to obtain multiple storyboard segments.

[0185] Extraction module 303 is used to extract target image features from the first person subject in the video frame of the storyboard segment;

[0186] The determining module 304 is used to extract the target image features based on the first person subject in the video frame, and determine the target shot parameters corresponding to the video frame from a pre-configured shot parameter template library. The shot parameter template library includes a video frame template, a template image feature corresponding to the second person subject in the video frame template, and a correspondence between the shot parameters corresponding to the template image feature.

[0187] The shooting module 305 is used to shoot the first virtual character based on the target lens parameters.

[0188] Corresponding to the display control method provided in the embodiments of this application, the embodiments of this application also provide an electronic device for implementing the display control method, such as... Figure 4 As shown, the electronic device includes: a processor 401; and a memory 402 for storing a program for a display control method. After the device is powered on and the program for the display control method is run by the processor, the following steps are performed:

[0189] In response to a shooting instruction for a first virtual character in a virtual scene, a reference video is acquired, wherein the reference video is obtained by shooting the first character as the main subject;

[0190] The reference video is processed into multiple storyboard segments.

[0191] Extract target image features from the first person in the video frames of the storyboard segment;

[0192] Based on the target image features extracted from the first person in the video frame, the target shot parameters corresponding to the video frame are determined from a pre-configured shot parameter template library. The shot parameter template library includes a video frame template, a template image feature corresponding to the second person in the video frame template, and a correspondence between the shot parameters corresponding to the template image feature.

[0193] The first virtual character is photographed based on the target camera parameters.

[0194] Corresponding to the display control method provided in the embodiments of this application, the embodiments of this application also provide a computer-readable storage medium storing a program for the display control method, which is executed by a processor to perform the following steps:

[0195] In response to a shooting instruction for a first virtual character in a virtual scene, a reference video is acquired, wherein the reference video is obtained by shooting the first character as the main subject;

[0196] The reference video is processed into multiple storyboard segments.

[0197] Extract target image features from the first person in the video frames of the storyboard segment;

[0198] Based on the target image features extracted from the first person in the video frame, the target shot parameters corresponding to the video frame are determined from a pre-configured shot parameter template library. The shot parameter template library includes a video frame template, a template image feature corresponding to the second person in the video frame template, and a correspondence between the shot parameters corresponding to the template image feature.

[0199] The first virtual character is photographed based on the target camera parameters.

[0200] It should be noted that for a detailed description of the apparatus, electronic device and computer-readable storage medium provided in the embodiments of this application, please refer to the relevant description of the display control method embodiments provided in the embodiments of this application, which will not be repeated here.

[0201] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

[0202] In a typical configuration, an electronic device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0203] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0204] 1. Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable operations, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0205] 2. Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0206] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

Claims

1. A display control method characterized by comprising: The method comprises: in response to a shooting instruction for a first virtual role in a virtual scene, obtaining a reference video, the reference video being obtained by shooting a first character subject; performing a screenplay processing on the reference video to obtain a plurality of screenplay segments; extracting a target picture feature of the first character subject in a video frame in the screenplay segment, comprising: determining a target video frame every N video frames of the screenplay segment, wherein N is a positive integer greater than 1; extracting a target picture feature of the first character subject in the target video frame; determining a target lens parameter corresponding to the video frame from a preconfigured lens parameter template library based on the target picture feature extracted for the first character subject in the video frame, comprising: determining difference data between the target picture feature of the first character subject in the video frame and a template picture feature of a second character subject in a video frame template in the lens parameter template library; determining the target lens parameter corresponding to the video frame from the lens parameter template library based on the difference data; wherein the lens parameter template library comprises a corresponding relationship of a video frame template, a template picture feature corresponding to a second character subject in the video frame template, and a lens parameter corresponding to the template picture feature; shooting the first virtual role based on the target lens parameter; The method further comprises: for an intermediate video frame located between a first target video frame and a second target video frame, performing an interpolation processing on the target lens parameter corresponding to the first target video frame and the target lens parameter of the second target video frame to obtain a target lens parameter corresponding to the intermediate video frame, the first target video frame and the second target video frame being any two adjacent target video frames in the screenplay segment.

2. The method of claim 1, wherein, The determining of the target lens parameter corresponding to the video frame from the lens parameter template library based on the difference data comprises: determining a target lens rotation parameter corresponding to the video frame from the lens parameter template library based on the difference data; and / or, determining a target lens translation parameter corresponding to the video frame from the lens parameter template library based on the difference data.

3. The method of claim 2, wherein, The determining of the difference data between the target picture feature of the first character subject in the video frame and the template picture feature of the second character subject in the video frame template in the lens parameter template library comprises: determining a pose data difference value of head posture data of the first character subject in the video frame and head posture data of the second character subject in the video frame template in the lens parameter template library; The determining of the target lens rotation parameter corresponding to the video frame from the lens parameter template library based on the difference data comprises: taking a rotation parameter corresponding to the video frame template with the smallest pose data difference value in the lens parameter template library as the target lens rotation parameter corresponding to the video frame.

4. The method of claim 2, wherein, The determining of the difference data between the target picture feature of the first character subject in the video frame and the template picture feature of the second character subject in the video frame template in the lens parameter template library comprises: determining a pose data difference between head pose data of the first character subject in the video frame and head pose data of a second character subject in the video frame template in the lens parameter template library; determining an area difference between a face area of the first character subject in the video frame and a face area of a second character subject in the video frame template in the lens parameter template library; determining a coordinate difference between a body position coordinate of the first character subject in the video frame and a body position coordinate of a second character subject in the video frame template in the lens parameter template library; determining a depth data difference between depth data of the first character subject in the video frame and depth data of a second character subject in the video frame template in the lens parameter template library.

5. The method of claim 4, wherein, The determining of the target lens translation parameter corresponding to the video frame from the lens parameter template library based on the difference data comprises: determining a target video frame template from the lens parameter template library according to at least one of the pose data difference, the area difference, the coordinate difference and the depth data difference; determining a translation parameter corresponding to the target video frame template as the target lens translation parameter corresponding to the video frame.

6. The method of claim 4, wherein, The extracting of the target picture feature for the first character subject in the video frame in the split shot segment comprises one or more of the following steps: extracting a face position feature of the first character subject for the first character subject in the video frame; extracting a head pose feature of the first character subject for the first character subject in the video frame; extracting a body position feature of the first character subject for the first character subject in the video frame; extracting a depth feature of the first character subject for the first character subject in the video frame.

7. The method of claim 1, wherein, The extracting of the target picture feature for the first character subject in the video frame in the split shot segment comprises: extracting a target picture feature for the first character subject in the video frame in the split shot segment through a pre-configured deep learning model; The method comprises: accelerating the feature extraction process of the deep learning model through a pre-configured acceleration model.

8. The method of claim 1, wherein, The shooting of the first virtual character based on the target lens parameter comprises: determining an expression and / or action of the first character subject in the reference video; controlling the first virtual character to present a virtual expression according to the expression of the first character subject, and / or controlling the first virtual character to present a virtual action according to the action of the first character subject; shooting the first virtual character through a virtual lens based on the target lens parameter in the process of the first virtual character presenting a virtual expression and / or a virtual action.

9. The method of claim 2, wherein, The target lens translation parameter comprises one or more of the following parameters: transition Transition_x, Transition_y, Transition_z and field of view angle FOV, and the target lens rotation parameter comprises one or more of the following parameters: pitch Pitch, yaw Yaw and roll Roll.

10. A display control device characterized by comprising: The device comprises: An acquisition module configured to acquire a reference video in response to a shooting instruction for a first virtual character in a virtual scene, the reference video being obtained by shooting a first person subject; A script module configured to perform script processing on the reference video to obtain a plurality of script segments; An extraction module configured to extract target picture features for the first person subject in video frames in the script segments, including: determining target video frames every N video frames for the script segments, where N is a positive integer greater than 1; and extracting target picture features for the first person subject in the target video frames; A determination module configured to determine target lens parameters corresponding to the video frames based on the target picture features extracted for the first person subject in the video frames, including: determining difference data between the target picture features of the first person subject in the video frames and template picture features of a second person subject in a video frame template in a lens parameter template library; and determining the target lens parameters corresponding to the video frames from the lens parameter template library based on the difference data; wherein the lens parameter template library includes a video frame template, template picture features corresponding to a second person subject in the video frame template, and a corresponding relationship between the template picture features and lens parameters corresponding to the template picture features; A shooting module configured to shoot the first virtual character based on the target lens parameters; The extraction module is further configured to perform interpolation processing on the target lens parameters corresponding to the first target video frame and the target lens parameters of the second target video frame to obtain target lens parameters corresponding to intermediate video frames between the first target video frame and the second target video frame, the first target video frame and the second target video frame being any two adjacent target video frames in the script segments.

11. An electronic device, comprising: Comprise: A processor; And A memory for storing a data processing program, after the electronic device is powered on and the program is run by the processor, the method as claimed in any one of claims 1-9 is executed.

12. A computer-readable storage medium, characterized in that, A data processing program is stored, the program is run by the processor, and the method as claimed in any one of claims 1-9 is executed.

Images