Display control method and device, electronic equipment and readable storage medium

Abstract

The invention discloses a display control method and device and electronic equipment, and the method comprises the steps: obtaining a reference video in response to a shooting instruction for a first virtual character in a virtual scene, and enabling the reference video to be obtained through the shooting of a first character main body; performing lens splitting processing on the reference video to obtain a plurality of lens segments; target picture features are extracted for a first person subject in the video frame in the sub-lens segment; determining a target shot parameter corresponding to the video frame from a pre-configured shot parameter template library based on a target picture feature extracted for a first person subject in the video frame; and shooting the first virtual character based on the target lens parameter. Through the method, a user can shoot the virtual object without having complicated mirror movement knowledge, and a video with a mirror movement effect is obtained.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a display control method, device, electronic device, and computer-readable storage medium. Background Art

[0002] In virtual reality technology, a virtual lens set in a virtual scene is usually used to shoot the virtual scene.

[0003] At present, a virtual object and at least one camera movement control control corresponding to the virtual object are usually displayed in a display interface of a virtual scene, and the user controls the virtual lens based on the at least one camera movement control control corresponding to the virtual object. For example, the user adjusts the lens by sliding a slider. However, this has high requirements on the user's camera movement level, and the user needs to have certain photography knowledge to shoot a video with camera movement effects that conforms to the rhythm changes of the plot. Therefore, there is an urgent need for a shooting method that can shoot a video with camera movement effects without the user having complex camera movement knowledge. Summary of the invention

[0004] In view of this, the present application provides a display control method, device, electronic device and computer-readable storage medium, which enable users to shoot videos with camera movement effects without requiring users to have complex camera movement knowledge.

[0005] In a first aspect, an embodiment of the present application provides a display control method, the method comprising:

[0006] In response to a shooting instruction for a first virtual character in a virtual scene, a reference video is acquired, where the reference video is shot for the first character subject;

[0007] Performing storyboard processing on the reference video to obtain a plurality of storyboard segments;

[0008] Extracting target screen features for the first character subject in the video frame in the storyboard segment;

[0009] Based on the target picture features extracted from the first human subject in the video frame, determine the target lens parameters corresponding to the video frame from a preconfigured lens parameter template library, wherein the lens parameter template library includes a video frame template, a template picture feature corresponding to the second human subject in the video frame template, and a lens parameter corresponding to the template picture feature;

[0010] The first virtual character is photographed based on the target lens parameters.

[0011] In a second aspect, an embodiment of the present application provides a display control device, the device comprising:

[0012] An acquisition module, configured to acquire a reference video in response to a shooting instruction for a first virtual character in a virtual scene, wherein the reference video is obtained by shooting a first character subject;

[0013] A storyboard module, used for storyboarding the reference video to obtain a plurality of storyboard segments;

[0014] An extraction module, configured to extract target screen features from the first character subject in the video frame in the storyboard segment;

[0015] A determination module, configured to determine target lens parameters corresponding to the video frame from a preconfigured lens parameter template library based on extracting the target screen features from the first human subject in the video frame, wherein the lens parameter template library includes a video frame template, a template screen feature corresponding to the second human subject in the video frame template, and a lens parameter corresponding to the template screen feature;

[0016] A shooting module is used to shoot the first virtual character based on the target lens parameters.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0018] Processor; and

[0019] The memory is used to store a data processing program. After the electronic device is powered on and the program is run by the processor, the method of the first aspect is executed.

[0020] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a data processing program, which is executed by a processor to perform the method of the first aspect.

[0021] The present application provides a display control method, which includes: in response to a shooting instruction for a first virtual character in a virtual scene, obtaining a reference video, the reference video being obtained by shooting a first character subject; performing storyboard processing on the reference video to obtain multiple storyboard segments; extracting target screen features for the first character subject in a video frame in the storyboard segment; based on the target screen features extracted for the first character subject in the video frame, determining the target lens parameters corresponding to the video frame from a preconfigured lens parameter template library; and shooting the first virtual character based on the target lens parameters. Through this method, the user can shoot a virtual object without having complex camera movement knowledge, and obtain a video with camera movement effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0023] Figure 1 A first flow chart of a display control method provided in an embodiment of the present application;

[0024] Figure 2 A second schematic diagram of the display control method provided in an embodiment of the present application;

[0025] Figure 3 A schematic diagram of the structure of a display control device provided in an embodiment of the present application;

[0026] Figure 4 A structural block diagram of an electronic device for implementing a display control method is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0027] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0028] It should be noted that the terms "first", "second", "third", etc. in the claims, description and drawings of the present application are used to distinguish similar objects and are not used to describe a specific order or sequence. The data used in this way are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including", "having" and their variants are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] It should be understood that in the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" is merely a way to describe the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "Including A, B and / or C" means including any one, any two, or any three of A, B, and C.

[0030] It should be understood that in the embodiments of the present application, "B corresponding to A", "B corresponding to A", "A corresponds to B", or "B corresponds to A" means that B is associated with A, and B can be determined according to A. Determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0031] Based on the problems existing in the above-mentioned related technologies, the embodiments of the present application provide a display control method, device, electronic device and computer-readable storage medium.

[0032] The display control method provided in the embodiment of the present application can be executed by an electronic device, which can be a terminal or a server. The terminal can be a terminal device such as a smart phone, a tablet computer, or a laptop computer. The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. It is understandable that the present application does not specifically limit the execution subject of the display control method.

[0033] In an optional embodiment, when the display control method is run on a terminal device, the terminal device stores an application program that implements the display control method. The terminal device interacts with the user through a graphical user interface. The terminal device may provide the graphical user interface to the user in a variety of ways, for example, it may be rendered and displayed on a display screen of the terminal device, or the graphical user interface may be presented through holographic projection.

[0034] In an optional embodiment, when the display control method is run on a server, the method can be implemented and executed based on a cloud service system. A cloud service system refers to a service method based on cloud computing. A cloud service system includes a server and a client device. The operating body of the application that implements the display control method and the display screen presentation body are separated, and the storage and operation of the display control method are completed on the server. The video screen presentation is completed on the client, and the client is mainly used for receiving, sending and presenting video data. For example, the client can be a display device with a data transmission function close to the user side, such as a mobile terminal, a television, a computer, a handheld computer, a personal digital assistant, a head-mounted display device (head display device), etc., but the electronic device that implements the display control method is a server in the cloud. For example, the user operates the client to send an instruction to the server, and the server controls the operation of the display control method according to the instruction, encodes and compresses the video screen and other data, returns it to the client through the network, and finally, decodes and outputs the video screen through the client.

[0035] It should be noted that in the embodiment of the present application, the execution subject of the display control method can be a terminal device or a server, wherein the terminal device can be a local terminal device or a client device in the aforementioned cloud service system. The embodiment of the present application does not limit the type of the execution subject.

[0036] The technical solution of the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments described below are used to explain the technical solution of the present application and are not intended to be used as a limitation for actual use.

[0037] Before describing the implementation methods of the present application in detail, the relevant concepts of the present application are first further introduced.

[0038] Image features: refers to specific attributes or patterns in images or videos that can be recognized and processed by computer vision algorithms. These features are used to describe the visual characteristics of objects, scenes, people, etc., and are the basis for computers to understand and analyze visual data. Different image features are suitable for different types of tasks, such as object detection, classification, tracking, matching, etc. Visual features can be color features, texture features, shape features, local features, global features, and motion features.

[0039] Transition_x is used to represent the translation parameter of the lens in the x direction;

[0040] Transition_y is used to represent the translation parameter of the lens in the y direction;

[0041] Transition_z is used to represent the translation parameter of the lens in the z direction;

[0042] Pitch is used to control the vertical viewing angle of the camera, which determines the degree to which the camera looks up or down.

[0043] Yaw is used to control the horizontal viewing angle of the camera, which determines the degree to which the camera looks left or right.

[0044] Roll is used to control the rotation of the lens and determines whether the lens image remains horizontal;

[0045] FOV (Field of View) is used to indicate the field of view of a lens, usually expressed in degrees.

[0046] Before describing the implementation methods of the present application in detail, the related technologies of the present application are first further introduced.

[0047] In the related art, in virtual reality technology, a virtual lens set in a virtual scene is usually used to shoot the virtual scene.

[0048] At present, a virtual object and at least one camera movement control control corresponding to the virtual object are usually displayed in a display interface of a virtual scene, and the user controls the virtual lens based on the at least one camera movement control control corresponding to the virtual object. For example, the user adjusts the lens by sliding a slider. That is, the user moves the lens to shoot the virtual scene. However, this has high requirements on the user's camera movement level, and the user needs to have certain photography knowledge to shoot a video with lens changes that conform to the rhythm of the plot.

[0049] In order to solve the problems existing in the related art, the present application embodiment provides a display control method, such as Figure 1 As shown, Figure 1 This is a flow chart of an example of a display control method provided in an embodiment of the present application. It should be noted that the steps shown may be performed in a logical order different from that shown in the flow chart of the method. The method may include the following steps S101 to S105.

[0050] Step S101: in response to a shooting instruction for a first virtual character in a virtual scene, a reference video is obtained, where the reference video is shot for a first character subject.

[0051] Virtual scenes refer to three-dimensional environments created by computer graphics technology, which users can interact with through various devices. These scenes are widely used in video games, VR (Virtual Reality), AR (Augmented Reality), simulation training, architectural design visualization, and film special effects production.

[0052] In an optional embodiment, the virtual scene may be a virtual scene provided in a game, and the first virtual character may be a virtual character in the game. The reference video may be a portion of a film or TV series or a portion of a game video, etc. This application is described by taking the virtual scene as a game scene as an example.

[0053] In an optional embodiment, in response to a shooting instruction for the first virtual character in the virtual scene, the user may trigger an import control in the game interface to import a reference video into an application that implements the display control method, so that the first virtual character can be subsequently shot based on the lens parameters corresponding to the reference video.

[0054] It should be noted that the first character entity can be a virtual character in a game, a real person character in a film or TV series, and of course, a virtual character in an animation, and this application does not limit this.

[0055] In an optional embodiment, when a plurality of first candidate first human subjects are included in the reference video, the first human subject needs to be determined.

[0056] In an optional embodiment, target detection is performed on a reference video to obtain a plurality of first candidate first human subjects in the reference video;

[0057] Based on the area and position of each first candidate first person subject in each video frame, the corresponding second candidate first person subject in each video frame is determined, and the second candidate first person subject can be the first candidate first person subject with the largest area in the video frame and / or the closest to the center of the video frame; the second candidate first person subject with the largest proportion in the reference video is determined as the first person subject. Of course, other methods can also be used to determine the first person subject, which is not limited in this application.

[0058] In an optional embodiment, target detection is performed on a reference video to obtain multiple first candidate first human subjects in the reference video, including: performing face detection on the reference video to obtain candidate faces; performing human body detection on the reference video to obtain candidate human bodies; matching the candidate faces and candidate human bodies through a nearest neighbor matching method to obtain a first candidate first human subject.

[0059] Considering that there may be multiple objects in a video frame, only extracting features from the subject in the video frame can improve processing efficiency and accuracy, and maximize the use of limited computing resources while ensuring performance. By focusing only on the first subject, the amount of data that needs to be processed can be greatly reduced, thereby reducing the computational complexity; unnecessary background information processing is reduced, allowing the algorithm to complete feature extraction and subsequent analysis tasks faster. Focusing on the first subject means that the algorithm can capture the features of the object more accurately and avoid being disturbed by irrelevant background; in complex and changing environments, such as crowded scenes or multi-target interactions, only extracting features from the first subject can make the algorithm more stable and less susceptible to changes in the surrounding environment; when the first subject is partially obscured, if the target area is determined in advance, useful features can still be extracted from the visible part without re-searching the entire picture. Only feature information related to the first subject needs to be saved, so the required storage space can be greatly reduced.

[0060] It should be noted that the first person subject is a person subject bound to the virtual lens through the target number parameter of the virtual lens. That is, the reference video is obtained by shooting the first person subject using the virtual lens.

[0061] Step S102: performing storyboard processing on the reference video to obtain a plurality of storyboard segments.

[0062] Considering that the lens parameters corresponding to different scenes vary greatly, in order to ensure the accuracy of the target lens parameters subsequently obtained, the embodiment of the present application performs storyboard processing on the reference video so as to obtain the corresponding target lens parameters in units of storyboard segments subsequently.

[0063] It should be noted that storyboarding a reference video usually refers to dividing it into a number of shots or scenes with independent meanings based on the video content. This process can be done manually in video editing software or with the assistance of automated tools.

[0064] In an optional embodiment, a video scene detection tool such as PySceneDetect is used to perform storyboard processing on the reference video.

[0065] It should be noted that the principles of the PySceneDetect video scene detection tool are: content flicker detection and threshold detection. Content flicker detection compares the histogram differences of adjacent video frames. When the difference exceeds the set threshold, it is considered that a scene switch has occurred. Threshold detection is based on the change in pixel values ​​of adjacent frames. When the difference exceeds the set threshold, it is considered that a scene switch has occurred. This tool combines these two detection methods to segment the storyboard.

[0066] In an optional embodiment, the reference video is storyboarded to obtain storyboard segment 1, storyboard segment 2, ..., storyboard segment m, each storyboard segment having an independent shot or scene.

[0067] In this step, storyboarding the input video to obtain multiple storyboard segments is a basic step to achieve high-quality video processing. This process provides the necessary material preparation for subsequent video processing, allowing the camera movement transition to be controlled at a more detailed level. Through storyboarding, the video can be broken down into small sections or scenes with independent meanings, which enables the rhythm of the story to be more accurately controlled during subsequent video processing, and then the most appropriate editing points can be selected according to the content of each storyboard segment, thereby ensuring a smooth and rhythmic narrative.

[0068] Step S103: extracting target screen features for the first person subject in the video frame in the storyboard segment.

[0069] In an optional embodiment, for the storyboard segment, a target video frame is determined every N video frames, where N is a positive integer greater than 1; and target picture features are extracted for the first human subject in the target video frame.

[0070] In an optional example, for storyboard segment A, the storyboard segment A has 30 video frames, and the target video frame is determined every 3 frames, and the target video frames are video frame 1, video frame 5, video frame 9, video frame 13, video frame 17, video frame 21, video frame 25, and video frame 29. Feature extraction is performed on video frame 1, video frame 5, video frame 9, video frame 13, video frame 17, video frame 21, video frame 25, and video frame 29 to obtain target screen features.

[0071] In an optional embodiment, the reference video is storyboarded to obtain a plurality of storyboard segments, and then the target screen features can be extracted from the first person subject in the video frame of each storyboard segment in units of storyboard segments, thereby obtaining the corresponding target lens parameters. In this way, it is avoided that the lens parameters vary greatly due to cross-scene, thereby resulting in a low accuracy of the determined target lens parameters.

[0072] In an optional example, the reference video includes video frame 1, video frame 2, video frame 3, video frame 4, video frame 5, video frame 6, video frame 7 and video frame 8. Storyboard segment 1 includes video frame 1, video frame 2, video frame 3 and video frame 4, storyboard segment 2 includes video frame 5, video frame 6, video frame 7, and storyboard segment 3 includes video frame 8 and video frame 9.

[0073] If storyboard processing is not performed in advance, the target video frame is determined every 3 video frames, and the target screen features are extracted for the first character subject in the target video frame: the target screen features need to be extracted for the first character subject in video frame 1, video frame 5 and video frame 9.

[0074] When determining the target lens parameters of the non-target video frame later, when performing interpolation processing, the target lens parameters of video frame 2, video frame 3 and video frame 4 will be determined based on the target lens parameters of video frame 1 and video frame 5. Obviously, video frame 1 and video frame 5 belong to video frames in different scenes, so the accuracy of the determined target lens parameters is low. Similarly, video frame 5 and video frame 9 also belong to video frames in different scenes, and there will also be a situation where the accuracy of the determined target lens parameters is low. However, pre-screening the reference video can avoid the above problems.

[0075] Extracting features every N frames can significantly reduce computational complexity and required hardware resources; it reduces the amount of data that needs to be processed, thereby speeding up the entire processing process; in many cases, the changes between consecutive frames are very small and almost repetitive, and extracting features every N frames can avoid excessive processing of this similar information and focus on capturing more meaningful changes.

[0076] In an optional embodiment, extracting target picture features for a first person subject in a video frame includes one or more of the following steps:

[0077] Extracting facial position features of a first person subject in a video frame;

[0078] Extracting head posture features of a first person subject in a video frame;

[0079] Extracting a human body position feature of a first human subject in a video frame;

[0080] A depth feature of a first human subject is extracted for a first human subject in a video frame.

[0081] In an optional example, the facial position feature of the first character subject can be expressed as (x1, y1, x2, y2), the head posture feature of the first character subject can be expressed as H = (r1, r2, r3), the body position feature of the first character subject can be expressed as (a1, b1, a2, b2), and the depth feature of the first character subject can be expressed as R = (3, H, W).

[0082] Among them, x1, y1, x2, y2 in the facial position feature are the face position coordinates;

[0083] The first component r1 in the head posture feature: the angle of up and down tilt, also known as the pitch angle. A positive value indicates that the head is raised up, and a negative value indicates that the head is lowered;

[0084] The second component r2 in the head posture feature: the angle of left and right rotation, also known as the yaw angle. A positive value indicates that the head is turned to the right, and a negative value indicates that the head is turned to the left;

[0085] The third component r3 in the head posture feature: the left-right tilt angle, also known as the roll angle. A positive value indicates that the head is tilted to the right, and a negative value indicates that the head is tilted to the left;

[0086] a1, b1, a2, b2 in the human body position feature are the human body position coordinates;

[0087] The first component 3 in the depth feature represents three color channels (RGB);

[0088] The second component H in the depth feature represents the height (number of lines) of the virtual character in the video frame;

[0089] The third component W in the depth feature represents the width (number of columns) of the virtual character in the video frame.

[0090] In an optional embodiment, features of the first human subject are extracted using preconfigured face detection models such as Retinaface and ArcFace to obtain facial position features and head posture features of the first human subject.

[0091] In an optional embodiment, a preconfigured human body detection model such as yolov is used to extract features of the first human body to obtain human body position features of the first human body.

[0092] In an optional embodiment, feature extraction is performed on the first human subject using a depth detection model such as DepthAnything to obtain depth features of the first human subject.

[0093] In an optional embodiment, extracting target screen features for a first person subject in a video frame in a storyboard segment includes:

[0094] Extract target screen features for the first person subject in the video frame in the storyboard clip through a preconfigured deep learning model;

[0095] The method includes: accelerating the feature extraction process of the deep learning model through a preconfigured acceleration model.

[0096] In an optional example, the TensorRT model is used to accelerate and optimize the above-mentioned model, and the reasoning process of the deep learning model is accelerated through optimization, quantization and efficient kernel execution. After the model is accelerated, the reasoning speed of the model can often be increased by 10 to 20 times, depending on the complexity of the model structure, which can greatly save computing resources and time.

[0097] Step S104: Based on the target picture features extracted for the first human subject in the video frame, determine the target lens parameters corresponding to the video frame from a preconfigured lens parameter template library, wherein the lens parameter template library includes the video frame template, the template picture features corresponding to the second human subject in the video frame template, and the correspondence between the lens parameters corresponding to the template picture features.

[0098] Based on the description of step S103 above, it can be known that the picture features in the embodiment of the present application may include one or more of the following features: facial position features, head posture features, body position features, and depth features of the subject.

[0099] In an optional embodiment, based on the target picture features extracted from the first human subject in the video frame, determining the target lens parameters corresponding to the video frame from a preconfigured lens parameter template library includes:

[0100] Determine difference data between target picture features of a first human subject in a video frame and template picture features of a second human subject in a video frame template in a lens parameter template library; and determine target lens parameters corresponding to the video frame from the lens parameter template library based on the difference data.

[0101] Different difference data can be obtained based on different picture features, and different target lens parameters are determined based on different difference data.

[0102] In an optional embodiment, taking the facial position feature including the facial area of ​​the human subject, the head posture feature including the head posture data including the pitch angle, the yaw angle and the roll angle, the body position feature including the body position coordinates, and the depth feature including the depth data as an example, a detailed description is given of determining the target lens parameters corresponding to the video frame from the preconfigured lens parameter template library based on the target picture features extracted for the first human subject in the video frame.

[0103] In an optional embodiment, determining target lens parameters corresponding to the video frame from a lens parameter template library based on the difference data includes: determining target lens rotation parameters corresponding to the video frame from the lens parameter template library based on the difference data; and / or determining target lens translation parameters corresponding to the video frame from the lens parameter template library based on the difference data.

[0104] The target lens translation parameters may include one or more of the following parameters: Transition_x, Transition_y, Transition_z, and FOV; the target lens rotation parameters may include one or more of the following parameters: Pitch, Yaw, and Roll.

[0105] In an optional embodiment, determining difference data between target picture features of a first human subject in a video frame and template picture features of a second human subject in a video frame template in a lens parameter template library includes: determining a posture data difference between head posture data of the first human subject in the video frame and head posture data of the second human subject in the video frame template in the lens parameter template library;

[0106] Determining target lens rotation parameters corresponding to the video frame from a lens parameter template library based on the difference data includes: taking the rotation parameters corresponding to the video frame template with the smallest posture data difference in the lens parameter template library as the target lens rotation parameters corresponding to the video frame.

[0107] In an optional example, determining a posture data difference between head posture data of a first human subject in a video frame and head posture data of a second human subject in a video frame template in a lens parameter template library includes:

[0108] The posture data difference between the head posture data of the first person subject in the video frame and the head posture data of the second person subject in the video frame template in the lens parameter template library is determined using the following formula:

[0109] Headpose_delta=MSE(H, H[m]);

[0110] Among them, Headpose_delta is used to represent the posture data difference between the head posture data of the first human subject in the video frame and the head posture data of the second human subject in the video frame template in the lens parameter template library, H is used to represent the head posture data of the first human subject in the video frame, H[m] is used to represent the head posture data of the second human subject in the mth video frame template in the lens parameter template library, and MSE represents the mean square error.

[0111] In an optional embodiment, determining difference data between target picture features of a first human subject in a video frame and template picture features of a second human subject in a video frame template in a lens parameter template library includes:

[0112] Determine a posture data difference between head posture data of a first human subject in a video frame and head posture data of a second human subject in a video frame template in a lens parameter template library;

[0113] Determine an area difference between a face area of ​​a first human subject in a video frame and a face area of ​​a second human subject in a video frame template in a lens parameter template library;

[0114] Determine the coordinate difference between the human body position coordinates of the first human subject in the video frame and the human body position coordinates of the second human subject in the video frame template in the lens parameter template library;

[0115] Determine a depth data difference between depth data of a first human subject in a video frame and depth data of a second human subject in a video frame template in a lens parameter template library.

[0116] It should be noted that, if in the embodiment of the present application, it is necessary to determine both the target lens translation parameters and the target lens rotation parameters, and the posture data difference between the head posture data of the first human subject in the video frame and the head posture data of the second human subject in the video frame template in the lens parameter template library is needed when determining the target lens translation parameters and the target lens rotation parameters, then the step of "determining the posture data difference between the head posture data of the first human subject in the video frame and the head posture data of the second human subject in the video frame template in the lens parameter template library" only needs to be executed once, and there is no need to repeat it.

[0117] In an optional example, determining the difference between the face area of ​​a first human subject in a video frame and the face area of ​​a second human subject in a video frame template in a lens parameter template library includes:

[0118] The following formula is used to determine the face area of ​​the first human subject in the video frame and the face area of ​​the second human subject in the video frame template in the lens parameter template library;

[0119] area = (y2-y1)*(x2-x1);

[0120] Among them, area is used to represent the face area of ​​the first human subject in the video frame and the face area of ​​the second human subject in the video frame template in the lens parameter template library, and (x1, y1) and (x2, y2) are used to represent the position coordinates of the face.

[0121] The following formula is used to determine the difference between the face area of ​​the first subject in the video frame and the face area of ​​the second subject in the video frame template in the lens parameter template library:

[0122] area_delta=abs(area-area[m]);

[0123] Among them, area_delta is used to represent the difference between the facial area of ​​the first human subject in the video frame and the facial area of ​​the second human subject in the video frame template in the lens parameter template library, area[m] is used to represent the facial area of ​​the second human subject in the mth video frame template in the lens parameter template library, area is used to represent the facial area of ​​the first human subject in the video frame, and abs() represents the absolute value.

[0124] In an optional example, determining the coordinate difference between the human body position coordinates of a first human subject in a video frame and the human body position coordinates of a second human subject in a video frame template in a lens parameter template library includes:

[0125] The following formula is used to determine the coordinate difference between the human body position coordinates of the first human subject in the video frame and the human body position coordinates of the second human subject in the video frame template in the lens parameter template library:

[0126] position_delta=abs((a1-a1[m])+(b1-b1[m])+(a2-a2[m])+(b2-b2[m]));

[0127] Among them, position_delta is used to represent the coordinate difference between the body position coordinates of the first human subject in the video frame and the body position coordinates of the second human subject in the video frame template in the lens parameter template library, m is used to represent the mth video frame template in the lens parameter template library, abs() represents the absolute value, a1, a2, b2, b2, a1[m], a2[m], a2, b1[m] and b2[m] are used to represent the body position coordinates.

[0128] In an optional example, determining a depth data difference between depth data of a first human subject in a video frame and depth data of a second human subject in a video frame template in a lens parameter template library includes:

[0129] The depth data difference between the depth data of the first human subject in the video frame and the depth data of the second human subject in the video frame template in the lens parameter template library is determined using the following formula:

[0130] depth_delta=MSE(R, R[m]);

[0131] Among them, depth_delta is used to represent the depth data difference between the depth data of the first human subject in the video frame and the depth data of the second human subject in the video frame template in the lens parameter template library, R is used to represent the depth data of the first human subject in the video frame, R[m] is used to represent the depth data of the second human subject in the video frame template in the lens parameter template library, and MSE represents the mean square error.

[0132] In an optional embodiment, determining a target lens shift parameter corresponding to a video frame from a lens parameter template library based on the difference data includes:

[0133] Determine a target video frame template from a lens parameter template library according to at least one difference among a posture data difference, an area difference, a coordinate difference, and a depth data difference;

[0134] The translation parameters corresponding to the target video frame template are determined as the target lens translation parameters corresponding to the video frame.

[0135] In an optional example, when the target video frame template is determined from the lens parameter template library according to the area difference, the coordinate difference and the depth data difference, the target video frame template is determined from the lens parameter template library using the following formula:

[0136] L=alpha1*area_delta+alpha2*position_delta+alpha3*depth_delta;

[0137] Among them, L represents the sum of the differences corresponding to the video frame templates in the lens parameter template library, alpha1, alpha2, and alpha3 are hyperparameters, area_delta is used to represent the area difference between the face area of ​​the first human subject in the video frame and the face area of ​​the second human subject in the video frame template in the lens parameter template library, position_delta is used to represent the coordinate difference between the body position coordinates of the first human subject in the video frame and the body position coordinates of the second human subject in the video frame template in the lens parameter template library, and depth_delta is used to represent the depth data difference between the depth data of the first human subject in the video frame and the depth data of the second human subject in the video frame template in the lens parameter template library.

[0138] Afterwards, the video frame template with the lowest sum of differences in the lens parameter template library is determined as the target video frame template, and the translation parameters corresponding to the target video frame template are determined as the target lens translation parameters corresponding to the video frame.

[0139] It should be noted that the formula L = alpha1*area_delta+alpha2*position_delta+alpha3*

[0140] Depth_delta can be adaptively changed according to the difference data, and this application does not limit this. When the target video frame template is determined from the lens parameter template library based on only one type of difference data, the target video frame template can be directly determined from the lens parameter template library based on the difference value corresponding to the difference data, without using the above formula for calculation.

[0141] In an optional embodiment, the method further includes the following steps: for an intermediate video frame located between the first target video frame and the second target video frame, interpolating the target lens parameters corresponding to the first target video frame and the target lens parameters of the second target video frame to obtain the target lens parameters corresponding to the intermediate video frame; the first target video frame and the second target video frame are any two adjacent target video frames in the storyboard segment.

[0142] In an optional example, for the intermediate video frames between any two adjacent target video frames, such as video frame 2, video frame 3, and video frame 4 between video frame 1 and video frame 5, and video frame 6, video frame 7, and video frame 8 between video frame 5 and video frame 9, the target lens parameters of any two adjacent target video frames are interpolated to obtain the target lens parameters corresponding to the intermediate video frames.

[0143] In an optional embodiment, the lens parameters matched by the previous video frame are: before_cam_params=[Transition_x1, Transition_y1, Transition_z1, Pitch1, Yaw1, Roll1, fov1];

[0144] The lens parameters of the next frame of video frame matching are:

[0145] after_cam_params=[Transition_x2, Transition_y2, Transition_z2, Pitch2, Yaw2, Roll2, fov2];

[0146] For N video frames separated in between, the lens parameters of each video frame can be:

[0147] Cam(i)=ratio*before_cam_params+(1-ratio)*after_cam_params, i=(1,...,N-1);

[0148] Here, i is used to represent the i-th video frame, and ratio=1-(i / N) is used to represent the proportional coefficient.

[0149] By processing only some frames, the consumption of computing resources can be significantly reduced. Because every frame does not need to be analyzed and processed in detail, this can speed up processing and reduce hardware requirements. Interpolation algorithms can help ensure that the transition between non-key frames is smoother and more natural, thereby maintaining the consistency and fluidity of the entire sequence. Since not all frames are saved or transmitted in full, but rely on interpolation to generate intermediate frames, the amount of data can be reduced, thereby saving storage space and bandwidth.

[0150] In an optional embodiment, pre-configuring the lens parameter template library includes the following steps:

[0151] Step 1: Set multiple sets of lens parameters. Lens parameters include one or more of the following parameters: Transition_x, Transition_y, Transition_z, FOV, Pitch, Yaw and Roll.

[0152] FOV is an angular measurement of the range that an optical system can observe. FOV values ​​vary in different application areas, such as photography, film shooting, virtual reality, augmented reality, surveillance cameras, and games. In the application scenarios of photography and videography, if the lens is a standard lens, the FOV value can be set to a value of 40-55 degrees to simulate the natural perspective of the human eye. If the lens is a wide-angle lens, the FOV value can be set to a value greater than 60 degrees to capture a wider scene. If the lens is a telephoto lens, the FOV value can be set to a value less than 30 degrees, which is suitable for long-distance shooting.

[0153] When setting multiple sets of lens parameters, it is necessary to consider whether the second person subject can be placed in the picture under the lens parameters. Only video frame templates in which the second person subject is placed in the picture can be stored in the lens parameter template library.

[0154] It should be noted that different FOVs correspond to different ranges of Transition_x, Transition_y, and Transition_z. For example, when FOV is 30, the range of Transition_x, Transition_y, and Transition_z is (0, 10); when FOV is 35, the range of Transition_x, Transition_y, and Transition_z is (0, 8); when FOV is 38, the range of Transition_x, Transition_y, and Transition_z is (0, 7). That is, when selecting FOV, you need to select the corresponding Transition_x, Transition_y, and Transition_z within the above range to avoid shooting video frame templates that cannot be stored in the lens parameter template library.

[0155] It should be noted that the numerical values ​​given in the examples of the present application are exemplary.

[0156] In an optional embodiment, as shown in Table 1, six groups of lens parameters are listed as examples in Table 1:

[0157] FOV 30 30 30 30 40 60 Transition_x 0 0 1 4 1 0 Transition_y 0 0 2 5 2 0 Transition_z 2 2 3 6 3 2 Pitch 30 31 40 37 67 45 Yaw 30 31 45 28 68 78 Roll 60 31 49 91 79 89

[0158] Table 1

[0159] Step 2: Setting a lens based on the determined multiple sets of lens parameters, shooting with the lens, and randomly sampling the shot video images to obtain multiple video frame templates.

[0160] Each time a group of lens parameters is set, shooting is performed using the lens that sets the group of lens parameters, and the shot video images are randomly sampled to obtain a video frame template corresponding to the group of lens parameters.

[0161] Step 3: Extract template image features of the video frame template.

[0162] The process of extracting the template picture features of the video frame template in this step is similar to the process of extracting the target picture features for the first character subject in the video frame in the storyboard segment in step S103, and this application will not elaborate on this.

[0163] Step 4: Record the corresponding relationship between the video frame template, the template picture feature corresponding to the video frame template, and the lens parameter corresponding to the template picture feature, and store the corresponding relationship in a lens parameter template library.

[0164] In an optional embodiment, the lens parameter template library may also store a correspondence between a video frame template, a depth map of a second human subject in the video frame template, template picture features corresponding to the video frame template, and lens parameters corresponding to the template picture features. The depth map of the second human subject in the video frame template is used to obtain depth data of the second human subject in the video frame template.

[0165] Step S105: photographing the first virtual character based on the target lens parameters.

[0166] In an optional embodiment, photographing the first virtual character based on the target lens parameter includes:

[0167] Determine the expression and / or action of the first human subject in the reference video; control the first virtual character to present a virtual expression according to the expression of the first human subject, and / or control the first virtual character to present a virtual action according to the action of the first human subject; while the first virtual character presents the virtual expression and / or the virtual action, shoot the first virtual character through a virtual lens based on target lens parameters.

[0168] It should be noted that the expressions include at least one or more of the following expressions: sad, happy, crying, angry, etc., and the actions include one or more of the following actions: running, jumping, shooting, etc.

[0169] In the embodiment of the present application, the method obtains a reference video, extracts target picture features from the reference video, and matches appropriate lens parameters from a lens parameter template library based on the extracted target picture features.

[0170] The first virtual character is photographed based on the target lens parameters. Through this method, the virtual object can be photographed without the user having complex camera movement knowledge, and a video with camera movement effect can be obtained.

[0171] The embodiment of the present application also provides a task logic diagram of a display control method, the video method comprising the following steps:

[0172] Step 1: Obtain a video frame template and store it in a lens parameter template library;

[0173] Step 2: Obtain template picture features of the video frame template from the lens parameter template library;

[0174] Step 3: Get reference video;

[0175] Step 4: Perform storyboard processing on the reference video to obtain storyboard clips;

[0176] Step 5: Accelerate the model;

[0177] Step 6: Extract the target image features of the target video frame every N frames in the storyboard clip through models such as face detection, human body detection, and depth estimation;

[0178] Step 7: Match the target picture features of the target video frame with the template picture features of the video frame template in the lens parameter template library to obtain the target lens parameters corresponding to the target video frame;

[0179] Step 8: For the non-target video frame, perform lens parameter interpolation processing based on the target lens parameters corresponding to the target video frame to obtain the target lens parameters of the non-target video frame;

[0180] Step 9: Output the target lens parameters of the reference video;

[0181] Step 10: Use the target lens parameters of the reference video to shoot the virtual object to obtain a video frame with a camera movement effect.

[0182] Corresponding to the display control method provided in the embodiment of the present application, the embodiment of the present application also provides a display control device 300, such as Figure 3As shown, the device comprises:

[0183] The acquisition module 301 is used to acquire a reference video in response to a shooting instruction for a first virtual character in a virtual scene, where the reference video is obtained by shooting a first character subject;

[0184] A storyboard module 302, configured to perform storyboard processing on the reference video to obtain a plurality of storyboard segments;

[0185] An extraction module 303 is used to extract target screen features for the first character subject in the video frame in the storyboard segment;

[0186] A determination module 304 is used to determine the target lens parameters corresponding to the video frame from a preconfigured lens parameter template library based on extracting the target screen features from the first human subject in the video frame, wherein the lens parameter template library includes a video frame template, a template screen feature corresponding to the second human subject in the video frame template, and a lens parameter corresponding to the template screen feature;

[0187] The shooting module 305 is used to shoot the first virtual character based on the target lens parameters.

[0188] Corresponding to a display control method provided in an embodiment of the present application, an embodiment of the present application also provides an electronic device for implementing the display control method, such as Figure 4 As shown, the electronic device includes: a processor 401; and a memory 402, which is used to store a program of a display control method. After the device is powered on and the program of the display control method is run by the processor, the following steps are performed:

[0189] In response to a shooting instruction for a first virtual character in a virtual scene, a reference video is acquired, where the reference video is shot for the first character subject;

[0190] Performing storyboard processing on the reference video to obtain a plurality of storyboard segments;

[0191] Extracting target screen features for the first character subject in the video frame in the storyboard segment;

[0192] Based on the target picture features extracted from the first human subject in the video frame, determine the target lens parameters corresponding to the video frame from a preconfigured lens parameter template library, wherein the lens parameter template library includes a video frame template, a template picture feature corresponding to the second human subject in the video frame template, and a lens parameter corresponding to the template picture feature;

[0193] The first virtual character is photographed based on the target lens parameters.

[0194] Corresponding to the display control method provided in the embodiment of the present application, the embodiment of the present application further provides a computer-readable storage medium storing a program of the display control method, the program is executed by a processor to perform the following steps:

[0195] In response to a shooting instruction for a first virtual character in a virtual scene, a reference video is acquired, where the reference video is shot for the first character subject;

[0196] Performing storyboard processing on the reference video to obtain a plurality of storyboard segments;

[0197] Extracting target screen features for the first character subject in the video frame in the storyboard segment;

[0198] Based on the target picture features extracted from the first human subject in the video frame, determine the target lens parameters corresponding to the video frame from a preconfigured lens parameter template library, wherein the lens parameter template library includes a video frame template, a template picture feature corresponding to the second human subject in the video frame template, and a lens parameter corresponding to the template picture feature;

[0199] The first virtual character is photographed based on the target lens parameters.

[0200] It should be noted that for the detailed description of the device, electronic device and computer-readable storage medium provided in the embodiments of the present application, reference can be made to the relevant description of the display control method embodiment provided in the embodiments of the present application, which will not be repeated here.

[0201] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

[0202] In a typical configuration, an electronic device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0203] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0204] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable operations, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0205] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0206] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

Claims

1. A display control method, characterized in that: The method comprises: In response to a shooting instruction for a first virtual character in a virtual scene, a reference video is acquired, where the reference video is shot for the first character subject; Performing storyboard processing on the reference video to obtain a plurality of storyboard segments; Extracting target screen features for the first character subject in the video frame in the storyboard segment; Based on the target picture features extracted from the first human subject in the video frame, determine the target lens parameters corresponding to the video frame from a preconfigured lens parameter template library, wherein the lens parameter template library includes a video frame template, a template picture feature corresponding to the second human subject in the video frame template, and a lens parameter corresponding to the template picture feature; The first virtual character is photographed based on the target lens parameters.

2. The method according to claim 1, characterized in that The step of determining target lens parameters corresponding to the video frame from a preconfigured lens parameter template library based on the target picture features extracted from the first human subject in the video frame includes: Determine difference data between target picture features of the first human subject in the video frame and template picture features of the second human subject in the video frame template in the lens parameter template library; The target lens parameter corresponding to the video frame is determined from the lens parameter template library based on the difference data.

3. The method according to claim 2, characterized in that The determining, based on the difference data, from the lens parameter template library, the target lens parameter corresponding to the video frame comprises: Determining a target lens rotation parameter corresponding to the video frame from the lens parameter template library based on the difference data; and / or, The target lens shift parameter corresponding to the video frame is determined from the lens parameter template library based on the difference data.

4. The method according to claim 3, characterized in that The determining of difference data between target picture features of the first subject in the video frame and template picture features of the second subject in the video frame template in the lens parameter template library includes: Determine a posture data difference between the head posture data of the first human subject in the video frame and the head posture data of the second human subject in the video frame template in the lens parameter template library; The determining the target lens rotation parameter corresponding to the video frame from the lens parameter template library based on the difference data includes: The rotation parameter corresponding to the video frame template with the smallest difference in the posture data in the lens parameter template library is used as the target lens rotation parameter corresponding to the video frame.

5. The method according to claim 3, characterized in that: The determining of difference data between target picture features of the first subject in the video frame and template picture features of the second subject in the video frame template in the lens parameter template library includes: Determine a posture data difference between the head posture data of the first human subject in the video frame and the head posture data of the second human subject in the video frame template in the lens parameter template library; Determine an area difference between a face area of ​​the first human subject in the video frame and a face area of ​​a second human subject in the video frame template in the lens parameter template library; Determine the coordinate difference between the human body position coordinates of the first human subject in the video frame and the human body position coordinates of the second human subject in the video frame template in the lens parameter template library; Determine a depth data difference between the depth data of the first human subject in the video frame and the depth data of the second human subject in the video frame template in the lens parameter template library.

6. The method according to claim 5, characterized in that The determining, based on the difference data, from the lens parameter template library, the target lens shift parameter corresponding to the video frame comprises: Determine a target video frame template from the lens parameter template library according to at least one difference among the posture data difference, the area difference, the coordinate difference and the depth data difference; The translation parameters corresponding to the target video frame template are determined as the target lens translation parameters corresponding to the video frame.

7. The method according to claim 5, characterized in that The step of extracting target screen features from the first character subject in the video frame in the storyboard segment comprises one or more of the following steps: Extracting facial position features of the first human subject in the video frame; Extracting head posture features of the first human subject in the video frame; Extracting a body position feature of the first human subject in the video frame; A depth feature of the first human subject is extracted for the first human subject in the video frame.

8. The method according to claim 1, characterized in that The step of extracting target screen features from the first character subject in the video frame in the storyboard segment includes: Determine a target video frame for every N video frames of the storyboard segment, where N is a positive integer greater than 1; Extracting target picture features for the first human subject in the target video frame; The method further comprises: For an intermediate video frame located between a first target video frame and a second target video frame, target lens parameters corresponding to the first target video frame and target lens parameters of the second target video frame are interpolated to obtain target lens parameters corresponding to the intermediate video frame, wherein the first target video frame and the second target video frame are any two adjacent target video frames in the storyboard segment.

9. The method according to claim 1, characterized in that: The step of extracting target screen features from the first character subject in the video frame in the storyboard segment includes: Extracting target screen features for the first character subject in the video frame in the storyboard segment through a preconfigured deep learning model; The method comprises: The feature extraction process of the deep learning model is accelerated by a preconfigured acceleration model.

10. The method according to claim 1, characterized in that The photographing the first virtual character based on the target lens parameter includes: Determining the expression and / or action of the first human subject in the reference video; Controlling the first virtual character to present a virtual expression according to the expression of the first character subject, and / or controlling the first virtual character to present a virtual action according to the action of the first character subject; During the process of the first virtual character presenting the virtual expression and / or virtual action, the first virtual character is photographed by a virtual lens based on the target lens parameters.

11. The method according to claim 1, characterized in that: The target lens translation parameters include one or more of the following parameters: transition_x, transition_y, transition_z and field of view FOV, and the target lens rotation parameters include one or more of the following parameters: pitch, yaw and roll.

12. A display control device, characterized in that: The device comprises: An acquisition module, configured to acquire a reference video in response to a shooting instruction for a first virtual character in a virtual scene, wherein the reference video is obtained by shooting a first character subject; A storyboard module, used for storyboarding the reference video to obtain a plurality of storyboard segments; An extraction module, configured to extract target screen features from the first character subject in the video frame in the storyboard segment; A determination module, configured to determine target lens parameters corresponding to the video frame from a preconfigured lens parameter template library based on extracting the target screen features from the first human subject in the video frame, wherein the lens parameter template library includes a video frame template, a template screen feature corresponding to the second human subject in the video frame template, and a lens parameter corresponding to the template screen feature; A shooting module is used to shoot the first virtual character based on the target lens parameters.

13. An electronic device, characterized in that: include: processor; as well as The memory is used to store a data processing program. After the electronic device is powered on and the program is run by the processor, the method according to any one of claims 1 to 11 is executed.

14. A computer-readable storage medium, characterized in that: A data processing program is stored, and the program is run by a processor to execute the method according to any one of claims 1 to 11.

Images