Video processing method, device, computer equipment, storage medium, program product

Abstract

The present disclosure relates to a video processing method, device, computer equipment, storage medium, and program product. The method includes: obtaining a panoramic video and the position information of a panoramic camera during the process of shooting the panoramic video; constructing a moving track map of the panoramic camera during the process of shooting the panoramic video based on the position information; determining the camera movement points during the process of shooting the panoramic video based on the moving track map; inputting the camera movement points and the panoramic video into a pre-trained camera movement type determination model, and outputting the video type and camera movement type of the panoramic video through the camera movement type determination model; matching the camera movement effect from a pre-set camera movement rule library based on the camera movement type and video type; and making adjustments based on the camera movement effect at the camera movement points of the panoramic video to generate a target video. The present method can be used to edit the panoramic video.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing technology, and in particular to a video processing method, apparatus, computer equipment, storage medium, and program product. Background Art

[0002] With the continuous development of various interactive platforms, more and more users will edit and adjust videos after shooting them to achieve satisfactory results. Common video editing and video editing work usually adopts manual operation, but the efficiency of traditional manual video editing is low. It needs to rely on the editor's rich editing experience and long-term accumulated video editing thinking to obtain better video editing effects. In order to further overcome this problem, automated editing came into being. Multifunctional video editing systems gradually replaced manual editing and became the mainstream operation of video editing.

[0003] However, traditional automatic editing cannot handle panoramic videos well, and there is currently no automatic editing method for videos of motion scenes. Summary of the invention

[0004] Based on this, it is necessary to provide a video processing method, device, computer equipment, storage medium, and program product to address the above technical problems.

[0005] In a first aspect, the present disclosure provides a video processing method, the method comprising:

[0006] Acquire a panoramic video and position information of a panoramic camera during shooting of the panoramic video;

[0007] Constructing a movement trajectory map of the panoramic camera during the process of shooting the panoramic video based on the position information;

[0008] Based on the movement trajectory map, determining the camera movement point in the process of shooting the panoramic video;

[0009] Inputting the camera movement point and the panoramic video into a pre-trained camera movement type determination model, and outputting the video type and camera movement type of the panoramic video via the camera movement type determination model;

[0010] Based on the camera movement type and the video type, matching the camera movement effect from a preset camera movement rule library;

[0011] At the camera movement point of the panoramic video, adjustments are made based on the camera movement effect to generate a target video.

[0012] In one embodiment, the method further comprises:

[0013] Acquiring the posture information of the panoramic camera during the process of shooting the panoramic video;

[0014] Calculating, based on the position information and / or posture information of the panoramic camera, a speed corresponding to each frame of the panoramic image obtained by the panoramic camera during the process of shooting the panoramic video;

[0015] Determining a field of view angle of each panoramic image frame in the panoramic video based on a speed corresponding to each panoramic image frame and a preset speed threshold;

[0016] Correspondingly, at the camera movement point of the panoramic video, adjustment is performed based on the camera movement effect to generate a target video, including:

[0017] Based on the field of view angle, the panoramic video is adjusted, and at the camera movement point of the adjusted panoramic video, a secondary adjustment is performed based on the camera movement effect to generate a target video.

[0018] In one of the embodiments, determining the field of view angle of each panoramic image frame in the panoramic video based on the speed corresponding to each panoramic image frame and a preset speed threshold includes:

[0019] In response to the speed corresponding to the panoramic image being greater than or equal to a preset speed threshold, determining a speed direction indicated by the speed corresponding to the panoramic image;

[0020] Based on the speed direction, determining a field of view angle of the panoramic image having a speed greater than or equal to a preset speed threshold;

[0021] In response to a speed corresponding to the panoramic image being less than a preset speed threshold, identifying a target local area in the panoramic image having a speed less than the preset speed threshold using a pointing attention model;

[0022] Based at least on the target local area, a field of view angle of the panoramic image having a speed less than a preset speed threshold is determined.

[0023] In one embodiment, the step of adjusting the panoramic video based on the field of view angle, and performing secondary adjustment based on the camera movement effect at the camera movement point of the adjusted panoramic video to generate a target video includes:

[0024] Classifying each panoramic image frame in the panoramic video by using the field of view angle to obtain at least one group of panoramic classification images, wherein the panoramic images contained in each group of the panoramic classification images have the same field of view angle;

[0025] In response to the absence of the camera movement point in the panoramic classification image, adjusting the playback speed of the panoramic classification image without the camera movement point according to a preset speed multiple, and adjusting the panoramic classification image according to the field of view angle corresponding to the panoramic classification image without the camera movement point;

[0026] In response to the presence of the camera movement point in the panoramic classification image, adjusting the panoramic classification image according to the field of view angle of the panoramic classification image where the camera movement point exists, and adjusting the panoramic classification image according to the camera movement effect at the camera movement point;

[0027] Based on the adjusted panoramic classification image, a target video is generated.

[0028] In one embodiment, the method further comprises:

[0029] Get panoramic training video;

[0030] Determining camera movement points in the panoramic training video, and determining a camera movement type and a video type that match the panoramic training video;

[0031] Based on the camera movement points in the panoramic training video, the camera movement type and the video type that match the panoramic training video, a neural network model is trained to obtain a camera movement type determination model.

[0032] In one of the embodiments, determining the camera movement point in the process of shooting the panoramic video based on the movement trajectory map includes:

[0033] Based on the movement trajectory map, determining a turning point at which the panoramic camera turns at an angle greater than a preset angle during the process of shooting the panoramic video;

[0034] The time interval between the turning points is determined, and the turning point whose time interval is greater than the preset time is determined as the camera movement point.

[0035] In a second aspect, the present disclosure further provides a video processing device. The device comprises:

[0036] A video acquisition module, used to acquire panoramic video and position information of the panoramic camera during the process of shooting the panoramic video;

[0037] A trajectory map determination module, used to construct a movement trajectory map of the panoramic camera during the process of shooting the panoramic video based on the position information;

[0038] A camera movement point determination module, used to determine the camera movement points in the process of shooting the panoramic video based on the movement trajectory map;

[0039] A model processing module, used for inputting the camera movement point and the panoramic video into a pre-trained camera movement type determination model, and outputting the video type and camera movement type of the panoramic video via the camera movement type determination model;

[0040] A camera movement effect matching module, used for matching camera movement effects from a preset camera movement rule library based on the camera movement type and video type;

[0041] The video generation module is used to make adjustments based on the camera movement effect at the camera movement point of the panoramic video to generate a target video.

[0042] In a third aspect, the present disclosure further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in any of the above method embodiments are implemented.

[0043] In a fourth aspect, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0044] In a fifth aspect, the present disclosure further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0045] In the above embodiments, a moving track map of the panoramic camera in the process of shooting the panoramic video is constructed based on the position information; based on the moving track map, the camera movement point in the process of shooting the panoramic video is determined. By constructing the moving track map, each position of the camera in the shooting process can be accurately understood to ensure that each picture of the video reflects the real scene changes. Determining the camera movement point helps to optimize the lens movement, capture the best visual effect, and improve the overall quality and viewing quality of the video. And a good camera movement point design can provide a smoother visual experience, reduce the dizziness when watching, and enhance the audience's immersion. The camera movement point and the panoramic video are input into the pre-trained camera movement type determination model, and the video type and camera movement type of the panoramic video are output through the camera movement type determination model; through the camera movement type determination model, the camera movement point and type of the video can be automatically analyzed to improve the efficiency of production and reduce manual intervention. Based on the camera movement type and video type, the camera movement effect is matched from the pre-set camera movement rule library; the camera movement effect is matched from the camera movement rule library so that the camera movement effect of the video matches the video type and the camera movement type, and the viewing experience is optimized. At the camera movement points of the panoramic video, adjustments are made based on the camera movement effects to generate a target video. Applying appropriate camera movement effects at the camera movement points can significantly improve the production level of the video and make the final target video more professional. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 A schematic diagram of an application environment of a video processing method in an embodiment;

[0048] Figure 2 is a schematic flow chart of a video processing method in one embodiment;

[0049] Figure 3 A schematic flow chart of the steps of determining the viewing angle in one embodiment;

[0050] Figure 4 Schematic diagram of the process of step S306 in one embodiment;

[0051] Figure 5 is a schematic diagram of a target local area and a field of view angle in one embodiment;

[0052] Figure 6A schematic diagram of a process for generating a target video in one embodiment;

[0053] Figure 7 A flowchart of the steps of training a model in one embodiment;

[0054] Figure 8 This is a schematic diagram of the process of step S206 in one embodiment;

[0055] Fig. 9 A schematic block diagram of the structure of a video processing method device in one embodiment;

[0056] Fig.10 Schematic diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0058] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0059] In this article, the term "and / or" is only a description of the association relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship.

[0060] The present disclosure provides a video processing method which can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 or the panoramic camera 106. The terminal 102 can obtain the panoramic video in the server 104, and the terminal 102 can also obtain the panoramic video in the panoramic camera 106. The terminal 102 can also obtain the position information of the panoramic camera during the panoramic video shooting process. The terminal 102 can construct a moving track map of the panoramic camera 106 during the panoramic video shooting process based on the position information. The terminal 102 determines the camera movement point during the panoramic video shooting process based on the moving track map. The terminal 102 inputs the camera movement point and the panoramic video into the pre-trained camera movement type determination model, and outputs the video type and camera movement type of the panoramic video through the camera movement type determination model. The terminal 102 matches the camera movement effect from the pre-set camera movement rule library based on the camera movement type and the video type. The terminal 102 adjusts the camera movement at the camera movement of the panoramic video based on the camera movement effect to generate a target video. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0061] In one embodiment, Figure 2 As shown, a video processing method is provided, which is applied to Figure 1 The terminal 102 in the example is used as an example to illustrate, and the following steps are included:

[0062] S202, obtaining a panoramic video and position information of a panoramic camera during shooting of the panoramic video.

[0063] It should be noted that in some embodiments of the present disclosure, the panoramic video is usually taken by a panoramic camera, not synthesized by some technology, so as to ensure that the panoramic video will have corresponding location information. The location information can usually be obtained by using GPS or other positioning systems in the panoramic camera. The location information can usually include: longitude, latitude and timestamp.

[0064] Specifically, a panoramic video can be shot with a panoramic camera, and the panoramic camera needs to record corresponding location information during the process of shooting the panoramic video. The terminal can be directly connected to the panoramic camera to obtain the panoramic video shot by the panoramic camera and the location information of the panoramic camera when shooting the panoramic video. In addition, the terminal can also be connected to a server (the server can be a cloud server), and the panoramic camera can upload the panoramic video shot and the location information when shooting the panoramic video to the server, and the terminal can obtain the panoramic video and corresponding location information in the server.

[0065] S204: constructing a movement track map of the panoramic camera during the process of shooting the panoramic video based on the position information.

[0066] Specifically, the location information can be cleaned first to remove outliers (e.g., obviously wrong location information). Then the location information is sorted according to the timestamp to ensure that the trajectory is arranged in chronological order. The GPS coordinate points in the adjacent location information are connected to form a movement trajectory map.

[0067] In some exemplary embodiments, the movement trajectory map may also be established using SLAM (Simultaneous Localization and Mapping) technology. SLAM (Simultaneous Localization and Mapping) is a technology used in fields such as mobile robots and autonomous driving.

[0068] S206: Determine the camera movement points during the process of shooting the panoramic video based on the movement trajectory map.

[0069] The camera movement point may generally be a time point in the panoramic video at which the camera movement is required.

[0070] Specifically, according to the moving trajectory of the panoramic camera in the moving trajectory map, a turning point can be selected from the moving trajectory, or a midpoint of a trajectory that has not turned for a long time can be selected as a camera movement point.

[0071] S208, inputting the camera movement points and the panoramic video into a pre-trained camera movement type determination model, and outputting the video type and camera movement type of the panoramic video via the camera movement type determination model.

[0072] Among them, the camera movement type determination model generally refers to a model used to select and determine the camera movement type in the fields of film and television production, video production, animation, etc. Camera movement refers to the movement and change of the lens during the shooting process, which can affect the narrative style, emotional expression and visual experience of the audience through different camera movement techniques. Camera movement type refers to the different lens movement and change methods used in film and television or video production to enrich the visual performance and enhance the narrative effect. Common camera movement types include: panning lens, zooming lens, zooming lens, etc. Video types can include: mountain bike, kart, off-road vehicle, bicycle and other video types with fast scene switching, and can also include slow speed scene switching video types such as jogging, marathon and other slow speeds. In addition, the camera movement type determination model can be obtained by training a neural network model using training data. The training data can include: panoramic training data, and camera movement points pre-set in the panoramic training data, camera movement types and video types of the panoramic training data. Neural network models can include: convolutional neural networks (CNNs), regional convolutional neural networks (R-CNNs), generative adversarial networks (GANs), etc.

[0073] Specifically, the camera movement points and the panoramic video may be input into a pre-trained camera movement type determination model, and the camera movement type determination model may output the video type and camera movement type of the panoramic video.

[0074] S210: Match camera movement effects from a preset camera movement rule library based on the camera movement type and the video type.

[0075] Among them, the camera movement rule library can store a variety of different types of camera movement effects, as well as the mapping relationship between camera movement effects and video types and / or camera movement types. Camera movement effects refer to the use of camera movement (lens movement, rotation and adjustment) to change the visual performance of the picture in film and television production and photography to achieve a specific narrative, emotional or aesthetic effect. Camera movement effects can enhance the expressiveness of the story and affect the audience's emotional response and understanding. Camera movement effects can include: 1. Dolly In: The camera moves forward and closer to the subject. It can make the audience focus on a specific object or emotion and enhance the sense of urgency. 2. Dolly Out: The camera moves backward and away from the subject. It is often used to show the environment or background to provide more context to the audience. 3. Pan: The camera rotates horizontally or vertically to scan the scene. It is often used to show large scenes or follow moving objects. 4. Tilt: The camera tilts up and down, usually used to show tall buildings or vertical movement. 5. Crane Shot: The camera moves on a high platform to create a dramatic effect, usually used for the opening or transition of important scenes. 6. Tracking Shot: The camera follows the moving object, often used to express dynamic scenes, such as chases or dialogues, to enhance the sense of the scene. 7. Close-Up: Focus on the facial expressions of characters or the details of objects to emphasize emotions or importance. Since the present disclosure processes panoramic videos, it is possible to magnify and move a certain target in the panoramic video to achieve different camera movement effects.

[0076] Specifically, the camera movement type and the video type may be used to perform matching in a camera movement rule library to obtain a camera movement effect that matches the camera movement type and the video type.

[0077] In some exemplary embodiments, for example, if the camera movement type is a zoom shot and the video type is a bicycle riding scene, the camera movement effect matched in the camera movement rule library may be an effect of zooming in on a local area of ​​the bicycle (such as a wheel or pedal or other area) according to a preset ratio for a close-up. For another example, if the camera movement type is a panning shot and the video type is also a bicycle riding scene, the camera movement effect matched in the camera movement rule library may be an effect of moving along with the entire bicycle. It should be noted that the camera movement effects matched are different according to different camera movement types and video types, and in some embodiments of the present disclosure, there is no absolute restriction on the camera movement effects.

[0078] S212, at the camera movement point of the panoramic video, make adjustments based on the camera movement effect to generate a target video.

[0079] Specifically, after the camera movement point is determined, adjustments can be made at the camera movement point of the panoramic video according to information indicated by the camera movement effect, and the target video is generated after the adjustments.

[0080] In the above-mentioned video processing method, a moving track map of the panoramic camera in the process of shooting the panoramic video is constructed based on the position information; based on the moving track map, the camera movement point in the process of shooting the panoramic video is determined. By constructing the moving track map, each position of the camera in the shooting process can be accurately understood, ensuring that each picture of the video reflects the real scene changes. Determining the camera movement point helps to optimize the lens movement, capture the best visual effect, and improve the overall quality and viewing quality of the video. And a good camera movement point design can provide a smoother visual experience, reduce the dizziness when watching, and enhance the audience's immersion. The camera movement point and the panoramic video are input into a pre-trained camera movement type determination model, and the video type and camera movement type of the panoramic video are output through the camera movement type determination model; through the camera movement type determination model, the camera movement point and type of the video can be automatically analyzed, the production efficiency is improved, and manual intervention is reduced. Based on the camera movement type and video type, the camera movement effect is matched from the pre-set camera movement rule library; the camera movement effect is matched from the camera movement rule library so that the camera movement effect of the video matches the video type and the camera movement type, and the viewing experience is optimized. At the camera movement points of the panoramic video, adjustments are made based on the camera movement effects to generate a target video. Applying appropriate camera movement effects at the camera movement points can significantly improve the production level of the video and make the final target video more professional.

[0081] In one embodiment, Figure 3 As shown, the method also includes:

[0082] S302, obtaining posture information of the panoramic camera during the process of shooting the panoramic video.

[0083] The posture information may generally be IMU information, which may include, for example, three-axis acceleration, three-axis angular velocity data, and other information.

[0084] Specifically, when the panoramic camera is shooting a panoramic video, the inertial measurement unit in the panoramic camera can record the posture information of the panoramic camera. The terminal can also obtain the posture information of the panoramic camera when shooting the panoramic video.

[0085] S304: Calculate, based on the position information and / or posture information of the panoramic camera, a speed corresponding to each frame of the panoramic image obtained by the panoramic camera during the process of shooting the panoramic video.

[0086] Specifically, in order to improve the calculation accuracy, Kalman filtering or other filters can be applied to reduce the noise of the position information and attitude information respectively. Then the displacement can be calculated based on the position information. The speed is calculated based on the timestamp information and displacement corresponding to the position information. The acceleration data of the attitude information can also be used to perform numerical integration to calculate the speed.

[0087] In some exemplary embodiments, the distance between two pieces of position information may be calculated using the Haversine Formula.

[0088]

[0089] in, is the radius of the Earth, is the angular distance between the position information.

[0090] Velocity formula: Velocity can be calculated by the ratio of displacement to time:

[0091]

[0092] in, It's distance. is the time interval.

[0093] You can also use the acceleration data of the attitude information to perform linear acceleration integration to calculate the speed. Specifically, you can use the following formula to calculate the speed:

[0094]

[0095] in, For The acceleration of time.

[0096] In addition, the position information and attitude information can be fused and the extended Kalman filter (EKF) can be used to combine the information of the two to obtain a more accurate velocity estimate.

[0097] S306: Determine the field of view angle of each panoramic image frame in the panoramic video based on the speed corresponding to each panoramic image frame and a preset speed threshold.

[0098] The field of view angle may generally be a viewing angle for viewing a panoramic image. Different viewing angles may result in different effects of the displayed panoramic image.

[0099] Specifically, the speed corresponding to each frame of the panoramic image may be determined, the speed may be compared with a preset speed threshold, and the field of view angle of the panoramic image may be determined according to the comparison result.

[0100] Correspondingly, at the camera movement point of the panoramic video, adjustment is performed based on the camera movement effect to generate a target video, including:

[0101] Adjusting the panoramic video based on the field of view angle;

[0102] At the adjusted camera movement point of the panoramic video, a secondary adjustment is performed based on the camera movement effect to generate a target video.

[0103] Specifically, after determining the field of view angle of each panoramic image frame, the panoramic video can be adjusted according to the field of view angle, and then the panoramic video with the adjusted field of view angle can be adjusted again according to the camera movement point and the camera movement effect, so as to generate the target video.

[0104] In this embodiment, by using position information and / or posture information to calculate the speed, using the speed and speed threshold to determine the field of view angle, and using the field of view angle to adjust the panoramic video, it is possible to better follow the moving object or event, thereby improving visual coherence.

[0105] In one embodiment, Figure 4 As shown, the determining of the field of view angle of each panoramic image frame in the panoramic video based on the speed corresponding to each panoramic image frame and a preset speed threshold includes:

[0106] S402: In response to the speed corresponding to the panoramic image being greater than or equal to a preset speed threshold, determine a speed direction indicated by the speed corresponding to the panoramic image.

[0107] S404: Based on the speed direction, determine the field of view angle of the panoramic image whose speed is greater than or equal to a preset speed threshold.

[0108] The speed direction may be the moving direction of the panoramic camera during the process of shooting the panoramic video. The field of view angle may generally be the angle for viewing the panoramic image, that is, the angle of the center of the panoramic image.

[0109] Specifically, the speed corresponding to each frame of the panoramic image is first determined, and when the speed is greater than a preset speed threshold, the speed direction of the speed greater than the speed threshold is determined. Using the speed direction, the field of view angle of the panoramic direction where the speed is greater than the speed threshold is determined. The speed direction can usually be the center direction of the panoramic image, so the field of view angle determined using the speed direction can be the center of the panoramic image.

[0110] S406, in response to the speed corresponding to the panoramic image being less than a preset speed threshold, using the pointing attention model to identify a target local area in the panoramic image having a speed less than the preset speed threshold.

[0111] S408: Determine, based at least on the target local area, a field of view angle of the panoramic image having a speed less than a preset speed threshold.

[0112] Among them, the Pointer-Generator Network is a neural network model that combines the pointing mechanism and the generating mechanism. The target local area can be an object or an area of ​​interest in the panoramic image.

[0113] Specifically, when the speed corresponding to the panoramic image is less than a preset speed threshold, the pointing attention model can be used to identify the target local area in the panoramic image whose speed is less than the preset threshold. The field of view angle is set in the target local area. For example, the field of view angle can be determined in the target local area according to a preset rule, such as a fixed rule such as slow horizontal rotation.

[0114] In some exemplary embodiments, Figure 5 As shown, for example, if three panoramic images with a speed less than the speed threshold are identified, and the target local areas in the three panoramic images are all area A, then Figure 5 In the manner shown in a, the field of view angle is determined in the horizontal direction in the target local area. Figure 5 In the method shown in b, the field of view angle is determined in the vertical direction in the target local area.

[0115] In this embodiment, the viewing angles of the panoramic image are differentiated by speed, so that the motion scene of the current panoramic image can be identified, and different viewing angles can be determined according to different scenes.

[0116] In one embodiment, Figure 6 As shown, the panoramic video is adjusted based on the field of view angle, and a secondary adjustment is performed at the camera movement point of the adjusted panoramic video based on the camera movement effect to generate a target video, including:

[0117] S502, classifying each panoramic image frame in the panoramic video by using a field of view angle to obtain at least one group of panoramic classification images, wherein the panoramic images included in each group of the panoramic classification images have the same field of view angle.

[0118] Specifically, after the field of view angle is determined, panoramic images with the same field of view angle may be grouped together according to the field of view angle, or panoramic images with the same and continuous field of view angle may be grouped together. The panoramic images grouped together may be panoramic classification images.

[0119] S504, in response to the absence of the camera movement point in the panoramic classification image, adjusting the playback speed of the panoramic classification image without the camera movement point according to a preset speed multiple, and adjusting the panoramic classification image according to the field of view angle corresponding to the panoramic classification image without the camera movement point.

[0120] Specifically, when there is no camera movement in the panoramic classification image, the playback speed of the panoramic classification image without the camera movement point can be adjusted according to the speed multiple. The speed multiple can be 3 times or 5 times, and the specific speed multiple can be determined according to actual needs. Then, the field of view angle of any frame of the panoramic classification image can be determined, and the panoramic classification image can be adjusted according to the field of view angle at the same time.

[0121] S506: In response to the presence of the camera movement point in the panoramic classification image, the panoramic classification image is adjusted according to the field of view angle of the panoramic classification image where the camera movement point exists, and the panoramic classification image is adjusted according to the camera movement effect at the camera movement point.

[0122] Specifically, when there is a camera movement point in the panoramic classification image, each frame of the panoramic image in the panoramic classification image is adjusted according to the field of view angle of the panoramic classification image with the camera movement point, and adjustments are made at the camera movement point according to the camera movement effect.

[0123] S508: Generate a target video based on the adjusted panoramic classification image.

[0124] Specifically, the target video may be generated by combining the adjusted panoramic classification image with the camera movement point and the adjusted panoramic classification image without the camera movement point.

[0125] In this embodiment, the panoramic images are classified according to the viewing angle, and then different panoramic classified images are adjusted according to whether there is a camera movement point, which can improve the viewing effect of the target video.

[0126] In one embodiment, Figure 7 As shown, the method also includes:

[0127] S602, obtaining a panoramic training video.

[0128] S604: Determine the camera movement points in the panoramic training video, and determine the camera movement type and video type that match the panoramic training video.

[0129] S606: Based on the camera movement points in the panoramic training video, the camera movement type and the video type that match the panoramic training video, a neural network model is trained to obtain a camera movement type determination model.

[0130] The panoramic training video may generally be a panoramic video used in training. In some embodiments of the present disclosure, the neural network model may be a network model capable of analyzing and processing video content.

[0131] Specifically, a panoramic training video can be obtained, and the camera movement points in the panoramic training video can be determined in the manner mentioned in the above embodiment, or in other ways (such as manual setting). In this embodiment, there is no restriction on how the camera movement points are determined. Since the panoramic training video is determined, the camera movement type and video type of the panoramic training video can also be determined. Then, the camera movement points in the panoramic training video, the camera movement type and video type that match the panoramic training video are used as training data to train the neural network model, and the parameters in the neural network model are updated, thereby obtaining a camera movement type determination model.

[0132] In one embodiment, Figure 8 As shown, the step of determining the camera movement point in the process of shooting the panoramic video based on the movement trajectory map includes:

[0133] S702, based on the movement trajectory map, determining a turning point at which the panoramic camera turns at an angle greater than a preset angle during the process of shooting the panoramic video.

[0134] S704, determining the time interval between the turning points, and determining the turning point whose time interval is greater than a preset time as the camera movement point.

[0135] Specifically, whether turning indoors or outdoors, generally entering a new scene (such as a new road, a new room), and this scene switching generally requires the addition of special effects or cloud effects. For example, when a racing car turns, the lens should focus on presenting the close-up of the body drifting and the wheels rubbing on the ground, so it is necessary to identify the turning scene first. Even if some cameras are integrated with gyroscopes, due to the uncertainty of the direction and jitter factors during the travel process, the turning scene cannot be directly identified by gyroscope data. Therefore, in the moving trajectory map, a turning point where the angle of the moving trajectory is greater than the preset angle can be found. The moving trajectory can usually be the trajectory of a panoramic camera in a panoramic video. After determining the turning point, if there are continuous turning points between a certain time period, in order to ensure that the user's experience or freshness is not poor due to frequent addition of camera effects when watching the video, the time interval between the turning points can be determined first, and the turning point with a time interval greater than the preset time is determined as a camera point.

[0136] In some exemplary embodiments, for example, the preset angle may be 60°, and there are 6 turning points where the panoramic camera turns at an angle greater than 60°, namely c1, c2...c6, wherein the preset time may be 10s, and the turning points greater than the preset time may be c1, c3 and c5, and c1, c3 and c5 may be determined as camera movement points.

[0137] In this embodiment, the camera movement point can be accurately identified through the turning point, thereby improving the accuracy of the camera movement effect adjustment.

[0138] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0139] Based on the same inventive concept, the embodiment of the present disclosure also provides a video processing device for implementing the above-mentioned video processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more video processing device embodiments provided below can refer to the limitations on the video processing method above, and will not be repeated here.

[0140] In one embodiment, Fig. 9 As shown, a video processing device 800 is provided, including: a video acquisition module 802, a trajectory map determination module 804, a camera point determination module 806, a model processing module 808, a camera effect matching module 810 and a video generation module 812, wherein:

[0141] The video acquisition module 802 is used to acquire the panoramic video and the position information of the panoramic camera during the process of shooting the panoramic video;

[0142] A track map determination module 804 is used to construct a movement track map of the panoramic camera during the process of shooting the panoramic video based on the position information;

[0143] A camera movement point determination module 806, configured to determine the camera movement points during the process of shooting the panoramic video based on the movement trajectory map;

[0144] A model processing module 808, configured to input the camera movement point and the panoramic video into a pre-trained camera movement type determination model, and output the video type and camera movement type of the panoramic video via the camera movement type determination model;

[0145] A camera movement effect matching module 810, configured to match camera movement effects from a preset camera movement rule library based on the camera movement type and the video type;

[0146] The video generation module 812 is used to make adjustments based on the camera movement effect at the camera movement point of the panoramic video to generate a target video.

[0147] In one embodiment of the device, the device further comprises:

[0148] A posture information acquisition module, used to acquire the posture information of the panoramic camera during the process of shooting the panoramic video;

[0149] A speed calculation module, used for calculating the speed corresponding to each frame of the panoramic image obtained by the panoramic camera in the process of shooting the panoramic video based on the position information and / or posture information of the panoramic camera;

[0150] A viewing angle determination module, configured to determine the viewing angle of each panoramic image frame in the panoramic video based on a speed corresponding to each panoramic image frame and a preset speed threshold;

[0151] The video generation module 812 is further used to adjust the panoramic video based on the field of view angle, and perform secondary adjustment based on the camera movement effect at the camera movement point of the adjusted panoramic video to generate a target video.

[0152] In one embodiment of the device, the field of view angle determination module is also used to determine the speed direction indicated by the speed corresponding to the panoramic image in response to the speed corresponding to the panoramic image being greater than or equal to a preset speed threshold; based on the speed direction, determine the field of view angle of the panoramic image whose speed is greater than or equal to the preset speed threshold; in response to the speed corresponding to the panoramic image being less than the preset speed threshold, use the pointing attention model to identify a target local area in the panoramic image whose speed is less than the preset speed threshold; and determine the field of view angle of the panoramic image whose speed is less than the preset speed threshold at least based on the target local area.

[0153] In one embodiment of the device, the video generation module 812 is further used to classify each panoramic image frame in the panoramic video by using the field of view angle to obtain at least one group of panoramic classified images, and the panoramic images contained in each group of the panoramic classified images have the same field of view angle; in response to the absence of the camera movement point in the panoramic classified images, the playback speed of the panoramic classified images without the camera movement point is adjusted according to a preset speed multiple, and the panoramic classified images are adjusted according to the field of view angle corresponding to the panoramic classified images without the camera movement point; in response to the presence of the camera movement point in the panoramic classified images, the panoramic classified images are adjusted according to the field of view angle of the panoramic classified images with the camera movement point, and the panoramic classified images are adjusted according to the camera movement effect at the camera movement point; based on the adjusted panoramic classified images, a target video is generated.

[0154] In one embodiment of the device, the device further comprises:

[0155] The model training module is used to obtain a panoramic training video; determine the camera movement points in the panoramic training video, and determine the camera movement type and video type that match the panoramic training video; based on the camera movement points in the panoramic training video, the camera movement type and video type that match the panoramic training video, train a neural network model to obtain a camera movement type determination model.

[0156] In one embodiment of the device, the camera movement point determination module 806 is further used to determine, based on the movement trajectory map, a turning point at which the panoramic camera turns at an angle greater than a preset angle during the process of shooting the panoramic video; determine the time interval between the turning points, and determine the turning point at which the time interval is greater than the preset time as the camera movement point.

[0157] Each module in the above video processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0158] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.10As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as mobile trajectory maps and panoramic videos. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video processing method is implemented.

[0159] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present disclosure, and does not constitute a limitation on the computer device to which the scheme of the present disclosure is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0160] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in any of the above method embodiments when executing the computer program.

[0161] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0162] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in any of the above method embodiments when executed by a processor.

[0163] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided by the present disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided by the present disclosure may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided by the present disclosure may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, etc., but are not limited to this.

[0164] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0165] The above-described embodiments only express several implementation methods of the present disclosure, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present disclosure, and these all belong to the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the attached claims.

Claims

1. A video processing method, characterized in that: The method comprises: Acquire a panoramic video and position information of a panoramic camera during shooting of the panoramic video; Constructing a movement trajectory map of the panoramic camera during the process of shooting the panoramic video based on the position information; Determining the camera movement points in the process of shooting the panoramic video based on the movement trajectory map; determining the camera movement points in the process of shooting the panoramic video based on the movement trajectory map includes: According to the moving track of the panoramic camera in the moving track map, a turning point is selected from the moving track, or a midpoint of the track that has not turned within a preset time as a camera movement point, wherein the camera movement point is a time point in the panoramic video where the camera needs to be moved; Inputting the camera movement point and the panoramic video into a pre-trained camera movement type determination model, and outputting the video type and camera movement type of the panoramic video via the camera movement type determination model; Based on the camera movement type and the video type, matching the camera movement effect from a preset camera movement rule library; At the camera movement point of the panoramic video, adjustments are made based on the camera movement effect to generate a target video.

2. The method according to claim 1, characterized in that The method further comprises: Acquiring the posture information of the panoramic camera during the process of shooting the panoramic video; Calculating, based on the position information and / or posture information of the panoramic camera, a speed corresponding to each frame of the panoramic image obtained by the panoramic camera during the process of shooting the panoramic video; Determining a field of view angle of each panoramic image frame in the panoramic video based on a speed corresponding to each panoramic image frame and a preset speed threshold; Correspondingly, at the camera movement point of the panoramic video, adjustment is performed based on the camera movement effect to generate a target video, including: Based on the field of view angle, the panoramic video is adjusted, and at the camera movement point of the adjusted panoramic video, a secondary adjustment is performed based on the camera movement effect to generate a target video.

3. The method according to claim 2, characterized in that The determining the field of view angle of each panoramic image frame in the panoramic video based on the speed corresponding to each panoramic image frame and a preset speed threshold comprises: In response to the speed corresponding to the panoramic image being greater than or equal to a preset speed threshold, determining a speed direction indicated by the speed corresponding to the panoramic image; Based on the speed direction, determining a field of view angle of the panoramic image having a speed greater than or equal to a preset speed threshold; In response to a speed corresponding to the panoramic image being less than a preset speed threshold, identifying a target local area in the panoramic image having a speed less than the preset speed threshold using a pointing attention model; Based at least on the target local area, a field of view angle of the panoramic image having a speed less than a preset speed threshold is determined.

4. The method according to claim 2, characterized in that: The step of adjusting the panoramic video based on the field of view angle, and performing secondary adjustment based on the camera movement effect at the camera movement point of the adjusted panoramic video to generate a target video includes: Classifying each panoramic image frame in the panoramic video by using the field of view angle to obtain at least one group of panoramic classification images, wherein the panoramic images contained in each group of the panoramic classification images have the same field of view angle; In response to the absence of the camera movement point in the panoramic classification image, adjusting the playback speed of the panoramic classification image without the camera movement point according to a preset speed multiple, and adjusting the panoramic classification image according to the field of view angle corresponding to the panoramic classification image without the camera movement point; In response to the presence of the camera movement point in the panoramic classification image, adjusting the panoramic classification image according to the field of view angle of the panoramic classification image where the camera movement point exists, and adjusting the panoramic classification image according to the camera movement effect at the camera movement point; Based on the adjusted panoramic classification image, a target video is generated.

5. The method according to claim 1, characterized in that The method further comprises: Get panoramic training video; Determining camera movement points in the panoramic training video, and determining a camera movement type and a video type that match the panoramic training video; Based on the camera movement points in the panoramic training video, the camera movement type and the video type that match the panoramic training video, a neural network model is trained to obtain a camera movement type determination model.

6. The method according to claim 1, characterized in that The determining, based on the movement trajectory map, the camera movement point in the process of shooting the panoramic video includes: Based on the movement trajectory map, determining a turning point at which the panoramic camera turns at an angle greater than a preset angle during the process of shooting the panoramic video; The time interval between the turning points is determined, and the turning point whose time interval is greater than the preset time is determined as the camera movement point.

7. A video processing device, characterized in that: The device comprises: A video acquisition module, used to acquire panoramic video and position information of the panoramic camera during the process of shooting the panoramic video; A trajectory map determination module, used to construct a movement trajectory map of the panoramic camera during the process of shooting the panoramic video based on the position information; A camera movement point determination module, used to determine the camera movement points in the process of shooting the panoramic video based on the movement trajectory map; The camera movement point determination module is further used to select a turning point from the moving trajectory of the panoramic camera in the moving trajectory map, or a midpoint of the trajectory that has not turned within a preset time as a camera movement point, wherein the camera movement point is a time point in the panoramic video where the camera needs to be moved; A model processing module, used for inputting the camera movement point and the panoramic video into a pre-trained camera movement type determination model, and outputting the video type and camera movement type of the panoramic video via the camera movement type determination model; A camera movement effect matching module, used for matching camera movement effects from a preset camera movement rule library based on the camera movement type and video type; The video generation module is used to make adjustments based on the camera movement effect at the camera movement point of the panoramic video to generate a target video.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images