Video processing methods, apparatus, electronic devices and storage media

Abstract

This application discloses a video processing method, apparatus, electronic device, and storage medium. The method includes: a control device acquiring preview posture data corresponding to a user previewing a panoramic image and sending the preview posture data to a recording device; then acquiring panoramic image data sent by the recording device, the panoramic image data being generated by the recording device based on the recorded panoramic image data and the preview posture data; further extracting the preview posture data and panoramic image data from the panoramic image data; and finally playing the video corresponding to the panoramic image data based on the preview posture data. This enables the video to be played from the user's preview angle when playing the video corresponding to the panoramic image data.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a video processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] In existing technologies, after a drone captures panoramic video through a camera, it directly transmits the panoramic video to other electronic devices. After these devices acquire the panoramic video, since the panoramic video does not include the user's viewing angle during the recording process, the user needs to adjust the viewing angle manually to view the video from the corresponding angle. Summary of the Invention

[0003] This application provides a video processing method, apparatus, electronic device, and storage medium. The video processing method can play video corresponding to panoramic image data at the preview angle when a user previews a panoramic view.

[0004] In a first aspect, embodiments of this application provide a video processing method, including:

[0005] Acquire the preview posture data corresponding to the user's panoramic view and send the preview posture data to the recording device;

[0006] Acquire panoramic image data sent by the recording device. The panoramic image data is generated by the recording device based on the recorded panoramic image data and the preview posture data.

[0007] Extract preview pose data and panoramic image data from panoramic image data;

[0008] Play the video corresponding to the panoramic image data based on the preview posture data.

[0009] Secondly, embodiments of this application provide a video processing apparatus, including:

[0010] The sending module is used to obtain the preview posture data corresponding to the user's preview of the panoramic image and send the preview posture data to the recording device;

[0011] The acquisition module is used to acquire panoramic image data sent by the recording device. The panoramic image data is generated by the recording device based on the recorded panoramic image data and the preview posture data.

[0012] The extraction module is used to extract preview pose data and panoramic image data from panoramic image data;

[0013] The playback module is used to play the video corresponding to the panoramic image data based on the preview posture data.

[0014] Thirdly, embodiments of this application provide an electronic device, including a memory storing executable program code and a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute the steps in the video processing method or data processing method provided in embodiments of this application.

[0015] Fourthly, embodiments of this application provide a storage medium storing multiple instructions adapted for loading by a processor to execute steps in the video processing method or data processing method provided in embodiments of this application.

[0016] In this embodiment, the control device can acquire preview posture data corresponding to the user's panoramic view and send the preview posture data to the recording device; then, it acquires panoramic image data sent by the recording device, which is generated by the recording device based on the recorded panoramic image data and preview posture data; next, it extracts preview posture data and panoramic image data from the panoramic image data; finally, it plays the video corresponding to the panoramic image data based on the preview posture data. This enables the video to be played from the user's preview angle when playing the video corresponding to the panoramic image data. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of a data transmission scenario between a recording device and a control device provided in an embodiment of this application.

[0019] Figure 2 This is a schematic diagram of the three-dimensional space provided in the embodiments of this application.

[0020] Figure 3 This is a first flowchart illustrating the video processing method provided in the embodiments of this application.

[0021] Figure 4 This is a schematic diagram of the second process of the video processing method provided in the embodiments of this application.

[0022] Figure 5 This is a schematic diagram of the third process of the video processing method provided in the embodiments of this application.

[0023] Figure 6 This is a first flowchart illustrating the data processing method provided in the embodiments of this application.

[0024] Figure 7This is a schematic diagram of the structure of the video processing device provided in the embodiments of this application.

[0025] Figure 8 This is a schematic diagram of the structure of the data processing device provided in the embodiments of this application.

[0026] Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] In related technologies, after a drone captures panoramic video through a camera, it directly transmits the panoramic video to other electronic devices. After the other electronic devices acquire the panoramic video, since the panoramic video does not include the user's viewing angle during the recording process, the user needs to adjust the viewing angle manually to view the video from the corresponding angle.

[0029] To address the aforementioned technical problems, this application provides a video processing method, apparatus, electronic device, and storage medium. This video processing method can be applied to VR (Virtual Reality) glasses, as well as to various electronic devices such as smartphones and computers.

[0030] Please see Figure 1 , Figure 1 This is a schematic diagram of a data transmission scenario between a recording device and a control device provided in an embodiment of this application.

[0031] In this embodiment of the application, the recording device can record panoramic image data, the control device can preview the panoramic image recorded by the recording device, and the control device can also be used to play the panoramic image data recorded by the recording device.

[0032] like Figure 1 As shown, S1 is a control device and S2 is a recording device. The control device can be VR glasses, and the recording device can be a drone. The drone is equipped with a 360-degree panoramic camera, and the recording device can record panoramic images in a real three-dimensional space.

[0033] It should be noted that the control device can also be a mobile phone, computer, or other electronic device with computing capabilities. The recording device can also be an electronic device such as a panoramic camera. Figure 1The drones and VR glasses shown are merely examples and should not be considered as limitations on this application.

[0034] Please continue reading. Figure 1 In practical applications, once the control device and the recording device establish a communication connection, the recording device can transmit the recorded panoramic image to the control device. The user can then preview the panoramic image through the control device. The control device will obtain the user's preview posture data when the user previews the panoramic image and then send the preview posture data to the recording device.

[0035] For example, the panoramic view previewed by the user on the control device might be a low-resolution image. Similarly, the panoramic image data recorded by the recording device might correspond to the original panoramic view. However, if the recording device transmits this original panoramic view data to the control device, the large data size would cause significant latency during transmission, hindering real-time previewing for the user.

[0036] Therefore, the recording device can convert the acquired panoramic image data into lower-resolution preview panoramic image data, and then send the preview panoramic image data to the control device. After receiving the preview panoramic image data, the control device can preview the panoramic scene based on the preview panoramic image data, and send the preview data of the user previewing the panoramic scene to the recording device. For example, the control device can use sensors such as gyroscopes and accelerometers to obtain the preview posture data of the user when previewing the panoramic scene.

[0037] After receiving the preview posture data, the recording device will generate panoramic image data based on the recorded panoramic image data and the preview posture data, and then send the panoramic image data to the control device.

[0038] After receiving the panoramic image data, the control device extracts the panoramic image data to obtain preview posture data and panoramic image data. Finally, it plays the video corresponding to the panoramic image data based on the preview posture data.

[0039] The control device can determine the user's preview angle based on the preview posture data, and then play the video corresponding to the panoramic image data based on the preview angle.

[0040] For information regarding the content of the panoramic view previewed by the control equipment based on the preview panoramic image data, please refer to the following: Figure 2 , Figure 2 This is a schematic diagram of the three-dimensional space provided in the embodiments of this application.

[0041] The control device can first determine the spatial center point A2, and then establish a three-dimensional space A1. This three-dimensional space corresponds to the three-dimensional space where the recording device is located, and the position of the center point in this three-dimensional space corresponds to the position of the recording device in the actual three-dimensional space. During the process of establishing the three-dimensional space, the north direction obtained by the recording device through the magnetometer can be determined as the standard direction corresponding to the spatial center point, and then the three-dimensional space can be established using this standard direction.

[0042] This three-dimensional space is a virtual space. When the recording device transmits the preview panoramic image data to the control device, the control device can render the preview panoramic image data in the three-dimensional space to obtain the preview panoramic image.

[0043] When the preview panoramic image data transmitted by the recording device is a data stream, a panoramic video is rendered in three-dimensional space.

[0044] When a user previews a panoramic view using a control device, the device can use sensors to record the user's corresponding preview posture data. For example, the user's preview posture data can be determined by data recorded by sensors such as gyroscopes and accelerometers.

[0045] For example, when a user wears a control device, the device can acquire the user's preview posture data. For instance, if a user turns their head to look in a certain direction, the control device can record the preview posture data corresponding to that direction. This preview posture data can be Euler angle data. Euler angles are a set of three independent angular parameters used to uniquely determine the position of a rigid body rotating at a fixed point; they consist of the nutation angle θ, the precession angle ψ, and the rotation angle φ.

[0046] like Figure 2 As shown, during the rendering of a panoramic image or video, the control device can determine the corresponding viewing angle B1 and viewing distance C1. Under these viewing angles and distances, the device then identifies the user-previewed image within the panoramic image or video. This image is then displayed on the control device's display area. The viewing distance C1 is the distance between the spatial center point A2 and spatial point A3.

[0047] For a more detailed understanding of the video processing method provided in the embodiments of this application, please refer to [link / reference]. Figure 3 , Figure 3 This is a first flowchart illustrating the video processing method provided in this application embodiment. The video processing method may include the following steps:

[0048] 110. Obtain the preview posture data corresponding to the user's panoramic view and send the preview posture data to the recording device.

[0049] In some implementations, when a user uses a control device to preview the panoramic image sent by the recording device, the control device can record the user's preview posture data and send the preview posture data to the recording device.

[0050] For example, control devices can use gyroscopes, accelerometers, etc. to acquire the user's corresponding preview posture data. For instance, when the control device is worn or held by the user, the corresponding sensor data can be acquired through the sensor, and then the sensor data can be used to determine the user's preview posture data. For example, preview posture data includes Euler angle data, angular velocity data, acceleration data, etc.

[0051] For example, when a user uses a mouse or touch screen to swipe the screen to preview a panoramic view, the control device can convert the user's mouse movement information or finger swipe information into preview posture data corresponding to the user's preview of the panoramic view. For example, the mouse movement information or finger swipe information can be used to determine the direction selected by the user when previewing the panoramic view, and thus determine the direction as the preview posture data.

[0052] In some implementations, after the control device obtains the user's preview posture data, it can associate the preview posture data with a specific time, and then send the preview posture data after associating the time to the recording device.

[0053] 120. Acquire panoramic image data sent by the recording device. The panoramic image data is generated by the recording device based on the recorded panoramic image data and the preview posture data.

[0054] In some implementations, after the control device sends the preview posture data to the recording device, the recording device generates panoramic image data based on the recorded panoramic image data and the preview posture data. Panoramic image data can be understood as data containing both image data and posture data.

[0055] The control device can receive panoramic image data sent by the recording device via wireless communication.

[0056] 130. Extract preview pose data and panoramic image data from panoramic image data.

[0057] After acquiring panoramic image data, the control device can analyze the panoramic image data to obtain panoramic image data and preview posture data.

[0058] For example, if the panoramic image data and preview posture data are panoramic image data generated through a preset encoding format, the control device can use the corresponding decoding method to decode the panoramic image data to obtain the panoramic image data and preview posture data.

[0059] It should be noted that the control device can also determine the time information corresponding to the preview posture data and the time information corresponding to the panoramic image data, and then match the preview posture data and panoramic image data at the same time.

[0060] In some implementations, when the control device parses the panoramic image data, it can smooth the panoramic image data to filter out useless or noisy data.

[0061] 140. Play the video corresponding to the panoramic image data based on the preview posture data.

[0062] In some implementations, when a user needs to play a video corresponding to panoramic image data, the control device can first determine the starting point of the playback time, then determine the preview posture data and panoramic image data corresponding to the playback starting point based on the starting point of the time, and then render the image corresponding to the panoramic image data according to the preview posture data in chronological order, thereby rendering the image from the user's preview angle.

[0063] When an electronic device continuously displays multiple frames of images from different preview angles, the control device plays the video corresponding to the panoramic image data.

[0064] In other words, in this embodiment of the application, the control device can determine the user's preview angle based on the preview posture data, and then play the video corresponding to the panoramic image data based on the preview angle.

[0065] It should be noted that the device used to play the panoramic video data is not limited to control devices, but can also be other electronic devices with playback capabilities.

[0066] Please refer to the following: Figure 2 ,like Figure 2 As shown, when the preview posture data is Euler angle data, the control device can determine the user's preview angle in three-dimensional space based on the Euler angle data, and determine the image corresponding to the preview angle.

[0067] When the display area of ​​the control device shows the images corresponding to multiple preview angles, it enables the playback of the video corresponding to the panoramic image data at the user's preview angle.

[0068] In this embodiment, the control device can acquire preview posture data corresponding to the user's panoramic view and send the preview posture data to the recording device; then, it acquires panoramic image data sent by the recording device, which is generated by the recording device based on the recorded panoramic image data and preview posture data; next, it extracts preview posture data and panoramic image data from the panoramic image data; finally, it plays the video corresponding to the panoramic image data based on the preview posture data. This enables the video to be played from the user's preview angle when playing the video corresponding to the panoramic image data.

[0069] Please continue reading. Figure 4 , Figure 4 This is a second flowchart illustrating the video processing method provided in this application embodiment. The video processing method may include the following steps:

[0070] 201. Obtain preview posture data.

[0071] For example, control devices can use gyroscopes, accelerometers, etc. to acquire the user's corresponding preview posture data. For instance, when the control device is worn or held by the user, the corresponding sensor data can be acquired through the sensor, and then the sensor data can be used to determine the user's preview posture data. For example, preview posture data includes Euler angle data, angular velocity data, acceleration data, etc.

[0072] For example, when a user uses a mouse or touch screen to swipe the screen to preview a panoramic view, the control device can convert the user's mouse movement information or finger swipe information into preview posture data corresponding to the user's preview of the panoramic view. For example, the mouse movement information or finger swipe information can be used to determine the direction selected by the user when previewing the panoramic view, and thus determine the direction as the preview posture data.

[0073] 202. Determine the difference between the first preview pose data corresponding to the current video frame and the second preview pose data corresponding to the previous video frame.

[0074] In some implementations, the control device can determine the first preview pose data corresponding to the current video frame and the second preview pose data corresponding to the previous video frame, and then determine the difference between the first preview pose data corresponding to the current video frame and the second preview pose data corresponding to the previous video frame.

[0075] When the preview attitude data is Euler angle data, then the first preview attitude data is the first Euler angle data, and the second attitude data is the second Euler angle data.

[0076] Specifically, the control device converts the first Euler angle data of the current video frame into a first spatial vector, converts the second Euler angle data of the previous video frame into a second spatial vector, and finally determines the difference between the first and second spatial vectors. This difference between the first and second spatial vectors is the difference between the first preview pose data corresponding to the current video frame and the second preview pose data corresponding to the previous video frame.

[0077] 203. If the difference is within the preset difference range, then stop sending the first preview posture data to the recording device.

[0078] In some implementations, the control device can set a preset difference range. When the difference between the first preview posture data corresponding to the current video frame and the second preview posture data corresponding to the previous video frame is within the preset difference range, it is considered that the user's preview posture change is very small, such as the user's head change is very small.

[0079] For example, if the preset difference range is 0 to 5 degrees, and the difference between the first preview posture data corresponding to the current video frame and the second preview posture data corresponding to the previous video frame is 4 degrees, then it is determined that the difference between the first preview posture data corresponding to the current video frame and the second preview posture data corresponding to the previous video frame is within the preset difference range.

[0080] At this point, the control device can stop sending preview data of the current video frame to the recording device, reducing data transmission to the recording device. This results in a smaller amount of panoramic image data generated by the recording device based on the preview pose data and panoramic image data. It also reduces the transmission of useless data, improving the data transmission efficiency between the data control device and the recording device.

[0081] 204. If the difference is outside the preset difference range, the first preview posture data will be sent to the recording device.

[0082] In some implementations, if the difference is outside the preset difference range, it indicates that the user's preview posture has changed significantly compared to the previous video frame. In this case, the first preview posture data can be sent to the recording device.

[0083] 205. Obtain panoramic image data sent by the recording device based on the preview posture data.

[0084] In some implementations, after the recording device receives the first preview pose data, the recording device can determine the timestamp corresponding to the first preview pose data, and then associate it with panoramic image data at the same time based on the timestamp.

[0085] The recording device can associate multiple frames of preview pose data with their corresponding panoramic image data to generate panoramic image data.

[0086] For example, the preview pose data corresponding to time 10:01 is associated with its corresponding panoramic image data, and the preview pose data corresponding to time 10:02 is associated with its corresponding panoramic image data. By associating the preview pose data and the panoramic image data in this way, panoramic image data is obtained.

[0087] The control device can receive panoramic image data sent by the recording device via wireless communication.

[0088] 206. Obtain the edited posture data generated after editing the preview posture data.

[0089] In some implementations, after the control device receives panoramic image data, the user can edit the preview pose data in the panoramic image data to generate edited pose data. For example, after selecting a keyframe, the user can edit the preview pose data corresponding to the keyframe to obtain edited pose data.

[0090] For example, the first keyframe corresponds to the first target preview pose data. After the user edits the first target preview pose data, the first edited pose data corresponding to the first keyframe is obtained. The second keyframe corresponds to the second target preview pose data. After the user edits the second target preview pose data, the second edited pose data corresponding to the second keyframe is obtained.

[0091] In some implementations, users can also change other parameters of the panoramic image data, such as changing the viewing angle corresponding to the keyframe after the user selects a keyframe.

[0092] 207. Play the video corresponding to the panoramic image data based on the edited posture data and the preview posture data.

[0093] In some implementations, the control device can determine the preview angle of the image corresponding to the key frame using the edited pose data corresponding to the key frame, determine the preview angle of the image corresponding to other frames in the panoramic image data using the preview pose data, and finally play the image corresponding to the key frame using the preview angle of the image corresponding to the key frame, and play the image corresponding to other frames using the preview angle of the image corresponding to other frames.

[0094] In some implementations, the control device can also smooth the posture data corresponding to each frame of panoramic image data in the panoramic image data using edited posture data and previewed posture data, and then play the corresponding picture of each frame of panoramic image data in the panoramic image data according to the smoothed posture data.

[0095] This enables the playback of videos corresponding to panoramic image data based on edited pose data and preview pose data.

[0096] To learn more about the video playback of panoramic image data based on edited pose data and preview pose data in this embodiment, please continue reading. Figure 5 , Figure 5 This is a schematic diagram of the third process of the video processing method provided in this application embodiment. The video processing method may further include the following steps:

[0097] 301. Determine the first keyframe corresponding to the panoramic image data and the first target preview pose data corresponding to the first keyframe.

[0098] In some implementations, taking a user-edited first keyframe as an example, the control device determines the first keyframe of the panoramic image data edited by the user and the first target preview pose data corresponding to the first keyframe, wherein the first target preview pose data is data extracted from the panoramic image data by the control device. The first keyframe is the keyframe corresponding to the panoramic image data selected by the user in the panoramic image data.

[0099] 302. Obtain the first edited posture data generated after editing the preview posture data of the first target.

[0100] For example, if the preview pose data is Euler angle data, after the user determines the first keyframe, the Euler angle data of the first target corresponding to the first keyframe is edited to obtain the edited Euler angle data corresponding to the first keyframe.

[0101] 303. Determine whether there is a second keyframe within the preset time range corresponding to the first keyframe.

[0102] For example, the preset time range corresponding to the first keyframe is the five seconds before the first keyframe and the five seconds after the first keyframe. The control device can determine whether there is a second keyframe that the user has selected to edit within this preset time range.

[0103] If a second keyframe exists within the preset time range corresponding to the first keyframe, proceed to step 304. If no second keyframe exists within the preset time range corresponding to the first keyframe, proceed to step 306.

[0104] 304. If there is a second keyframe within the preset time range corresponding to the first keyframe, then obtain the second edited pose data corresponding to the second keyframe.

[0105] For example, if there is a second keyframe within the preset time range corresponding to the first keyframe, the control device can determine the second target preview posture data corresponding to the second keyframe, and then obtain the second edited posture data generated after the user edits the second target preview posture data.

[0106] 305. Play the video corresponding to the panoramic image data between the first keyframe and the second keyframe based on the first edited pose data and the second edited pose data, and play the video corresponding to the panoramic image data outside the preset time range based on the preview pose data.

[0107] Specifically, the control device can perform interpolation calculations on the pose data corresponding to each frame of panoramic image data between the first keyframe and the second keyframe based on the first edited pose data and the second edited pose data, to obtain the interpolated pose data corresponding to each frame of panoramic image data between the first keyframe and the second keyframe; then, it can play the video corresponding to the panoramic image data between the first keyframe and the second keyframe based on the interpolated pose data.

[0108] For example, if there are three frames between the first and second keyframes, these three frames are designated as frame A, frame B, and frame C in chronological order. The control device can calculate the attitude data corresponding to frames A, B, and C based on the first and second edited attitude data, thereby obtaining the interpolated attitude data corresponding to frames A, B, and C.

[0109] Then, based on the interpolated pose data, the viewing angles corresponding to frames A, B, and C are determined. Finally, the images corresponding to frames A, B, and C are played according to these viewing angles. This achieves the playback of video corresponding to the panoramic image data between the first and second keyframes based on the interpolated pose data.

[0110] In some implementations, panoramic image data outside a preset time range can be used to play corresponding videos by extracting preview pose data from the panoramic image data.

[0111] For example, panoramic image data up to 5 seconds before the first keyframe can be used to play the corresponding video using the preview pose data.

[0112] By using the above method, the attitude data of the entire panoramic image data can be smoothly processed, so that when playing the video corresponding to the panoramic image data, there will be a smoother playback effect with a smoother transition of viewing angles, without any stuttering between video frames.

[0113] 306. If there is no second keyframe within the preset time range corresponding to the first keyframe, then determine the first video frame corresponding to the start time and the second video frame corresponding to the end time within the preset time range.

[0114] For example, the preset time range corresponding to the first keyframe is the five seconds before the first keyframe and the five seconds after the first keyframe. If there is no second keyframe within the preset time range corresponding to the first keyframe, the control device determines the first video frame corresponding to the five seconds before the first keyframe and the second video frame corresponding to the five seconds after the first keyframe.

[0115] 307. Based on the preview pose data corresponding to the first video frame and the preview pose data corresponding to the second video frame, perform interpolation calculation on the pose data corresponding to the first key frame to obtain the first interpolated pose data corresponding to the first key frame.

[0116] For example, the control device first obtains the preview pose data corresponding to the first video frame and the preview pose data corresponding to the second video frame from the panoramic image data. Then, it performs interpolation calculation on the pose data corresponding to the first key frame using the preview pose data corresponding to the first video frame and the preview pose data corresponding to the second video frame, thereby calculating the first interpolated pose data corresponding to the first key frame.

[0117] The first edited pose data edited by the user on the first keyframe is not used.

[0118] 308. Display the image corresponding to the first keyframe based on the first interpolated pose data and the panoramic image data corresponding to the first keyframe.

[0119] For example, the control device determines the viewing angle corresponding to the first keyframe by using the first interpolated attitude data, then determines the panoramic image data corresponding to the first keyframe in the panoramic image data, and finally displays the image of the panoramic image data corresponding to the first keyframe by using the viewing angle corresponding to the first keyframe.

[0120] In this embodiment, after the control device acquires the panoramic image data, it can play the video corresponding to the panoramic image data based on the preview posture data in the panoramic image data, using the angle at which the user previewed the panoramic view. If the user edits the preview posture data in the panoramic image data, the video corresponding to the panoramic image data can be played together based on the edited posture data and the preview posture data.

[0121] In some implementations, if it is necessary to play the edited panoramic image data on other devices, the file format of the edited panoramic image data can be converted to a common format, such as mp4. Then, when other devices play the edited panoramic image data, they can directly play the video corresponding to the panoramic image data by using the angles corresponding to the edited posture data and the preview posture data.

[0122] In this embodiment, the control device acquires preview pose data, then determines the difference between the first preview pose data corresponding to the current video frame and the second preview pose data corresponding to the previous video frame. If the difference is within a preset range, the device stops sending the first preview pose data to the recording device. If the difference is outside the preset range, the device sends the first preview pose data to the recording device, then acquires the panoramic image data sent by the recording device based on the preview pose data, acquires the edited pose data generated after editing the preview pose data, and finally plays the video corresponding to the panoramic image data based on the edited pose data and the preview pose data. Thus, the video corresponding to the panoramic image data is played using the angles corresponding to the edited pose data and the preview pose data.

[0123] Please see Figure 6 , Figure 6 This is a first flowchart illustrating a data processing method provided in an embodiment of this application. The data processing method is for a recording device and may include the following steps:

[0124] 410. Collect panoramic image data.

[0125] The recording device is equipped with a 360-degree panoramic camera, which captures panoramic image data, which is raw data.

[0126] The recording device can convert panoramic image data into lower-resolution preview panoramic image data, and then send the preview panoramic image data to the control device, so that the user can preview the panoramic scene corresponding to the preview panoramic image data through the control device.

[0127] 420. Receive preview attitude data sent by the control device.

[0128] In some implementations, the recording device can also acquire preview posture data sent by the control device, which contains corresponding time information. For example, preview posture data at different times may correspond to a timestamp.

[0129] 430. Associate panoramic image data and preview pose data from the same time period to generate panoramic image data.

[0130] In some implementations, the recording device can associate panoramic image data and preview pose data at the same time based on the time information of the preview pose data and the time information of the panoramic image data to generate an associated file; the associated file is then placed at the end of the panoramic image data to generate panoramic image data.

[0131] 440. Send the panoramic image data to the control device.

[0132] The recording device can transmit panoramic image data to the control device via wireless communication.

[0133] In this embodiment, the recording device acquires panoramic image data, receives preview posture data sent by the control device, associates the panoramic image data and preview posture data from the same time period to generate panoramic image data, and finally sends the panoramic image data to the control device. This achieves the association between the user's preview posture data and the panoramic image data.

[0134] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of the video processing apparatus provided in an embodiment of this application. The video processing apparatus 500 may include:

[0135] The sending module 510 is used to acquire the preview posture data corresponding to the user's preview of the panoramic image and send the preview posture data to the recording device.

[0136] The sending module 510 is also used to determine the difference between the first preview posture data corresponding to the current video frame and the second preview posture data corresponding to the previous video frame;

[0137] If the difference is outside the preset difference range, the first preview posture data will be sent to the recording device.

[0138] If the difference is within the preset difference range, then stop sending the first preview posture data to the recording device.

[0139] The sending module 510 is also used to convert the first Euler angle data of the current video frame into a first spatial vector and convert the second Euler angle data of the previous video frame into a second spatial vector;

[0140] Determine the difference between the first spatial vector and the second spatial vector.

[0141] The acquisition module 520 is used to acquire panoramic image data sent by the recording device. The panoramic image data is generated by the recording device based on the recorded panoramic image data and the preview posture data.

[0142] Extraction module 530 is used to extract preview pose data and panoramic image data from panoramic image data.

[0143] The playback module 540 is used to play the video corresponding to the panoramic image data based on the preview posture data.

[0144] The playback module 540 is also used to acquire edited posture data generated after editing the preview posture data; and to play the video corresponding to the panoramic image data based on the edited posture data and the preview posture data.

[0145] The playback module 540 is also used to determine the first keyframe corresponding to the panoramic image data and the first target preview pose data corresponding to the first keyframe;

[0146] Obtain the first edited pose data generated after editing the preview pose data of the first target.

[0147] The playback module 540 is also used to determine whether there is a second keyframe within the preset time range corresponding to the first keyframe;

[0148] If there is a second keyframe within the preset time range corresponding to the first keyframe, then obtain the second edited pose data corresponding to the second keyframe.

[0149] Based on the first edited pose data and the second edited pose data, play the video corresponding to the panoramic image data between the first keyframe and the second keyframe; based on the preview pose data, play the video corresponding to the panoramic image data outside the preset time range.

[0150] The playback module 540 is also used to perform interpolation calculation on the posture data corresponding to each frame of panoramic image data between the first keyframe and the second keyframe based on the first edited posture data and the second edited posture data, so as to obtain the interpolated posture data corresponding to each frame of panoramic image data between the first keyframe and the second keyframe.

[0151] Play the video corresponding to the panoramic image data between the first and second keyframes based on the interpolated pose data.

[0152] After determining whether there is a second keyframe within the preset time range corresponding to the first keyframe, the playback module 540 is further configured to determine the first video frame corresponding to the start time and the second video frame corresponding to the end time within the preset time range if there is no second keyframe within the preset time range corresponding to the first keyframe.

[0153] Based on the preview pose data corresponding to the first video frame and the preview pose data corresponding to the second video frame, interpolation calculation is performed on the pose data of the first key frame to obtain the first interpolated pose data corresponding to the first key frame.

[0154] The panoramic image data corresponding to the first keyframe is displayed based on the first interpolated pose data and the panoramic image data corresponding to the first keyframe.

[0155] In this embodiment, the control device can acquire preview posture data corresponding to the user's panoramic view and send the preview posture data to the recording device; then, it acquires panoramic image data sent by the recording device, which is generated by the recording device based on the recorded panoramic image data and preview posture data; next, it extracts preview posture data and panoramic image data from the panoramic image data; finally, it plays the video corresponding to the panoramic image data based on the preview posture data. This enables the video to be played from the user's preview angle when playing the video corresponding to the panoramic image data.

[0156] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of the data processing apparatus provided in an embodiment of this application. The data processing apparatus 600 may include:

[0157] The acquisition module 610 is used to acquire panoramic image data.

[0158] The receiving module 620 is used to receive preview attitude data sent by the control device.

[0159] The association module 630 is used to associate panoramic image data and preview pose data at the same time to generate panoramic image data.

[0160] The association module 630 is also used to associate the initial panoramic image data and the preview pose data at the same time to generate an association file;

[0161] Add the associated file to the end of the panoramic image data to generate panoramic image data.

[0162] The data transmission module 640 is used to send panoramic image data to the control device.

[0163] In this embodiment, the recording device acquires panoramic image data, receives preview posture data sent by the control device, associates panoramic image data and preview posture data from the same time period to generate panoramic image data, and finally sends the panoramic image data to the control device. This achieves the association between the user's preview posture data and the initial panoramic image data.

[0164] Accordingly, embodiments of this application also provide an electronic device, such as... Figure 9 As shown, the electronic device may include a memory 701 with one or more computer-readable storage media, an input unit 702, a display unit 703, a sensor 704, a processor 705 including one or more processing cores, and a power supply 706, among other components. Those skilled in the art will understand that... Figure 9The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0165] The memory 701 can be used to store software programs and modules. The processor 705 executes various functional applications and data processing by running the software programs and modules stored in the memory 701. The memory 701 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device (such as audio data, telephone directory, etc.). In addition, the memory 701 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 701 may also include a memory controller to provide access to the memory 701 for the processor 705 and the input unit 702.

[0166] Input unit 702 can be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, in one embodiment, input unit 702 may include a touch-sensitive surface and other input devices. A touch-sensitive surface, also known as a touch display or touchpad, can collect user touch operations on or near it (e.g., user operations using fingers, styluses, or any suitable object or accessory on or near the touch-sensitive surface) and drive corresponding connection devices according to a pre-set program. Optionally, the touch-sensitive surface may include a touch detection device and a touch controller. The touch detection device detects the user's touch orientation and the signal generated by the touch operation, transmitting the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 705, and can receive and execute commands from the processor 705. Furthermore, various types of touch-sensitive surfaces, such as resistive, capacitive, infrared, and surface acoustic wave, can be used. In addition to the touch-sensitive surface, input unit 702 may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0167] Display unit 703 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of electronic devices. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Display unit 703 may include a display panel, optionally configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar form. Furthermore, a touch-sensitive surface may cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it transmits the information to processor 705 to determine the type of touch event. Subsequently, processor 705 provides corresponding visual output on the display panel according to the type of touch event. Although in Figure 9 In this context, the touch-sensitive surface and the display panel are two separate components for implementing input and output functions. However, in some embodiments, the touch-sensitive surface and the display panel can be integrated to achieve both input and output functions.

[0168] Electronic devices may also include at least one sensor 704, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel according to the ambient light level, and the proximity sensor can turn off the display panel and / or backlight when the electronic device is moved to the ear. As a type of motion sensor, a gravity acceleration sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that identify the posture of electronic devices (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometers, taps), etc. Other sensors that may be configured in electronic devices, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0169] The processor 705 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 701, and by calling data stored in the memory 701, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. Optionally, the processor 705 may include one or more processing cores; preferably, the processor 705 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 705.

[0170] The electronic device also includes a power supply 706 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 705 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 706 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0171] Although not shown, the electronic device may also include a camera, Bluetooth module, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 705 in the electronic device loads the computer program stored on the memory 701. By loading the computer program, the processor 705 realizes various functions:

[0172] Acquire the preview posture data corresponding to the user's panoramic view and send the preview posture data to the recording device;

[0173] Acquire panoramic image data sent by the recording device. The panoramic image data is generated by the recording device based on the recorded panoramic image data and the preview posture data.

[0174] Extract preview pose data and panoramic image data from panoramic image data;

[0175] Play the video corresponding to the panoramic image data based on the preview posture data.

[0176] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0177] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the video processing methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0178] Acquire the preview posture data corresponding to the user's panoramic view and send the preview posture data to the recording device;

[0179] Acquire panoramic image data sent by the recording device. The panoramic image data is generated by the recording device based on the recorded panoramic image data and the preview posture data.

[0180] Extract preview pose data and panoramic image data from panoramic image data;

[0181] Play the video corresponding to the panoramic image data based on the preview posture data.

[0182] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0183] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0184] Since the instructions stored in the storage medium can execute the steps of any of the video processing methods provided in the embodiments of this application, the beneficial effects that any of the video processing methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0185] The foregoing has provided a detailed description of a video processing method, apparatus, electronic device, and storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A video processing method, characterized in that, include: Acquire the preview posture data corresponding to the user's preview of the panoramic image, and send the preview posture data to the recording device; Acquire panoramic image data sent by the recording device, wherein the panoramic image data is generated by the recording device based on the recorded panoramic image data and the preview pose data; Extract the preview pose data and the panoramic image data from the panoramic image data; Play the video corresponding to the panoramic image data based on the preview posture data.

2. The video processing method according to claim 1, characterized in that, The step of acquiring the preview posture data corresponding to the user's panoramic preview and sending the preview posture data to the recording device includes: Determine the difference between the first preview pose data corresponding to the current video frame and the second preview pose data corresponding to the previous video frame; If the difference is outside the preset difference range, the first preview posture data is sent to the recording device; If the difference is within the preset difference range, then the transmission of the first preview posture data to the recording device is stopped.

3. The video processing method according to claim 2, characterized in that, The preview pose data includes Euler angle data, and determining the difference between the first preview pose data corresponding to the current video frame and the second preview pose data corresponding to the previous video frame includes: Convert the first Euler angle data of the current video frame into a first spatial vector, and convert the second Euler angle data of the previous video frame into a second spatial vector; Determine the difference between the first spatial vector and the second spatial vector.

4. The video processing method according to claim 1, characterized in that, Playing the video corresponding to the panoramic image data based on the preview pose data includes: Obtain the edited pose data generated after editing the preview pose data; Play the video corresponding to the panoramic image data based on the edited pose data and the preview pose data.

5. The video processing method according to claim 4, characterized in that, The step of obtaining the edited pose data generated after editing the preview pose data includes: Determine the first keyframe corresponding to the panoramic image data and the first target preview pose data corresponding to the first keyframe; Obtain the first edited posture data generated after editing the preview posture data of the first target.

6. The video processing method according to claim 5, characterized in that, Playing the video corresponding to the panoramic image data based on the edited pose data and the preview pose data includes: Determine whether there is a second keyframe within the preset time range corresponding to the first keyframe; If there is a second keyframe within the preset time range corresponding to the first keyframe, then the second edited pose data corresponding to the second keyframe is obtained; Play the video corresponding to the panoramic image data between the first keyframe and the second keyframe based on the first edited pose data and the second edited pose data, and play the video corresponding to the panoramic image data outside the preset time range based on the preview pose data.

7. The video processing method according to claim 6, characterized in that, Playing the video corresponding to the panoramic image data between the first keyframe and the second keyframe based on the first edited pose data and the second edited pose data includes: Based on the first edited pose data and the second edited pose data, interpolation calculation is performed on the pose data corresponding to each frame of panoramic image data between the first keyframe and the second keyframe to obtain the interpolated pose data corresponding to each frame of panoramic image data between the first keyframe and the second keyframe. Play the video corresponding to the panoramic image data between the first keyframe and the second keyframe based on the interpolated pose data.

8. The video processing method according to claim 6, characterized in that, After determining whether there is a second keyframe within the preset time range corresponding to the first keyframe, the method further includes: If there is no second keyframe within the preset time range corresponding to the first keyframe, then the first video frame corresponding to the start time and the second video frame corresponding to the end time within the preset time range are determined. Based on the preview pose data corresponding to the first video frame and the preview pose data corresponding to the second video frame, interpolation calculation is performed on the pose data of the first key frame to obtain the first interpolated pose data corresponding to the first key frame. The panoramic image data corresponding to the first keyframe is displayed based on the first interpolated pose data and the panoramic image data corresponding to the first keyframe.

9. A video processing apparatus, characterized in that, include: The sending module is used to acquire the preview posture data corresponding to the user's preview of the panoramic image, and send the preview posture data to the recording device; The acquisition module is used to acquire panoramic image data sent by the recording device, wherein the panoramic image data is generated by the recording device based on the recorded panoramic image data and the preview posture data; The extraction module is used to extract the preview pose data and the panoramic image data from the panoramic image data; The playback module is used to play the video corresponding to the panoramic image data based on the preview posture data.

10. An electronic device, characterized in that, include: A memory storing executable program code, and a processor coupled to the memory; The processor calls the executable program code stored in the memory to perform the steps in the video processing method as described in any one of claims 1 to 8.

11. A storage medium, characterized in that, The storage medium stores a plurality of instructions adapted for loading by a processor to execute the steps of the video processing method according to any one of claims 1 to 10.

Images