A video playing method, device, equipment and storage medium

Abstract

Embodiments of the present application provide a video playing method and device, equipment and storage medium, and relate to the technical field of multimedia. The method comprises: collecting head movement data of a viewing object by a wearable device, then mapping the head movement data to picture offset information under a video picture visual angle by coordinate system conversion, and then performing picture compensation on original video frames to be played on a smart device by using the picture offset information to obtain target video frames. The wearable device and the smart device cooperate to perform anti-shake processing on the video picture, effectively compensate for video jitter caused by head shaking of the viewing object, and finally output a smooth, stable and clear video stream, thereby improving the video viewing experience. The head movement data collected by the wearable device directly reflects the change of the head visual angle of the viewing object, so that when the picture offset information obtained by mapping the head movement data is used for video picture compensation, the accuracy of the picture compensation can be effectively improved, and the anti-shake processing effect of the video picture is improved.

Description

Technical Field

[0001] This application relates to the field of multimedia technology, and in particular to a video playback method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of computer and internet technologies, various applications with video playback functions have emerged one after another, and correspondingly, viewers have increasingly higher requirements for the video viewing experience.

[0003] In practical applications, when a device is playing video, the video may shake due to device movement, resulting in a blurry video for the viewer. To address this, related technologies utilize internal motion sensors to detect the device's own movement. Based on this movement, the video playback is compensated accordingly; for example, if the device is detected moving to the right, the video image is shifted to the left relative to the device's screen to display a stable video image.

[0004] However, in some scenarios, video shakiness is not caused by device movement, but by the movement of the viewer. In such cases, the viewer will still see a blurry video, thus affecting the viewing experience.

[0005] For example, imagine placing a smartphone on a treadmill and playing a video on it. A viewer watches the video while running; the smartphone remains stationary and therefore doesn't correct the video image. However, to the viewer, the video appears shaky, blurry, and thus negatively impacts the viewing experience.

[0006] Therefore, a new video playback method needs to be designed to overcome the above-mentioned shortcomings. Summary of the Invention

[0007] This application provides a video playback method, apparatus, device, and storage medium for stabilizing video images in scenarios where the viewed object is moving, thereby improving the video viewing experience.

[0008] On one hand, embodiments of this application provide a video playback method, the method comprising:

[0009] The system acquires the original video frame to be played on the smart device, and acquires the head motion data of the viewing object of the original video frame; wherein the head motion data is collected from the wearable device used by the viewing object, and the wearable device is equipped with a device coordinate system.

[0010] Based on the head motion data, the head offset information of the viewing object in the device coordinate system is obtained;

[0011] The head offset information is mapped from the device coordinate system to the screen coordinate system used by the smart device to obtain the screen offset information of the smart device;

[0012] Using the aforementioned image offset information, image compensation is performed on the original video frame to obtain and display the corresponding target video frame.

[0013] On one hand, embodiments of this application provide a video playback device, the device comprising:

[0014] The acquisition module is used to acquire the original video frame to be played on the smart device, and to acquire the head motion data of the viewing object of the original video frame; wherein, the head motion data is collected from the wearable device used by the viewing object, and the wearable device is equipped with a device coordinate system;

[0015] The conversion module is used to obtain the head offset information of the viewing object in the device coordinate system based on the head motion data;

[0016] The mapping module is used to map the head offset information from the device coordinate system to the screen coordinate system used by the smart device to obtain the screen offset information of the smart device.

[0017] The compensation module is used to perform image compensation on the original video frame using the image offset information, so as to obtain and display the corresponding target video frame.

[0018] Optionally, the mapping module is specifically used for:

[0019] Based on the screen resolution of the smart device and a preset head offset threshold, the scaling coefficient between the device coordinate system and the screen coordinate system is obtained.

[0020] Based on the aforementioned proportional mapping coefficient, the head offset information is mapped to the screen offset information of the smart device.

[0021] Optionally, the head offset information includes: pitch angle and yaw angle; the image offset information includes: horizontal offset and vertical offset;

[0022] Optionally, the mapping module is specifically used for:

[0023] The product of the scaling factor and the pitch angle is used as the vertical offset.

[0024] The product of the scaling factor and the yaw angle is used as the horizontal offset.

[0025] Optionally, the image offset information includes: horizontal offset and vertical offset;

[0026] The compensation module is specifically used for:

[0027] The original video frame is compensated horizontally using the horizontal offset, and the original video frame is compensated vertically using the vertical offset, so as to obtain and display the corresponding target video frame.

[0028] Optionally, the compensation module is specifically used for:

[0029] Based on the horizontal and vertical offsets, construct a coordinate transformation matrix;

[0030] For multiple pixels in the original video frame, the following steps are performed: the original coordinates of a pixel are transformed using the coordinate transformation matrix to obtain the target coordinates of the pixel;

[0031] Based on the target coordinates of each of the multiple pixels, the target video frame is obtained and displayed;

[0032] Optionally, the compensation module is specifically used for:

[0033] The original video frame is enlarged according to a preset ratio to obtain an intermediate video frame;

[0034] Using the aforementioned image offset information, image compensation is performed on the intermediate video frames to obtain and display the corresponding target video frames.

[0035] Optionally, the head motion data includes: head angular velocity; the head offset information includes: head offset angle;

[0036] The conversion module is specifically used for:

[0037] The time interval for acquiring head movement data by the wearable device;

[0038] Based on the head angular velocity and the time interval, the head offset angle of the viewing object in the device coordinate system is obtained.

[0039] Optionally, the acquisition module is further configured to:

[0040] Before acquiring the original video frame to be played on the smart device and acquiring the head motion data of the viewing object of the original video frame, in response to the mode selection operation triggered in the video application, the video playback mode of the video application is set to anti-shake mode, the video application runs on the smart device, and is used to render and display the target video frame.

[0041] Optionally, the acquisition module is further configured to:

[0042] In response to a mode switching operation triggered in the video application, the video playback mode of the video application is switched to non-stabilized mode;

[0043] The original video frames are rendered and displayed in the video application.

[0044] On one hand, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the aforementioned video playback method, including:

[0045] The processor acquires the original video frames to be played on the smart device, as well as the head motion data of the viewing object in the original video frames; the head motion data is collected from the wearable device used by the viewing object and stored in the memory, and the wearable device has a device coordinate system set up.

[0046] The processor obtains the head offset information of the viewed object in the device coordinate system based on head motion data;

[0047] The processor maps the head offset information from the device coordinate system to the screen coordinate system used by the smart device to obtain the screen offset information of the smart device.

[0048] The processor uses the image offset information to perform image compensation on the original video frames, and obtains and displays the corresponding target video frames.

[0049] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the video playback method described above.

[0050] On one hand, embodiments of this application provide a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the video playback method described above.

[0051] In this embodiment, head motion data of the viewer is collected by a wearable device. Then, the head motion data is mapped to the image offset information under the video viewpoint through coordinate system transformation. The image offset information is then used to compensate the original video frame to be played on the smart device to obtain the corresponding target video frame. This can compensate for the video jitter caused by the shaking of the viewer's head and finally output a smooth, stable and clear video stream, thereby improving the video viewing experience.

[0052] Secondly, wearable devices and smart devices work together to perform dynamic image stabilization on video footage. So, even when the smart device is stationary and the viewer is in motion, the original video frames to be played on the smart device can be compensated to obtain and display the corresponding target video frames. This solves the problem of blurry video caused by head shaking in motion scenarios, allowing the viewer to see clear and stable video footage even when in motion, thus improving the video viewing experience in motion scenarios.

[0053] In addition, head movement data of the viewer collected by wearable devices directly reflects the changes in the viewer's head perspective. Therefore, the image offset information obtained by mapping the head movement data under the video view perspective is more correlated with the actual changes in the viewer's head perspective. Thus, when using the obtained image offset information to compensate for the video image, the accuracy of image compensation can be effectively improved, thereby enhancing the anti-shake processing effect of the video image. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 A schematic diagram of a system architecture provided in an embodiment of this application;

[0056] Figure 2 A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0057] Figure 3 A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0058] Figure 4 A flowchart illustrating a video playback method provided in an embodiment of this application;

[0059] Figure 5A A schematic diagram illustrating the process from video acquisition to video playback provided in an embodiment of this application;

[0060] Figure 5B A flowchart illustrating a video post-processing method provided in an embodiment of this application;

[0061] Figure 6 A schematic diagram of the application homepage of a video application provided in an embodiment of this application;

[0062] Figure 7A schematic diagram of a video playback interface provided in an embodiment of this application;

[0063] Figure 8 A schematic diagram of a device coordinate system provided in an embodiment of this application;

[0064] Figure 9 A schematic diagram of a screen coordinate system provided in an embodiment of this application;

[0065] Figure 10 A flowchart illustrating a coordinate transformation method provided in an embodiment of this application;

[0066] Figure 11 A flowchart illustrating a coordinate transformation method for related technologies;

[0067] Figure 12 A flowchart illustrating a coordinate transformation method provided in an embodiment of this application;

[0068] Figure 13 A flowchart illustrating a video playback method provided in an embodiment of this application;

[0069] Figure 14 This is a schematic diagram of the structure of a video playback device provided in an embodiment of this application;

[0070] Figure 15 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0071] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0072] For ease of understanding, the terms used in the embodiments of this invention are explained below.

[0073] A video frame, also known as a still image, is a still image in a video and is the basic unit of video. A video is composed of consecutive video frames, each representing a complete image at a specific point in time. Video frames are created by continuously capturing or generating a series of successive image frames. During video playback, these video frames are played back continuously at a certain rate, thus producing dynamic video effects.

[0074] Virtual Reality (VR) is a technology that primarily uses computer technology, integrating 3D graphics, multimedia, simulation, and display technologies to create a realistic 3D virtual world with multiple sensory experiences, including visual, tactile, and olfactory sensations. This allows viewers to feel as if they are actually in the virtual world. VR technology is already widely used, for example in video playback and gaming. In VR video playback, for instance, VR glasses typically have a built-in VR player, allowing viewers to watch VR videos played by the VR player.

[0075] Augmented Reality (AR) is an interactive technology that overlays virtual information onto the real environment to enhance the user's perception.

[0076] Headphone motion sensor: A miniature motion sensing module integrated into smart headphones (such as True Wireless Stereo (TWS) and over-ear headphones). It uses technologies such as inertial measurement unit (IMU), optical sensors or bioelectrical signal detection to capture head or ear movements (such as posture changes, acceleration, angular velocity, etc.) in real time and convert the data into an interpretable motion trajectory to drive human-computer interaction, health monitoring, and dynamic scene compensation (such as video stabilization and AR / VR display optimization).

[0077] Image stabilization: refers to eliminating or reducing image shake caused by device movement or object motion. It involves real-time correction of the image through hardware (such as optical image stabilization, OIS) or algorithms (such as electronic image stabilization, EIS) to output a stable and smooth video stream.

[0078] Video post-processing refers to a series of optimization and enhancement operations performed on video footage after encoding and decoding, aiming to improve visual quality, correct defects, or add specific effects.

[0079] Affine transformation: a two-dimensional geometric transformation, also known as affine mapping, refers to the transformation of a vector space into another vector space by performing a linear transformation followed by a translation. In the embodiments of this application, a combination of translation, rotation, scaling, and shearing is used to perform spatial mapping on the entire video image.

[0080] It is understood that in the specific embodiments of this application, data such as head motion data are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0081] In practical applications, when a device is playing video, the video may shake due to device movement, resulting in a blurry video for the viewer. To address this, related technologies utilize internal motion sensors to detect the device's own movement. Based on this movement, the video playback is compensated accordingly; for example, if the device is detected moving to the right, the video image is shifted to the left relative to the device's screen to display a stable video image.

[0082] However, in some scenarios, video shakiness is not caused by device movement, but by the movement of the viewer. In such cases, the device does not correct the video image, so the viewer will still see a blurry video, thus affecting the viewing experience.

[0083] For example, imagine placing a smartphone on a treadmill and playing a video on it. A viewer watches the video while running; the smartphone remains stationary and therefore doesn't correct the video image. However, to the viewer, the video appears shaky, blurry, and thus negatively impacts the viewing experience.

[0084] In view of this, embodiments of this application provide a video playback method applied to a smart device, the method specifically including the following steps:

[0085] The system acquires the original video frames to be played on the smart device, as well as the head motion data of the viewing object in the original video frames; the head motion data is collected from the wearable device used by the viewing object, and the wearable device has a device coordinate system.

[0086] Based on head motion data, the head offset information of the viewed object in the device coordinate system is obtained; then the head offset information is mapped from the device coordinate system to the screen coordinate system used by the smart device to obtain the screen offset information of the smart device; then the screen offset information is used to perform screen compensation on the original video frame to obtain and display the corresponding target video frame.

[0087] In this embodiment, head motion data of the viewer is collected by a wearable device. Then, the head motion data is mapped to the image offset information under the video viewpoint through coordinate system transformation. The image offset information is then used to compensate the original video frame to be played on the smart device to obtain the corresponding target video frame. This can compensate for the video jitter caused by the shaking of the viewer's head and finally output a smooth, stable and clear video stream, thereby improving the video viewing experience.

[0088] Secondly, wearable devices and smart devices work together to perform dynamic image stabilization on video footage. So, even when the smart device is stationary and the viewer is in motion, the original video frames to be played on the smart device can be compensated to obtain and display the corresponding target video frames. This solves the problem of blurry video caused by head shaking in motion scenarios, allowing the viewer to see clear and stable video footage even when in motion, thus improving the video viewing experience in motion scenarios.

[0089] In addition, head movement data of the viewer collected by wearable devices directly reflects the changes in the viewer's head perspective. Therefore, the image offset information obtained by mapping the head movement data under the video view perspective is more correlated with the actual changes in the viewer's head perspective. Thus, when using the obtained image offset information to compensate for the video image, the accuracy of image compensation can be effectively improved, thereby enhancing the anti-shake processing effect of the video image.

[0090] The following is a brief introduction to the system architecture diagram applicable to the technical solutions of the embodiments of this application. It should be noted that the system architecture diagram described below is only used to illustrate the embodiments of this application and is not intended to limit the scope of the application.

[0091] refer to Figure 1 This is a system architecture diagram applicable to the embodiments of this application. The system architecture includes at least: smart device 101, wearable device 102 and server 103. The number of smart devices 101 can be one or more, the number of wearable devices 102 can also be one or more, and the number of servers 103 can also be one or more. This application does not specifically limit the number of smart devices 101, wearable devices 102 and servers 103.

[0092] The smart device 101 has a pre-installed target application with video playback functionality. This target application can be a client application, a web application, a mini-program application, etc. The smart device 101 can be a smartphone, tablet, laptop, desktop computer, smart home appliance, smart voice interaction device, smart in-vehicle device, etc., but is not limited to these.

[0093] When the smart device 101 displays video footage in the target application, the wearable device 102 simultaneously plays audio associated with the video footage. The wearable device 102 can be a smart headset, a smart helmet, an extended reality (XR) device, a smart bracelet, etc. XR includes various forms such as augmented reality (AR), VR, and mixed reality (MR); correspondingly, XR devices can be AR devices, VR devices, MR devices, etc.

[0094] The smart device 101 and the wearable device 102 can communicate directly or indirectly through one or more networks. This network can be a wired network or a wireless network; for example, the wireless network could be Bluetooth Low Energy (BLE) or a Wireless-Fidelity (WIFI) network.

[0095] Server 103 is the backend server of the target application. Server 103 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0096] The video playback method in this application embodiment can be executed by the smart device 101, the server 103, or by the interaction between the smart device 101 and the server 103.

[0097] Both server 103 and smart device 101 may include one or more processors, memory, and interactive I / O interfaces. Furthermore, server 103 may be configured with a database for video data, etc. The memory of server 103 and smart device 101 may also store program instructions required for execution in the video playback method provided in this application embodiment. These program instructions, when executed by the processor, can be used to implement the video playback process provided in this application embodiment.

[0098] It should be noted that when the video playback method provided in this application embodiment is executed by either server 103 or smart device 101 alone, the system architecture of this application may also include only a single device, either server 103 or smart device 101. Alternatively, server 103 and smart device 101 may be considered as the same device. Of course, in practical applications, when the video playback method provided in this application embodiment is executed by both server 103 and smart device 101, server 103 and smart device 101 may also be the same device. That is, server 103 and smart device 101 may be different functional modules of the same device, or virtual devices virtualized by the same physical device.

[0099] In this embodiment, the smart device 101 and the server 103 can communicate directly or indirectly through one or more networks. This network can be a wired network or a wireless network; for example, a wireless network can be a mobile cellular network or a Wi-Fi network, or other possible networks. This embodiment does not limit the types of networks used.

[0100] The following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of this application are applicable. It should be noted that the application scenarios described below are only for illustrating the embodiments of this application and are not intended to limit the scope. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0101] The solution provided in this application can be applied to video playback in various application scenarios, such as movie and TV video playback, live video playback, game video playback, short video playback, and VR video playback. This solution can also be used as a foundational technology in various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0102] The following provides an illustrative example of the application scenarios to which the solution provided in this application is applicable.

[0103] Application Scenario 1: Video playback.

[0104] See Figure 2 A smartphone 202, equipped with a screen coordinate system, is placed on a treadmill 201. The viewer wears smart headphones 203, which also have a device coordinate system and are connected to the smartphone 202 via Bluetooth. While the viewer is running, a video application on the smartphone 202 plays a movie. The viewer watches the video on the smartphone while running, and simultaneously listens to audio associated with the video through the smart headphones 203. Throughout this process, the smartphone 202 remains stationary.

[0105] When the smartphone 202 receives the original video frame to be played from the server, it obtains the head motion data of the viewer in the device coordinate system from the smart earphone 203. Then, based on the head motion data, it obtains the head offset information of the viewer in the device coordinate system (e.g., rotating 2 degrees to the right). The head offset information is then mapped from the device coordinate system to the screen coordinate system used by the smart device to obtain the smartphone's screen offset information (e.g., shifting 10 pixels to the right). Using the screen offset information, the original video frame is compensated to obtain the target video frame, which is then displayed on the video playback interface 204 of the video application, thereby effectively reducing the visual blurring of the video image caused by head shaking during running.

[0106] Application Scenario 2: Live video playback.

[0107] See Figure 3 A tablet computer 302 is placed on a desktop 301. The tablet computer 302 has a screen coordinate system set up. A live streaming application on the tablet computer 302 plays a weight loss exercise video by a fitness streamer. The viewer wears a smart headset 303, which has a device coordinate system set up. The smart headset 303 is connected to the tablet computer 302 via Bluetooth.

[0108] The viewer follows along with a weight loss exercise video played on tablet computer 302, while simultaneously listening to audio commentary and background music from a fitness instructor via smart headphones 303. During this process, tablet computer 302 remains stationary.

[0109] When a viewer follows along with a weight loss exercise, the smart earphone 303 collects the viewer's head movement data in the device coordinate system and sends the head movement data to the tablet computer 302. The tablet computer 302 receives the original video frame to be played from the server, as well as the head movement data sent by the smart earphone 303. Based on the head movement data, it obtains the viewer's head offset information in the device coordinate system (e.g., a 3-degree right rotation). This head offset information is then mapped from the device coordinate system to the screen coordinate system used by the smart device to obtain the smartphone's screen offset information (e.g., a 20-pixel right shift). Using this screen offset information, the original video frame is compensated to obtain the target video frame, which is then displayed on the live streaming application's interface 304. This way, even if the viewer's head shakes while following along with the weight loss exercise, they can still see a clear and stable video image, thus solving the problem of blurry video caused by head shaking during exercise.

[0110] It should be noted that the embodiments of this application are not limited to the application scenarios exemplified above, but may also be other application scenarios, which are not specifically limited in this application.

[0111] based on Figure 1The system architecture diagram shown in this application illustrates a flowchart of a video playback method. Figure 4 As shown, the process of this method is executed by a computer device, which can be... Figure 1 The smart device 101 and / or server 103 shown include the following steps:

[0112] Step 401: Obtain the original video frame to be played on the smart device, and obtain the head motion data of the viewing object of the original video frame.

[0113] Specifically, a video consists of a series of consecutive video frames, each representing a complete image of the video at a certain point in time. Multiple video frames are played continuously at a certain rate, thus creating dynamic video effects.

[0114] According to the principle of visual persistence, when a light pulse stimulates the human eye, the establishment and disappearance of vision require a certain process, that is, it has a certain degree of inertia. When the repetition frequency of the light pulse is not high enough, the human eye perceives a flickering sensation of alternating brightness and darkness. Only when there are continuous light pulses, that is, when the image changes more than 24 frames per second, the human eye's flickering sensation is greatly reduced and it is difficult to distinguish a single static image. What appears to be a smooth and continuous video effect is thus a continuous series of images, which constitutes video.

[0115] The following describes the process from video acquisition to video playback. (See [link / reference]) Figure 5A This includes the following steps:

[0116] Raw video data is captured using devices such as cameras, camcorders, and smartphones, or imported from storage media. The raw data is then preprocessed to ensure its quality and integrity. Preprocessing includes operations such as noise reduction, color correction, and stabilization.

[0117] Next, the raw data is encoded according to a specific video encoding standard to obtain compressed video data, thereby reducing the video file size and improving transmission efficiency. The video encoding standards include H.264, H.265, VP9, ​​etc.

[0118] The compressed video data is encapsulated with multiple media streams such as audio data and subtitles to obtain the video's encapsulation format data (i.e., packaged into a complete container file). Video encapsulation formats include MP4, AVI, and MKV. The processed encapsulation format data is then saved to storage media or transmitted over a network for playback and viewing on different devices.

[0119] After acquiring the video's encapsulation format data, the smart device decapsulates the data to obtain compressed video data, audio data, subtitles, and other data. The compressed video data is then decoded back into the original video data, and the audio data is decoded into audio samples for subsequent processing and playback.

[0120] In the embodiments of this application, see Figure 5B The decoded video data includes one or more original video frames to be played. The smart device performs post-processing on the original video frames to obtain the target video frame, and then renders the target video frame onto the screen for playback. The solution in this embodiment occurs during the post-processing stage, specifically through a motion stabilization module in the video application installed on the smart device. That is, the acquired original video frames and the head movement data of the viewer sent by the wearable device are input into the motion stabilization module for processing to obtain a stable target video frame, which is then rendered onto the screen for playback.

[0121] It should be noted that the embodiments of this application can perform image stabilization processing on each original video frame to be played, or can perform image stabilization processing on a portion of the original video to be played while not performing image stabilization processing on the other portion of the original video to be played. This application does not make specific limitations in this regard.

[0122] In this embodiment of the application, the viewing object wears a wearable device, which has a device coordinate system. For example, the wearable device is set as a smart headset. In the device coordinate system of the smart headset, the X-axis is horizontal to the right (pointing to the right side of the viewing object when worn), the Y-axis is vertically upward (pointing to the top of the head), and the Z-axis is horizontal forward (pointing to the front of the viewing object).

[0123] Wearable devices periodically collect head motion data of the viewed object in the device's coordinate system via an inertial measurement unit, while smart devices collect this head motion data from the wearable device. This head motion data can include angular velocity collected by a gyroscope, acceleration collected by an accelerometer, and so on.

[0124] In this embodiment, the stabilization function is implemented through the motion stabilization processing module in the video application. This is a software stabilization function, and it supports determining whether to enable the stabilization mode based on the selection of the viewing object.

[0125] In some embodiments, before acquiring the original video frame to be played on the smart device and the head motion data of the viewing object of the original video frame, the video playback mode of the video application is set to anti-shake mode in response to a mode selection operation triggered in the video application.

[0126] Specifically, the above mode selection operation can be either setting the video playback mode for the first time or switching between previously set video playback modes. The previously set video playback mode can be the system default or a mode previously set by the viewer. Mode selection can be achieved through tapping the screen, swiping the screen, voice input, or gesture input. The following sections provide a detailed description of several mode setting scenarios:

[0127] Upon initial launch of the video application, a mode selection interface can be displayed within the application, offering multiple selectable video playback modes. Viewers can then trigger a mode selection action within this interface, choosing a stabilization mode from the available options. In response to this selection action, the video application's playback mode is automatically set to the stabilization mode. Subsequent launches of the video application will then reuse the previously configured stabilization mode.

[0128] Alternatively, you can display a mode selection interface in the video application each time it is launched, and then, in response to the mode selection operation triggered in the video application, set the video playback mode of the video application to the anti-shake mode.

[0129] You can also display a mode selection interface in the video application when the viewing object triggers video playback, and then, in response to the mode selection operation triggered in the video application, set the video playback mode of the video application to the anti-shake mode.

[0130] After setting the video playback mode to anti-shake mode using any of the above methods, the original video frames are converted into target video frames with stable images by executing steps 401 to 404 in the embodiments of this application.

[0131] For example, see Figure 6 In response to the video application launch operation triggered by the viewing object A, the smart device displays the video application's homepage 601 and shows the mode selection interface 602 in the video application as a pop-up window. The mode selection interface 602 includes two selectable video playback modes: stabilized mode and non-stabilized mode.

[0132] Viewer A selects "Anti-shake Mode" on the mode selection interface 602. The smart device responds to Viewer A's mode selection operation by setting the video playback mode of the video application to "Anti-shake Mode" and displays the selected "Anti-shake Mode" in area 603 on the application's homepage 601.

[0133] In this embodiment of the application, by providing multiple video playback modes for viewers to choose from, each viewer can select whether to watch the video in anti-shake mode according to their actual needs, thereby improving the user experience of the video application.

[0134] In some embodiments, in response to a mode switching operation triggered in the video application, the video playback mode of the video application is switched to a non-stabilized mode, and then the original video frames are rendered and displayed in the video application.

[0135] Specifically, mode switching can be achieved through tapping the screen, swiping the screen, voice input, gesture input, etc. Taking one of these methods, the video application displays the current video playback mode (i.e., stabilization mode) along with a mode switching button. When the viewer taps the mode switching button, a mode selection interface is displayed in the video application, offering multiple selectable video playback modes.

[0136] When the viewer selects a non-stabilized mode in the mode selection interface, in response to the mode switching operation triggered in the video application, the video application's video playback mode is switched to non-stabilized mode. In this case, the smart device directly renders the original video frames to be played onto the screen in the video application, without using the motion stabilization module in the video application to perform stabilization on the original video frames.

[0137] For example, see Figure 7 The current video playback mode (i.e., anti-shake mode) is displayed in the video playback interface 700 of the video application, and the mode switching button 701 is also displayed.

[0138] Viewer A clicks the mode switch button 701, triggering the display of the mode selection interface 702 in the video application. The mode selection interface 702 includes: stabilized mode and non-stabilized mode, where stabilized mode is shown as selected. Viewer A selects non-stabilized mode in the mode selection interface 702. In response to viewer A's mode switch operation, the smart device sets the video playback mode of the video application to non-stabilized mode.

[0139] In this embodiment of the application, by providing multiple video playback modes for viewers to choose from, each viewer can select whether to watch the video in anti-shake mode according to their actual needs, thereby improving the user experience of the video application.

[0140] It should be noted that viewers can trigger the mode switching operation again on any interface of the video application to switch from non-stabilized mode back to stabilized mode. This application does not impose specific restrictions on this.

[0141] Step 402: Based on head motion data, obtain the head offset information of the viewed object in the device coordinate system.

[0142] Specifically, according to a preset mapping relationship, head motion data in the device coordinate system is converted into head offset information in the device coordinate system. Head offset information refers to the head attitude angle and / or head displacement of the viewed object in the device coordinate system. Head attitude angles include at least one of the following: pitch angle, yaw angle, and roll angle. Head displacement includes at least one of the following: head displacement along the X, Y, and Z axes.

[0143] See Figure 8 The X-axis of the device coordinate system is horizontal to the right (pointing to the right side of the viewer when worn), the Y-axis is vertically upward (pointing to the top of the viewer's head), and the Z-axis is horizontal forward (pointing to the front of the viewer).

[0144] Pitch angle is the angle at which the head nods up and down (i.e., the angle of rotation around the X-axis); a positive pitch is when the head is tilted down (chin down), and a negative pitch is when the head is tilted up (chin up). Yaw angle is the angle at which the head turns left and right (i.e., the angle of rotation around the Y-axis); a positive yaw is when the head turns right, and a negative yaw is when the head turns left. Roll angle is the angle at which the head tilts to the side (i.e., the angle of rotation around the Z-axis); a positive roll is when the head rolls to the right, and a negative roll is when the head rolls to the left.

[0145] In some embodiments, head motion data includes: head angular velocity; head offset information includes: head offset angle; in this case, the head offset information of the viewing object is obtained in the following manner:

[0146] The time interval for the wearable device to collect head motion data is obtained; then, based on the head angular velocity and the time interval, the head offset angle of the viewed object in the device coordinate system is obtained.

[0147] Specifically, the time interval for collecting head motion data refers to the difference between the time of the current head motion data collection and the time of the previous head motion data collection. In practical applications, head motion data can be collected at fixed intervals, in which case the time interval for collecting head motion data is a fixed value; of course, the time interval for collecting head motion data can also be set to a non-fixed value; for example, when the wearable device is in power-saving mode, the frequency of collecting head motion data is reduced, that is, the time interval for collecting head motion data is increased.

[0148] Each time head motion data is collected, it corresponds to a raw video frame to be played. No other raw video frames may be played between two adjacent times of collecting head motion data, meaning that stabilization is performed on each raw video frame. Of course, other raw video frames may also be played between two adjacent times of collecting head motion data, meaning that stabilization is performed once every certain number of raw video frames, rather than stabilizing each raw video frame.

[0149] In this embodiment, the product of the head angular velocity and the time interval is used as the head offset angle of the viewing object in the device coordinate system.

[0150] Head angular velocity includes ω x ω y ω z At least one of them, where ω x This represents the angular velocity of rotation about the X-axis, i.e., the angular velocity of head nodding up and down; in Figure 8 In the device coordinate system shown, when the viewed object nods downwards, the corresponding ω x It is a positive number; when the object being viewed looks up, the corresponding ω is... x It is a negative number.

[0151] ω y This represents the angular velocity of rotation about the Y-axis, i.e., the angular velocity of left and right rotation; in Figure 8 In the device coordinate system shown, when the viewing object turns its head to the right, the corresponding ω y It is a positive number; when the object being viewed turns its head to the left, the corresponding ω is... y It is a negative number.

[0152] ω z This represents the angular velocity of rotation around the Z-axis, i.e., the angular velocity of head tilt. Figure 8 In the device coordinate system shown, when the viewing object tilts to the right, the corresponding ω z It is a positive number; when the object being viewed is tilted to the left, the corresponding ω is... z It is a negative number.

[0153] Head offset angles include pitch angle θ, yaw angle σ, and roll angle. At least one of them. The angular velocity ω of the up-and-down nodding motion. x Multiplying the angle by the time interval t yields the pitch angle θ; the angular velocity ω of the left and right rotations is also calculated. y Multiplying this by the time interval t yields the yaw angle σ; the angular velocity ω of the head tilt is then calculated. z Multiplying this by the time interval t yields the roll angle.

[0154] Alternatively, the head angular velocity can be integrated over the time interval to obtain the head offset angle of the viewed object in the device coordinate system, which will not be elaborated here.

[0155] In this embodiment, the head angular velocity of the viewing object is collected by a wearable device, and then the head offset angle of the viewing object is obtained by combining the head angular velocity and the time interval of collecting the head angular velocity. The head offset angle directly reflects the change in the viewing object's head perspective. Therefore, when compensating for the video image based on the head offset angle, the accuracy of image compensation can be effectively improved.

[0156] In some embodiments, head motion data includes head angular velocity and head acceleration; head offset information includes head offset angle; in this case, the head offset angle of the viewing object is obtained in the following manner:

[0157] Head acceleration includes: a x a y a z At least one of them, where a x This represents acceleration in the X-axis direction; a y This represents acceleration in the Y-axis direction; a z This represents the acceleration in the Z-axis direction.

[0158] Based on gravitational acceleration g and acceleration a z The pitch angle θ is obtained as shown in the following formula (1):

[0159]

[0160] Based on gravitational acceleration g and acceleration a x Obtain the roll angle Specifically, as shown in the following formula (2):

[0161]

[0162] Since the yaw angle σ cannot be directly obtained through acceleration, the angular velocity ω acquired by the gyroscope is used instead. y Multiply by the time interval t to obtain the yaw angle σ.

[0163] In this embodiment, the head acceleration and head angular velocity of the viewing object are collected by a wearable device. Then, based on the head acceleration and head angular velocity, the head offset angle of the viewing object is obtained. This head offset angle directly reflects the change in the viewing angle of the viewing object's head. Therefore, when compensating for the video image based on the head offset angle, the accuracy of image compensation can be effectively improved.

[0164] Step 403: Map the head offset information from the device coordinate system to the screen coordinate system used by the smart device to obtain the screen offset information of the smart device.

[0165] Specifically, a screen coordinate system is a coordinate system that describes the position of pixels or graphic elements on the screen, including the origin and coordinate axes.

[0166] For example, see Figure 9The screen coordinate system uses the top-left corner (0,0) of screen 901 as the origin, with the positive X-axis pointing horizontally to the right and the positive Y-axis pointing vertically downwards. The unit of the screen coordinate system is pixels (px). Additionally, the screen coordinate system includes a valid coordinate range; pixels outside this range cannot be displayed on the screen. The valid coordinate range is determined by the screen resolution. For example, when the screen resolution is 1920×1080, the valid coordinate range along the X-axis is [0, 1919], and the valid coordinate range along the Y-axis is [0, 1079].

[0167] In this application, the screen coordinate system is the coordinate system under the viewpoint of the video screen, which is closely related to the viewing perspective of the viewing object; at the same time, the screen offset information of the smart device also refers to the offset of the video screen under the screen coordinate system.

[0168] In some embodiments, a scaling factor between the device coordinate system and the screen coordinate system is obtained based on the screen resolution of the smart device and a preset head offset threshold; then, based on the scaling factor, the head offset information is mapped to the screen offset information of the smart device.

[0169] Specifically, in order to avoid distortion of the compensated video image, the viewing angle rotation range is preset, that is, the head offset threshold, also known as the maximum offset angle of the head, for example, set to 30 degrees.

[0170] The screen width is obtained from the screen resolution, and then the ratio between the screen width and twice the maximum offset angle is used as the scaling factor, as shown in the following formula (3):

[0171] k=w / (2*m)……………………(3)

[0172] Where k represents the scaling factor, w represents the screen width, and m represents the header offset threshold. The scaling factor represents the number of offset pixels corresponding to a unit offset angle in device coordinates in screen coordinates.

[0173] Of course, besides using screen width to obtain the scaling factor, screen height can also be used. The scaling factor will differ for smart devices with different resolutions.

[0174] Next, based on the scaling factor, the head offset information is transformed from the device coordinate system to the screen coordinate system to obtain the screen offset information of the smart device; the screen offset information refers to the number of pixels offset in the original video frame in the screen coordinate system.

[0175] In this embodiment, head motion data is mapped to image offset information from the video viewpoint using a scaling mapping coefficient. This allows for image compensation of the original video frames using the image offset information, thereby reducing video jitter caused by head shaking and improving the video viewing experience.

[0176] It should be noted that, in the embodiments of this application, each obtained head offset angle can be mapped to the screen offset information of the smart device to trigger compensation of the video screen; alternatively, the head offset angle can be mapped to the screen offset information of the smart device only when it is greater than a preset angle to trigger compensation of the video screen. This can greatly reduce power consumption while achieving image stabilization.

[0177] In some embodiments, head offset information includes pitch angle and yaw angle; image offset information includes horizontal offset and vertical offset.

[0178] Both horizontal and vertical offsets are measured in pixels (px). The horizontal offset is the number of pixels offset in the X-axis direction of the screen coordinate system. If the horizontal offset is greater than 0, it is the number of pixels offset in the positive X-axis direction; if the horizontal offset is less than 0, it is the number of pixels offset in the negative X-axis direction.

[0179] The vertical offset is the number of pixels offset in the Y-axis direction in the screen coordinate system; if the vertical offset is greater than 0, the vertical offset is the number of pixels offset in the positive Y-axis direction; if the vertical offset is less than 0, the vertical offset is the number of pixels offset in the negative Y-axis direction.

[0180] The product of the scaling factor and the pitch angle is taken as the vertical offset, as shown in the following formula (4):

[0181] Δy=k*θ……………………(4)

[0182] Where Δy represents the vertical offset, θ represents the pitch angle, and k represents the scaling factor.

[0183] The product of the scaling factor and the yaw angle is used as the horizontal offset, as shown in the following formula (5):

[0184] Δx=k*σ……………………(5)

[0185] Where Δx represents the horizontal offset, σ represents the yaw angle, and k represents the scaling factor.

[0186] For example, if the screen resolution is set to 1920×1080 and the head offset threshold is 30 degrees, then the scaling factor k = 32. Assuming the tilt angle of the viewing object is -1 degree (i.e., tilting the head up 1 degree), then according to formula (4), the vertical offset in the screen coordinate system is -32px.

[0187] Assuming the yaw angle of the object being viewed is 5 degrees (5 degrees to the right), then according to formula (5), the horizontal offset in the screen coordinate system is 160px.

[0188] In this embodiment, the pitch angle is mapped to the vertical offset in the video viewpoint and the yaw angle is mapped to the horizontal offset in the video viewpoint using a scaling mapping coefficient, so as to obtain comprehensive image offset information in the screen coordinate system and improve the accuracy of subsequent image compensation.

[0189] Step 404: Use the image offset information to perform image compensation on the original video frame to obtain and display the corresponding target video frame.

[0190] Specifically, when the viewer's head turns to the right, the video image on the screen also needs to be panned to the right to ensure that the video image is stable and not shaky from the viewer's perspective; similarly, when the viewer's head turns to the left, the video image on the screen also needs to be panned to the left; when the viewer looks up, the video image on the screen also needs to be panned upward; and when the viewer nods downward, the video image on the screen also needs to be panned downward.

[0191] In other words, in this embodiment, when using image offset information to compensate for the original video frame, a forward compensation method is used, meaning the direction of head rotation is the same as the direction of image compensation. After obtaining the target video frame through image compensation, the target video frame is rendered onto the screen for playback.

[0192] In this embodiment, head motion data of the viewer is collected by a wearable device. Then, the head motion data is mapped to the image offset information under the video viewpoint through coordinate system transformation. The image offset information is then used to compensate the original video frame to be played on the smart device to obtain the corresponding target video frame. This can compensate for the video jitter caused by the shaking of the viewer's head and finally output a smooth, stable and clear video stream, thereby improving the video viewing experience.

[0193] Secondly, wearable devices and smart devices work together to perform dynamic image stabilization on video footage. So, even when the smart device is stationary and the viewer is in motion, the original video frames to be played on the smart device can be compensated to obtain and display the corresponding target video frames. This solves the problem of blurry video caused by head shaking in motion scenarios, allowing the viewer to see clear and stable video footage even when in motion, thus improving the video viewing experience in motion scenarios.

[0194] In addition, head movement data of the viewer collected by wearable devices directly reflects the changes in the viewer's head perspective. Therefore, the image offset information obtained by mapping the head movement data under the video view perspective is more correlated with the actual changes in the viewer's head perspective. Thus, when using the obtained image offset information to compensate for the video image, the accuracy of image compensation can be effectively improved, thereby enhancing the anti-shake processing effect of the video image.

[0195] In some embodiments, the image offset information includes a horizontal offset and a vertical offset. The horizontal offset is used to perform horizontal image compensation on the original video frame; and the vertical offset is used to perform vertical image compensation on the original video frame, thereby obtaining and displaying the corresponding target video frame.

[0196] Specifically, the meanings of horizontal and vertical offsets differ in different screen coordinate systems.

[0197] by Figure 9 For example, in the screen coordinate system shown, the horizontal offset refers to the offset (in pixels) in the X-axis direction. When the horizontal offset is greater than zero, it refers to the offset in the positive X-axis direction; when the horizontal offset is less than zero, it refers to the offset in the negative X-axis direction.

[0198] Vertical offset refers to the offset in the Y-axis direction. When the vertical offset is greater than zero, it refers to the offset in the positive Y-axis direction; when the vertical offset is less than zero, it refers to the offset in the negative Y-axis direction.

[0199] When using a horizontal offset to compensate for the horizontal image of the original video frame, if the horizontal offset is greater than zero, the original video frame is moved horizontally in the positive X-axis direction by the horizontal offset; if the horizontal offset is less than zero, the original video frame is moved horizontally in the negative X-axis direction by the horizontal offset.

[0200] When using a vertical offset to compensate for the vertical image of the original video frame, if the vertical offset is greater than zero, the original video frame is moved vertically in the positive Y-axis direction by the vertical offset; if the vertical offset is less than zero, the original video frame is moved vertically in the negative Y-axis direction by the vertical offset.

[0201] After image compensation is completed in both the horizontal and vertical directions, the target video frame can be obtained.

[0202] In this embodiment, head motion data is mapped to horizontal and vertical offsets from the video viewpoint through coordinate system transformation. Then, the horizontal and vertical offsets are used to perform horizontal and vertical image compensation on the original video frame to be played, thereby obtaining the corresponding target video frame. This can compensate for video jitter caused by the shaking of the viewer's head, thereby improving the video viewing experience.

[0203] In some embodiments, a coordinate transformation matrix is ​​constructed based on horizontal and vertical offsets. For multiple pixels in the original video frame, the following steps are performed: the original coordinates of a pixel are transformed using the coordinate transformation matrix to obtain the target coordinates of that pixel. Based on the target coordinates of the multiple pixels, the target video frame is obtained and displayed.

[0204] Specifically, the coordinate transformation matrix can be an affine transformation matrix or any other matrix capable of performing coordinate transformations. The affine transformation matrix is ​​obtained based on the horizontal and vertical offsets, as shown in the following formula (6):

[0205]

[0206] Where M represents the affine transformation matrix, Δx represents the horizontal offset, and Δy represents the vertical offset.

[0207] Each pixel in the original video frame corresponds to a pixel coordinate in the screen coordinate system, namely the original coordinate (x0, y0). For the original coordinate (x0, y0) of each pixel, an affine transformation matrix is ​​used to increase the horizontal offset Δx in the horizontal direction and the vertical offset Δy in the vertical direction to obtain the target coordinate (x1, y1) of the pixel, as shown in the following formula (7):

[0208]

[0209] The target video frame is obtained and displayed based on the target coordinates of multiple pixels. In specific implementation, the target video frame is rendered according to the target coordinates of multiple pixels and displayed on the screen of the smart device.

[0210] For example, see Figure 10 If the screen resolution of a smart device is set to 1920×1080, then in the screen coordinate system of the smart device (specifically as follows): Figure 9 As shown in the figure, the effective coordinate range in the X-axis direction is [0, 1919], and the effective coordinate range in the Y-axis direction is [0, 1079].

[0211] Based on the previous example, the vertical offset in the screen coordinate system is -32px, and the horizontal offset in the screen coordinate system is 160px. Therefore, the following affine transformation matrix is ​​constructed:

[0212]

[0213] The original video frame resolution is set to 1609×889. For the pixel with original coordinates (150, 100) in original video frame 1001, after applying an affine transformation matrix, the target coordinates are obtained as (310, 68). This means that in the screen coordinate system, this pixel is shifted 160 pixels to the right and 32 pixels upwards. The target coordinates of other pixels in the original video frame can be obtained using the same method, which will not be elaborated here.

[0214] Finally, the target video frame 1002 is constructed using the pixels within the effective coordinate range of the screen coordinate system.

[0215] In this embodiment, an affine transformation matrix is ​​constructed using the horizontal and vertical offsets obtained from the video frame viewpoint through mapping. Then, the coordinates of each pixel in the original video frame are transformed using the affine transformation matrix to obtain the target video frame after image compensation and play it, thereby improving the efficiency of video image compensation and enhancing the video viewing experience.

[0216] In some embodiments, after transforming the original coordinates of a pixel using an affine transformation matrix to obtain the target coordinates of the pixel, the target coordinates of some pixels will exceed the effective coordinate range of the screen coordinate system, and these pixels will not be displayed; at the same time, it will also result in some positions within the effective coordinate range having no effective pixels, resulting in blank areas (i.e., black borders) in the video displayed on the screen.

[0217] For example, see Figure 11 Set the screen resolution of the smart device to 1920×1080, the resolution of the original video frame to 1920×1080, the vertical offset in the screen coordinate system to -32px, and the horizontal offset in the screen coordinate system to 160px.

[0218] In the screen coordinate system, the original coordinates of each pixel in the original video frame 1101 are shifted 160 pixels to the right and 32 pixels upward to obtain the target coordinates of each pixel. At this time, the pixels with horizontal coordinates of [1920, 2079] are outside the effective pixel range of the X-axis, so these pixels will not be displayed; correspondingly, within the effective pixel range of the X-axis, the area with horizontal coordinates between [0, 160] has no effective pixels, so a black border 1102 is displayed.

[0219] Similarly, pixels with a ordinate of [-32, 0] are outside the effective pixel range of the Y-axis, so these pixels will not be displayed; correspondingly, within the effective pixel range of the Y-axis, the area with a ordinate of [1047, 1079] has no effective pixels, so a black border of 1102 is displayed.

[0220] Finally, based on the black border 1102 and the original video frame 1101 within the effective pixel range, the final displayed video image 1103 is obtained. This results in an incomplete video image display and the appearance of black borders, thus affecting the video viewing experience.

[0221] Therefore, in this embodiment of the application, the original video frame is magnified according to a preset ratio to obtain an intermediate video frame; then, the intermediate video frame is compensated using image offset information to obtain and display the corresponding target video frame.

[0222] Specifically, after the original video frame is enlarged according to a preset ratio, the resulting intermediate video frame will exceed the effective pixel range. In this way, when the intermediate video frame is compensated by using image translation information, even if the intermediate video frame is moved, the pixels within the effective pixel range can be guaranteed as much as possible to remain the pixels in the original video frame.

[0223] The method of using image offset information to compensate for intermediate video frames is the same as the method of using image offset information to compensate for original video frames, and will not be described again here.

[0224] For example, see Figure 12 Set the screen resolution of the smart device to 1920×1080, the resolution of the original video frame to 1920×1080, the vertical offset in the screen coordinate system to -32px, and the horizontal offset in the screen coordinate system to 160px.

[0225] In the screen coordinate system, the original video frame 1201 is enlarged according to the preset ratio of 1.2 to obtain the resolution of the intermediate video frame: 2304×1296, where the horizontal coordinate range is [-225, 2079] and the horizontal coordinate range is [-80, 1215].

[0226] By shifting the original coordinates of each pixel in the intermediate video frame 1202 160 pixels to the right and 32 pixels upward, the target coordinates of each pixel are obtained. At this point, the area with horizontal coordinates between [0, 160] is still a pixel in the intermediate video frame 1202, so no black border will be displayed.

[0227] Similarly, within the effective pixel range of the Y-axis, the area with the vertical coordinate between [1047, 1079] is still a pixel in the intermediate video frame 1202, so no black border will be displayed.

[0228] Finally, based on the pixels located within the effective pixel range, the final target video frame 1203 is obtained.

[0229] In this embodiment of the application, after the original video frame is magnified according to a preset ratio, the resulting intermediate video frame will exceed the effective pixel range. Therefore, when the intermediate video frame is compensated by using image translation information, even if the intermediate video frame is moved, the pixels within the effective pixel range can still be kept as the same as the pixels in the original video frame as possible, thereby reducing the black borders in the video and improving the video playback effect.

[0230] It should be noted that besides enlarging the original video frames to reduce black borders in the video, other methods can be used. For example, adding a solid color, gradient, or dynamic background (such as a video wall) to cover the black border area; or leaving 5% to 10% edge redundancy in the frame during shooting (to avoid the subject being too close to the edge of the frame), providing cropping space for dynamic cropping of black and white pixels in post-production; or filling the black border with adjacent pixels or a blur effect.

[0231] In some embodiments, in order to further reduce jitter and stuttering and improve the smoothness of video footage, dynamic frame interpolation can be performed between the target video frame and the previous video frame, or between the target video frame and the next video frame.

[0232] Taking the target video frame and the subsequent video frame as an example, motion estimation is performed on both to obtain motion information between them. This motion information includes object movement, camera changes, and other details. Next, based on this motion information, intermediate transition frames are generated. For example, a weighted mixture of the target and subsequent video frames can be used to obtain intermediate transition frames. These intermediate transition frames are then inserted between the target and subsequent video frames to improve the smoothness of video playback and the stability of the video footage.

[0233] To better explain the embodiments of this application, a video playback method provided by the embodiments of this application is described below with reference to a specific implementation scenario. The process of this method can be as follows: Figure 1 The actions can be executed by the smart device 101 shown, or by the server 103, or by the smart device 101 interacting with the server 103, including the following steps, such as... Figure 13 As shown:

[0234] A smartphone 202 is placed on a treadmill 201. The smartphone 202 has a screen coordinate system and a video application installed, which includes a motion stabilization module for image stabilization. The viewer wears smart headphones 203, which has a device coordinate system and uses a motion sensor to collect head angular velocity. The smart headphones 203 and smartphone 202 are connected via Bluetooth.

[0235] While the subject is running, a video application on smartphone 202 plays a movie video. The subject watches the video on smartphone 202 while running, and simultaneously listens to audio associated with the video through smart headphones 203. During this process, smartphone 202 does not move.

[0236] The smartphone 202's video application acquires the original video frame 1301 to be played, and obtains the head angular velocity of the viewing object in the device coordinate system from the smart earphone 203. Then, in the motion stabilization module, the pitch and yaw angles are obtained based on the time interval between the head angular velocity and the head angular velocity acquired by the smart earphone 202. An affine transformation matrix is ​​then constructed based on the pitch and yaw angles. Using the affine transformation matrix, the original coordinates of each pixel in the original video frame are transformed to obtain the target video frame 1302. The target video frame 1302 is then rendered and displayed in the video application.

[0237] In this embodiment, the wearable device and the smart device work together to perform image stabilization on the video. Even when the smart device is stationary and the viewer is in motion, the original video frame to be played on the smart device can be compensated to obtain and display the corresponding target video frame. This solves the problem of blurry video caused by head shaking in motion scenarios, allowing the viewer to see a clear and stable video image even in motion, thus improving the video viewing experience in motion scenarios.

[0238] Secondly, the head movement data of the viewer collected by wearable devices directly reflects the changes in the viewer's head perspective. Therefore, the image offset information obtained by mapping the head movement data under the video image perspective is more correlated with the actual changes in the viewer's head perspective. Thus, when using the obtained image offset information to compensate for the video image, the accuracy of image compensation can be effectively improved, thereby enhancing the anti-shake processing effect of the video image.

[0239] Based on the same technical concept, this application provides a structural schematic diagram of a video playback device, such as... Figure 14 As shown, the video playback device 1400 includes:

[0240] The acquisition module 1401 is used to acquire the original video frame to be played on the smart device, and to acquire the head motion data of the viewing object of the original video frame; wherein, the head motion data is collected from the wearable device used by the viewing object, and the wearable device is equipped with a device coordinate system.

[0241] The conversion module 1402 is used to obtain the head offset information of the viewing object in the device coordinate system based on the head motion data;

[0242] Mapping module 1403 is used to map the head offset information from the device coordinate system to the screen coordinate system used by the smart device to obtain the screen offset information of the smart device;

[0243] The compensation module 1404 is used to perform image compensation on the original video frame using the image offset information, so as to obtain and display the corresponding target video frame.

[0244] Optionally, the mapping module 1403 is specifically used for:

[0245] Based on the screen resolution of the smart device and a preset head offset threshold, the scaling coefficient between the device coordinate system and the screen coordinate system is obtained.

[0246] Based on the aforementioned proportional mapping coefficient, the head offset information is mapped to the screen offset information of the smart device.

[0247] Optionally, the head offset information includes: pitch angle and yaw angle; the image offset information includes: horizontal offset and vertical offset;

[0248] Optionally, the mapping module 1403 is specifically used for:

[0249] The product of the scaling factor and the pitch angle is used as the vertical offset.

[0250] The product of the scaling factor and the yaw angle is used as the horizontal offset.

[0251] Optionally, the image offset information includes: horizontal offset and vertical offset;

[0252] The compensation module 1404 is specifically used for:

[0253] The original video frame is compensated horizontally using the horizontal offset, and the original video frame is compensated vertically using the vertical offset, so as to obtain and display the corresponding target video frame.

[0254] Optionally, the compensation module 1404 is specifically used for:

[0255] Based on the horizontal and vertical offsets, construct a coordinate transformation matrix;

[0256] For multiple pixels in the original video frame, the following steps are performed: the original coordinates of a pixel are transformed using the coordinate transformation matrix to obtain the target coordinates of the pixel;

[0257] Based on the target coordinates of each of the multiple pixels, the target video frame is obtained and displayed;

[0258] Optionally, the compensation module 1404 is specifically used for:

[0259] The original video frame is enlarged according to a preset ratio to obtain an intermediate video frame;

[0260] Using the aforementioned image offset information, image compensation is performed on the intermediate video frames to obtain and display the corresponding target video frames.

[0261] Optionally, the head motion data includes: head angular velocity; the head offset information includes: head offset angle;

[0262] The conversion module 1402 is specifically used for:

[0263] The time interval for acquiring head movement data by the wearable device;

[0264] Based on the head angular velocity and the time interval, the head offset angle of the viewing object in the device coordinate system is obtained.

[0265] Optionally, the acquisition module 1401 is further configured to:

[0266] Before acquiring the original video frame to be played on the smart device and acquiring the head motion data of the viewing object of the original video frame, in response to the mode selection operation triggered in the video application, the video playback mode of the video application is set to anti-shake mode, the video application runs on the smart device, and is used to render and display the target video frame.

[0267] Optionally, the acquisition module 1401 is further configured to:

[0268] In response to a mode switching operation triggered in the video application, the video playback mode of the video application is switched to non-stabilized mode;

[0269] The original video frames are rendered and displayed in the video application.

[0270] In this embodiment, head motion data of the viewer is collected by a wearable device. Then, the head motion data is mapped to the image offset information under the video viewpoint through coordinate system transformation. The image offset information is then used to compensate the original video frame to be played on the smart device to obtain the corresponding target video frame. This can compensate for the video jitter caused by the shaking of the viewer's head and finally output a smooth, stable and clear video stream, thereby improving the video viewing experience.

[0271] Secondly, wearable devices and smart devices work together to stabilize video footage. Even when the smart device is stationary and the viewer is in motion, the original video frames to be played on the smart device can be compensated to obtain and display the corresponding target video frames. This solves the problem of blurry video caused by head shaking in motion scenarios, allowing the viewer to see clear and stable video footage even when in motion, thus improving the video viewing experience in motion scenarios.

[0272] In addition, head movement data of the viewer collected by wearable devices directly reflects the changes in the viewer's head perspective. Therefore, the image offset information obtained by mapping the head movement data under the video view perspective is more correlated with the actual changes in the viewer's head perspective. Thus, when using the obtained image offset information to compensate for the video image, the accuracy of image compensation can be effectively improved, thereby enhancing the anti-shake processing effect of the video image.

[0273] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0274] Based on the same technical concept, embodiments of this application provide a computer device, which can be... Figure 1 The smart devices and / or servers shown, such as Figure 15 As shown, it includes at least one processor 1501 and a memory 1502 connected to at least one processor. In this embodiment, the specific connection medium between the processor 1501 and the memory 1502 is not limited. Figure 15 Taking the connection between processor 1501 and memory 1502 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc.

[0275] In this embodiment of the application, the memory 1502 stores instructions that can be executed by at least one processor 1501. By executing the instructions stored in the memory 1502, at least one processor 1501 can perform the steps of the video playback method described above.

[0276] As one embodiment, the processor 1501 acquires the original video frame to be played on the smart device, and acquires the head motion data of the viewing object of the original video frame; wherein, the head motion data is collected from the wearable device used by the viewing object and stored in the memory 1502, and the wearable device is provided with a device coordinate system;

[0277] The processor 1501 obtains the head offset information of the viewed object in the device coordinate system based on head motion data;

[0278] The processor 1501 maps the head offset information from the device coordinate system to the screen coordinate system used by the smart device to obtain the screen offset information of the smart device;

[0279] The processor 1501 uses the image offset information to perform image compensation on the original video frame, and obtains and displays the corresponding target video frame.

[0280] The processor 1501 is the control center of the computer device, capable of connecting to various parts of the device via various interfaces and lines. It achieves video stabilization by running or executing instructions stored in the memory 1502 and accessing data stored in the memory 1502. Optionally, the processor 1501 may include one or more processing units. The processor 1501 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the processor 1501. In some embodiments, the processor 1501 and the memory 1502 may be implemented on the same chip; in other embodiments, they may be implemented on separate chips.

[0281] Processor 1501 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0282] Memory 1502, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 1502 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 1502 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer device, but is not limited thereto. Memory 1502 in the embodiments of this application may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0283] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the video playback method described above.

[0284] Based on the same inventive concept, this application provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the above-described video playback method.

[0285] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0286] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer apparatus or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0287] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer device or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0288] These computer program instructions may also be loaded onto a computer device or other programmable data processing equipment to cause a series of operational steps to be performed on the computer device or other programmable equipment to produce a process implemented by the computer device, thereby providing instructions that execute on the computer device or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0289] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0290] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A video playback method, characterized in that, include: The system acquires the original video frame to be played on the smart device, and acquires the head motion data of the viewing object of the original video frame; wherein the head motion data is collected from the wearable device used by the viewing object, and the wearable device is equipped with a device coordinate system. Based on the head motion data, the head offset information of the viewing object in the device coordinate system is obtained; The head offset information is mapped from the device coordinate system to the screen coordinate system used by the smart device to obtain the screen offset information of the smart device; Using the aforementioned image offset information, image compensation is performed on the original video frame to obtain and display the corresponding target video frame.

2. The method as described in claim 1, characterized in that, The step of mapping the head offset information from the device coordinate system to the screen coordinate system used by the smart device to obtain the screen offset information of the smart device includes: Based on the screen resolution of the smart device and a preset head offset threshold, the scaling coefficient between the device coordinate system and the screen coordinate system is obtained. Based on the aforementioned proportional mapping coefficient, the head offset information is mapped to the screen offset information of the smart device.

3. The method as described in claim 2, characterized in that, The head offset information includes: pitch angle and yaw angle; the image offset information includes: horizontal offset and vertical offset; The step of mapping the head offset information to the screen offset information of the smart device based on the proportional mapping coefficient includes: The product of the scaling factor and the pitch angle is used as the vertical offset. The product of the scaling factor and the yaw angle is used as the horizontal offset.

4. The method as described in claim 1, characterized in that, The image offset information includes: horizontal offset and vertical offset; the step of using the image offset information to perform image compensation on the original video frame to obtain and display the corresponding target video frame includes: The original video frame is compensated horizontally using the horizontal offset, and the original video frame is compensated vertically using the vertical offset, so as to obtain and display the corresponding target video frame.

5. The method as described in claim 4, characterized in that, The step of performing horizontal image compensation on the original video frame using the horizontal offset and vertical image compensation on the original video frame using the vertical offset to obtain and display the corresponding target video frame includes: Based on the horizontal and vertical offsets, construct a coordinate transformation matrix; For multiple pixels in the original video frame, the following steps are performed: the original coordinates of a pixel are transformed using the coordinate transformation matrix to obtain the target coordinates of the pixel; The target video frame is obtained and displayed based on the target coordinates of each of the multiple pixels.

6. The method as described in claim 1, characterized in that, The step of using the image offset information to perform image compensation on the original video frame to obtain and display the corresponding target video frame includes: The original video frame is enlarged according to a preset ratio to obtain an intermediate video frame; Using the aforementioned image offset information, image compensation is performed on the intermediate video frames to obtain and display the corresponding target video frames.

7. The method as described in any one of claims 1 to 6, characterized in that, The head motion data includes: head angular velocity; the head offset information includes: head offset angle; The step of obtaining the head offset information of the viewing object in the device coordinate system based on the head motion data includes: The time interval for acquiring head movement data by the wearable device; Based on the head angular velocity and the time interval, the head offset angle of the viewing object in the device coordinate system is obtained.

8. The method according to any one of claims 1 to 6, characterized in that, Before acquiring the original video frame to be played on the smart device and the head motion data of the viewer of the original video frame, the method further includes: In response to a mode selection operation triggered in a video application, the video playback mode of the video application is set to anti-shake mode. The video application runs on the smart device and is used to render and display the target video frame.

9. The method as described in claim 8, characterized in that, Also includes: In response to a mode switching operation triggered in the video application, the video playback mode of the video application is switched to non-stabilized mode; The original video frames are rendered and displayed in the video application.

10. A video playback device, characterized in that, include: The acquisition module is used to acquire the original video frame to be played on the smart device, and to acquire the head motion data of the viewing object of the original video frame; wherein, the head motion data is collected from the wearable device used by the viewing object, and the wearable device is equipped with a device coordinate system; The conversion module is used to obtain the head offset information of the viewing object in the device coordinate system based on the head motion data; The mapping module is used to map the head offset information from the device coordinate system to the screen coordinate system used by the smart device to obtain the screen offset information of the smart device. The compensation module is used to perform image compensation on the original video frame using the image offset information, so as to obtain and display the corresponding target video frame.

11. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, It stores a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes a computer program stored on a computer-readable storage medium, the computer program including program instructions that, when executed by a computer device, cause the computer device to perform the steps of the method according to any one of claims 1-9.

Images