Video data transmission method and system, and electronic device and storage medium

Abstract

Disclosed in the present invention is a video data transmission method for a network device. The method comprises: parsing a video frame to be processed, so as to determine a video structure type of the video frame; encoding the video frame on the basis of the video structure type, and determining a data insertion point of the video frame; acquiring gyroscope data, and inserting the gyroscope data into the data insertion point, so as to generate a video data stream including the gyroscope data; and sending to a video terminal the video data stream including the gyroscope data. Thus, the method achieves the synchronization between video stream data and gyroscope data under the framework of a standard protocol. Further exemplarily disclosed in the present invention are an electronic device including gyroscope data, a system, and a storage medium.

Description

Video data transmission method, system, electronic device and storage medium Technical Field

[0001] Exemplary embodiments of the present invention relate to the field of video coding and decoding, and more particularly to a video transmission method, system, electronic device, and storage medium with gyroscope data. Background Art

[0002] With the rapid development of video applications, various video terminals are experiencing numerous issues when playing video streaming data, such as difficulty rotating live images and even high-frequency jitter. This significantly hinders video viewing, compression encoding, content analysis, and detail recognition. To improve the quality of rotated live video, image angle adjustment technology has been widely adopted by various video terminals.

[0003] Video rotation technology can be roughly divided into mechanical, optical, electronic and digital methods. The most typical technology is to use gyroscopes to obtain the changes in the position and posture of the video terminal, and then apply them to image post-processing to eliminate the problem of unstable video images caused by the changes in the position and posture of the video terminal. Among them, there are two common processing methods for using gyroscopes to eliminate image jitter: (1) Separate transmission: transmit the device's video stream and gyroscope angle separately. However, this method has a significant disadvantage, that is, the gyroscope data and video stream data are not synchronized. On the mobile phone side, it is necessary to rotate the device image in real time according to the gyroscope angle. However, due to the lag in data transmission, the gyroscope angle may not correspond to the image in time, resulting in untimely processing. (2) Combined transmission: transmit the device's video stream and gyroscope angle together, but add an extension header before the video data for processing. This solves the problem of data synchronization, but introduces new challenges, that is, it is impossible to use standard protocols such as RTP to transmit code streams, resulting in third-party players being unable to play normally.

[0004] To overcome these issues, a technical solution is needed that can synchronize the device's video stream and gyroscope data within the framework of a standard protocol. This will make real-time image rotation more efficient and reliable, while ensuring compatibility. Summary of the Invention

[0005] Based on this, the present invention provides a video transmission method, device and storage medium with gyroscope data to at least solve the problems existing in the prior art, that is, how to achieve synchronization between the device video stream and gyroscope data while being able to implement it within the framework of the standard protocol.

[0006] In order to achieve the above-mentioned purpose, according to a first aspect of the present invention, a video data transmission method for a network device is provided, comprising: parsing a video frame to be processed, and determining a video structure type of the video frame; encoding the video frame based on the video structure type, and determining a data insertion point of the video frame; acquiring gyroscope data, and inserting the gyroscope data into the data insertion point to generate a video data stream having the gyroscope data; and sending the video data stream having the gyroscope data to a video terminal.

[0007] In some embodiments, determining the data insertion point of the video frame includes: when the video structure type is the H.264 protocol or the H.265 protocol, determining that the data insertion point of the data stream is located in the supplemental enhancement information unit; or when the video structure type is the MJPEG protocol, determining that the data insertion point of the data stream is located before the end marker.

[0008] In some embodiments, before sending the video data stream having the gyroscope data to the video terminal, the method further includes: synchronizing the timestamp of the gyroscope data with the timestamp of the video frame.

[0009] In some embodiments, the gyroscope data includes: gyroscope rotational angular velocity information and / or gyroscope movement acceleration information.

[0010] In some embodiments, the gyroscope data includes: respectively corresponding the gyroscope data to each frame data in the video frame.

[0011] In some embodiments, obtaining gyroscope data includes: using a gravity sensor to read an acceleration value to obtain gyroscope movement acceleration information; and / or using an angular velocity sensor to read an angular velocity value to obtain gyroscope movement acceleration information.

[0012] In a second aspect, the present invention also provides a video data transmission method for a video terminal, comprising: receiving a video data stream having gyroscope data sent by a network device; parsing the video data stream to obtain the gyroscope data; and playing the video data stream based on the gyroscope data; wherein the video data stream is a data stream having the gyroscope data at a data insertion point of the network device based on a video structure type.

[0013] In a third aspect, the present invention also provides a system for video data transmission, comprising: a network device, configured to parse a video frame to be processed and determine a video structure type of the video frame; encode the video frame based on the video structure type and determine a data insertion point of the video frame; obtain gyroscope data and insert the gyroscope data into the data insertion point to generate a video data stream having the gyroscope data; send the video data stream having the gyroscope data to a video terminal; and a terminal, configured to receive a video data stream having gyroscope data sent by the network device; parse the video data stream to obtain the gyroscope data; and play the video data stream based on the gyroscope data; wherein the video data stream is a data stream having the gyroscope data at a data insertion point based on the video structure type of the network device.

[0014] In a fourth aspect, the present invention further provides an electronic device comprising: a memory for storing a computer program product; and a processor for executing the computer program product stored in the memory, wherein the computer program product implements the above method when executed.

[0015] In a fifth aspect, the present invention further provides a non-transitory computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above method.

[0016] The present invention generates a video data stream containing the gyroscope data by inserting the gyroscope data into the data insertion point of the data stream of the predetermined protocol, thereby achieving synchronization between the device video stream and the gyroscope data while complying with the framework of the standard protocol. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without exceeding the scope of protection required by the present invention.

[0018] FIG1 is a schematic flow chart showing a method for transmitting video data in a network device according to an exemplary embodiment of the present invention;

[0019] FIG2 is a schematic flow chart showing a method for transmitting video data for a video terminal according to an exemplary embodiment of the present invention;

[0020] FIG3 shows a schematic diagram of an electronic device according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] According to a typical embodiment of the present invention, FIG1 shows a flow chart of a method 100 for transmitting video data on a network device according to an exemplary embodiment of the present invention. The method 100 provides a method for transmitting video data on a network device, including:

[0024] Step 102: Determine the video structure type of the video frame, that is, parse the video frame to be processed and determine the video structure type of the video frame.

[0025] In some exemplary embodiments of the present invention, a video frame to be processed may be first acquired, and then the acquired video frame data stream may be sent to an image signal processor (ISP) for the ISP to perform typical ISP functions such as linearity correction, noise reduction, bad pixel removal, interpolation, white balancing, and color conversion. Furthermore, in this embodiment, the video frame to be processed may be parsed to determine the video structure type of the video frame; the video structure type may include, but is not limited to, a standard video codec protocol, such as the H.264 protocol, the H.265 protocol, or the MJPEG protocol.

[0026] Step 104: Determine a data insertion point, that is, encode the video frame based on the video structure type and determine a data insertion point of the video frame.

[0027] In some exemplary embodiments of the present invention, the data insertion point can be adapted according to different video structure types. For example, when the video structure type is the H.264 protocol or the H.265 protocol, it can be determined that the data insertion point of the data stream is located in the supplemental enhancement information unit (Supplemental Enhancement Information); or when the video structure type is the MJPEG protocol, it can be determined that the data insertion point of the data stream is located before the end marker (FFD9).

[0028] Step 106: inserting gyroscope data, that is, acquiring gyroscope data, and inserting the gyroscope data into the data insertion point to generate a video data stream having the gyroscope data.

[0029] In some exemplary embodiments of the present invention, when the predetermined protocol is the MJPEG standard, the data format of the gyroscope may include the following fields: a type field (iva_type), an identification field (is_end_flag), a length field (iva_len) and a content field (iva_data); wherein the type field (iva_type) indicates the type of data carried, so as to facilitate the expansion of multiple data types according to the needs of the application scenario; the identification field (is_end_flag) indicates whether it is necessary to continue parsing, so as to facilitate the expansion of multiple data according to the needs of the application scenario; the length field (iva_len) indicates the data length; and the content field (iva_data) indicates the data length content.

[0030] Gyroscope data may include: gyroscope angular velocity information and / or gyroscope motion acceleration information. The gyroscope angular velocity information is specifically the angular velocity information recorded by the gyroscope corresponding to the video frame data at a certain time point (moment) on the three-dimensional spatial axis. The angular velocity varies over time, and by integrating the angular velocity, the three-dimensional rotation angle corresponding to different time points can be obtained. Furthermore, by filtering the gyroscope angular velocity information, it is possible to separate and determine the intentional movement of the mobile terminal (generally corresponding to low-frequency components) and random movement caused by jitter, etc. (generally corresponding to high-frequency components). The gyroscope motion acceleration information is specifically the motion acceleration information recorded by the gyroscope in the mobile terminal corresponding to the video frame data at a certain time point (moment). By determining the motion acceleration, the posture and position changes corresponding to the video frame data during the recording process, as well as the impact of these posture and position changes on the video image, can be more accurately determined. Compared with the gyroscope angular velocity information, the gyroscope motion acceleration information focuses more on the local displacement of the mobile terminal.

[0031] In some exemplary embodiments of the present invention, obtaining gyroscope data may include: using a gravity sensor to read an accelerometer value to obtain gyroscope movement acceleration information; and / or using an angular velocity sensor to read an angular velocity value to obtain gyroscope movement acceleration information.

[0032] Compared to the MJPEG standard, the Supplemental Enhancement Information (SEI) unit is a concept within the codestream, providing a method for adding information to the video codestream. It is a feature of the H.264 / H.265 video compression standard. In some exemplary embodiments of the present invention, gyroscope data can be written to the SEI, and the data format definition is consistent with MJPEG.

[0033] Step 108: Send the video data stream to the video terminal, that is, send the video data stream with the gyroscope data to the video terminal.

[0034] In some exemplary embodiments of the present invention, the video data stream having the gyroscope data is sent to the video terminal, so that the video terminal plays the video frame after parsing the received video data stream to obtain the gyroscope data, and further comprises: according to the H.264 protocol or the H.265 protocol, the video data stream having the gyroscope data is packetized to obtain multiple groups of data packets, and the multiple groups of data packets are sequentially sent to the video terminal, so that the video terminal receives the multiple groups of data packets and merges the multiple groups of data packets to obtain the video frame in compliance with the H.264 protocol. or H.265 protocol, parse the received video data stream to obtain the gyroscope data, and then play the video frame; or, according to the MJPEG protocol, packetize the video data stream containing the gyroscope data to obtain multiple data packets, and send the multiple data packets in sequence to the video terminal, so that the video terminal receives the multiple data packets and merges the multiple data packets to obtain a data stream compliant with the MJPEG protocol, and then parses the received video data stream to obtain the gyroscope data, and then plays the first video frame. In some exemplary embodiments of the present invention, the method may further include: respectively corresponding the gyroscope data to each frame of the video frame, so that each frame of the data is synchronized with the gyroscope data.

[0035] In some exemplary embodiments of the present invention, before sending the video data stream having the gyroscope data to the video terminal, the method further includes: synchronizing the timestamp of the gyroscope data with the timestamp of the video frame.

[0036] FIG2 shows a flow chart of a video data transmission method 200 for a video terminal according to an exemplary embodiment of the present invention. The method provides a video data transmission method for a video terminal, comprising:

[0037] Step 202: receiving a video data stream having gyroscope data, that is, the video terminal receives the video data stream having gyroscope data sent by the network device;

[0038] Step 204: parsing the video data stream, that is, parsing the video data stream to obtain the gyroscope data;

[0039] Step 206: Play the video data stream, that is, play the video data stream based on the gyroscope data; wherein the video data stream is a data stream having the gyroscope data at a data insertion point based on a video structure type of the network device.

[0040] The present invention also discloses a system for video data transmission, characterized in that it includes: a network device, used to parse the video frame to be processed and determine the video structure type of the video frame; encode the video frame based on the video structure type and determine the data insertion point of the video frame; obtain gyroscope data and insert the gyroscope data into the data insertion point to generate a video data stream with the gyroscope data; send the video data stream with the gyroscope data to a video terminal; and a terminal, used to receive the video data stream with the gyroscope data sent by the network device; parse the video data stream to obtain the gyroscope data; play the video data stream based on the gyroscope data; wherein the video data stream is a data stream having the gyroscope data at the data insertion point of the network device based on the video structure type.

[0041] Figure 3 shows a schematic diagram of an electronic device 300 according to an exemplary embodiment of the present invention. The electronic device 300 may include: at least one processor 302; and at least one memory 304 including computer program code. The at least one memory 304 and the computer program product 306 are configured to utilize the at least one processor 302 to cause the electronic device 300 to execute: The electronic device may implement the steps of the video data transmission method for a network device or a video terminal according to the above-described embodiment of the present invention.

[0042] The present invention also discloses a computer-readable storage medium storing a computer program. The computer program can implement the steps of the video data transmission method for a network device or a video terminal in the above embodiment of the present invention.

[0043] The processor may be a CPU or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0044] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the task generation method of each embodiment of the present invention described above and / or other desired functions.

[0045] In one example, the electronic device may further include an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0046] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0047] The output device can output various information to the outside, including determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0048] Of course, for simplicity, the present invention only shows some of the components related to the present invention in the electronic device, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may further include any other appropriate components according to specific application scenarios.

[0049] In addition to the above-mentioned methods and devices, an embodiment of the present invention may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to perform the steps of the task generation method according to various embodiments of the present invention described in the above part of this specification.

[0050] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0051] In the several embodiments provided herein, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is merely a logical functional division. Actual implementations may employ other divisions, such as combining or integrating multiple units or components into another system, or omitting or disabling certain features. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through interfaces, or indirect coupling or communication connection between units or modules, either electrical or otherwise.

[0052] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected based on actual needs to achieve the purpose of the solution provided in this embodiment.

[0053] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0054] In addition, an embodiment of the present invention may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps of the task generation method according to various embodiments of the present invention described in the above part of this specification.

[0055] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0056] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk, etc. Various media that can store program codes.

[0057] The basic principles of the present invention have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in the present invention are merely illustrative and non-limiting, and should not be construed as necessarily possessed by each embodiment of the present invention. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, and are not intended to be limiting. These details do not necessarily limit the present invention to being implemented using these specific details.

[0058] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A video data transmission method for a network device, characterized in that, including: Analyze a video frame to be processed and determine the video structure type of the video frame; Encode the video frame based on the video structure type and determine the data insertion point of the video frame; Obtain gyroscope data and insert the gyroscope data into the data insertion point to generate a video data stream with the gyroscope data; and Send the video data stream with the gyroscope data to a video terminal.

2. The video data transmission method for a network device according to claim 1, wherein Determining the data insertion point of the video frame includes: When the video structure type is the H.264 protocol or the H.265 protocol, determining that the data insertion point of the data stream is located in the supplementary enhancement information unit; or When the video structure type is the MJPEG protocol, determining that the data insertion point of the data stream is located before the end flag.

3. The video data transmission method for a network device according to claim 1, wherein Before sending the video data stream with the gyroscope data to the video terminal, it further includes: Synchronize the timestamp of the gyroscope data with the timestamp of the video frame.

4. The video data transmission method for a network device according to any one of claims 1 to 3, wherein, The gyroscope data includes: gyroscope rotation angular velocity information and / or gyroscope movement acceleration information.

5. The video data transmission method for a network device according to claim 4, wherein It further includes: Correspond the gyroscope data to each frame of data in the video frame respectively.

6. The video data transmission method for a network device according to claim 5, wherein Obtaining gyroscope data includes: Use a gravity sensor to read the accelerometer value to obtain gyroscope movement acceleration information; and / or Use an angular velocity sensor to read the angular velocity value to obtain gyroscope movement acceleration information.

7. A video data transmission method for a video terminal, characterized in that, including: Receive a video data stream with gyroscope data sent by a network device; Analyze the video data stream to obtain the gyroscope data; Play the video data stream based on the gyroscope data; wherein The video data stream is a data stream in which the network device has the gyroscope data at the data insertion point based on the video structure type.

8. A system for video data transmission, characterized in that, including: A network device for analyzing a video frame to be processed and determining the video structure type of the video frame; Encode the video frame based on the video structure type and determine the data insertion point of the video frame; Obtain gyroscope data and insert the gyroscope data into the data insertion point to generate a video data stream with the gyroscope data; Send the video data stream with the gyroscope data to a video terminal; and A terminal for receiving a video data stream with gyroscope data sent by the network device; Analyze the video data stream to obtain the gyroscope data; Play the video data stream based on the gyroscope data; wherein the video data stream is a data stream in which the network device has the gyroscope data at the data insertion point based on the video structure type.

9. An electronic device, characterized in that, including: A memory for storing a computer program product; and A processor for executing the computer program product stored in the memory, and when the computer program product is executed, implementing the method according to any one of claims 1 to 7 above.

10. A non-transitory computer-readable storage medium, on which computer-readable instructions are stored, and when the instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 7.

Images