Data transmission method and device, storage medium, and electronic device

Abstract

The present invention discloses a data transmission method and device, a storage medium, and an electronic device, which belong to the field of cloud services. The method includes: obtaining panoramic video source data to be transmitted; detecting the motion trajectory of a target object in the panoramic video source data, and obtaining the viewing angle switching direction of the playback end; if the motion trajectory matches the viewing angle switching direction, encoding the target source data in the panoramic video source data to generate panoramic video target data, wherein the target source data is data including the target object; transmitting the panoramic video target data to the playback end. The present invention solves the technical problem that the related technology is prone to freeze when switching the viewing angle of the panoramic video, avoids the switching delay caused by the temporary transmission of high-definition data when switching the viewing angle, reduces the encoding bit rate while maintaining a good viewing experience, and reduces the transmission cost.

Description

Technical Field

[0001] The present invention relates to the field of cloud services, and in particular to a data transmission method and device, a storage medium, and an electronic device. Background Art

[0002] In the related technology, when data is transmitted between a VR (Virtual Reality) server and a client, Omaf-dash (Omni-directional Media Application Format) can be used to encode and transmit the tile (image block) of the panoramic video. The Omaf-dash protocol cuts the video into multiple tiles in the spatial domain, and in the time domain, at a certain time interval, each tile corresponding to a certain length of video is encoded and packaged into a fragmented MP4 file.

[0003] In the related art, transmitting high-definition bit streams will incur a large transmission cost, and when switching the viewing angle to play the key area, the transmission is slow due to the large amount of data temporarily retransmitted, resulting in a poor viewing experience.

[0004] With respect to the above-mentioned problems existing in the related technologies, no effective solutions have been found so far. Summary of the invention

[0005] The embodiments of the present invention provide a data transmission method and device, a storage medium, and an electronic device.

[0006] According to one aspect of an embodiment of the present invention, a data transmission method is provided, comprising: obtaining panoramic video source data to be transmitted; detecting a motion trajectory of a target object in the panoramic video source data, and obtaining a viewing angle switching direction of a playback end; if the motion trajectory matches the viewing angle switching direction, encoding target source data in the panoramic video source data to generate panoramic video target data, wherein the target source data is data including the target object; and transmitting the panoramic video target data to the playback end.

[0007] Furthermore, detecting the motion trajectory of the target object in the panoramic video source data includes: locating a first target object matching a preset picture in multiple image frames of the panoramic video source data; generating candidate areas containing the first target object in the multiple image frames respectively; detecting the motion trajectories of the multiple candidate areas, and determining the motion trajectories of the multiple candidate areas as the motion trajectory of the first target object.

[0008] Furthermore, detecting the motion trajectory of the target object in the panoramic video source data includes: detecting M second target objects in the panoramic video source data; sorting the M second target objects based on the area of ​​the region frame of the second target object and / or the distance of the second target object from the center point to obtain a target sequence; selecting a number of N second target objects that meet predetermined conditions in the target sequence, and detecting the motion trajectories of the N second target objects in the panoramic video source data, wherein N≤M, M, and N are all positive integers.

[0009] Furthermore, obtaining the perspective switching direction of the playback end includes: receiving perspective switching information sent by the playback end, wherein the perspective switching information is used to indicate the current perspective switching mode of the playback end and the perspective moving direction in the current perspective switching mode; parsing the perspective switching direction of the playback end based on the perspective switching information.

[0010] Further, parsing the perspective switching direction of the playback end based on the perspective switching information includes: determining the perspective movement range of the playback end based on the perspective switching mode, wherein the perspective switching mode includes: gaze mode, directional movement mode, and flashing mode; locating the perspective movement direction within the perspective movement range, and determining the perspective movement direction as the perspective switching direction of the playback end.

[0011] Furthermore, obtaining the perspective switching direction of the playback end includes: continuously detecting the playback perspective of the playback end according to a preset period; if it is continuously detected that the playback perspective of the playback end is moving in the same direction within the preset period, the average moving direction of the playback perspective within the preset period is output as the perspective switching direction of the playback end.

[0012] Furthermore, encoding the target source data in the panoramic video source data to generate the panoramic video target data includes: extracting the target source data from the panoramic video source data; encoding the target source data in the panoramic video source data according to a first quantization parameter QP value to obtain first sub-data, and encoding the source data in the panoramic video source data except the target source data according to a second QP value to obtain second sub-data, wherein the first QP value is less than the second QP value; and combining the first sub-data and the second sub-data to obtain the panoramic video target data.

[0013] Furthermore, before encoding the target source data in the panoramic video source data to generate the panoramic video target data, the method also includes: determining whether the angle difference between the motion trajectory and the perspective switching direction is less than a preset threshold; if the angle difference between the motion trajectory and the perspective switching direction is less than the preset threshold, determining that the motion trajectory matches the perspective switching direction; if the angle difference between the motion trajectory and the perspective switching direction is greater than or equal to the preset threshold, determining that the motion trajectory does not match the perspective switching direction.

[0014] According to one aspect of an embodiment of the present invention, another data transmission method is provided, comprising: sending perspective switching information to a server, wherein the perspective switching information is used to indicate a current perspective switching mode of a playback end and a perspective movement direction in the current perspective switching mode; receiving panoramic video target data transmitted by the server, wherein the panoramic video target data is generated by the server according to the perspective switching information and a motion trajectory of a target object in panoramic video source data; and playing the panoramic video target data locally.

[0015] According to another aspect of an embodiment of the present invention, a data transmission device is provided, comprising: an acquisition module, used to acquire panoramic video source data to be transmitted; a processing module, used to detect the motion trajectory of a target object in the panoramic video source data, and to acquire the viewing angle switching direction of a playback end; an encoding module, used to encode the target source data in the panoramic video source data to generate panoramic video target data if the motion trajectory matches the viewing angle switching direction, wherein the target source data is data including the target object; and an output module, used to transmit the panoramic video target data to the playback end.

[0016] Furthermore, the processing module includes: a positioning unit, used to locate a first target object matching a preset picture in multiple image frames of the panoramic video source data; a generation unit, used to respectively generate candidate areas containing the first target object in the multiple image frames; and a first detection unit, used to detect the moving trajectories of the multiple candidate areas, and determine the moving trajectories of the multiple candidate areas as the motion trajectories of the first target object.

[0017] Furthermore, the processing module includes: a second detection unit, used to detect M second target objects in the panoramic video source data; a sorting unit, used to sort the M second target objects based on the area of ​​the region frame of the second target object and / or the distance of the second target object from the center point to obtain a target sequence; a third detection unit, used to select a number of N second target objects that meet predetermined conditions in the target sequence, and detect the motion trajectories of the N second target objects in the panoramic video source data, wherein N≤M, M, N are both positive integers.

[0018] Furthermore, the processing module includes: a receiving unit, used to receive perspective switching information sent by the playback end, wherein the perspective switching information is used to indicate the current perspective switching mode of the playback end and the perspective movement direction in the current perspective switching mode; and a parsing unit, used to parse the perspective switching direction of the playback end based on the perspective switching information.

[0019] Further, the analysis unit includes: a determination subunit, used to determine the perspective movement range of the playback end based on the perspective switching mode, wherein the perspective switching mode includes: a gaze mode, a directional movement mode, and a flashing mode; a positioning subunit, used to locate the perspective movement direction within the perspective movement range, and determine the perspective movement direction as the perspective switching direction of the playback end.

[0020] Furthermore, the processing module includes: a fourth detection unit, used to continuously detect the playback perspective of the playback end according to a preset period; an output unit, used to output the average moving direction of the playback perspective within the preset period as the perspective switching direction of the playback end if it is continuously detected that the playback perspective of the playback end is moving in the same direction within the preset period.

[0021] Furthermore, the encoding module includes: an extraction unit, used to extract the target source data from the panoramic video source data; an encoding unit, used to encode the target source data in the panoramic video source data according to a first quantization parameter QP value to obtain first sub-data, and encode the source data in the panoramic video source data except the target source data according to a second QP value to obtain second sub-data, wherein the first QP value is less than the second QP value; a combining unit, used to combine the first sub-data and the second sub-data to obtain the panoramic video target data.

[0022] Furthermore, the device also includes: a judgment module, which is used to judge whether the angle difference between the motion trajectory and the perspective switching direction is less than a preset threshold before the encoding module encodes the target source data in the panoramic video source data to generate panoramic video target data; a determination module, which is used to determine that the motion trajectory matches the perspective switching direction if the angle difference between the motion trajectory and the perspective switching direction is less than the preset threshold; and determine that the motion trajectory does not match the perspective switching direction if the angle difference between the motion trajectory and the perspective switching direction is greater than or equal to the preset threshold.

[0023] According to another aspect of an embodiment of the present invention, another data transmission device is provided, comprising: a sending module, used to send perspective switching information to a server, wherein the perspective switching information is used to indicate a current perspective switching mode of a playback terminal and a perspective movement direction in the current perspective switching mode; a receiving module, used to receive panoramic video target data transmitted by the server, wherein the panoramic video target data is generated by the server according to the perspective switching information and a motion trajectory of a target object in panoramic video source data; and a playing module, used to play the panoramic video target data locally.

[0024] According to another aspect of an embodiment of the present invention, a storage medium is further provided. The storage medium includes a stored program. When the program is run, the above steps are executed.

[0025] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; wherein: the memory is used to store computer programs; and the processor is used to execute the steps in the above method by running the program stored in the memory.

[0026] The embodiment of the present invention also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps in the above method.

[0027] Through the present invention, panoramic video source data to be transmitted is obtained, the motion trajectory of the target object is detected in the panoramic video source data, and the viewing angle switching direction of the playback end is obtained. If the motion trajectory matches the viewing angle switching direction, the target source data in the panoramic video source data is encoded to generate panoramic video target data, and the panoramic video target data is transmitted to the playback end. By pre-detecting the motion trajectory of the target object and the viewing angle switching direction of the playback end in the transmission stage before switching, the target source data whose motion trajectory matches the viewing angle switching direction is encoded, and the video encoding bit rate is adjusted based on the motion trajectory of the object in the video and the playback viewing angle. The resolution of the picture corresponding to the panoramic video target data is greater than the resolution of the picture corresponding to other data in the panoramic video source data except the panoramic video target data. The technical problem that the related technology is prone to freeze when switching the viewing angle of the panoramic video is solved, and the switching delay caused by the temporary transmission of high-definition data when the viewing angle is switched is avoided. The encoding bit rate is reduced while maintaining a better viewing experience, thereby reducing the transmission cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0029] Figure 1 is a hardware structure block diagram of a server according to an embodiment of the present invention;

[0030] Figure 2 is a flow chart of a data transmission method according to an embodiment of the present invention;

[0031] Figure 3 Schematic diagram of the principles of three viewing modes according to an embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of a panoramic video encoding principle based on the characteristics of the human eye according to an embodiment of the present invention;

[0033] Figure 5 is a flow chart of another data transmission method according to an embodiment of the present invention;

[0034] Figure 6 is a structural block diagram of a data transmission device according to an embodiment of the present invention;

[0035] Figure 7 It is a structural block diagram of an electronic device implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] Example 1

[0039] The method embodiment provided in the first embodiment of the present invention can be executed in a computer, a VR device, a server or a similar data storage device. Taking running on a server as an example, Figure 1 FIG. 1 is a hardware structure diagram of a server according to an embodiment of the present invention. Figure 1 As shown, the server may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the server may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0040] The memory 104 can be used to store server programs, for example, software programs and modules of application software, such as a server program corresponding to a data transmission method in an embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the server program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0041] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0042] In this embodiment, a data transmission method is provided. Figure 2 is a flow chart of a data transmission method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0043] Step S202, obtaining panoramic video source data to be transmitted;

[0044] In this embodiment, the panoramic video source data is the original data before encoding, and needs to be encoded at the server end before being transmitted to the playback end (panoramic video presentation end).

[0045] Step S204, detecting the motion trajectory of the target object in the panoramic video source data, and obtaining the viewing angle switching direction of the playback end;

[0046] Optionally, the motion trajectory of the target object is the dynamic moving direction of the target object in the panoramic video. For example, the panoramic video source data includes three image frames, which are frame 11, frame 22, and frame 33 in time sequence. The positions of the target object in frames 11, 22, and 33 are the lower left corner, the center point, and the upper right corner, respectively. Then, the motion trajectory of the target object in the panoramic video source data is: lower left corner-center point-upper right corner.

[0047] Generally, in directional movement mode and gaze mode, there is a greater probability of paying more attention to moving objects in the video. The moving objects in the video can be located through target detection algorithm and tracking algorithm, and the bit rate allocation of video encoding can be adjusted according to the movement of objects in the video and the viewing mode, so that a lower bit rate can be used to obtain a better viewing experience.

[0048] Step S206, if the motion trajectory matches the view switching direction, encoding the target source data in the panoramic video source data to generate panoramic video target data, wherein the target source data is data including the target object;

[0049] Step S208, transmitting the panoramic video target data to the playback end.

[0050] Optionally, the playback end may be a VR device (such as glasses, a helmet, etc.), a naked-eye display device, etc.

[0051] Through the above steps, the panoramic video source data to be transmitted is obtained, the motion trajectory of the target object is detected in the panoramic video source data, and the viewing angle switching direction of the playback end is obtained. If the motion trajectory matches the viewing angle switching direction, the target source data in the panoramic video source data is encoded to generate panoramic video target data, and the panoramic video target data is transmitted to the playback end. By pre-detecting the motion trajectory of the target object and the viewing angle switching direction of the playback end in the transmission stage before switching, the target source data whose motion trajectory matches the viewing angle switching direction is encoded, and the video encoding bit rate is adjusted based on the motion trajectory of the object in the video and the playback viewing angle. The resolution of the picture corresponding to the panoramic video target data is greater than the resolution of the picture corresponding to other data in the panoramic video source data except the panoramic video target data. The technical problem that the related technology is prone to freeze when switching the viewing angle of the panoramic video is solved, and the switching delay caused by the temporary transmission of high-definition data when the viewing angle is switched is avoided. The encoding bit rate is reduced while maintaining a good viewing experience, thereby reducing the transmission cost.

[0052] In an embodiment, one or more target objects in a panoramic video may be detected.

[0053] In one embodiment, detecting the motion trajectory of a target object in panoramic video source data includes: locating a first target object that matches a preset image in multiple image frames of the panoramic video source data; generating candidate areas containing the first target object in the multiple image frames respectively; detecting the motion trajectories of the multiple candidate areas, and determining the motion trajectories of the multiple candidate areas as the motion trajectory of the first target object.

[0054] In one example, a segmentation algorithm is first used to generate small regions on each of multiple image frames. These small regions are the most basic sub-regions. Then, regions are merged based on the similarity between these regions. The criteria for measuring similarity can be color, texture, size, etc. Such small regions are continuously superimposed until they are all merged into one piece. Then, a circumscribed rectangle is made for each region, and multiple region boxes that may be objects are obtained. The regions of the region boxes are candidate regions. Then, different candidate regions are enlarged to the same size, and the convolution network is collected to classify the candidate regions to find the candidate regions that meet the conditions. For the convolution network, the output vector dimension is required to be the same, so that the support vector machine (SVM) can classify. In order to ensure the same output dimension, the input image size of the convolution network must be consistent. Objects are classified by SVM, so that the first target object is detected from the image frame, and the probability of this object matching the preset image is predicted. After detection, the candidate regions are further classified and calibrated to obtain the final detection result, and the candidate regions with a probability greater than the threshold value are selected as the final output candidate regions.

[0055] Optionally, the first target object may be a protagonist portrait in a panoramic video. For example, in a movie, the protagonist is often the focus of the user. The preset image is the base image of the person that the user pays most attention to in the panoramic video (such as the protagonist). When watching a panoramic video, the user will pay more attention to the movement of the protagonist in the plot scene. The movement trajectory of the protagonist in the panoramic video is likely to be the actual perspective switching trajectory of the user.

[0056] In another implementation of the embodiment, detecting the motion trajectory of the target object in the panoramic video source data includes: detecting M second target objects in the panoramic video source data; sorting the M second target objects based on the area of ​​the region frame of the second target object and / or the distance of the second target object from the center point to obtain a target sequence; selecting a number of N second target objects that meet predetermined conditions in the target sequence, and detecting the motion trajectories of the N second target objects in the panoramic video source data, wherein N≤M, M, and N are all positive integers.

[0057] In one example, a panoramic video includes 5 elements, which may be props, characters, scenery, etc. in the video, namely object 1, object 2, object 3, object 4, and object 5. A multi-target tracking algorithm is used to detect 5 second target objects in the panoramic video source data, wherein the region frame areas of object 1, object 2, object 3, object 4, and object 5 are 1.0, 2.2, 0.8, 3.2, and 1.2, respectively. Based on the region frame area, they are sorted to obtain a target sequence: object 4, object 2, object 5, object 1, and object 3. Then, TOP2 is selected in the target sequence, and object 4 and object 2 are taken as two second target objects that meet predetermined conditions. The motion trajectories of object 4 and object 2 are detected in the panoramic video source data.

[0058] In this embodiment, the perspective switching direction can be obtained at the playback end or the server end. In an implementation scenario, the perspective switching direction is calculated at the playback end, the perspective switching information is obtained, and then sent to the server end. Obtaining the perspective switching direction of the playback end includes: receiving the perspective switching information sent by the playback end, wherein the perspective switching information is used to indicate the current perspective switching mode of the playback end and the perspective movement direction under the current perspective switching mode; parsing the perspective switching direction of the playback end based on the perspective switching information.

[0059] Optionally, analyzing the perspective switching direction of the playback end based on the perspective switching information includes: determining the perspective movement range of the playback end based on the perspective switching mode, wherein the perspective switching mode includes: gaze mode, directional movement mode, and flashing mode; locating the perspective movement direction within the perspective movement range, and determining the perspective movement direction as the perspective switching direction of the playback end.

[0060] Among them, the viewing angle movement range can be an angle range or an area range. The viewing angle movement range of the gaze mode is 0 degrees (in place), or a first circular area with a radius less than a first threshold value. The viewing angle movement range of the directional movement mode is 20 degrees (can be a value within 10 to 180 degrees), or a fan-shaped area with a central angle of 20 degrees. The viewing angle movement range of the flashing mode is 360 degrees, or a second circular area with a radius greater than a second threshold value. By first determining a viewing angle movement range and then locating the viewing angle movement direction within the viewing angle movement range, the locating time of the viewing angle movement direction can be shortened. In addition, if the viewing angle movement direction is not within the viewing angle movement range of the viewing angle switching mode, it means that the user has temporarily changed the gaze mode or the user has performed an action unrelated to watching the video. It is judged that the viewing angle switching information obtained in the current cycle is invalid and the viewing angle switching information needs to be obtained again.

[0061] In one implementation of this embodiment, the server calculates the perspective switching direction based on the playback perspective fed back by the playback end in real time. Obtaining the perspective switching direction of the playback end includes: continuously detecting the playback perspective of the playback end according to a preset period; if the playback perspective of the playback end is continuously detected to be moving in the same direction within the preset period, the average moving direction of the playback perspective within the preset period is output as the perspective switching direction of the playback end.

[0062] In this embodiment, the perspective switching mode corresponds to the user's viewing mode, that is, the state of viewing the panoramic video, such as gaze mode, directional movement mode and flashing mode. In one example, the perspective switching mode includes gaze mode, directional movement mode and flashing mode. Figure 3 It is a schematic diagram of the principle of three viewing modes of an embodiment of the present invention. The gaze mode refers to a viewing mode in which the viewing angle does not change for a long time, indicating that the viewer is watching the content of a fixed viewing angle; the directional movement mode can also be understood as a tracking mode, which means that the viewing angle moves in a certain direction, indicating that the viewer is tracking a moving object in the video or scanning the entire video; the flashing mode means that the viewing angle has no fixed moving direction, indicating that the viewer may be browsing the content of the panoramic video randomly. This embodiment performs the bit rate allocation in the encoding process based on the three viewing modes combined with the characteristics of the human eye, and allocates a higher bit rate to the area that the human eye pays attention to. For example, in the gaze mode, the human eye can better pay attention to the details in the field of vision. Therefore, in the case of the gaze mode, a higher bit rate is allocated to the current focus area for encoding, retaining more details; in the directional movement mode, the user pays more attention to the details of the object consistent with the moving direction, but the area inconsistent with the moving direction will produce motion blur. Therefore, in the case of directional movement, a lower bit rate is allocated to the area inconsistent with the moving direction, which saves the bit rate while not affecting the actual viewing experience; in the flashing mode, the human eye can basically not pay attention to the details, and a lower bit rate can be allocated to the entire picture.

[0063] Figure 4This is a schematic diagram of the panoramic video encoding principle based on the characteristics of the human eye in an embodiment of the present invention. Based on the panoramic video encoding algorithm based on the characteristics of the human eye, the panoramic video can adopt a video playback format based on fov (Field of Vision, viewing angle). By adding preprocessing of the panoramic video source data, the trajectory of the moving object in the panoramic video is obtained through the target detection algorithm or the multi-target tracking algorithm. In addition, the playback end also adds a viewing mode detection function. Each time the playback end returns information about the change of viewing angle, it also returns information about the current viewing mode (such as marking through a flag bit, 0 represents the gaze mode, 1 represents the directional movement mode, and 2 represents the flashing mode). The target detection algorithm and the multi-target tracking algorithm select appropriate deep learning-based models according to the requirements for real-time and computing power. The process of the playback end judging the viewing mode includes: setting a duration time stay , as a preset period, when no FOV change is detected after this fixed time, it is considered to be in the gaze mode. When the FOV change is continuously detected within a fixed time, and along the same direction (an allowable offset range can be set, such as the maximum angle difference in all directions does not exceed 20 degrees, the angle difference can be a set threshold, and if it is less than the set threshold, the direction is considered to be consistent, otherwise the direction is considered inconsistent), it is considered to be in the directional movement mode. If it does not belong to the above two situations, it is considered to be in the flashing mode. Further, in the directional movement mode, take the current time stay The average fov moving direction mv avg As the viewing angle switching direction of the player.

[0064] In one implementation of this embodiment, encoding target source data in the panoramic video source data to generate the panoramic video target data includes: extracting the target source data from the panoramic video source data; encoding the target source data in the panoramic video source data according to a first quantization parameter QP value to obtain first sub-data, and encoding source data other than the target source data in the panoramic video source data according to a second QP value to obtain second sub-data, wherein the first QP value is less than the second QP value; combining the first sub-data and the second sub-data to obtain the panoramic video target data. Since the first QP value is less than the second QP value, the first sub-data encoded by the first QP value is high-definition data, and the second sub-data encoded by the second QP value is low-definition data.

[0065] In an example, high-definition data refers to video data with a resolution greater than or equal to 720p and a bit rate greater than or equal to 3072 kbps, and video data with a resolution less than 720p or a bit rate less than 3072 kbps is low-definition data.

[0066] In one implementation of the present embodiment, after obtaining the panoramic video source data to be transmitted, it also includes: determining the perspective switching mode of the playback end, wherein the perspective switching mode is used to indicate the perspective switching range of the playback end; if the playback end is in the first perspective switching mode, the panoramic video source data is encoded based on the first bit rate, and if the playback end is in the second perspective switching mode, the panoramic video source data is encoded based on the second bit rate, wherein the perspective switching range of the first perspective switching mode is smaller than the perspective switching range of the second perspective switching mode, the first bit rate is greater than the second bit rate, and optionally, the first perspective switching mode is a gaze mode, and the second perspective switching mode is a flashing mode.

[0067] In one example, a QP (Quantizer Parameter) reduction value qp is set. cut When in gaze mode, the tiles of the playback perspective are encoded using normal QP; when in directional movement, the tiles where the objects whose movement trajectories match the perspective switching direction are encoded using normal QP, and all coding tree units (CTUs) in the unmatched tiles are encoded using QP+qp cut Encode to save bit rate; for flash mode, all CTUs of all tiles use QP+qp cut Encode, qp cut Is a positive number.

[0068] In this embodiment, before encoding the target source data in the panoramic video source data to generate the panoramic video target data, it also includes: judging whether the angle difference between the motion trajectory and the viewing angle switching direction is less than a preset threshold; if the angle difference between the motion trajectory and the viewing angle switching direction is less than the preset threshold, determining that the motion trajectory matches the viewing angle switching direction; if the angle difference between the motion trajectory and the viewing angle switching direction is greater than or equal to the preset threshold, determining that the motion trajectory does not match the viewing angle switching direction.

[0069] Optionally, taking the preset threshold of 20 degrees as an example, by matching the moving trajectory of the object in the video, if the difference is less than 20 degrees, it is considered that the moving direction is consistent.

[0070] Figure 5 is a flowchart of another data transmission method according to an embodiment of the present invention, and Figure 2 The embodiment shown corresponds to the video playback client corresponding to the server, such as Figure 5 As shown, the process includes the following steps:

[0071] Step S502: sending perspective switching information to the server, wherein the perspective switching information is used to indicate the current perspective switching mode of the playback end and the perspective moving direction in the current perspective switching mode;

[0072] Step S504, receiving panoramic video target data transmitted by the server, wherein the panoramic video target data is generated by the server according to the viewing angle switching information and the motion trajectory of the target object in the panoramic video source data;

[0073] Step S506: Play the panoramic video target data locally.

[0074] A new viewing mode detection step is added on the playback end. Each time the viewing angle movement direction is returned, the current viewing mode information must also be returned and sent to the server. The server extracts the panoramic video target data from the panoramic video source data to be transmitted based on the viewing angle switching information and the motion trajectory of the target object in the panoramic video source data.

[0075] Based on the scheme of this embodiment, a panoramic video encoding algorithm based on the characteristics of the human eye is proposed. The panoramic video includes a gaze mode, a directional movement mode, and a fast flashing mode. Since the sensitivity of the human eye to details is different in different modes, there is a greater probability of paying more attention to the moving objects in the video in the directional movement mode and the gaze mode. The moving objects in the video can be located through the target detection algorithm and the tracking algorithm. The bit rate of the video encoding is adjusted based on the movement of the objects in the video and the viewing mode, and a lower bit rate can be used to obtain a better viewing experience. This embodiment allocates bit rates based on the mode of video content and viewing angle changes, reduces the bit rate while maintaining a good viewing experience, and reduces the transmission cost.

[0076] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0077] Example 2

[0078] In the present embodiment, a data transmission device is also provided to implement the above-mentioned embodiments and preferred implementation modes, which have been described and will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0079] Figure 6is a structural block diagram of a data transmission device according to an embodiment of the present invention. Figure 6 As shown, the device includes: an acquisition module 60, a processing module 62, an encoding module 64, and an output module 66, wherein:

[0080] An acquisition module 60, used to acquire panoramic video source data to be transmitted;

[0081] A processing module 62, configured to detect a motion trajectory of a target object in the panoramic video source data, and obtain a viewing angle switching direction of a playback end;

[0082] The encoding module 64 is used to encode the target source data in the panoramic video source data to generate panoramic video target data if the motion trajectory matches the view switching direction, wherein the target source data is data including the target object;

[0083] The output module 66 is used to transmit the panoramic video target data to the playback end.

[0084] Optionally, the processing module includes: a positioning unit, used to locate a first target object matching a preset picture in multiple image frames of the panoramic video source data; a generation unit, used to respectively generate candidate areas containing the first target object in the multiple image frames; and a first detection unit, used to detect the moving trajectories of the multiple candidate areas, and determine the moving trajectories of the multiple candidate areas as the motion trajectories of the first target object.

[0085] Optionally, the processing module includes: a second detection unit, used to detect M second target objects in the panoramic video source data; a sorting unit, used to sort the M second target objects based on the area of ​​the region frame of the second target object and / or the distance of the second target object from the center point to obtain a target sequence; a third detection unit, used to select a number of N second target objects that meet predetermined conditions in the target sequence, and detect the motion trajectories of the N second target objects in the panoramic video source data, wherein N≤M, M, N are both positive integers.

[0086] Optionally, the processing module includes: a receiving unit, used to receive perspective switching information sent by the playback end, wherein the perspective switching information is used to indicate a current perspective switching mode of the playback end and a perspective moving direction in the current perspective switching mode; and a parsing unit, used to parse the perspective switching direction of the playback end based on the perspective switching information.

[0087] Optionally, the parsing unit includes: a determination subunit, used to determine the perspective movement range of the playback end based on the perspective switching mode, wherein the perspective switching mode includes: a gaze mode, a directional movement mode, and a flashing mode; a positioning subunit, used to locate the perspective movement direction within the perspective movement range, and determine the perspective movement direction as the perspective switching direction of the playback end.

[0088] Optionally, the processing module includes: a fourth detection unit, used to continuously detect the playback perspective of the playback end according to a preset period; an output unit, used to output the average moving direction of the playback perspective within the preset period as the perspective switching direction of the playback end if it is continuously detected that the playback perspective of the playback end is moving in the same direction within the preset period.

[0089] Optionally, the encoding module includes: an extraction unit, used to extract the target source data from the panoramic video source data; an encoding unit, used to encode the target source data in the panoramic video source data according to a first quantization parameter QP value to obtain first sub-data, and encode the source data in the panoramic video source data except the target source data according to a second QP value to obtain second sub-data, wherein the first QP value is less than the second QP value; and a combining unit, used to combine the first sub-data and the second sub-data to obtain the panoramic video target data.

[0090] Optionally, the device also includes: a judgment module, used to judge whether the angle difference between the motion trajectory and the perspective switching direction is less than a preset threshold before the encoding module encodes the target source data in the panoramic video source data to generate panoramic video target data; a determination module, used to determine that the motion trajectory matches the perspective switching direction if the angle difference between the motion trajectory and the perspective switching direction is less than a preset threshold; and determine that the motion trajectory does not match the perspective switching direction if the angle difference between the motion trajectory and the perspective switching direction is greater than or equal to a preset threshold.

[0091] This embodiment also provides another data transmission device, including: a sending module, used to send perspective switching information to a server, wherein the perspective switching information is used to indicate the current perspective switching mode of the playback end and the perspective movement direction in the current perspective switching mode; a receiving module, used to receive panoramic video target data transmitted by the server, wherein the panoramic video target data is generated by the server according to the perspective switching information and the motion trajectory of the target object in the panoramic video source data; and a playing module, used to play the panoramic video target data locally.

[0092] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0093] Example 3

[0094] The embodiment of the present application also provides an electronic device, Figure 7 is a structural diagram of an electronic device according to an embodiment of the present invention, such as Figure 7 As shown, it includes a processor 71, a communication interface 72, a memory 73 and a communication bus 74, wherein the processor 71, the communication interface 72 and the memory 73 communicate with each other through the communication bus 74, and the memory 73 is used to store computer programs; the processor 71 is used to implement the following steps when executing the program stored in the memory 73: obtaining panoramic video source data to be transmitted; detecting the motion trajectory of the target object in the panoramic video source data, and obtaining the viewing angle switching direction of the playback end; if the motion trajectory matches the viewing angle switching direction, encoding the target source data in the panoramic video source data to generate panoramic video target data, wherein the target source data is data including the target object; transmitting the panoramic video target data to the playback end.

[0095] Furthermore, detecting the motion trajectory of the target object in the panoramic video source data includes: locating a first target object matching a preset picture in multiple image frames of the panoramic video source data; generating candidate areas containing the first target object in the multiple image frames respectively; detecting the motion trajectories of the multiple candidate areas, and determining the motion trajectories of the multiple candidate areas as the motion trajectory of the first target object.

[0096] Furthermore, detecting the motion trajectory of the target object in the panoramic video source data includes: detecting M second target objects in the panoramic video source data; sorting the M second target objects based on the area of ​​the region frame of the second target object and / or the distance of the second target object from the center point to obtain a target sequence; selecting a number of N second target objects that meet predetermined conditions in the target sequence, and detecting the motion trajectories of the N second target objects in the panoramic video source data, wherein N≤M, M, and N are all positive integers.

[0097] Furthermore, obtaining the perspective switching direction of the playback end includes: receiving perspective switching information sent by the playback end, wherein the perspective switching information is used to indicate the current perspective switching mode of the playback end and the perspective moving direction in the current perspective switching mode; parsing the perspective switching direction of the playback end based on the perspective switching information.

[0098] Further, parsing the perspective switching direction of the playback end based on the perspective switching information includes: determining the perspective movement range of the playback end based on the perspective switching mode, wherein the perspective switching mode includes: gaze mode, directional movement mode, and flashing mode; locating the perspective movement direction within the perspective movement range, and determining the perspective movement direction as the perspective switching direction of the playback end.

[0099] Furthermore, obtaining the perspective switching direction of the playback end includes: continuously detecting the playback perspective of the playback end according to a preset period; if it is continuously detected that the playback perspective of the playback end is moving in the same direction within the preset period, the average moving direction of the playback perspective within the preset period is output as the perspective switching direction of the playback end.

[0100] Furthermore, encoding the target source data in the panoramic video source data to generate the panoramic video target data includes: extracting the target source data from the panoramic video source data; encoding the target source data in the panoramic video source data according to a first quantization parameter QP value to obtain first sub-data, and encoding the source data in the panoramic video source data except the target source data according to a second QP value to obtain second sub-data, wherein the first QP value is less than the second QP value; and combining the first sub-data and the second sub-data to obtain the panoramic video target data.

[0101] Furthermore, before encoding the target source data in the panoramic video source data to generate the panoramic video target data, the method also includes: determining whether the angle difference between the motion trajectory and the perspective switching direction is less than a preset threshold; if the angle difference between the motion trajectory and the perspective switching direction is less than the preset threshold, determining that the motion trajectory matches the perspective switching direction; if the angle difference between the motion trajectory and the perspective switching direction is greater than or equal to the preset threshold, determining that the motion trajectory does not match the perspective switching direction.

[0102] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0103] The communication interface is used for communication between the above terminal and other devices.

[0104] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0105] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0106] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which instructions are stored. When the computer-readable storage medium is executed on a computer, the computer executes the data transmission method described in any one of the above embodiments.

[0107] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the data transmission method described in any one of the above embodiments.

[0108] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0109] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

[0110] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A data transmission method, characterized in that: include: Obtaining panoramic video source data to be transmitted; Detecting the motion trajectory of the target object in the panoramic video source data, and obtaining the viewing angle switching direction of the playback end; If the motion trajectory matches the view switching direction, encoding the target source data in the panoramic video source data to generate panoramic video target data, wherein the target source data is data including the target object; Transmitting the panoramic video target data to the playback end; Wherein, before encoding the target source data in the panoramic video source data to generate the panoramic video target data, the method also includes: judging whether the angle difference between the motion trajectory and the perspective switching direction is less than a preset threshold; if the angle difference between the motion trajectory and the perspective switching direction is less than the preset threshold, determining that the motion trajectory matches the perspective switching direction; if the angle difference between the motion trajectory and the perspective switching direction is greater than or equal to the preset threshold, determining that the motion trajectory does not match the perspective switching direction.

2. The method according to claim 1, characterized in that: Detecting the motion trajectory of the target object in the panoramic video source data includes: Locating a first target object matching a preset picture in a plurality of image frames of the panoramic video source data; Generating candidate regions including the first target object in the multiple image frames respectively; The moving trajectories of the plurality of candidate areas are detected, and the moving trajectories of the plurality of candidate areas are determined as the moving trajectories of the first target object.

3. The method according to claim 1, characterized in that Detecting the motion trajectory of the target object in the panoramic video source data includes: Detecting M second target objects in the panoramic video source data; Sort the M second target objects based on the area of ​​the region frame of the second target object and / or the distance of the second target object from the center point to obtain a target sequence; A number of N second target objects meeting a predetermined condition are selected in the target sequence, and motion trajectories of the N second target objects are detected in the panoramic video source data, wherein N≤M, M, and N are both positive integers.

4. The method according to claim 1, characterized in that: Obtaining the viewing angle switching direction of the player includes: Receiving perspective switching information sent by the playback end, wherein the perspective switching information is used to indicate a current perspective switching mode of the playback end and a perspective moving direction in the current perspective switching mode; The perspective switching direction of the playback end is analyzed based on the perspective switching information.

5. The method according to claim 4, characterized in that Analyzing the perspective switching direction of the playback end based on the perspective switching information includes: Determining the viewing angle movement range of the playback end based on the viewing angle switching mode, wherein the viewing angle switching mode includes: a gaze mode, a directional movement mode, and a flashing mode; The viewing angle moving direction is located within the viewing angle moving range, and the viewing angle moving direction is determined as the viewing angle switching direction of the playback end.

6. The method according to claim 1, characterized in that Obtaining the viewing angle switching direction of the player includes: Continuously detecting the playback viewing angle of the playback end according to a preset period; If it is continuously detected that the viewing angle of the playback end moves in the same direction within the preset period, the average moving direction of the viewing angle within the preset period is output as the viewing angle switching direction of the playback end.

7. The method according to claim 1, characterized in that Encoding the target source data in the panoramic video source data to generate the panoramic video target data includes: Extracting the target source data from the panoramic video source data; encoding target source data in the panoramic video source data according to a first QP value to obtain first sub-data, and encoding source data other than the target source data in the panoramic video source data according to a second QP value to obtain second sub-data, wherein the first QP value is less than the second QP value; The first sub-data and the second sub-data are combined to obtain the panoramic video target data.

8. A data transmission method, characterized in that: include: Sending perspective switching information to the server, wherein the perspective switching information is used to indicate the current perspective switching mode of the playback end and the perspective movement direction in the current perspective switching mode; Receiving panoramic video target data transmitted by the server, wherein the panoramic video target data is generated by the server according to the viewing angle switching information and a motion trajectory of a target object in the panoramic video source data; Playing the panoramic video target data locally; Among them, before the server encodes the target source data in the panoramic video source data to generate the panoramic video target data, the method also includes: judging whether the angle difference between the motion trajectory and the perspective switching direction is less than a preset threshold; if the angle difference between the motion trajectory and the perspective switching direction is less than the preset threshold, determining that the motion trajectory matches the perspective switching direction; if the angle difference between the motion trajectory and the perspective switching direction is greater than or equal to the preset threshold, determining that the motion trajectory does not match the perspective switching direction.

9. A data transmission device, characterized in that: include: An acquisition module, used to acquire panoramic video source data to be transmitted; A processing module, used to detect the motion trajectory of the target object in the panoramic video source data, and obtain the viewing angle switching direction of the playback end; an encoding module, configured to encode target source data in the panoramic video source data to generate panoramic video target data if the motion trajectory matches the view switching direction, wherein the target source data is data including the target object; An output module, used for transmitting the panoramic video target data to the playback end; The device further includes: a judgment module, used for judging whether the angle difference between the motion trajectory and the perspective switching direction is less than a preset threshold before the encoding module encodes the target source data in the panoramic video source data to generate the panoramic video target data; a determination module, used for determining that the motion trajectory matches the perspective switching direction if the angle difference between the motion trajectory and the perspective switching direction is less than the preset threshold; and determining that the motion trajectory does not match the perspective switching direction if the angle difference between the motion trajectory and the perspective switching direction is greater than or equal to the preset threshold.

10. A data transmission device, characterized in that: include: A sending module, used to send perspective switching information to the server, wherein the perspective switching information is used to indicate the current perspective switching mode of the playback end and the perspective moving direction in the current perspective switching mode; A receiving module, configured to receive the panoramic video target data transmitted by the server, wherein the panoramic video target data is generated by the server according to the viewing angle switching information and the motion trajectory of the target object in the panoramic video source data; A playback module, used for playing the panoramic video target data locally; Among them, the server includes: a judgment module, which is used to judge whether the angle difference between the motion trajectory and the perspective switching direction is less than a preset threshold before the encoding module encodes the target source data in the panoramic video source data to generate the panoramic video target data; a determination module, which is used to determine that the motion trajectory matches the perspective switching direction if the angle difference between the motion trajectory and the perspective switching direction is less than the preset threshold; if the angle difference between the motion trajectory and the perspective switching direction is greater than or equal to the preset threshold, determine that the motion trajectory does not match the perspective switching direction.

11. A storage medium, characterized in that: The storage medium includes a stored program, wherein the program executes the steps of any one of the methods of claims 1 to 8 when executed.

12. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein: The processor, the communication interface, and the memory communicate with each other via a communication bus; wherein: Memory, used to store computer programs; A processor, configured to execute the steps of the method according to any one of claims 1 to 8 by running a program stored in a memory.

Images