Video processing method based on video synthesis platform, video synthesis platform and device

Abstract

The application provides a video processing method based on a video synthesis platform, a video synthesis platform and equipment, and the method comprises the following steps: determining a first time difference and at least one second time difference through a master control module, and sending the first time difference to an input module and sending each second time difference to a corresponding decoding module; the first time difference is the time difference of the input module relative to the master control module, and the second time difference is the time difference of the decoding module relative to the master control module; determining the sending time sequence value of the collected video data according to the first time difference through the input module, and sending the video data and the corresponding sending time sequence value to each decoding module; for any decoding module, determining the current video data to be displayed according to the second time difference and the sending time sequence value of the video data through the decoding module, and sending the current video data to be displayed. Thus, the synchronous sending display of the video data in the video synthesis platform can be realized, and the synchronous display effect of the video data is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video data processing, in particular to a video processing method based on a video synthesis platform, a video synthesis platform and equipment. BACKGROUND

[0002] The video synthesis platform is a kind of telecom-grade rack-type video processing synthesis platform product, which usually includes a main control, an input board card and a decoding output board card. The input board card is mainly used for collecting video data, and the decoding output board card is used for receiving video data and decoding and displaying the video data.

[0003] However, since the start-up time of the main control, the multiple input board cards and the multiple decoding output board cards is not fixed, there is a time difference between the various board cards, which will cause different decoding output board cards to display the same frame of video data at different times, resulting in the problem of asynchronous display. In addition, in the related video synthesis platform technology, when the network stream and the local stream need to be processed and displayed at the same time, the network stream and the local stream will interfere with each other, thereby affecting the efficiency of the display. SUMMARY

[0004] The present application aims to at least solve one of the technical problems in the related art. To this end, the present application provides a video processing method based on a video synthesis platform, a video synthesis platform and equipment, to realize the synchronous display of video data in the video synthesis platform and ensure the synchronous display effect of the video data.

[0005] The present application provides a video processing method based on a video synthesis platform, the video synthesis platform including a main control module, an input module and at least one decoding module connected to each other, the method comprising:

[0006] determining a first time difference and at least one second time difference by the main control module, and sending the first time difference to the input module and each second time difference to the corresponding decoding module; the first time difference is the time difference of the input module relative to the main control module, and the second time difference is the time difference of the decoding module relative to the main control module;

[0007] determining the sending time sequence value of the collected video data by the input module according to the first time difference, and sending the video data and the corresponding sending time sequence value to each decoding module;

[0008] for any decoding module, determining the current video data to be displayed by the decoding module according to the second time difference and the sending time sequence value of the video data, and displaying the current video data to be displayed.

[0009] In some possible implementation manners, the determining, by the master module, of the first time difference and the at least one second time difference comprises: obtaining a master starting duration, a master clock period of the master module, an input starting duration, an input clock period of the input module, a decoding starting duration, and a decoding clock period of each decoding module; determining a master timing value according to the master starting duration and the master clock period, determining an input timing value according to the input starting duration and the input clock period, and determining the first time difference according to the master timing value and the input timing value; determining a decoding timing value according to the decoding starting duration and the decoding clock period, and determining the second time difference according to the master timing value and the decoding timing value.

[0010] In some possible implementation manners, the determining, by the input module, of the sending timing value of the collected video data according to the first time difference comprises: determining a synchronized input timing value according to the first time difference and the input timing value; and determining the sending timing value of the video data according to the synchronized input timing value.

[0011] In some possible implementation manners, each decoding module comprises at least two first buffer units, and different first buffer units are used to buffer different frame video data; and the determining, by the decoding module, of the current video data to be displayed according to the second time difference and the sending timing value of the video data comprises: for any decoding module, determining a synchronized decoding timing value according to the second time difference and the decoding timing value; determining the current video data to be displayed according to the synchronized decoding timing value and the sending timing value, and calling the current video data to be displayed from a first buffer unit in which the current video data to be displayed is located.

[0012] In some possible implementation manners, the determining, according to the synchronized decoding timing value and the sending timing value, of the current video data to be displayed comprises: in a case where the sending timing value meets a preset condition, determining the video data corresponding to the sending timing value as the current video data to be displayed; and the preset condition comprises that the sending timing value is adjacent to the synchronized decoding timing value, and the sending timing value is less than or equal to the synchronized decoding timing value.

[0013] In some possible implementation manners, the determining, by the decoding module, of the current video data to be displayed according to the second time difference and the sending timing value of the video data comprises: adjusting initial phases of the decoding modules to be consistent; and determining, by the adjusted decoding modules, the current video data to be displayed according to the second time difference and the sending timing value of the video data.

[0014] In some possible implementation manners, the decoding module further includes a second buffer unit and a third buffer unit, and the method further includes: receiving the video data and the corresponding sending time sequence value by the second buffer unit; and sending the video data and the corresponding sending time sequence value to the first buffer unit for buffering by the third buffer unit.

[0015] In some possible implementation manners, the number of decoding modules is at least two, the video synthesis platform further includes a first switch and a second switch, the video data includes network video data and local video data, and the method further includes: receiving the network video data by the first switch, sending the network video data to any decoding module, and forwarding the network video data to other decoding modules by the decoding module; and sending the local video data acquired by the input module to each decoding module by the second switch.

[0016] The application further provides a video synthesis platform, including: a master control module, an input module, and at least one decoding module connected with each other; the master control module is configured to determine a first time difference and at least one second time difference, send the first time difference to the input module, and send each second time difference to the corresponding decoding module; the first time difference is a time difference of the input module relative to the master control module, and the second time difference is a time difference of the decoding module relative to the master control module; the input module is configured to determine a sending time sequence value of acquired video data according to the first time difference, and send the video data and the corresponding sending time sequence value to each decoding module; and the decoding module is configured to determine current video data to be displayed according to the second time difference and the sending time sequence value of the video data, and send the current video data to be displayed.

[0017] In some possible implementation manners, the number of the decoding modules is at least two, the video synthesis platform further includes a first switch and a second switch, and the video data includes network video data and local video data; the first switch is configured to receive the network video data, and send the network video data to any one of the decoding modules and forward the network video data to other decoding modules through the decoding module; the second switch is configured to send the local video data acquired by the input module to each decoding module; the input module includes a video acquisition unit, a first video processing unit and a video sending unit connected in sequence; the decoding module includes a video receiving unit, a second video processing unit, a synchronization control unit and a display sending unit connected in sequence, and the video receiving unit is connected with the video sending unit; the decoding module further includes a code stream receiving unit, a code stream sending unit and a decoding unit, the code stream receiving unit is connected with the code stream sending unit, the code stream receiving unit is connected with the decoding unit, the decoding unit is connected with the second video processing unit, and the code stream sending unit is connected with the code stream receiving unit of the next decoding module.

[0018] The application further provides an electronic device, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the video processing method based on the video synthesis platform according to any one of the above when executing the program.

[0019] The application further provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executable on the processor to implement the video processing method based on the video synthesis platform according to any one of the above.

[0020] The application further provides a computer program product, which includes a computer program, and the computer program is executable on the processor to implement the video processing method based on the video synthesis platform according to any one of the above.

[0021] The video processing method based on the video synthesis platform, the video synthesis platform and the device provided by the application determine the first time difference between the input module and the master control module, and determine the second time difference between each decoding module and the master control module; the input module determines the sending time sequence value of the video data sent by the input module according to the first time difference, and the input module sends the video data and the sending time sequence value to each decoding module; each decoding module synchronizes the processing time sequence of the decoding module with the master control module through the corresponding second time difference, and determines the current video data to be displayed according to the synchronized processing time sequence and the sending time sequence value of the video data, so that the synchronous processing of the video data by each module in the video synthesis platform is realized, and the synchronous display of each decoding module is realized, and the synchronous display effect of the video data is ensured. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0023] Figure 1 The scene schematic diagram of the video processing method based on the video synthesis platform provided by the present application;

[0024] Figure 2 The flowchart of the video processing method of an embodiment of the present application;

[0025] Figure 3 The flowchart of the video processing method of an embodiment of the present application;

[0026] Figure 4 The flowchart of the video processing method of an embodiment of the present application;

[0027] Figure 5 The structural schematic diagram of the input module and the decoding module of an embodiment of the present application;

[0028] Figure 6 The flowchart of the video processing method of an embodiment of the present application;

[0029] Figure 7 The schematic diagram of the first cache unit of one specific example of the present application;

[0030] Figure 8a The timing schematic diagram of the unadjusted initial phase of each decoding module of one specific example of the present application;

[0031] Figure 8b The timing schematic diagram of the adjusted initial phase of each decoding module of one specific example of the present application;

[0032] Figure 9 The flowchart of the video processing method of an embodiment of the present application;

[0033] Figure 10 The timing diagram of one specific example of the present application;

[0034] Figure 11 The structural schematic diagram of the video synthesis platform of an embodiment of the present application;

[0035] Figure 12 The structural schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0036] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0037] For the convenience of understanding, the embodiments of the present application provide a scene example of a video processing method based on a video synthesis platform, which is applied in an application environment as shown in the figure, wherein the application environment includes a video synthesis platform, an IP camera (IPC) and an external switch, and the video synthesis platform establishes a communication connection with the IP camera through the external switch. Figure 1

[0038] In the scene example, the video data includes two types, one is network video data, and the other is local video data. The difference between the two types of video data is that the network video data needs to be obtained from external devices through the Internet, while the local video data is video data pre-cached in the local memory.

[0039] In the scene example, the video synthesis platform includes a main control module, an input module and a decoding module, and further includes a first switch and a second switch. The main control module can realize communication with the input module and the decoding module through the first switch and the second switch.

[0040] Specifically, the main control module of the video synthesis platform can be set in multiple. When any one of the main control modules appears abnormal, the remaining main control modules can back up and display the video data, so as to ensure the stability of the display. In the scene example, two main control modules are set in the video synthesis platform, and the two main control modules are the same in function and structure.

[0041] ​In the scene example, the input module can be an input card, and the input card is provided with an FPGA (Field Programmable Gate Array) chip. The decoding module can be a decoding card, and the decoding card is provided with a DSP (Digital Signal Processing) chip. The product corresponding to the video synthesis platform can be provided with multiple card slots for inserting cards, and each card slot can be used for inserting an input card or a decoding card. When the input card or the decoding card is inserted into the card slot, the input card or the decoding card can establish communication with the main control module. In addition, since the number of input cards and decoding cards inserted into the product corresponding to the video synthesis platform is not fixed, the number of input modules and the number of decoding modules are also not fixed, and the number of input modules and the number of decoding modules can be selected by the staff according to actual needs.

[0042] In the scene example, the first switch can be an Ethernet switch, for example, a Gigabit Ethernet switch (GESW). The second switch can be a bus switch, for example, a Peripheral Component Interconnect Express switch (PCIe SW). The first switch and the second switch can be arranged on the same hardware link.

[0043] In the scene example, the network video data can be obtained by a network camera, and the network camera sends the network video data to the video synthesis platform through an external switch. The first switch inside the video synthesis platform sends the network video data to the decoding module, and the decoding module decodes and displays the network video data. For details, refer to the data flow direction of the solid arrow in Figure 1 .

[0044] In the scene example, the local video data can be obtained by the input module from the local device, and the input module sends the local video data to each decoding module through the second switch, and each decoding module decodes and displays the local video data. For details, refer to the data flow direction of the dashed arrow in Figure 1 .

[0045] Based on the scene example of the video processing method based on the video synthesis platform in the foregoing embodiment, the video processing method based on the video synthesis platform in the embodiment of the application is described in detail.

[0046] Figure 2FIG. 1 is a flowchart of a video processing method according to an embodiment of the present application. The video processing method based on a video synthesis platform can be applied to the video synthesis platform of the aforementioned embodiments, which comprises a master control module, an input module and at least one decoding module connected with each other. As shown in FIG. 1, the video processing method based on the video synthesis platform can comprise the following steps: Figure 2

[0047] Step 210: determining a first time difference and at least one second time difference by the master control module, and sending the first time difference to the input module and the second time differences to the corresponding decoding modules; the first time difference is the time difference of the input module relative to the master control module, and the second time difference is the time difference of the decoding module relative to the master control module.

[0048] First of all, it should be noted that the number of master control modules, input modules and decoding modules is not limited in the present embodiment.

[0049] In the present embodiment, the master control module comprises a processor CPU. When the master control module, the input module and the decoding modules are started, the CPU of the master control module can first acquire the starting time length of each input module, the starting time length of each decoding module and the starting time length of the master control module itself. Then, the CPU of the master control module determines the first time difference of the input module relative to the master control module according to the starting time length of the master control module and the starting time length of the input module, and determines the second time difference of each decoding module relative to the master control module according to the starting time length of the master control module and the starting time length of each decoding module.

[0050] After obtaining the first time difference and the second time differences, the master control module sends the first time difference to the input module and the second time differences to the corresponding decoding modules. After receiving the first time difference, the input module can achieve time synchronization with the master control module according to the first time difference. Similarly, after receiving the corresponding second time difference, each decoding module can achieve time synchronization with the master control module according to the second time difference, thereby achieving time synchronization between each decoding module and the input module.

[0051] Step 220: determining the sending time sequence value of the collected video data by the input module according to the first time difference, and sending the video data and the corresponding sending time sequence value to each decoding module.

[0052] It should be noted that the input module processes and sends the collected video data according to the internal input time sequence. For example, the input time sequence can be the time sequence of the frame synchronization pulse signal. Generally, the frame synchronization pulse signal is generated once every 1 / 30 second (33.3 milliseconds), i.e. the input module can perform video data sending operation every 33.3 ms. ​

[0053] When the input module receives the first time difference sent by the master module, the input module can synchronize the original input timing according to the first time difference; after synchronization, the input module sends a frame of video data to each decoding module, and sends the current synchronized input timing value as the sending timing value to each decoding module.

[0054] Step 230: For any decoding module, the decoding module determines the current video data to be displayed according to the second time difference and the sending timing value of the video data, and sends the current video data to be displayed for display.

[0055] It should be noted that each decoding module processes the video data according to the internal decoding timing.

[0056] When the decoding module receives the second time difference sent by the master module, the decoding module can synchronize the original decoding timing according to the second time difference; based on the synchronized decoding timing and the sending timing value of the video data, the decoding module determines the current video data to be displayed, and sends the current video data to be displayed to the display screen.

[0057] Figure 3 Figure 2 is a flowchart of a video processing method according to an embodiment of the present application. As shown in Figure 2, in some embodiments, the method can include the following steps: Figure 3

[0058] Step 310: Obtain the master control start duration, the master control clock period of the master module, the input start duration, the input clock period of the input module, the decoding start duration of each decoding module, and the decoding clock period.

[0059] Step 320: Determine the master control timing value according to the master control start duration and the master control clock period, determine the input timing value according to the input start duration and the input clock period, and determine the first time difference according to the master control timing value and the input timing value.

[0060] Step 330: Determine the decoding timing value according to the decoding start duration and the decoding clock period, and determine the second time difference according to the master control timing value and the decoding timing value.

[0061] It should be noted that the input start duration refers to the start duration of the input module, and the decoding start duration refers to the start duration of the decoding module.

[0062] ​Specifically, when the master module, the input module and each decoding module are started, the master starting time length, the master clock period, the input starting time length, the input clock period, the decoding starting time length of each decoding module and the decoding clock period can be obtained by the CPU of the master module. After the CPU of the master module obtains the above parameters, the first time difference and the second time difference are calculated according to the above parameters.

[0063] In the embodiment, the starting time length of each module can be converted into a time sequence value in TICK (time base) according to the starting time length and the clock period of each module, so as to convert the starting time length into an integer.

[0064] Taking the master clock period as 1 / 60 second, the input clock period as 1 / 30 second and the decoding clock period as 1 / 60 second as an example, the calculation method of the first time difference and the second time difference is introduced in detail.

[0065] It can be understood that if the master clock period is 1 / 60 second, that is, the master module outputs 60 pulses per second, when the master starting time length (in seconds) is obtained, the number of pulses output by the master module in the time length can be determined according to the master starting time length, and the pulses are arranged in time sequence, so as to obtain the current master time sequence value. Therefore, the master time sequence value can be calculated by the following formula:

[0066]

[0067] Wherein, Time Tick is the master time sequence value, S is second, and NS is nanosecond. As an example, if the current master starting time length is 90 seconds, then Time Tick = 90*60 = 5400 (Tick), at this time, the master time sequence value of the master module is 5400 (Tick).

[0068] Similarly, if the input clock period is 1 / 30 second, that is, the input module outputs 30 pulses per second, when the input starting time length (in seconds) is obtained, the number of pulses output by the input module in the time length can be determined according to the input starting time length, and the pulses are arranged in time sequence, so as to obtain the current input time sequence value, which is recorded as Tick_IN in the embodiment. As an example, if the current input starting time length of the input module is 60 seconds, then the input time sequence value Tick_IN = 60*30 = 1800 (Tick).

[0069] After obtaining the master time sequence value and the input time sequence value, since the master clock period is 1 / 60 second and the input clock period is 1 / 30 second, the clock periods of the master time sequence value and the input time sequence value need to be unified, and then the first time difference is calculated. Specifically, the first time difference can be calculated by the following formula:

[0070] Delta Tick_IN = Time Tick - Tick_IN * 2

[0071] Wherein, Delta Tick_IN is the first time difference. As an example, if the current master module startup duration is 90 seconds, and the current input module input startup duration is 60 seconds, then Delta Tick_IN = (90*60) - (60*30*2) = 1800(Tick).

[0072] It is worth mentioning that the master module can directly obtain the frame synchronization pulse signal count inside the input module as the input timing value, and further calculate the first time difference, because the input module is to process the video data sent with the timing of the frame synchronization pulse signal as the input timing.

[0073] Similarly, if the decoding clock period is 1 / 60 seconds, which means that the input module outputs 60 pulses per second, when the decoding startup duration is obtained (in seconds), the number of pulses output by the decoding module within this duration can be determined according to the decoding startup duration, and these pulses are arranged in time sequence, that is, the current decoding timing value is obtained, which is denoted as Tick_OUT in this embodiment. As an example, if the input startup duration of the current decoding module is 50 seconds, then the input timing value Tick_OUT = 50*60 = 3000(Tick).

[0074] After obtaining the master timing value and the decoding timing value, the second time difference can be calculated by the following formula:

[0075] Delta Tick_OUT = Time Tick - Tick_OUT

[0076] Wherein, Delta Tick_OUT is the second time difference. As an example, if the current master module startup duration is 90 seconds, and the current decoding module input startup duration is 50 seconds, then Delta Tick_OUT = (90*60) - (50*60) = 2400(Tick).

[0077] It is worth mentioning that the master module can directly obtain the Vblank (pulse voltage) interrupt count inside the decoding module as the decoding timing value, and further calculate the second time difference, Vblank interrupt once, which means that the decoding module has completed a display, and at the same time means that the next frame of video data can be displayed. Therefore, the decoding timing value can be calculated, and the Vblank interrupt count can be directly used to represent the decoding timing value.

[0078] Figure 4Fig. 3 is a flowchart illustrating a third embodiment of the video processing method of the present application. As shown in Fig. 3, in some embodiments, the step of determining the sending timing value of the collected video data according to the first time difference by the input module in step 220 can include the following steps: Figure 4

[0079] Step 410: determining the synchronized input timing value according to the first time difference and the input timing value.

[0080] Step 420: determining the sending timing value of the video data according to the synchronized input timing value.

[0081] To more clearly explain the process of sending the video data and the sending timing value from the input module to the decoding module, the specific structure of the input module will be introduced first. Figure 5 Fig. 4 is a structural diagram of the input module and the decoding module of the present application. As shown in Fig. 4, the input module of the present embodiment includes a video collection unit, a first video processing unit and a video sending unit connected in sequence. Figure 5

[0082] The video collection unit is used to collect video data and send the video data to the first video processing unit. The first video processing unit will perform pre-processing such as cropping on the video data to obtain YUV data (color coding data). The first video processing unit will send the obtained YUV data to the video sending unit, and finally the video sending unit will send the YUV data to each decoding module.

[0083] When the master module calculates the first time difference, the master module will send the first time difference to the input module, and the video sending unit of the input module will record the first time difference Delta Tick_IN. When the input module sends a frame of video data to each decoding module, it will send the current synchronized input timing value of the input module as the sending timing value to each decoding module. As an example, if the master clock period is 1 / 60 second and the input clock period is 1 / 30 second, the synchronized input timing value can be calculated by the following formula:

[0084] Synchronization Tick_IN = Tick_IN * 2 + Delta Tick_IN

[0085] wherein Synchronization Tick_IN is the synchronized input timing value.

[0086] ​​Specifically, the video sending unit of the input module sends the data packets in the order of a frame header packet, video data, and a frame tail packet to the decoding module. The frame tail packet carries a timestamp of when the video data is sent by the input module. In this embodiment, the timestamp carried in the frame tail packet is the input timing value after the current synchronization (i.e., the sending timing value).

[0087] It is considered that the process of processing and sending the video data by the input module requires a certain time. Therefore, in some embodiments, a redundant value can be added to the input timing value after the current synchronization to obtain the sending timing value. The size of the redundant value can be set artificially by the staff according to actual needs, and the minimum redundant value is 1.

[0088] As an example, if the master clock period is 1 / 60 second, the input clock period is 1 / 30 second, the starting duration of the current master module is 90 seconds, and the input starting duration of the current input module is 60 seconds, the first time difference can be calculated as 1800 (Tick), the current input timing value is 1800 (Tick), and further, the input timing value after the synchronization Synchronization Tick_IN = 1800*2+1800 = 5400 (Tick) can be calculated. If the redundant value is set to 1, the sending timing value carried in the frame tail packet by the video sending unit of the input module is 5401 (Tick).

[0089] Continuing to refer to Figure 5 In some embodiments, the decoding module can include a video receiving unit, a second video processing unit, a synchronization control unit, and a display sending unit connected in sequence. The video receiving unit is connected with the video sending unit of the input module, and is configured to receive the video data and the corresponding sending timing value sent by the video sending unit of the input module. Further, the video receiving unit sends the video data and the corresponding sending timing value to the second video processing unit.

[0090] After the second video processing unit processes a frame of video data, the frame of video data is sent to the synchronization control unit. In some embodiments, each decoding module includes at least two first buffer units, and different first buffer units are used to buffer different frames of video data. In this embodiment, each first buffer unit is arranged in the synchronization control unit.

[0091] Figure 6 A flowchart of the video processing method of the embodiment of the present application is shown in FIG. 4. As shown in FIG. 4, in some embodiments, the step of determining the current video data to be displayed by the decoding module according to the second time difference and the sending timing value of the video data in step 230 can include the following steps: Figure 6

[0092] ​Step 610: determining a synchronized decoding time value according to the second time difference and the decoding time value for any decoding module.

[0093] Step 620: determining current video data to be displayed according to the synchronized decoding time value and the sending time value, and retrieving the current video data to be displayed from a first cache unit in which the current video data to be displayed is located.

[0094] When the decoding module receives the second time difference sent by the master module, the decoding module can record the second time difference Delta Tick OUT, and then determine the current synchronized decoding time value according to the second time difference and the decoding time value. Taking the master clock period as 1 / 60 second and the decoding clock period as 1 / 60 second as an example, the synchronized decoding time value can be calculated by the following formula:

[0095] Synchronization Tick OUT = Tick OUT + Delta Tick OUT

[0096] wherein Synchronization Tick OUT is the synchronized decoding time value.

[0097] As an example, if the master clock period is 1 / 60 second, the decoding clock period is 1 / 60 second, the current start-up time of the master module is 90 seconds, and the current input start-up time of the input module is 50 seconds, the first time difference can be calculated as 2400 (Tick), the current decoding time value can be calculated as 3000 (Tick), and further, the synchronized decoding time value can be calculated as 5400 (Tick).

[0098] The video receiving unit of the decoding module sends the video data and the corresponding sending time value to the second video processing unit for processing after receiving each frame of video data. After the second video processing unit finishes processing, the video data and the corresponding sending time value are sent to the synchronization control unit, and the synchronization control unit stores the processed video data with the sending time value in a first cache unit.

[0099] Similarly, considering that the processing of the video data by the decoding module needs time, in some embodiments, a redundancy value can also be added to the synchronized decoding time value, and the redundancy value is at least 1.

[0100] When the display sending unit of the decoding module is sending for display, the display sending unit determines the decoding timing value after the current synchronization, and finds the video data that meets the conditions in each first cache unit according to the decoding timing value after the current synchronization, and uses the video data that meets the conditions as the video data to be currently displayed. At this time, the display sending unit retrieves the video data to be currently displayed from the first cache unit where the video data to be currently displayed is located for display.

[0101] In some embodiments, determining the video data to be currently displayed based on the synchronized decoding timing value and the sending timing value in step 620 may include: when the sending timing value meets a preset condition, determining the video data corresponding to the sending timing value as the video data to be currently displayed; the preset condition includes that the sending timing value is adjacent to the synchronized decoding timing value, and the sending timing value is less than or equal to the synchronized decoding timing value.

[0102] Specifically, the display unit can find video data corresponding to the currently synchronized decoding timing value and less than or equal to the currently synchronized decoding timing value in each first cache unit according to the currently synchronized decoding timing value. The video data is the video data to be displayed.

[0103] As an example, Figure 7 As shown, if the number of first cache units is 6, the sending timing values ​​corresponding to the video data stored in these 6 first cache units are: 5400 (Tick), 5402 (Tick), 5404 (Tick), 5406 (Tick), 5408 (Tick) and 5410 (Tick). If the display unit determines that the decoding timing value after the current synchronization is 5405 (Tick), the video data corresponding to the sending timing value of 5404 (Tick) can be used as the video data to be displayed.

[0104] In some embodiments, the video receiving unit of each decoding module includes a second cache unit and a third cache unit. The video processing method based on the video integrated platform may also include: receiving video data and corresponding sending timing values ​​through the second cache unit; sending the video data and corresponding sending timing values ​​to the first cache unit for caching through the third cache unit.

[0105] Specifically, the video receiving unit is internally equipped with a second cache unit and a third cache unit. When the video sending unit of the input module sends a frame of video data to the decoding module, the video receiving unit of the decoding module stores this frame of video data in the second cache unit, then retrieves the previous frame of video data from the third cache unit and sends it to the second video processing unit. Using two caches for data reception and transmission effectively solves the tearing problem caused by a single cache and the high latency caused by multiple caches.

[0106] In some embodiments, the step of determining, by the decoding modules, the video data to be currently displayed according to the second time difference and the sending time sequence value of the video data in step 230 can further include: adjusting the initial phases of the decoding modules to be consistent; and determining the video data to be currently displayed according to the second time difference and the sending time sequence value of the video data by the adjusted decoding modules.

[0107] Since the initial phases of the original decoding processing time sequences of the decoding modules are inconsistent after the decoding modules are started, in the present embodiment, the initial phases of the original decoding processing time sequences of the decoding modules can be adjusted after the decoding modules are started, so that the initial phases of the original decoding processing time sequences of the decoding modules are kept consistent, thereby further ensuring the synchronization of the display of the video data by the decoding modules. Figure 8a A timing diagram showing the initial phases of the decoding modules not being adjusted is shown in FIG. 4A; Figure 8b A timing diagram showing the initial phases of the decoding modules being adjusted is shown in FIG. 4B.

[0108] Reference Figure 1 In some embodiments, the number of decoding modules is at least two, the video synthesis platform further includes a first switch and a second switch, and the video data includes network video data and local video data. In addition, as shown in FIG. 5, the decoding module further includes a code stream receiving unit, a code stream forwarding unit and a decoding unit, the code stream receiving unit is connected with the code stream forwarding unit and the decoding unit, and the decoding unit is connected with the second video processing unit. Figure 5

[0109] In the present embodiment, the first switch can be a GE SW, the second switch can be a PCIe SW, and the first switch and the second switch can be arranged on the same hardware link. The host module can realize data interaction with the decoding modules and the input module through the first switch or the second switch, and similarly, the input module can also realize data interaction with the decoding modules through the second switch.

[0110] Figure 9 FIG. 5 is a flowchart of a video processing method according to an embodiment of the present application. Figure 9 As shown in FIG. 5, in some embodiments, the video data includes network video data and local video data, when the network video data and the local video data need to be processed simultaneously and decoded and displayed by the decoding modules, the video processing method based on the video synthesis platform can further include the following steps:

[0111] Step 910: receiving the network video data through the first switch, sending the network video data to any decoding module, and forwarding the network video data to other decoding modules through the decoding module. ​

[0112] Step 920: sending the local video data acquired by the input module to each decoding module through the second switch.

[0113] Specifically, the first switch can receive network video data sent by an external network camera IPC, and after receiving the network video data, the first switch sends the network video data to any one decoding module; the stream receiving unit of the decoding module is configured to receive the network video data, and after receiving the network video data, the stream receiving unit sends the network video data to the stream sending unit; the stream sending unit directly forwards the network video data to the stream receiving unit of other decoding modules.

[0114] The second switch is configured to receive local video data sent by the video sending unit in the input module, and send the local video data to the video receiving unit in each decoding module.

[0115] It should be noted that when the existing video integrated platform needs to process network video data and local video data at the same time and send the video data to display through multiple decoding modules, the processing process is as follows: the first switch receives network video data and sends the network video data to the decoding module; after receiving the network video data, the decoding module performs scaling, cropping and other processing to obtain YNV data; after obtaining the YNV data, the decoding module needs to send the YNV data to other decoding modules through the second switch due to the large data volume of the YNV data, so the YNV data sent by the decoding module will occupy the bandwidth of the second switch. At the same time, the second switch also needs to send local video data to each decoding module, so the local video data and the network video data will interfere with each other, thereby affecting the display of the video data.

[0116] Therefore, in the present embodiment, after receiving the network video data, the stream receiving unit directly sends the network video data to the stream sending unit; and then the stream sending unit forwards the network video data to the stream receiving unit of other decoding modules through an internal protocol, so that the network video data will not occupy the bandwidth of the second switch, thereby realizing separate transmission of the local video data and the network video data, reducing the mutual interference between the local video data and the network video data, and effectively improving the transmission efficiency of the video data.

[0117] The synchronization display process of the multiple decoding modules on the same network video data is as follows: after each decoding module receives the second time difference sent by the master module, the code stream receiving unit of each decoding module can record the second time difference, and determine the synchronized decoding time sequence value according to the second time difference and the decoding time sequence value; the code stream receiving unit records the current synchronized decoding time sequence value in the private header of the network video data after receiving each frame of network video data. When the network video data is displayed, each display unit determines the network video data to be displayed according to the current synchronized decoding time sequence value.

[0118] For the convenience of understanding, the video processing method based on the video synthesis platform of the embodiment of the application is further introduced below by taking the example of the synchronization display of one local video data by two decoding modules (decoding module 1 and decoding module 2).

[0119] Firstly, the collection frame rate of the input module is 60 Hz, that is, one frame of local video data is collected every 16.6 ms (1 / 60 s), and the input clock period of the input module is configured as 1 / 30 s, that is, one frame of local video data is processed or sent every 33.3 ms. In addition, it is assumed that the current input start time of the input module is 60 s.

[0120] Secondly, the decoding clock period of the decoding module 1 and the decoding module 2 is configured as 1 / 60 s, that is, one frame of local video data is outputted and processed every 16.6 ms, but the start time of the decoding module 1 and the decoding module 2 is different, and in this example, the VBlank interrupt count can be directly obtained as the decoding time sequence value. In addition, it is assumed that the current input start time of the decoding module 1 is 50 s, and it is assumed that the current input start time of the decoding module 2 is 40 s.

[0121] Finally, the master clock period of the master module is configured as 1 / 60 s, and it is assumed that the current master start time of the master module is 90 s. Figure 10 For the timing diagram of one specific example of the application, the collection timing, processing timing of the input module and the processing timing of the decoding module can refer to the timing in Figure 10 .

[0122] Further, the master module determines a master timing value as 5400 (Tick) according to the master start-up time length 90s and the master clock cycle 1 / 60; determines an input timing value as 1800 (Tick) according to the input start-up time length 60s and the input clock cycle 1 / 30, in the present example, the input timing value can be directly obtained by counting the frame synchronization pulse as the input timing value; determines a decoding timing value of the decoding module 1 as 3000 (Tick) according to the decoding start-up time length 50s of the decoding module 1 and the decoding clock cycle 1 / 60; determines a decoding timing value of the decoding module 2 as 2400 (Tick) according to the decoding start-up time length 40s of the decoding module 2 and the decoding clock cycle 1 / 60, in the present example, the decoding timing value can be directly obtained by counting the VBlank interrupt of each decoding module as the decoding timing value.

[0123] Further, the master module determines a first time difference according to the master timing value and the frame synchronization pulse count; determines a second time difference according to the master timing value and the VBlank interrupt count. The master module further determines a synchronized input timing value according to the first time difference and the frame synchronization pulse count; determines a synchronized decoding timing value according to the second time difference and the VBlank interrupt count. The corresponding relationship between the synchronized input timing value and the frame synchronization pulse count, and the corresponding relationship between the synchronized decoding timing value and the VBlank interrupt count can refer to the corresponding relationship in Figure 10 .

[0124] As shown in Figure 10 , when the input module starts collecting local video data from the 9th frame, the input module triggers the frame synchronization pulse signal at the same time to process the local video data. When the frame synchronization pulse count is 1802 (Tick), the input module sends the 11th frame of local video data, and converts the frame synchronization pulse count 1802 (Tick) into a synchronized input timing value:

[0125] Synchronization Tick_IN = 1802 * 2 + 1800 = 5404 (Tick)

[0126] After obtaining the synchronized input timing value 5404 (Tick), a redundant value 1 is added to the synchronized input timing value 5404 (Tick) to obtain a sending timing value 5405 (Tick) corresponding to the 11th frame of local video data. Finally, the input module sends the 11th frame of local video data and the sending timing value 5405 (Tick) to the decoding module 1 and the decoding module 2, respectively, through the video sending unit.

[0127] Considering that the decoding module needs a certain time to process the video data, a redundancy value 1 is added to the sending time sequence of the local video data of the decoding module. Specifically, referring to Figure 10 , the decoding module 1 receives the 11th frame of local video data and a sending time sequence value 5405 (Tick), and when the synchronized decoding time sequence value of the decoding module 1 reaches 5406 (Tick), the decoding module 1 can send the 11th frame of local video data. Similarly, the decoding module 2 receives the 11th frame of local video data and a sending time sequence value 5405 (Tick), and when the synchronized decoding time sequence value of the decoding module 2 reaches 5406 (Tick), the decoding module 2 can send the 11th frame of local video data.

[0128] Thus, the decoding module 1 and the decoding module 2 send the same frame of local video data at the same time, realizing the synchronized sending of each decoding module and ensuring the synchronized display effect of the video data.

[0129] Figure 11 The structure of the video synthesis platform of the embodiment of the present application is shown in FIG. 11. As shown in FIG. 11, the video synthesis platform 1100 includes a main control module 1110, an input module 1120, and at least one decoding module 1130 connected with each other. Figure 11

[0130] The main control module 1110 is configured to determine a first time difference and at least one second time difference, send the first time difference to the input module 1120, and send each second time difference to the corresponding decoding module 1130. The first time difference is the time difference of the input module 1120 relative to the main control module 1110, and the second time difference is the time difference of the decoding module 1130 relative to the main control module 1110.

[0131] The input module 1120 is configured to determine the sending time sequence value of the collected video data according to the first time difference, and send the video data and the corresponding sending time sequence value to each decoding module 1130.

[0132] The decoding module 1130 is configured to determine the current video data to be displayed according to the second time difference and the sending time sequence value of the video data, and send the current video data to be displayed.

[0133] ​Thus, by determining the first time difference between the input module 1120 and the master module 1110, the second time difference between each decoding module 1130 and the master module 1110 is determined; the input module 1120 determines the transmission timing value of the video data according to the first time difference, and the input module 1120 transmits the video data and the transmission timing value to each decoding module 1130; each decoding module 1130 synchronizes the processing timing of the decoding module 1130 with the master module 110 through the corresponding second time difference, and determines the current video data to be displayed according to the synchronized processing timing and the transmission timing value of the video data, thereby realizing the synchronous processing of the video data by each module in the video synthesis platform 1100, and further realizing the synchronous display of each decoding module 1130, and ensuring the synchronous display effect of the video data.

[0134] In some embodiments, referring to Figure 5 and Figure 11 , the number of decoding modules is at least two, and the video synthesis platform further comprises a first switch and a second switch; the video data comprises network video data and local video data; the first switch is configured to receive the network video data and transmit the network video data to any decoding module, and forward the network video data to other decoding modules through the decoding module; the second switch is configured to transmit the local video data obtained by the input module to each decoding module; the input module comprises a video acquisition unit, a first video processing unit and a video transmission unit connected in sequence; the decoding module comprises a video receiving unit, a second video processing unit, a synchronization control unit and a display unit connected in sequence, and the video receiving unit is connected with the video transmission unit; the decoding module further comprises a code stream receiving unit, a code stream transmission unit and a decoding unit, the code stream receiving unit is connected with the code stream transmission unit, the code stream receiving unit is connected with the decoding unit, the decoding unit is connected with the second video processing unit, and the code stream transmission unit is connected with the code stream receiving unit of the next decoding module.

[0135] It should be noted that the details of the video synthesis platform in the present embodiment are not disclosed, please refer to the details disclosed in the embodiments of the video processing method based on the video synthesis platform in the present specification, which will not be repeated here.

[0136] Figure 12 An example of the physical structure of an electronic device is shown in the following figure: Figure 12As shown, the electronic device can include a processor 1210, a communications interface 1220, a memory 1230, and a communications bus 1240, wherein the processor 1210, the communications interface 1220, and the memory 1230 complete mutual communication through the communications bus 1240. The processor 1210 can invoke a logic instruction in the memory 1230 to execute a video processing method of a video synthesis platform, the method including determining a first time difference and at least one second time difference through a master module, and sending the first time difference to an input module and sending each second time difference to a corresponding decoding module; the first time difference is a time difference of the input module relative to the master module, and the second time difference is a time difference of the decoding module relative to the master module; determining a sending timing value of collected video data according to the first time difference through the input module, and sending the video data and the corresponding sending timing value to each decoding module; for any decoding module, determining current video data to be displayed according to the second time difference and the sending timing value of the video data through the decoding module, and sending the current video data to be displayed.

[0137] In addition, the logic instruction in the memory 1230 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0138] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer readable storage medium, and the computer program is executable by a processor to enable a computer to perform the video processing method of the video synthesis platform provided by the above-mentioned methods, which comprises determining a first time difference and at least one second time difference by a master module, and sending the first time difference to an input module and sending each second time difference to a corresponding decoding module; the first time difference is the time difference of the input module relative to the master module, and the second time difference is the time difference of the decoding module relative to the master module; determining the sending time sequence value of the collected video data according to the first time difference by the input module, and sending the video data and the corresponding sending time sequence value to each decoding module; for any decoding module, determining the current video data to be displayed according to the second time difference and the sending time sequence value of the video data by the decoding module, and sending the current video data to be displayed.

[0139] In another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, and the computer program is executable by a processor to implement the video processing method of the video synthesis platform provided by the above-mentioned methods, which comprises determining a first time difference and at least one second time difference by a master module, and sending the first time difference to an input module and sending each second time difference to a corresponding decoding module; the first time difference is the time difference of the input module relative to the master module, and the second time difference is the time difference of the decoding module relative to the master module; determining the sending time sequence value of the collected video data according to the first time difference by the input module, and sending the video data and the corresponding sending time sequence value to each decoding module; for any decoding module, determining the current video data to be displayed according to the second time difference and the sending time sequence value of the video data by the decoding module, and sending the current video data to be displayed.

[0140] The apparatus embodiments described above are only schematic, wherein the units illustrated as separate components can or can not be physically separate, and the components illustrated as units can or can not be physical units, i.e., can be located in one place or distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0141] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0142] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video processing method based on a video integration platform, characterized in that: The video integration platform includes a main control module, an input module and at least one decoding module connected to each other, and the method includes: Determining a first time difference and at least one second time difference by the main control module, and sending the first time difference to the input module, and sending each second time difference to a corresponding decoding module; the first time difference is the time difference between the input module and the main control module, and the second time difference is the time difference between the decoding module and the main control module; Determining, by the input module, a sending timing value of the collected video data according to the first time difference, and sending the video data and the corresponding sending timing value to each of the decoding modules; For any decoding module, the decoding module determines the video data to be currently displayed according to the second time difference and the sending timing value of the video data, and sends the video data to be currently displayed for display.

2. The video processing method based on the video integrated platform according to claim 1, characterized in that: The determining the first time difference and at least one second time difference by the main control module includes: Obtaining the master control startup duration and master control clock cycle of the master control module, the input startup duration and input clock cycle of the input module, and the decoding startup duration and decoding clock cycle of each decoding module; Determining a master timing value according to the master startup duration and the master clock cycle, determining an input timing value according to the input startup duration and the input clock cycle, and determining the first time difference according to the master timing value and the input timing value; A decoding timing value is determined according to the decoding startup duration and the decoding clock period, and the second time difference is determined according to the master control timing value and the decoding timing value.

3. The video processing method based on the video integrated platform according to claim 2, characterized in that: The determining, by the input module, a sending timing value of the collected video data according to the first time difference includes: determining a synchronized input timing value according to the first time difference and the input timing value; The sending timing value of the video data is determined according to the synchronized input timing value.

4. The video processing method based on the video integrated platform according to claim 2, characterized in that: Each of the decoding modules includes at least two first cache units, different first cache units are used to cache different frames of video data, and determining the current video data to be displayed by the decoding module according to the second time difference and the sending timing value of the video data includes: For any decoding module, determining a synchronized decoding timing value according to the second time difference and the decoding timing value; The video data to be currently displayed is determined according to the synchronized decoding timing value and the sending timing value, and the video data to be currently displayed is retrieved from the first cache unit where the video data to be currently displayed is located.

5. The video processing method based on the video integration platform according to claim 4, characterized in that: The determining the video data to be currently displayed according to the synchronized decoding timing value and the sending timing value includes: When the sending timing value meets a preset condition, the video data corresponding to the sending timing value is determined as the video data currently to be displayed; the preset condition includes that the sending timing value is adjacent to the synchronized decoding timing value, and the sending timing value is less than or equal to the synchronized decoding timing value.

6. The video processing method based on a video integration platform according to any one of claims 1 to 5, characterized in that: The step of determining, by the decoding module, the video data to be currently displayed according to the second time difference and the sending timing value of the video data includes: Adjusting the initial phases of the decoding modules to be consistent; The video data to be currently displayed is determined by the adjusted decoding module according to the second time difference and the sending timing value of the video data.

7. The video processing method based on the video integration platform according to claim 4, characterized in that: Each of the decoding modules further includes a second cache unit and a third cache unit, and the method further includes: receiving the video data and the corresponding sending timing value through the second buffer unit; The video data and the corresponding sending timing value are sent to the first cache unit through the third cache unit for cache.

8. The video processing method based on a video integration platform according to any one of claims 1 to 5, characterized in that: The number of the decoding modules is at least two, the video integrated platform further includes a first switch and a second switch, the video data includes network video data and local video data, and the method further includes: receiving the network video data through the first switch, sending the network video data to any one of the decoding modules, and forwarding the network video data to other decoding modules through the decoding module; The local video data acquired by the input module is sent to each of the decoding modules through the second switch.

9. A video integrated platform, characterized in that: It includes a main control module, an input module and at least one decoding module that are interconnected; The main control module is configured to determine a first time difference and at least one second time difference, and send the first time difference to the input module, and send each second time difference to a corresponding decoding module; the first time difference is a time difference between the input module and the main control module, and the second time difference is a time difference between the decoding module and the main control module; The input module is used to determine a sending timing value of the collected video data according to the first time difference, and send the video data and the corresponding sending timing value to each of the decoding modules; The decoding module is used to determine the video data to be currently displayed according to the second time difference and the sending timing value of the video data, and send the video data to be currently displayed for display.

10. The video integrated platform according to claim 9, characterized in that: The number of the decoding modules is at least two, the video integrated platform further comprises a first switch and a second switch, and the video data comprises network video data and local video data; The first switch is used to receive the network video data, send the network video data to any one of the decoding modules, and forward the network video data to other decoding modules through the decoding module; The second switch is used to send the local video data obtained by the input module to each of the decoding modules; The input module includes a video acquisition unit, a first video processing unit and a video sending unit connected in sequence; The decoding module includes a video receiving unit, a second video processing unit, a synchronization control unit and a display unit connected in sequence, and the video receiving unit is connected to the video sending unit; The decoding module also includes a code stream receiving unit, a code stream sending unit and a decoding unit, the code stream receiving unit is connected to the code stream sending unit, the code stream receiving unit is connected to the decoding unit, the decoding unit is connected to the second video processing unit, and the code stream sending unit is connected to the code stream receiving unit of the next decoding module.

Images