Video stitching method and system based on MIPI (Mobile Industry Processor Interface) and storage medium

Abstract

The invention relates to an MIPI-based video stitching method and system and a storage medium, and the method comprises the steps: S1, receiving at least two paths of MIPI video signals, carrying out the unpacking, and recovering a video time sequence; s2, caching the MIPI video signals after the video time sequence is recovered, and carrying out frame synchronization on the two paths of MIPI video signals; and S3, based on a video time sequence required by a screen, clock domain synchronization is performed on the two paths of MIPI video signals, and the two paths of MIPI video signals after clock domain synchronization are read out and subjected to video stitching in sequence according to a stitching sequence, so that data receiving and video stitching are realized, and meanwhile, the power consumption is low, the cost is low, and miniaturization is facilitated.

Description

Technical Field

[0001] The present application relates to the technical field of display and testing of liquid crystal modules, and in particular to a video splicing method, system and storage medium based on MIPI. Background Art

[0002] The video interface of the MIPI DSI protocol is widely used in mobile terminal devices due to its low power consumption and strong interoperability. In some small-screen applications, such as some electronic instrument products such as motorcycle dashboards and portable measuring instruments, some users have the demand for multi-channel image splicing display, and require certain real-time and reliability. Usually, FPGA is used with an external large-capacity SDRAM solution to achieve the synchronous display of two-channel video with its high-bandwidth video storage and processing capabilities. However, some FPGAs do not have the MIPI DSI protocol decoding function, and a protocol conversion chip needs to be added, which not only increases the cost, but also has a large circuit area, which is not conducive to miniaturization. Summary of the invention

[0003] The purpose of this application is to provide a MIPI-based video splicing method, system and storage medium to solve the problem that some existing FPGAs do not have MIPI DSI protocol decoding and need to add a protocol conversion chip.

[0004] In order to solve the above problems, the present application adopts the following technical solutions:

[0005] The first aspect of the present application provides a MIPI-based video splicing method, comprising:

[0006] S1. Receive at least two MIPI video signals, unpack them and restore the video timing;

[0007] S2, caching the MIPI video signal after the video timing is restored, and performing frame synchronization of the two MIPI video signals;

[0008] S3. Based on the video timing required by the screen, the two MIPI video signals are synchronized in the clock domain, and the two MIPI video signals after the clock domain synchronization are read out in sequence according to the splicing order and the video is spliced.

[0009] Further, S2, caching the MIPI video signal after the video timing is restored, and performing frame synchronization of the two MIPI video signals, specifically includes:

[0010] S21, writing the MIPI video signal after the video timing is restored into the wr_fifo module;

[0011] S22, when the psram controller is idle, the data of the wr_fifo module is read out;

[0012] S23, writing into the psram according to the packing format of the psram controller, and caching a whole frame to achieve frame synchronization of two MIPI video signals.

[0013] Further, S21, the step of writing the MIPI video signal after the video timing is restored into the wr_fifo module specifically includes:

[0014] S211, obtaining the MIPI video signal after the video timing is restored;

[0015] S212, writing the MIPI video signal into the wr_fifo module;

[0016] S213, determining whether the wr_fifo module is fully loaded with a line of video data;

[0017] S214, if the wr_fifo module is full of one line of video data, read out one line of video data from the wr_fifo module and pack and write it into the psram;

[0018] S215. If the wr_fifo module does not store a full line of video data, return to S211.

[0019] Further, S3, based on the video timing required by the screen, the two MIPI video signals are synchronized in the clock domain, and the two MIPI video signals after the clock domain synchronization are read out in sequence according to the splicing order, and the video splicing step specifically includes:

[0020] S31, obtaining the row synchronization signal of the timing_gen module;

[0021] S32, based on the row synchronization signal, read out the MIPI video signal corresponding to the row in the psram, and write them into the rd_fifo module respectively;

[0022] S33, determining whether the rd_fifo module is fully loaded with a row of video data;

[0023] S34, if the rd_fifo modules are all full of one line of video data, then splicing the video data of the rd_fifo modules;

[0024] S35. If the rd_fifo module does not store a full line of video data, return to S31.

[0025] Furthermore, in the frame synchronization of the two MIPI video signals, when a new frame of the built-in timing arrives, if the progress k of a currently stored frame is greater than the set threshold k0, the frame header stored this time starts to be output;

[0026] If k is less than k0, the output starts from the frame header of the previous frame; where 0<k≤1.

[0027] Furthermore, the following relationship is satisfied between the input frame rate fin, the output frame rate fout and the set threshold k0:

[0028] (1-k 0 )f out <f in <(2-k 0 )f out .

[0029] Furthermore, the time interval td from when the signal source completes a frame input to when it completes the frame output satisfies the following relationship:

[0030] 0 <t d <k 0 T in +T out

[0031] Among them, Tin and Tout are the frame periods of the system input and output video signals respectively.

[0032] The second aspect of the present application provides a MIPI-based video splicing system, including:

[0033] A receiving module, used to receive at least two MIPI video signals, unpack them and restore the video timing;

[0034] A frame synchronization module, used for caching the MIPI video signal after the video timing is restored, and performing frame synchronization of two MIPI video signals;

[0035] The splicing module is used to perform clock domain synchronization on the two MIPI video signals based on the video timing required by the screen, and to read out and splice the two MIPI video signals after clock domain synchronization in a splicing order.

[0036] Furthermore, the MIPI-based video splicing system includes a row determination module, and the row determination module is used to determine whether a row of video data is fully stored.

[0037] A third aspect of the present application provides a storage medium, wherein the storage medium stores a computer program, and wherein the computer program implements any of the above methods when executed by a processor.

[0038] Compared with the prior art, the beneficial effects of the present application are: by receiving at least two MIPI video signals, unpacking and recovering the video timing; caching and performing frame synchronization of the two MIPI video signals, the efficiency of video processing is improved. Based on the video timing required by the screen, the two MIPI video signals are synchronized in the clock domain, and the two MIPI video signals after clock domain synchronization are read out and video spliced ​​in sequence in order to achieve data reception and video splicing. Since no additional SDRAM memory is required to cache a large amount of video data, a more compact system design can be achieved, which is particularly suitable for small-screen application devices with limited space, low power consumption, low cost and easy to miniaturize. By caching and frame synchronization processing of video signals inside the FPGA, the real-time performance of video data and the reliability of display are ensured to meet the requirements of the application. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flowchart of a video stitching method based on MIP I provided in an embodiment of the present application;

[0040] Figure 2 A system block diagram of a video splicing method based on MIP I provided in an embodiment of the present application;

[0041] Figure 3 A flowchart of writing data into psram and reading data from psram provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of a frame synchronization readout judgment mechanism provided in an embodiment of the present application;

[0043] Figure 5 A schematic diagram of a MIP I DS I input interface provided in an embodiment of the present application;

[0044] Figure 6 A schematic diagram of an LVDS output interface provided in an embodiment of the present application; and

[0045] Figure 7 A schematic diagram of an FPGA BANK provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The specific implementation methods of the present application are described in detail below with reference to the accompanying drawings.

[0047] It should be noted that, in the absence of conflict, the embodiments and technical features in the embodiments of the present application can be combined with each other, and the detailed description in the specific implementation method should be understood as an explanation of the purpose of the present application and should not be regarded as an improper limitation on the present application.

[0048] It should be understood that the orientation or position relationship is based on the orientation or position relationship shown in the drawings. These orientation terms are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present application.

[0049] Figure 1 A flowchart of a video splicing method based on MIP I is provided in an embodiment of the present application. Figure 2 A system block diagram of a video splicing method based on MIP I provided in an embodiment of the present application, Figure 3 A flowchart of writing data into psram and reading data from psram provided in an embodiment of the present application, Figure 4 A schematic diagram of a frame synchronization readout judgment mechanism provided in an embodiment of the present application, Figure 5 A schematic diagram of a MIP I DS I input interface provided in an embodiment of the present application, Figure 6 A schematic diagram of an LVDS output interface provided in an embodiment of the present application, Figure 7 A schematic diagram of an FPGA BANK provided in an embodiment of the present application.

[0050] like Figures 1 to 7 As shown, the embodiment of the present application provides a video splicing method based on MIPI, including:

[0051] S1, receiving at least two MIP I video signals, and unpacking and recovering the video timing;

[0052] S2, caching the MIPI video signal after the video timing is restored, and performing frame synchronization of the two MIPI video signals;

[0053] S3. Based on the video timing required by the screen, the two MIPI video signals are synchronized in the clock domain, and the two MIPI video signals after the clock domain synchronization are read out in sequence according to the splicing order and the video is spliced.

[0054] Specifically, the FPGA chip of the MIPI DSI interface receives the two-way MIPI video signals. The MIPI DSI decoding function is performed to receive and unpack the video signal and restore the video timing. The received MIPI video signal is unpacked, the video data stream is decomposed into separate image frames, and the original video timing is restored for subsequent processing. Inside the FPGA, a dedicated video processing IP core is used to cache the MIPI video signal after the video timing is restored. The two-way MIPI video signals are frame-synchronized through the frame synchronization algorithm to ensure the synchronization of the two-way video images during display.

[0055] After the clock domain synchronization, the two MIPI video signals are read out in sequence according to the splicing order, and the two video signals are spliced ​​using the video splicing algorithm to generate a complete video image. The spliced ​​video image is output to the display screen to achieve the synchronous display of multiple video images. For example, the timing_gen module can generate a video timing that meets the screen requirements. The rd_ctr l module reads the image content of the psram according to the video timing generated by the timing_gen module, writes it into the rd_fifo, and performs clock domain synchronization. The mix_ctr l module then reads out the image content of rd_fifo_0 and rd_fifo_1 in sequence according to the splicing order to achieve the splicing function.

[0056] It should be noted that the MIPI-based video splicing method of the present application can realize video splicing of 2-way MIPI input and 1-way LVDS output. The overall architecture is as follows: Figure 2 As shown. Inside the FPGA, two built-in MIPI DS I modules are used to receive and unpack the two MIPI video signals respectively and restore the video timing. Since the PSRAM operating clock inside the FPGA does not match the pixel clock of the video stream, FIFO is required for synchronization when writing and reading data from the PSRAM. In particular, the splicing of the two MIPI video signals can be vertical splicing or horizontal splicing, and the splicing position can be swapped left and right or up and down. Through the FPGA with built-in flash, the power-on initialization time is as short as 50ms, which improves the efficiency of video splicing.

[0057] By receiving at least two MIPI video signals, unpacking and recovering the video timing; caching and performing frame synchronization of the two MIPI video signals, the efficiency of video processing is improved. Based on the video timing required by the screen, the two MIPI video signals are synchronized in the clock domain, and the two MIPI video signals after clock domain synchronization are read out and spliced ​​in sequence in order to achieve data reception and video splicing. Since no additional SDRAM memory is required to cache a large amount of video data, a more compact system design can be achieved, which is particularly suitable for small-screen application devices with limited space, with low power consumption, low cost and easy miniaturization. By caching and frame synchronization processing of video signals inside the FPGA, the real-time performance of video data and the reliability of display are ensured to meet the requirements of the application.

[0058] In some embodiments, S2, caching the MIPI video signal after the video timing is restored, and performing frame synchronization of the two MIPI video signals, specifically includes:

[0059] S21, writing the MIPI video signal after the video timing is restored into the wr_fifo module;

[0060] S22, when the psram controller is idle, the data of the wr_fifo module is read out;

[0061] S23, writing into the psram according to the packing format of the psram controller, and caching a whole frame to achieve frame synchronization of two MIPI video signals.

[0062] Specifically, inside the FPGA, the processed video data is written to a module called wr_fifo, wherein the wr_fifo module is responsible for temporarily storing the video data and waiting for subsequent read operations, thereby completing the writing of the MIPI video signal after the video timing is restored to the wr_fifo module. When the psram controller is idle, the data of the wr_fifo module is read out. After reading the data, the psram controller will write the video data to psram (psram0, psram1) according to a predetermined packaging format. The packaging format includes a specific data order, data size or other related parameters to ensure the correct storage of the video data and subsequent fast reading. It should be noted that when caching, it is necessary to ensure that a whole frame of caching is performed to achieve frame synchronization of two MIPI video signals.

[0063] For example, the wr_ctrl module stores the video signal recovered by the mipidsiPHY module in the wr_fifo module. When the psram controller is idle, the fifo data is read out and written into the psram according to the packing format of the psram controller to cache a whole frame, thereby achieving frame synchronization of the two-way video to ensure stable display of the video image.

[0064] In some embodiments, S21, the step of writing the MIPI video signal after the video timing is restored into the wr_fifo module, specifically includes:

[0065] S211, obtaining the MIPI video signal after the video timing is restored;

[0066] S212, writing the MIPI video signal into the wr_fifo module;

[0067] S213, determining whether the wr_fifo module is fully loaded with a line of video data;

[0068] S214, if the wr_fifo module is full of one line of video data, read out one line of video data from the wr_fifo module and pack and write it into the psram;

[0069] S215. If the wr_fifo module does not store a full line of video data, return to S211.

[0070] Specifically, see Figure 3 , the received MIPI video signal is decoded and processed by the video decoder to restore the original video timing so that the order of the video signal is consistent with the original acquisition order. The decoded video signal is written into the wr_fifo module at a certain rate, wherein the wr_fifo module has enough capacity to adapt to the temporary storage requirements of video data at the current resolution. During the process of writing data into the wr_fifo module, the system will continuously monitor to determine whether a row of video data has been fully stored. This is achieved by comparing the amount of data currently stored with the total amount of a row of data of the image. When it is detected that the wr_fifo module is fully stored with a row of video data, the system will start the data read operation and perform necessary packaging processing on the read video row data so as to facilitate storage in the PSRAM. Packaging may include encoding, compressing or other forms of packaging to ensure efficient data storage. If the wr_fifo module is not fully stored with a row of data, it means that the current video data input has not been completed, and the system will return to step S211 to continue processing new video data.

[0071] In some embodiments, S3, based on the video timing required by the screen, the two MIPI video signals are synchronized in the clock domain, and the two MIPI video signals after the clock domain synchronization are read out in sequence according to the splicing order, and the steps of video splicing specifically include:

[0072] S31, obtaining the row synchronization signal of the timing_gen module;

[0073] S32, based on the row synchronization signal, read out the MIPI video signal corresponding to the row in the psram, and write them into the rd_fifo module respectively;

[0074] S33, determining whether the rd_fifo module is fully loaded with a row of video data;

[0075] S34, if the rd_fifo modules are all full of one line of video data, then splicing the video data of the rd_fifo modules;

[0076] S35. If the rd_fifo module does not store a full line of video data, return to S31.

[0077] Specifically, the system obtains the line synchronization signal from the timing_gen module. The timing_gen module is a hardware module that generates accurate line synchronization signals to ensure that the readout of the video signal is synchronized with the refresh rate of the display screen. Under the control of the line synchronization signal, the system reads the MIPI video signal of the corresponding line from the PSRAM and writes these signals into two different rd_fifo modules respectively. The system checks whether each rd_fifo module has stored a full line of video data. When both rd_fifo modules are full of a line of video data, the system splices the video data in the two modules. The splicing process may include aligning the two video data streams, adjusting the timing, converting the color space, etc., to ensure that the quality of the final output video signal meets the requirements. If any rd_fifo module does not store a full line of video data, it means that the current video data reading has not been completed, and the system will return to step S31 to continue waiting and reading new video data. This process will be repeated until all video data are correctly read, spliced ​​and output.

[0078] In some embodiments, in the frame synchronization of the two MIPI video signals, when a new frame of the built-in timing arrives, if the progress k of the currently stored frame is greater than the set threshold k0, the frame header stored this time starts to be output;

[0079] If k is less than k0, the output starts from the frame header of the previous frame; where 0<k≤1.

[0080] Specifically, the progress k of storing a frame represents the degree parameter of the current frame storing a frame in the video, ranging from 0 to 1, 1 means completion of storage, 0 means start of storage, and the threshold k0 is a set value used to determine when to start outputting a new frame. At the beginning, the progress k is set to 0, indicating that the new frame has not yet started to be processed. The threshold k0 is set. This value can be adjusted according to actual needs. The system continuously monitors the two-way MIPI video signal and waits for the video data of a new frame. When the video data of a new frame is detected, the current progress k is checked. If k is greater than k0 (ie k>k0), it means that the current threshold has been passed, enough data is ready, and a new frame can be started. If k is less than or equal to k0 (ie k≤k0), it means that the current data is not ready yet and needs to be output from the frame header of the previous frame.

[0081] See also Figure 4, frame synchronization is achieved by adopting a scheme of caching two frames and reconstructing the video timing, so it has a high tolerance for the timing parameters of the front-end signal source. Even if the timing parameters of the signal source and the pixel clock frequency change greatly, it can still ensure a stable signal output to the screen. In the embodiment of the present application, the control module monitors the progress of storing a frame currently. Assuming that the progress of storing a frame currently is k (0<k≤1), when a new frame of the built-in timing arrives, if k is greater than the set threshold k0, the frame header stored this time will be output. If k is less than k0, the output will start from the frame header of the previous frame.

[0082] In some embodiments, the following relationship is satisfied between the input frame rate fi n, the output frame rate fout and the set threshold k0:

[0083] (1-k 0 )f out <f in <(2-k 0 )f out .

[0084] Specifically, to prevent screen tearing caused by the encounter of the read and write addresses, the input frame rate fi n and the output frame rate fout should satisfy the above relationship. For example, the frame rate output to the screen is 60Hz, k0 is 0.5, then this solution supports input frame rates in the range of 30Hz to 90Hz.

[0085] In some embodiments, the time interval td from when the signal source completes a frame input to when it completes the frame output satisfies the following relationship:

[0086] 0 <t d <k 0 T in +T out

[0087] Wherein, Tin and Tout are the frame periods of the system input and output video signals respectively.

[0088] Specifically, the time interval td from the signal source completing a frame input to the system completing the frame output has the above-mentioned value range. For example, k0 is 0.2, and the frame rates of the input and output video signals are both 60 Hz, then the maximum delay td is only 20 ms, which has a higher real-time performance.

[0089] It should be noted that in the video splicing method based on MIPI in the embodiment of the present application, relying on built-in MIPI decoding module and PSRAM large-capacity memory, a single chip is realized to solve data reception, splicing and point screen flow, with low power consumption, low cost and easy to miniaturize. The mode of caching 2 frames and built-in video timing is adopted, and the wide range refresh rate of compatible signal source is with low delay, high real-time performance, and at the same time, it has LVDS level and TTL level screen interface, and can be compatible with multiple screen resolutions, and has high flexibility and customizability. At the same time, the scheme of the embodiment of the present application has high flexibility, adapts to different image ratios and resolutions, and can support video signal splicing with a total bandwidth lower than 1080P@50Hz.

[0090] The second aspect of the present application provides a MIPI-based video splicing system, including:

[0091] A receiving module, used to receive at least two MIPI video signals, unpack them and restore the video timing;

[0092] A frame synchronization module, used for caching the MIP I video signal after the video timing is restored, and performing frame synchronization of the two MIP I video signals;

[0093] The splicing module is used to perform clock domain synchronization on the two MIP I video signals based on the video timing required by the screen, and to read out the two MIP I video signals after clock domain synchronization in a splicing order and perform video splicing.

[0094] Specifically, the receiving module receives at least two MIP I video signals, which come from different video sources, such as cameras or video players, and uses the high-bandwidth processing capability of FPGA to unpack the received video signals and restore the original video data and timing information. The frame synchronization module caches the MIP I video signals after the restored video timing to ensure the real-time and reliability of the video data. The frame synchronization logic is implemented to ensure that the frames of the two video signals are processed in the correct timing, for example, by comparing the frame headers of the two video frames. If the currently stored progress k is greater than the set threshold k0, a new frame is output; otherwise, the output starts from the frame header of the previous frame. The splicing module performs clock domain synchronization on the two video signals according to the display requirements of the screen to ensure the timing consistency of the video data, wherein the two video signals after clock domain synchronization are read out in sequence according to the splicing order, video splicing is performed, and the spliced ​​video signals are output, which can be displayed on a motorcycle dashboard, a portable measuring instrument or other electronic instrument products.

[0095] In some embodiments, the MIPI-based video splicing system includes a row determination module, and the row determination module is used to determine whether a row of video data is fully stored.

[0096] A third aspect of the present application provides a storage medium, wherein the storage medium stores a computer program, and wherein the computer program implements any of the above methods when executed by a processor.

[0097] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the same. Although the present application has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to replace some of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions claimed to be protected by the present application.

Claims

1. A video splicing method based on MIPI, characterized in that: include: Receive at least two MIPI video signals, unpack them and restore the video timing; Cache the MIPI video signal after the video timing is restored, and perform frame synchronization of the two MIPI video signals; Based on the video timing required by the screen, the two MIPI video signals are synchronized in the clock domain, and the two MIPI video signals after the clock domain synchronization are read out and spliced ​​in sequence according to the splicing order.

2. A MIPI-based video splicing method according to claim 1, characterized in that: S2, caching the MIPI video signal after the video timing is restored, and performing frame synchronization of the two MIPI video signals, specifically comprising: Write the MIPI video signal after the video timing is restored into the wr_fifo module; When the psram controller is idle, the data of the wr_fifo module is read out; The data is written into the psram according to the packing format of the psram controller, and a whole frame is cached to achieve frame synchronization of two MIPI video signals.

3. A MIPI-based video splicing method according to claim 2, characterized in that: S21, the step of writing the MIPI video signal after the video timing is restored into the wr_fifo module, specifically comprising: Obtain the MIPI video signal after restoring the video timing; Writing the MIPI video signal into the wr_fifo module; Determine whether the wr_fifo module is fully loaded with a line of video data; If the wr_fifo module is full of one line of video data, then read out one line of video data from the wr_fifo module and pack it into the psram; If the wr_fifo module does not store a full line of video data, the process returns to S211.

4. A MIPI-based video splicing method according to claim 2, characterized in that: Based on the video timing required by the screen, the two MIPI video signals are synchronized in the clock domain, and the two MIPI video signals after the clock domain synchronization are read out in sequence according to the splicing order, and the video splicing steps specifically include: Get the line synchronization signal of the timing_gen module; Based on the row synchronization signal, the MIPI video signal corresponding to the row in the psram is read out and written into the rd_fifo module respectively; Determine whether the rd_fifo module is fully loaded with a line of video data; If the rd_fifo modules are all full of one line of video data, the video data of the rd_fifo modules are spliced; If the rd_fifo module does not store a full line of video data, the process returns to S31.

5. The MIPI-based video splicing method according to claim 1, characterized in that: In the frame synchronization of the two MIPI video signals, when a new frame of the built-in timing arrives, if the progress k of the currently stored frame is greater than the set threshold k0, the frame header stored this time starts to be output; If k is less than k0, the output starts from the frame header of the previous frame; where 0<k≤1.

6. A MIPI-based video splicing method according to claim 5, characterized in that: The following relationship is satisfied between the input frame rate fin, the output frame rate fout and the set threshold k0: (1-k0)f out <f in <(2-k0)f out 。 7. A MIPI-based video splicing method according to claim 6, characterized in that: The time interval td from the signal source completing a frame input to completing the frame output satisfies the following relationship: 0<t d <k0T in +T out Among them, Tin and Tout are the frame periods of the system input and output video signals respectively.

8. A MIPI-based video splicing system, characterized in that: include: A receiving module, used to receive at least two MIPI video signals, unpack them and restore the video timing; A frame synchronization module, used for caching the MIPI video signal after the video timing is restored, and performing frame synchronization of two MIPI video signals; The splicing module is used to perform clock domain synchronization on the two MIPI video signals based on the video timing required by the screen, and to read out and splice the two MIPI video signals after clock domain synchronization in a splicing order.

9. A MIPI-based video splicing system according to claim 8, characterized in that: The MIPI-based video splicing system includes a row determination module, and the row determination module is used to determine whether a row of video data is fully stored.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images