MIPI-based optimized video splicing method, system and storage medium

Abstract

The invention discloses an optimized video splicing method based on MIPI, comprising the following steps: an FPGA main control module receives video data; performs format processing and video cutting on the video data to generate multiple blocks of image data; stores the multiple blocks of image data in sequence according to a storage format and calculates the pixel value of each block of image data, wherein the storage format includes a mirror format and a non-mirror format; determines whether to perform scaling processing on each block of image data according to configuration information of the external controller; if so, performs scaling processing on each block of image data in sequence according to its pixel value to obtain a target image pixel value corresponding to each block of image data; uses the same timing control module to convert each block of target image pixel value into a corresponding MIPI signal according to the MIPI protocol, and sends the signal to multiple modules for display at the same time, thereby greatly reducing FPGA resources and improving the operation efficiency of FPGA.

Description

Technical Field

[0001] The present invention relates to the technical field of display and testing of liquid crystal modules, and in particular to an optimized video splicing method, system and storage medium based on MIPI. Background Art

[0002] Currently, there are two main ways to implement MIPI video segmentation processing in the display industry. One is to transfer the segmented video data to the module for display through the RGB to MIPI chip, and the other is to process and cut the video data internally through the FPGA and transfer the data to the module for display through the MIPI protocol.

[0003] The first method is to realize the function of RGB to MIPI through hardware. Although this method is easy to develop, the development cost increases due to the use of multiple conversion chips. The second method is a more common method. This method requires FPGA to implement the MIPI protocol. Although this method has a low development cost, if multiple modules are used for display, multiple MIPI cores need to be instantiated, which occupies more internal resources of the FPGA and causes the system processing speed to be relatively slow. Summary of the invention

[0004] The object of the present invention is to provide a MIPI-based optimized video splicing method, system and storage medium.

[0005] In order to achieve the above-mentioned object of the invention, the present invention provides a MIPI-based optimized video splicing method, the method comprising the steps of:

[0006] As a further improvement of the present invention, the method also includes: an FPGA main control module receives video data; performs format processing and video cutting on the video data to generate multiple blocks of image data; stores the multiple blocks of image data in sequence according to a storage format and calculates the pixel value of each block of image data, wherein the storage format includes a mirror format and a non-mirror format; determines whether to perform scaling processing on each block of image data according to the configuration information of the external controller; if so, performs scaling processing on each block of image data in sequence according to its pixel value to obtain a target image pixel value corresponding to each block of image data; uses the same timing control module to convert each block of target image pixel value into a corresponding MIPI signal according to the MIPI protocol, and sends it to multiple modules for display at the same time.

[0007] As a further improvement of the present invention, the "storing the multiple blocks of image data in sequence according to the storage format and calculating the pixel value of each block of image data" specifically includes: determining the storage format corresponding to each block of image data according to the configuration information of the external controller; based on the storage format, calculating the storage address corresponding to each block of image data and reading the corresponding image data from the FPGA cache according to the storage address; storing the read image data in the memory of the FPGA plug-in.

[0008] As a further improvement of the present invention, the "calculating the storage address corresponding to each block of image data and reading the corresponding image data from the FPGA cache according to the storage address" specifically includes: determining the storage location of each block of image data according to the video data and the number of divided image data blocks; if the storage format corresponding to the image data block currently being read is a non-mirror format, reading the image data from the low address to the high address of the current image data block; if the storage format corresponding to the image data block currently being read is a mirror format, reading the image data from the high address to the low address of the current image data block.

[0009] As a further improvement of the present invention, the method also includes: using the read-write storage module in the FPGA to implement read-write operations between the FPGA and its external memory; the read-write storage module is composed of an interface read-write control logic with multiple read and one write ports and a controller of the external memory to cache the image data.

[0010] As a further improvement of the present invention, the "converting the pixel values ​​of each block of the target image into corresponding MIPI signals according to the MIPI protocol, and sending them to multiple modules for display at the same time" specifically includes: instantiating multiple data parallel-to-serial converters at the physical layer according to the multiple target image pixel values; converting the multiple target image pixel values ​​into corresponding MIPI signals according to the configuration information of the external controller and the MIPI protocol; and sending the multiple MIPI signals to multiple corresponding modules for display at the same time through the multiple data parallel-to-serial converters.

[0011] As a further improvement of the present invention, before the step of "simultaneously sending the multiple blocks of image data to multiple corresponding modules for display through the multiple data parallel-to-serial converters", the step also includes: sending the module register information that needs to be read to all modules by broadcasting; receiving feedback information from all modules and arbitrating according to the feedback information, screening out the module register information that meets the requirements and sending it to the external controller, so as to realize reading the register information of the corresponding module.

[0012] As a further improvement of the present invention, the method further includes: the scaling process adopts a bilinear interpolation method.

[0013] As a further improvement of the present invention, the method further includes: the storage formats corresponding to each block of image data are independent of each other.

[0014] The present invention also provides an optimized video splicing system based on MIPI, which includes an external controller module for configuring address information and parameter information required inside the FPGA; a communication module for realizing communication between the external control module and the FPGA and reading and writing data to the external memory; a global register configuration module for configuring the configuration of relevant timing parameters and MIPI interface related parameters during the image output; an image data packaging module for packaging according to the configuration information of the external register in a specified data format and transmitting it to the image data storage module; an image data storage module for storing the image data in the external memory in a specified storage format according to the configuration information of the external controller; an image data processing module for sequentially reading corresponding image data from the external memory according to the specified address and performing data processing; an image data cache module for caching the multiple blocks of image data to realize the read and write operations between the FPGA and the memory external to the FPGA; and an image data output module for outputting the multiple target image pixel values ​​generated by the image data processing module to the corresponding multiple modules according to the specified MIPI protocol.

[0015] The present invention also provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, any one of the above methods for optimizing video splicing based on MIPI is implemented.

[0016] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0017] The present invention mainly transmits multiple image data blocks to multiple modules for display according to the MIPI protocol. During the data transmission process, multiple MIPI cores share the same protocol layer, and instantiate multiple data parallel-to-serial converters in the physical layer to realize broadcast data transmission, receive feedback information from multiple modules, arbitrate the feedback information, select module register information that meets the requirements and send it to an external controller to control the image data block output to the display of the corresponding module. One protocol layer corresponds to multiple physical layers, so that one MIPI core can light up multiple modules at the same time, greatly reducing FPGA resources and improving the operating efficiency of FPGA; in addition, the screen displayed by multiple modules can realize independent adjustment, scaling and screen mirroring functions, so as to facilitate users to debug themselves, make the application more convenient, and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1It is a flowchart of the steps of the MIPI-based optimized video splicing method in an embodiment of the present invention.

[0019] Figure 2 It is a schematic diagram of an optimized video splicing system based on MIPI in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0021] It should be noted that the term "comprises" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In addition, the terms "first", "second", etc. are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0022] The embodiment of the present invention is directed to a MIPI-based optimized video splicing method, which is modified on the basis of MIPI to achieve one protocol layer corresponding to multiple physical layers, and send the cut multiple video data to multiple modules at the same time, thereby greatly reducing FPGA resources and improving FPGA operation rate. Based on conventional or no creative labor, the execution order of the steps in the method that do not have necessary causal relationships in logic is not limited to the execution order provided in the embodiments of the present application.

[0023] like Figure 1 As shown, the MIPI-based video optimization splicing system includes: an external control module (STM32), a communication module (FSMC), a global register configuration module (global_REG), an image data packaging module (DATA_STREAM), an image data storage module (PIC_Storage), an image data processing module (PIC_Process), an image data cache module (SDRAM_IF) and an image data output module (MIPI_IP).

[0024] The external control module configures related modules through the bus, mainly configuring some registers required inside the FPGA, such as image pixel frequency, resolution, timing parameters, some parameters required by the MIPI core, etc. This module notifies the FPGA through the bus to cache the image data output by the video receiving card into the memory plugged in the FPGA, and also transmits the address to be cached by the external memory to the register of the FPGA through the bus. In other words, the external controller module communicates with the FPGA through the communication module. In addition, the communication module can also read and write data to the external memory.

[0025] Furthermore, in this embodiment, the global register configuration module (global_REG) is mainly composed of a register unit module and a multi-channel interface transfer control module, wherein the register unit module is used to configure the timing parameters of the image output, and the interface transfer control module is a peripheral interface reserved for the convenience of the later expansion of the system. It is only necessary to connect the peripherals of the corresponding interface to the interface transfer module, such as the MIPI interface controller and the configurable clock generation module used by the controller. Specifically, in this embodiment, a 2-way axi-lite interface transfer control module is used as the interface transfer module.

[0026] The image data packaging module (DATA_STREAM) is mainly composed of two parts: input interface timing control and data packaging. Among them, the timing control is mainly to adjust the timing of the input signal, and the data packaging is mainly to transmit the received data to the image data storage module in the corresponding format for storage.

[0027] The image data storage module (PIC_Storage) is mainly used to store the image data generated by the image data packaging module in the FPGA external memory, such as DDR, in a mirror or non-mirror storage format according to the configuration of the external controller. The module mainly includes two parts, namely the address control part and the data control part. The module opens two cache areas on the external memory, assuming that they are cache area 1 and cache area 2. Specifically, cache area 1 is used to store a row of image data read from the low-power random access memory of the FPGA according to the storage format of different image data blocks. After caching a frame of complete image data, the image data processing module is notified to read the cached image data in the cache area. At the same time, the cache address is switched so that the write address signal points to cache area 2 to continue caching the next frame of image data, thereby realizing the rotation of read and write data between the two cache areas. Of course, the cache is not limited to two, but can also be multiple, and corresponding adjustments are made according to actual needs.

[0028] The image data processing module (PIC_Process) mainly realizes reading the cached image data from the external memory according to a specific address. In addition, this module will read the frame of image data from the corresponding address only after a frame of image data is completely cached in the external memory, thereby ensuring the integrity and accuracy of the read data.

[0029] The image data cache module (SDRAM_IF) is mainly used for caching image data. In this embodiment, the storage module is composed of an interface read-write control logic with multiple read and one write ports and an external memory controller. Specifically, the module can cache multiple blocks of image data generated by cutting in sequence, and finally achieve the effect of outputting multiple blocks of image data at the same time through the read-write control logic. In an embodiment of the present invention, when the obtained video data is cut into N blocks, 8 blocks of image data are generated. For example, N is equal to 8, then the interface read-write controller is set to the corresponding read-write port logic, that is, the AXI4 interface read-write control logic is set to 8-way read and 1-way write port, so that the cached 8 blocks of image data can be output to different modules at the same time.

[0030] The image data output module (MIPI_IP) is mainly responsible for outputting the multiple blocks of image data output from the image data cache module to the corresponding multiple modules according to the specified MIPI protocol to display the complete image data.

[0031] Specifically, in this embodiment, the external controller communicates with the FPGA through the communication module. The FPGA reads the video data according to the image address configured in the global register configuration module, performs format processing as required, and outputs it to the FPGA in the specified format, such as RGB or RRGGBB. Then, the video data of the specified output format is cut as required to generate multiple image data blocks. The image data packaging module organizes and packages the image data according to the configuration information of the global register, the specified data format and the delay signal, and transmits it to the image data storage module. The storage module determines whether a mirror operation is required according to the configuration of the FPGA by the external controller module, so as to store multiple image data blocks in the external memory. Finally, the image processing module reads data from the external memory according to the specified address, and finally caches the read data in the image data cache module in sequence, and then outputs it to the image data output module with the help of the read-write controller therein, and then outputs it to the corresponding multiple modules and displays it.

[0032] like Figure 2 As shown, the embodiment of the present invention also provides a MIPI-based optimized video splicing method, comprising the following steps:

[0033] Step 101: The FPGA main control module receives video data.

[0034] Step 102: format processing and video cutting are performed on the video data to generate multiple blocks of image data.

[0035] Step 103: Store the multiple blocks of image data in sequence according to the storage format and calculate the size of each block of image data.

[0036] Step 104: Determine whether to perform scaling processing on each block of image data according to the configuration information of the external controller.

[0037] Step 105: If yes, scaling the image data blocks in turn according to their sizes to obtain pixel values ​​of the image data blocks.

[0038] Step 106 , using the same timing control module, converting the pixel value of each block of image data into a corresponding MIPI signal according to the MIPI protocol, and simultaneously sending the signal to multiple modules for display.

[0039] In step 101, the external controller configures the register information required inside the FPGA, such as MIPI core related parameters and timing information, into the FPGA register in advance so that the FPGA can work normally. When the video receiving card receives the corresponding image data, the FPGA will load the address of the image data according to the global register configuration information and obtain the image data according to the address.

[0040] In step 102, the low-power random access memory in the FPGA will store each frame of image data in the external memory DDR first when reading it. Before writing it into the external memory DDR, the image data needs to be decoded and cut according to the specified format to generate multiple blocks of image data. Specifically, in this embodiment, the input image data format and the data storage format of the external memory DDR may be different. For example, the image data format is 24 bits, while the data storage format of DDR is 512 bits. Therefore, the image data needs to be processed according to the data format of the external memory DDR, and then the image data that meets the storage format is written into the external memory DDR to increase the transmission rate.

[0041] In step 103, it specifically includes:

[0042] Step 1031, determining the storage format corresponding to each block of image data according to the configuration information of the external controller.

[0043] Step 1032: Based on the storage format, calculate the storage address corresponding to each block of image data and read the corresponding image data from the FPGA cache according to the storage address.

[0044] Step 1033, storing the read image data in a memory external to the FPGA.

[0045] In the embodiment of the present invention, the FPGA determines the storage format corresponding to each image data block after cutting according to the configuration information of the external controller, wherein the storage format includes a mirror format and a non-mirror format, and the storage format corresponding to each image data block is independent of each other. If the image data block currently read is in a non-mirror format, the data in the image data block is read from a low address to a high address according to a normal reading method and is stored in sequence in an external memory DDR. On the contrary, if the image data block currently read is in a mirror format, the data in the image data block is read from a high address to a low address and is stored in sequence in an external memory DDR. While storing, the image size corresponding to each block of image data must also be calculated.

[0046] Here, it should be explained that before storage, the external memory DDR opens up two buffer areas in advance to realize rotation. When the vertical synchronization (vsync) signal of the next frame of data is received, it proves that the external memory has completely received a frame of image. At this time, the image data processing module will be notified to read the image data from the external memory DDR according to the corresponding address. At the same time, the external memory write address signal will also be transferred to the next buffer area to complete the address switching and realize the next buffer area to store the next frame of image data.

[0047] In step 104, after the external memory DDR stores a complete frame of image data, the image data processing module in the FPGA reads the frame of image data from the external memory according to the specified address. The frame of image data is composed of multiple blocks of image data. According to the configuration information of the external controller, it is determined whether to perform scaling processing on each block of image data, and the scaling ratio is calculated according to the size of each block of image data, wherein the scaling ratio of each block of image data is the same. Specifically, in the embodiment of the present invention, the bilinear interpolation method is used to implement the scaling of image data, and the formula is as follows:

[0048] f(x,y)=f(0,0)(1-x)(1-y)+f(1,0)x(1-y)+f(0,1)(1-x)y+f(1,1)xy, where x and y are coordinate points in the coordinate system, and f is the pixel value of a pixel point.

[0049] In the embodiment of the present invention, specifically, according to the configuration information of the external controller, the target image size displayed by each module is obtained, and then combined with the size of each data block in the current frame of image data, the corresponding position of a certain pixel point of the target image in the current image data is determined, and then the four pixels closest to the corresponding position and the corresponding weights are calculated, so as to calculate the scaled pixel value using the above formula to obtain the scaled pixel value of each block of image data. Then, the multiple blocks of image data generated after scaling are cached in sequence, and according to the configuration information of the external register, it is determined whether to perform data offset or frame processing, so as to cache multiple processed image data blocks.

[0050] In step 106, the multiple image data blocks and data valid signals generated in step 105 are transmitted to the same timing control module, which sends the data to a MIPI IP core module in real time according to the configuration of the external controller. The module uses the same protocol layer to convert the pixel value of each block of image data into a corresponding MIPI signal and sends it to multiple physical layers, so that one MIPI IP core can light up multiple modules at the same time. In an embodiment of the present invention, specifically, according to the MIPI protocol, N parallel-to-serial converters (OSERDESE) are instantiated at the physical layer according to the number of image data cuts, and multiple image data blocks and corresponding clock signals are sent to corresponding multiple modules at the same time to realize simultaneous transmission and display of data, so that a frame of image can be fully displayed.

[0051] In an embodiment of the present invention, before the module is displayed, different modules may have different display requirements, such as the brightness of the module display. In order to meet the requirements, the module register information to be read is first sent to multiple modules in a broadcast manner, and the directions of multiple data buses are turned before the module response is received, and the information fed back by multiple modules is received simultaneously. After receiving the feedback data, the data bus is turned again, thereby realizing the sending and receiving of data using the same set of buses. In addition, after receiving the feedback information of multiple modules, the received feedback information is arbitrated according to the requirements, and the module register information that meets the conditions is screened out and sent to the external controller, so as to facilitate the corresponding processing of the image data before display, and finally display it in the corresponding module, so that the image data finally displayed meets the requirements of each module; and the user can also independently adjust the relevant parameters of a module according to the requirements, which is easy to operate and improves the user experience.

[0052] An embodiment of the present invention provides an optimized video splicing system based on MIPI, including: an external controller module, used to configure address information and parameter information required inside FPGA; a communication module, used to realize communication between the external control module and FPGA and read and write data to an external memory; a global register configuration module, used to configure the configuration of relevant timing parameters and MIPI interface related parameters during the image output; an image data packaging module, used to package according to the configuration information of the external register in a specified data format and transmit it to the image data storage module; an image data storage module, used to store the image data in an external memory in a specified storage format according to the configuration information of the external controller; an image data processing module, used to read the corresponding image data in sequence from the external memory according to the specified address and perform data processing; an image data caching module, which sequentially caches each block of image data processed by the image data processing module; and an image data output module, used to output the cached multiple blocks of image data to corresponding multiple modules according to the specified MIPI protocol.

[0053] An embodiment of the present invention further provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the optimized video splicing method based on MIPI as described in any one of the above items is implemented.

[0054] In summary, the present invention provides an optimized video splicing method, system and storage medium based on MIPI. The technical solution of the present invention is to make modifications and optimizations on the basis of MIPI, so as to realize that one protocol layer corresponds to multiple physical layers, thereby greatly reducing FPGA resources, not only reducing costs, but also improving the FPGA operation rate. In addition, after the split screen is realized, the screens displayed by multiple modules are independent, with functions such as independent adjustment, screen mirroring or zooming, so that customers can debug conveniently and easily by themselves.

[0055] It should be understood that although this specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation mode may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

[0056] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. A MIPI-based optimized video splicing method, characterized in that: The method comprises: The FPGA main control module receives the video data; Performing format processing and video cutting on the video data to generate multiple blocks of image data; The plurality of blocks of image data are sequentially stored according to a storage format and a pixel value of each block of image data is calculated, wherein the storage format includes a mirror image format and a non-mirror image format; Determining whether to perform scaling processing on each block of image data according to configuration information of the external controller; If yes, scaling each block of image data is performed in turn according to the pixel value of each block of image data to obtain the target image pixel value corresponding to each block of image data; The same timing control module is used to convert the target image pixel value corresponding to each block of image data into a corresponding MIPI signal according to the MIPI protocol, and simultaneously send the signal to multiple modules for display.

2. The optimized video splicing method based on MIPI according to claim 1, characterized in that: The “storing the plurality of image data in sequence according to the storage format” specifically includes: Determine the storage format corresponding to each block of image data according to the configuration information of the external controller; Based on the storage format, calculate the storage address corresponding to each block of image data and read the corresponding image data from the FPGA cache according to the storage address; The read image data is stored in a memory external to the FPGA.

3. The optimized video splicing method based on MIPI according to claim 2, characterized in that: The "calculating the storage address corresponding to each block of image data and reading the corresponding image data from the FPGA cache according to the storage address" specifically includes: Determining the storage location of each block of image data according to the size of the video data and the number of divided image data blocks; If the storage format corresponding to the image data block currently being read is a non-mirror format, reading the image data from the lower address of the image data block currently being read toward the higher address; If the storage format corresponding to the image data block currently being read is the mirror format, the image data is read starting from the high address to the low address of the image data block currently being read.

4. The optimized video splicing method based on MIPI according to claim 2, characterized in that: The method further comprises: Use the read-write storage module in the FPGA to implement read-write operations between the FPGA and its external memory; The read-write storage module is composed of an interface read-write control logic of multiple read ports and one write port and a controller of an external memory to cache the image data.

5. The optimized video splicing method based on MIPI according to claim 1, characterized in that: The “converting the target image pixel value corresponding to each block of image data into a corresponding MIPI signal according to the MIPI protocol, and simultaneously sending it to multiple modules for display” specifically includes: Instantiating a plurality of data parallel-to-serial converters at a physical layer according to a target image pixel value corresponding to each block of image data; According to the configuration information of the external controller and the MIPI protocol, the target image pixel value corresponding to each block of image data is converted into a corresponding MIPI signal; The multiple MIPI signals are simultaneously sent to the multiple corresponding modules through the multiple data parallel-to-serial converters for display.

6. The optimized video splicing method based on MIPI according to claim 5, characterized in that: Before the step of "sending the plurality of MIPI signals to the plurality of corresponding modules through the plurality of data parallel-to-serial converters simultaneously for display", the method further comprises: Send the module register information that needs to be read to all modules through broadcasting; Receive feedback information from all modules and perform arbitration according to the feedback information, filter out module register information that meets the requirements and send it to the external controller to read the register information of the corresponding module.

7. The optimized video splicing method based on MIPI according to claim 1, characterized in that: The scaling process uses a bilinear interpolation method.

8. The optimized video splicing method based on MIPI according to claim 1, characterized in that: The storage formats corresponding to each block of image data are independent of each other.

9. An optimized video splicing system for executing the MIPI-based optimized video splicing method according to any one of claims 1 to 8, characterized in that: include: The external controller module is used to configure the address information and parameter information required inside the FPGA; A communication module, used to realize communication between the external controller module and the FPGA and to read and write data to the external memory; A global register configuration module, used to configure the relevant timing parameters during the image output and the configuration of the relevant parameters of the MIPI interface; The image data packaging module is used to package the data in a specified data format according to the configuration information of the external register and transmit it to the image data storage module; An image data storage module, used to store the image data in an external memory in a specified storage format according to the configuration information of the external controller; An image data processing module is used to read corresponding image data from an external memory in sequence according to a specified address and perform data processing; An image data cache module, used for caching the plurality of blocks of image data to implement read and write operations between the FPGA and a memory external to the FPGA; The image data output module is used to output the multiple target image pixel values ​​generated by the image data processing module to the corresponding multiple modules according to the specified MIPI protocol.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images