System, method and apparatus for video return using a multi-path display interface

Abstract

The application discloses a system, method and device for video back injection by using a multi-path display interface, and relates to the technical field of image processing. The system comprises a video back injection system and an automatic driving system. The video back injection system comprises a main chip, a plurality of serializers corresponding to the main chip, and a protocol data conversion unit. The protocol data conversion unit is configured to convert video stream data after color space conversion into protocol data of a long-distance transmission protocol, and transmit the protocol data to the automatic driving system. The main chip comprises a processor, a memory, a plurality of display units and a plurality of display interface controllers. A single display unit is connected to at least one display interface controller. The display unit supports DMA and color space conversion. The application can solve the strong dependence problem of the video injection system chip interface, and utilize the traditional display interface supporting device to expand the range of available chips of the video injection system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a system, method and device for video playback using a multi-channel display interface. BACKGROUND

[0002] Current camera video injection card devices usually provide video output results through a CSI TX (Camera Serial Interface Transmitter) and output to an acceptance device through a serial deserializer, which is usually an automatic driving system, thereby replacing the real vehicle camera information input. The advantage of this processing method is that the video injection card can accept pre-recorded data to replace real vehicle data for reducing the testing cost of the automatic driving system.

[0003] Referring to Figure 1 The system architecture diagram of the video injection card can be seen, and the serial chip MAX9295D only supports the CSI TX signal from the camera or the video injection system. It can be seen that the existing method relies on the multi-channel CSI TX capability, which greatly limits the chip selection range of the video injection / playback system. SUMMARY

[0004] The present application provides a system, method and device for video playback using a multi-channel display interface, which can solve the strong dependence on the video injection system chip interface and utilize the traditional display interface supported device to expand the range of available chips for the video injection system.

[0005] In a first aspect, the embodiments of the present application provide a system for video playback using a multi-channel display interface, which includes a video playback system and an automatic driving system. The video playback system includes a main chip, a plurality of serializers corresponding to the main chip for converting video stream data converted in a color space into protocol data of a long-distance transmission protocol and transmitting the protocol data to the automatic driving system.

[0006] In combination with the first aspect, in an implementation manner,

[0007] The main chip includes a processor, a memory, a plurality of display units and a plurality of display interface controllers, and a single display unit corresponds to at least one display interface controller.

[0008] The display unit supports DMA and color space conversion. The display unit is configured to acquire the video stream data stored in the memory and perform color space conversion operation to convert the color space of the video stream data into a color space required by the automatic driving system.

[0009] In combination with the first aspect, in an implementation, the serializer supports multiple data inputs, and a single serializer corresponds to multiple display interface controllers.

[0010] In combination with the first aspect, in an implementation,

[0011] The video feedback system and the autonomous driving system are further provided with multiple deserializers, the number of the deserializers is the same as that of the serializers and the deserializers one-to-one correspond to the serializers;

[0012] The deserializers are used to convert data of the long-distance transmission protocol into target format data, and the deserializers and the serializers comply with the same protocol.

[0013] In combination with the first aspect, in an implementation, the autonomous driving system is provided with multiple video input interfaces, and a single deserializer corresponds to at least one video input interface.

[0014] In the second aspect, an embodiment of the present application provides a method for video feedback using multiple display interfaces, which is based on the above-mentioned system implementation, and the method for video feedback using multiple display interfaces comprises the following steps.

[0015] Converting video stream data in BPC8 form into video stream data in BPC10 form based on color space conversion operation;

[0016] Converting the video stream data in BPC10 form into protocol data of the long-distance transmission protocol through format conversion operation, and transmitting the protocol data to the autonomous driving system for processing.

[0017] In combination with the second aspect, in an implementation, the step of converting video stream data in BPC8 form into video stream data in BPC10 form based on color space conversion operation specifically comprises the following steps.

[0018] A display unit in the main chip acquires video stream data in BPC8 form stored in a memory in the main chip;

[0019] The display unit converts the video stream data in BPC8 form into video stream data in BPC10 form based on color space conversion operation.

[0020] In combination with the second aspect, in an implementation, the step of converting the video stream data in BPC10 form into protocol data of the long-distance transmission protocol through format conversion operation, and transmitting the protocol data to the autonomous driving system for processing specifically comprises the following steps.

[0021] A display interface controller in the main chip outputs the video stream data in BPC10 form obtained after conversion to a display interface;

[0022] The serializer of the video back-annotation system converts the video stream data in the form of BPC 10 into protocol data of a long-distance transmission protocol, and transmits the protocol data to the autonomous driving system for processing after being processed by a deserializer.

[0023] In combination with the second aspect, in an implementation, the serializer outputs data through a GMSL cable.

[0024] In a third aspect, an embodiment of the present application provides a device for video back-annotation using a multi-channel display interface, the device for video back-annotation using a multi-channel display interface comprising a processor, a memory, and a program for video back-annotation using a multi-channel display interface stored in the memory and executable by the processor, wherein the program for video back-annotation using a multi-channel display interface, when executed by the processor, implements the steps of the method for video back-annotation using a multi-channel display interface described above.

[0025] The technical solutions provided in the embodiments of the present application have the following beneficial effects:

[0026] The technical solutions provided in the embodiments of the present application have the following beneficial effects: BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 FIG. 1 is a system architecture diagram of a video injection card;

[0028] Figure 2 FIG. 2 is a structural schematic diagram of a system for video back-annotation using a multi-channel display interface according to the present application;

[0029] Figure 3 FIG. 3 is a flowchart of a method for video back-annotation using a multi-channel display interface according to the present application;

[0030] Figure 4 FIG. 4 is a corresponding example diagram of the method for video back-annotation using a multi-channel display interface according to the present application;

[0031] Figure 5 FIG. 5 is a data conversion schematic diagram;

[0032] Figure 6 FIG. 6 is a hardware structure schematic diagram of a device for video back-annotation using a multi-channel display interface according to the present application. DETAILED DESCRIPTION

[0033] In order to make the personnel in the technical field better understand the scheme of the present application, the technical scheme in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0034] First of all, it needs to be pointed out that, Figure 1 In the figure, FMC represents an FPGA intermediate board card, POC represents a technology for transmitting data and power simultaneously through a single cable, I2C represents a synchronous serial bus, VPOC represents a circuit, ADDR represents a structure, MIPI represents a serial communication interface, Lane represents a lane, MAX related represents a serializer chip model, COAX represents a coaxial interface, GPIO represents a general-purpose input / output port, ADR represents an address register, and J1, J2 and J3 represent joints.

[0035] In order to make the purpose, technical scheme and advantages of the present application more clear, the embodiments of the present application will be further described in detail below in combination with the drawings.

[0036] In a first aspect, the embodiments of the present application provide a system for video back annotation using a multi-channel display interface, to reduce the dependence on an output interface, to provide multi-channel video stream output capability using a traditional display interface, to construct a video back annotation scheme through a display interface with a bridge, and to simulate the color space required by a camera input through color space conversion.

[0037] In an embodiment, reference is made to Figure 2 , Figure 2 The figure is a structural schematic diagram of the system for video back annotation using a multi-channel display interface of the present application. As shown in the figure, Figure 2 The system for video back annotation using a multi-channel display interface includes a main chip, and a plurality of serializers corresponding to the main chip to convert video stream data converted in color space into protocol data of a long-distance transmission protocol and transmit to an automatic driving system.

[0038] Further, the main chip includes a processor, a memory, a plurality of display units and a plurality of display interface controllers, and a single display unit corresponds to at least one display interface controller. In actual application, a display unit can support a plurality of display interface controllers, for example, 1 display unit supports the output of 2 display interface controllers.

[0039] The display unit supports DMA (Direct Memory Access) and color space conversion, and is configured to obtain the video stream data stored in the memory and perform color space conversion to convert the color space of the video stream data into a color space required by the autonomous driving system.

[0040] Further, the serializer supports multiple data inputs, and a single serializer corresponds to multiple display interface controllers. In actual applications, the serializer usually supports 2 or more inputs, for example, 2 display interface controllers can be solved by using one serializer.

[0041] Further, a plurality of deserializers are arranged between the video feedback system and the autonomous driving system, the number of the deserializers is the same as that of the serializers and the deserializers one-to-one correspond to the serializers; the deserializers are configured to convert the data of the long-distance transmission protocol into target format data (i.e. the format required in actual applications), and the deserializers and the serializers comply with the same protocol. A plurality of video input interfaces are arranged in the autonomous driving system, and a single deserializer corresponds to at least one video input interface. In actual applications, the deserializer can support 1-4 outputs.

[0042] Specifically, in actual applications, the display interface controller can be DSI, DP or HDMI, and a matching serializer can be provided. The processor is configured to control the entire process. The memory is configured to store the video stream data. The display unit has DMA capability and directly carries the video stream data from the memory to the display interface. The display interface controller supports common video interfaces such as DSI, DP and HDMI.

[0043] The serializer is configured to convert the data output by the display interface controller into long-distance transmission protocol data, for example, GMSL, and common chips such as MAX96789. The deserializer is configured to convert the data of the long-distance transmission protocol into the required format data, and common chips such as MXA96724 are required. The deserializer and the serializer comply with the same protocol, and can be interconnected. The video input interface in the autonomous driving system can be CSI RX.

[0044] That is, for the system for video feedback using multiple display interfaces according to the present application, in one possible implementation scenario, it can include a plurality of connected data transmission links, each data transmission link can include a display unit, a display interface controller, a serializer, a deserializer and a video input interface connected in sequence, and the display unit in each data transmission link is connected with the memory.

[0045] The system for video back-annotation using a multi-path display interface can solve the strong dependence on the chip interface of the video injection system, utilize the traditional display interface supported device, expand the range of available chips of the video injection system, change the strong dependence on the CSITX to no dependence or less dependence, and reuse some existing multi-path display systems to construct the video back-annotation system, thereby avoiding the dependence on the existing design and commercial scheme.

[0046] In a second aspect, the embodiments of the present application also provide a method for video back-annotation using a multi-path display interface, which is implemented based on the above-mentioned system. Specifically, the system for video back-annotation using a multi-path display interface includes a video back-annotation system and an automatic driving system. The video back-annotation system includes a main chip, a plurality of serializers corresponding to the main chip for converting the video stream data converted in color space into protocol data of a long-distance transmission protocol and transmitting the protocol data to the automatic driving system. The main chip includes a processor, a memory, a plurality of display units, and a plurality of display interface controllers, and a single display unit corresponds to at least one display interface controller. The display unit supports DMA and color space conversion, and is configured to acquire the video stream data stored in the memory and perform color space conversion to convert the color space of the video stream data into a color space required by the automatic driving system. The serializer supports multi-path data input, and a single serializer corresponds to a plurality of display interface controllers. A plurality of deserializers are further arranged between the video back-annotation system and the automatic driving system, and the number of the deserializers is the same as that of the serializers and each deserializer corresponds to one serializer. The deserializer is configured to convert the data of the long-distance transmission protocol into target format data, and the deserializer and the serializer comply with the same protocol. A plurality of video input interfaces are arranged in the automatic driving system, and a single deserializer corresponds to at least one video input interface. That is, for the system for video back-annotation using a multi-path display interface, in a possible implementation scenario, a plurality of connected data transmission links can be included, each data transmission link can include a display unit, a display interface controller, a serializer, a deserializer, and a video input interface connected in sequence, and the display unit in each data transmission link is connected with the memory.

[0047] In an embodiment, referring to Figure 3 , Figure 3 FIG. 1 is a flowchart of the method for video back-annotation using a multi-path display interface. As shown in FIG. 1, the method for video back-annotation using a multi-path display interface includes the following steps. Figure 3

[0048] S1: converting the video stream data in BPC8 form into video stream data in BPC10 form based on a color space conversion operation; wherein BPC represents color depth.

[0049] ​S2: convert the video stream data in the form of BPC10 into protocol data of a long-distance transmission protocol through a format conversion operation, and transmit to an autonomous driving system for processing.

[0050] Further, in an embodiment, the video stream data in the form of BPC8 is converted into video stream data in the form of BPC10 based on a color space conversion operation, specifically including:

[0051] S101: a display unit in the main chip acquires video stream data in the form of BPC8 stored in the internal memory of the main chip;

[0052] S102: based on a color space conversion operation, the display unit converts the video stream data in the form of BPC8 into video stream data in the form of BPC10.

[0053] Further, in an embodiment, the video stream data in the form of BPC10 is converted into protocol data of a long-distance transmission protocol through a format conversion operation, and transmitted to an autonomous driving system for processing, specifically including:

[0054] S201: the display interface controller in the main chip outputs the converted video stream data in the form of BPC10 to a display interface;

[0055] S202: the serializer of the video feedback system converts the video stream data in the form of BPC10 into protocol data of a long-distance transmission protocol, and transmits to an autonomous driving system for processing after being processed by the deserializer. The serializer outputs data through a GMSL cable.

[0056] The method for video feedback using a multi-display interface of the present application will be specifically described below in combination with Figure 4 and Figure 5 .

[0057] Referring to Figure 4 , a color space conversion technology combining software and hardware is used to map a target system, i.e., a color space required by an autonomous driving system, for example, input RGB888 (BPC8) and output RAW10 (BPP10). Figure 4 In the above, Memory represents an internal memory.

[0058] First, the original RGB888 format (24bit) is converted into the RAW format, such as BGGR. One way is to simply replace, for example, the first row of pixel points P1 (the first pixel point) is replaced from RGB to BBB, P2 (the second pixel point) is replaced from RGB to GGG, P3 (the third pixel point) is replaced from RGB to BBB, and so on. The first row is replaced with a BGBG format row. The pixel points of the second row are replaced as follows: P1 (the first of the second row) is replaced from RGB to GGG, P2 (the second of the second row) is replaced from RGB to RRR, P3 (the third of the second row) is replaced from RGB to GGG, and so on. The first row is replaced with a GRGR format row, and the following is obtained Figure 5 Bayer RAW format as shown.

[0059] Then the display unit obtains the Figure 5 content as shown, and performs a color space conversion operation to convert the video stream data in the BPC8 form into video stream data in the BPC10 form. Then the display interface controller outputs to the display interface in the BPP10 (10 bits per pixel) form. The serializer converts the data format into a long-distance transmission protocol format such as GMSL. Then the BPP10 format data is sent to the autonomous driving system through the GMSL (Gigabit Multimedia Serial Link) cable and the remote deserializer for processing.

[0060] The method for video back-annotation using the multi-channel display interface provided in the embodiments of the present application can convert the video stream data in the BPC8 form into video stream data in the BPC10 form based on a color space conversion operation, and then convert the video stream data in the BPC10 form into protocol data of a long-distance transmission protocol through a format conversion operation and transmit the protocol data to the autonomous driving system for processing. This can solve the strong dependence on the video injection system chip interface, and utilize the traditional display interface supported devices, thereby expanding the range of available chips of the video injection system, changing the strong dependence on the CSITX to no dependence or less dependence, and reusing some existing multi-channel display systems, thereby constructing a video back-annotation system and avoiding dependence on the existing design and commercial solutions.

[0061] In a third aspect, the embodiments of the present application provide a device for video back-annotation using a multi-channel display interface. The device for video back-annotation using a multi-channel display interface can be a personal computer (PC), a notebook computer, a server, or other devices with data processing functions.

[0062] Referring to Figure 6 , Figure 6This is a schematic diagram of the hardware structure of a device for video back-interpretation using a multi-channel display interface, as described in an embodiment of this application. In this embodiment, the device for video back-interpretation using a multi-channel display interface may include a processor, a memory, a communication interface, and a communication bus.

[0063] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0064] Communication interfaces include input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting internal devices within a device that uses a multi-display interface for video replay, as well as interfaces used for interconnecting the device with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0065] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0066] The processor can be a general-purpose processor, which can call a program stored in memory for video back-anchoring using a multiplexed display interface and execute the method for video back-anchoring using a multiplexed display interface provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the program for video back-anchoring using a multiplexed display interface is called can be referred to the various embodiments of the method for video back-anchoring using a multiplexed display interface in this application, and will not be repeated here.

[0067] Those skilled in the art will understand that Figure 6 The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0068] The terms “include,” “comprise,” “have,” and any variations thereof, in the Specification and in the Claims of the present application, and the above-mentioned drawings, are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or device that includes a list of steps or units is not limited to the listed steps or units, but can optionally further include steps or units not listed, or can optionally further include other steps or units inherent to such processes, methods, products, or devices. The terms “first,” “second,” and “third” and the like descriptions are used to distinguish different objects, and do not represent a sequence or limit the types of “first,” “second,” and “third.”

[0069] In the description of the embodiments of the present application, “exemplary”, “for example”, or “for instance” is used to represent an example, an illustration, or a description. Any embodiment or design scheme described as “exemplary”, “for example”, or “for instance” in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the words “exemplary”, “for example”, or “for instance” are intended to present the relevant concept in a specific manner.

[0070] In the description of the embodiments of the present application, unless otherwise specified, “ / ” represents the meaning of or, for example, A / B can represent A or B; “and / or” in the text only represents a description of the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. In addition, in the description of the embodiments of the present application, “multiple” means two or more than two.

[0071] In some of the processes described in the embodiments of the present application, a plurality of operations or steps are included in a specific order, but it should be understood that these operations or steps can be executed or performed in parallel or in an order different from that in which they appear in the embodiments of the present application. The serial number of the operation is only used to distinguish different operations, and the serial number itself does not represent any execution order. In addition, these processes can include more or fewer operations, and these operations or steps can be executed in sequence or in parallel, and these operations or steps can be combined.

[0072] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and a general hardware platform as required, of course, it can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, an optical disk) as described above, and includes a plurality of instructions for causing a terminal device to execute the methods described in the embodiments of the present application.

[0073] The preferred embodiments of the present application have been described above with the illustrated embodiments, and are not intended to limit the scope of patent protection for the present application. Any equivalent structure or equivalent process variations, which directly or indirectly apply to the content of the specification and drawings of the present application, are also included in the patent protection scope of the present application.

Claims

1. A system for video re-injection using a multi-channel display interface, characterized in that, The system for video back annotation using multiple display interfaces comprises a video back annotation system and an automatic driving system, the video back annotation system comprises a main chip, and a plurality of serializers corresponding to the main chip to convert video stream data after color space conversion into protocol data of a long-distance transmission protocol and transmit to the automatic driving system; The color space conversion operation converts the video stream data in BPC8 form into video stream data in BPC10 form, and then the format conversion operation converts the video stream data in BPC10 form into protocol data of the long-distance transmission protocol; The main chip comprises a processor, a memory, a plurality of display units and a plurality of display interface controllers, and a single display unit corresponds to at least one display interface controller; The display unit supports DMA and color space conversion, and the display unit is used to acquire the video stream data stored in the memory and perform color space conversion operation to convert the color space of the video stream data into the color space required by the automatic driving system; The serializer supports multiple data inputs, and a single serializer corresponds to a plurality of display interface controllers.

2. The system for video back annotation using multiple display interfaces according to claim 1, wherein: A plurality of deserializers are further arranged between the video back annotation system and the automatic driving system, the number of the deserializers is the same as that of the serializers and each deserializer corresponds to one serializer; The deserializer is used to convert the data of the long-distance transmission protocol into target format data, and the deserializer and the serializer comply with the same protocol.

3. A system for video return using a multi-path display interface as recited in claim 2, wherein: A plurality of video input interfaces are arranged in the automatic driving system, and a single deserializer corresponds to at least one video input interface.

4. A method for video feedback using a multi-path display interface, implemented based on the system of any one of claims 1 to 3, characterized in that, The method for video back annotation using multiple display interfaces comprises: Converting the video stream data in BPC8 form into video stream data in BPC10 form based on color space conversion operation; Converting the video stream data in BPC10 form into protocol data of a long-distance transmission protocol through format conversion operation and transmitting to the automatic driving system for processing.

5. The method for video playback using a multi-display interface according to claim 4, wherein, The color space conversion operation converts the video stream data in BPC8 form into video stream data in BPC10 form, specifically comprising: The display unit in the main chip acquires the video stream data in BPC8 form stored in the memory in the main chip; The display unit converts the video stream data in BPC8 form into video stream data in BPC10 form based on color space conversion operation.

6. The method for video playback with multi-display interface according to claim 5, wherein, The format conversion operation converts the video stream data in BPC10 form into protocol data of a long-distance transmission protocol and transmits to the automatic driving system for processing, specifically comprising: The display interface controller in the main chip outputs the video stream data in BPC10 form obtained after conversion to the display interface; The serializer of the video back annotation system converts the video stream data in BPC10 form into protocol data of a long-distance transmission protocol, and transmits to the automatic driving system for processing after being processed by the deserializer.

7. The method for video playback using a multi-display interface according to claim 6, wherein: The serializer outputs data through a GMSL cable.

8. A device for video back-monitoring using a multi-channel display interface, characterized in that, The device for video playback using a multi-path display interface comprises a processor, a memory, and a program for video playback using a multi-path display interface stored on the memory and executable by the processor, wherein the program for video playback using a multi-path display interface, when executed by the processor, implements the steps of the method for video playback using a multi-path display interface according to any one of claims 4 to 7.

Images