System, method and equipment for carrying out video reinjection by utilizing multi-path display interface

Abstract

The invention discloses a system, a method and equipment for video reinjection by utilizing a multi-path display interface, and relates to the technical field of image processing, the system comprises a video reinjection system and an automatic driving system, the video reinjection system comprises a main chip, and the main chip is connected with the automatic driving system. And a plurality of serializers corresponding to the main chip and used for converting the video stream data after color space conversion into protocol data of a long-distance transmission protocol and transmitting the protocol data to the automatic driving system. The main chip comprises a processor, a memory, a plurality of display units and a plurality of display interface controllers, and each display unit at least corresponds to one display interface controller; the display unit supports DMA and color space conversion. The problem of strong dependence on a video injection system chip interface can be solved, a traditional device supporting a display interface is utilized, and the range of available chips of the video injection system is expanded.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a system, method and device for video re-injection using multiple display interfaces. Background Art

[0002] Current camera video injection cards typically provide video output via the CSI TX (Camera Serial Interface Transmitter), which is then output via a serializer-deserializer (SERD) to a receiving device, typically an autonomous driving system, replacing the actual vehicle camera input. This approach allows the video injection card to accept pre-recorded data instead of actual vehicle data, reducing autonomous driving system testing costs.

[0003] See also Figure 1 Figure 2 shows the system architecture of a video injection card. The MAX9295D serializer chip only supports CSI TX signals from cameras or video injection systems. This shows that existing approaches rely on multi-channel CSI TX capabilities, significantly limiting the range of chip options for video injection / re-injection systems. Summary of the Invention

[0004] The present application provides a system, method and device for video injection using multiple display interfaces, which can solve the problem of strong dependence on the chip interface of the video injection system, utilize traditional devices that support display interfaces, and expand the range of available chips for the video injection system.

[0005] In a first aspect, an embodiment of the present application provides a system for video re-injection using a multi-channel display interface, wherein the system for video re-injection using a multi-channel display interface includes a video re-injection system and an autonomous driving system, wherein the video re-injection system includes a main chip, and multiple serializers corresponding to the main chip for converting video stream data after color space conversion into protocol data of a long-distance transmission protocol and transmitting the data to the autonomous driving system.

[0006] In conjunction with the first aspect, in one embodiment, The main chip includes a processor, a memory, multiple display units and multiple display interface controllers, and each display unit corresponds to at least one display interface controller; The display unit supports DMA and color space conversion. The display unit is used to obtain the video stream data stored in the memory and perform a color space conversion operation to convert the color space of the video stream data into the color space required by the autonomous driving system.

[0007] In combination with the first aspect, in one implementation, the serializer supports multiple data inputs, and a single serializer corresponds to multiple display interface controllers.

[0008] In conjunction with the first aspect, in one embodiment, A plurality of deserializers are further provided between the video re-injection system and the autonomous driving system, wherein the number of the deserializers is the same as the number of the serializers and corresponds one-to-one with the serializers; The deserializer is used to convert data of a long-distance transmission protocol into target format data, and the deserializer and the serializer follow the same protocol.

[0009] In combination with the first aspect, in one embodiment, the autonomous driving system is provided with multiple video input interfaces, and a single deserializer corresponds to at least one video input interface.

[0010] In a second aspect, an embodiment of the present application provides a method for video re-injection using multiple display interfaces. Based on the above-described system implementation, the method for video re-injection using multiple display interfaces includes: Based on the color space conversion operation, the video stream data in the BPC8 format is converted into the video stream data in the BPC10 format; Through the format conversion operation, the video stream data in the BPC10 format is converted into protocol data of the long-distance transmission protocol and transmitted to the autonomous driving system for processing.

[0011] In conjunction with the second aspect, in one embodiment, the color space conversion operation based on converting the video stream data in the BPC8 format into the video stream data in the BPC10 format specifically includes: The display unit in the main chip obtains the video stream data in the BPC8 format stored in the memory in the main chip; Based on the color space conversion operation, the display unit converts the video stream data in the BPC8 format into the video stream data in the BPC10 format.

[0012] In conjunction with the second aspect, in one embodiment, the format conversion operation converts the video stream data in the BPC10 format into protocol data of a long-distance transmission protocol and transmits it to the autonomous driving system for processing, specifically including: The display interface controller in the main chip outputs the converted video stream data in BPC10 format to the display interface; The serializer of the video injection system converts the video stream data in the BPC10 format into protocol data of the long-distance transmission protocol, and transmits it to the autonomous driving system for processing after being processed by the deserializer.

[0013] In combination with the second aspect, in one implementation, the serializer outputs data via a GMSL cable.

[0014] In a third aspect, an embodiment of the present application provides a device for video re-injection using a multi-channel display interface, wherein the device for video re-injection using a multi-channel display interface comprises a processor, a memory, and a program for video re-injection using a multi-channel display interface stored on the memory and executable by the processor, wherein when the program for video re-injection using a multi-channel display interface is executed by the processor, the steps of the above-mentioned method for video re-injection using a multi-channel display interface are implemented.

[0015] The beneficial effects of the technical solutions provided in the embodiments of the present application include: It can solve the problem of strong dependence on the chip interface of the video injection system and utilize traditional devices that support display interfaces, expanding the range of available chips for the video injection system, changing the strong dependence on CSI TX to no dependence or little dependence, and can reuse some existing multi-channel display systems to build a video injection system, avoiding dependence on existing designs and commercial solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 System architecture diagram for the video injection card; Figure 2 This is a schematic diagram of the structure of the system for video re-injection using multiple display interfaces in this application; Figure 3 This is a flowchart of a method for performing video re-injection using multiple display interfaces in this application; Figure 4 This is an example diagram corresponding to the method of using multiple display interfaces for video re-injection in this application; Figure 5 Schematic diagram of data conversion; Figure 6 This is a schematic diagram of the hardware structure of the device that uses multiple display interfaces for video re-injection in this application. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0018] First of all, it should be noted that Figure 1In the figure, FMC indicates FPGA middle layer card, POC indicates a technology that transmits data and power simultaneously through a single cable, I2C indicates synchronous serial bus, VPOC indicates a circuit, ADDR indicates structure, MIPI indicates serial communication interface, Lane indicates lane, MAX indicates serializer chip model, COAX indicates coaxial interface, GPIO indicates general input / output port, ADR indicates address register, and J1, J2, and J3 indicate connectors.

[0019] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0020] On the first aspect, an embodiment of the present application provides a system for video re-injection using multiple display interfaces to reduce dependence on output interfaces, use traditional display interfaces to provide multi-channel video stream output capabilities, construct a video re-injection solution by adding a display interface to a bridge chip, and convert the color space required by the analog camera input through color space conversion.

[0021] In one embodiment, referring to Figure 2 , Figure 2 This is a schematic diagram of the structure of the system using multiple display interfaces for video re-injection in this application. Figure 2 As shown, the system for video injection using multiple display interfaces includes: a main chip, and multiple serializers corresponding to the main chip to convert the video stream data after color space conversion into protocol data of a long-distance transmission protocol and transmit it to the autonomous driving system.

[0022] Furthermore, the main chip includes a processor, memory, multiple display units, and multiple display interface controllers, and each display unit corresponds to at least one display interface controller. In actual applications, the display unit can support multiple display interface controllers, for example, one display unit supports the output of two display interfaces.

[0023] The display unit supports DMA (Direct Memory Access) and color space conversion. The display unit is used to obtain video stream data stored in the memory and perform a color space conversion operation to convert the color space of the video stream data into a color space required by the autonomous driving system.

[0024] Furthermore, the serializer supports multiple data inputs, and a single serializer corresponds to multiple display interface controllers. In actual applications, the serializer usually supports two or more inputs. For example, two display interface controllers can be solved with one serializer.

[0025] Furthermore, multiple deserializers are deployed between the video re-injection system and the autonomous driving system. The number of deserializers matches the number of serializers and corresponds one-to-one with each serializer. These deserializers are used to convert data from the long-distance transmission protocol into the target format (i.e., the format required for actual application), and they follow the same protocol as the serializers. The autonomous driving system has multiple video input interfaces, with each deserializer corresponding to at least one video input interface. In actual applications, the deserializers can support one to four output channels.

[0026] Specifically, in practical applications, the display interface controller can be DSI, DP, or HDMI, providing a matching serializer. The processor controls the entire process. Memory is used to store video stream data. The display unit has DMA capabilities, directly transferring video stream data from memory to the display interface. The display interface controller supports common video interfaces such as DSI, DP, and HDMI.

[0027] The serializer converts the data output by the display interface controller into data using a long-distance transmission protocol, such as GMSL. Common chips include the MAX96789. The deserializer converts data from the long-distance transmission protocol into the required format. Common chips include the MXA96724. As long as the deserializer and serializer conform to the same protocol, they can interconnect. The video input interface in autonomous driving systems can be CSI RX.

[0028] That is, for the system of the present application that uses multiple display interfaces for video re-injection, in a possible implementation scenario, it may include multiple connected data transmission links, each data transmission link may include a display unit, a display interface controller, a serializer, a deserializer, and a video input interface connected in sequence, and the display unit in each data transmission link is connected to the memory.

[0029] The system for video injection using multiple display interfaces in the embodiment of the present application can solve the problem of strong dependence on the chip interface of the video injection system, utilize traditional devices that support display interfaces, expand the range of available chips for the video injection system, and change from strong dependence on CSI TX to no dependence or low dependence. In addition, some existing multi-channel display systems can be reused to build a video injection system, avoiding dependence on existing designs and commercial solutions.

[0030] In a second aspect, an embodiment of the present application also provides a method for video re-injection using multiple display interfaces, which is implemented based on the above-described system. Specifically, the system for video re-injection using multiple display interfaces includes a video re-injection system and an autonomous driving system. The video re-injection system includes a main chip and multiple serializers corresponding to the main chip to convert the video stream data after color space conversion into protocol data of a long-distance transmission protocol and transmit it to the autonomous driving system. The main chip includes a processor, memory, multiple display units, and multiple display interface controllers, and each display unit corresponds to at least one display interface controller; the display unit supports DMA and color space conversion, and the display unit is used to obtain the video stream data stored in the memory and perform color space conversion operations to convert the color space of the video stream data into the color space required by the autonomous driving system. The serializer supports multiple data inputs, and a single serializer corresponds to multiple display interface controllers. Multiple deserializers are also provided between the video re-injection system and the autonomous driving system. The number of the deserializers is the same as the number of serializers and corresponds one-to-one with the serializers; the deserializers are used to convert the data of the long-distance transmission protocol into target format data, and the deserializers and the serializers follow the same protocol. An autonomous driving system has multiple video input interfaces, and a single deserializer corresponds to at least one video input interface. In other words, a system utilizing multiple display interfaces for video re-injection, in one possible implementation scenario, may include multiple connected data transmission links. Each data transmission link may include a display unit, a display interface controller, a serializer, a deserializer, and a video input interface, connected in sequence. Each display unit in each data transmission link is connected to memory.

[0031] In one embodiment, referring to Figure 3 , Figure 3 This is a flow chart of the method for video re-injection using multiple display interfaces in this application. Figure 3 As shown, the method for performing video re-injection using multiple display interfaces includes: S1: Based on a color space conversion operation, convert the video stream data in the BPC8 format into the video stream data in the BPC10 format; where BPC represents color depth; S2: Through the format conversion operation, the video stream data in the BPC10 format is converted into the protocol data of the long-distance transmission protocol and transmitted to the autonomous driving system for processing.

[0032] Furthermore, in one embodiment, based on the color space conversion operation, the video stream data in the BPC8 format is converted into the video stream data in the BPC10 format, specifically including: S101: The display unit in the main chip obtains the video stream data in the BPC8 format stored in the memory in the main chip; S102: Based on the color space conversion operation, the display unit converts the video stream data in the BPC8 format into the video stream data in the BPC10 format.

[0033] Furthermore, in one embodiment, the video stream data in the BPC10 format is converted into protocol data of a long-distance transmission protocol through a format conversion operation, and transmitted to the autonomous driving system for processing, specifically including: S201: The display interface controller in the main chip outputs the converted video stream data in BPC10 format to the display interface; S202: The serializer of the video re-injection system converts the video stream data in BPC10 format into protocol data for a long-distance transmission protocol. The serializer processes the data through a deserializer and transmits it to the autonomous driving system for processing. The serializer outputs the data via a GMSL cable.

[0034] The following combination Figure 4 and Figure 5 , the method of video re-injection using multiple display interfaces in this application is specifically described.

[0035] See also Figure 4 , using the color space conversion technology combining software and hardware, the color space required by the target system, that is, the autonomous driving system, is mapped. For example: input RGB888 (BPC8) and output RAW10 (BPP10). Figure 4 In the example, Memory refers to memory.

[0036] First, convert the original RGB888 format (24bit) to a RAW format, such as BGGR. One way is to simply replace it. For example, the first row of pixels P1 (the first pixel) is replaced from RGB to BBB, P2 (the second pixel) is replaced from RGB to GGG, P3 (the third pixel) is replaced from RGB to BBB, and so on. The first row is replaced with a BGBG format row, and the second row of pixels is replaced as follows: P1 (the first pixel in the second row) is replaced from RGB to GGG, P2 (the second pixel in the second row) is replaced from RGB to RRR, P3 (the third pixel in the second row) is replaced from RGB to GGG, and so on. The first row is replaced with a GRGR format row, and we get Figure 5 BayerRAW format shown.

[0037] The display unit will then get Figure 5The content shown undergoes a color space conversion operation, converting video stream data in BPC8 format to BPC10 format. The display interface controller then outputs the data to the display interface in accordance with BPP10 (10 bits per pixel). The serializer converts the data format into a long-distance transmission protocol format such as GMSL. The BPP10 format data is then sent to the autonomous driving system for processing via a GMSL (Gigabit Multimedia Serial Link) cable and a remote deserializer.

[0038] The method for video injection using a multi-channel display interface in an embodiment of the present application converts video stream data in the BPC8 format into video stream data in the BPC10 format based on a color space conversion operation, and then converts the video stream data in the BPC10 format into protocol data of a long-distance transmission protocol through a format conversion operation, and transmits it to the autonomous driving system for processing. This can solve the problem of strong dependence on the chip interface of the video injection system, make use of traditional devices that support display interfaces, expand the range of available chips for the video injection system, and change from strong dependence on CSI TX to no dependence or less dependence, and can reuse some existing multi-channel display systems to build a video injection system, avoiding dependence on existing designs and commercial solutions.

[0039] In a third aspect, an embodiment of the present application provides a device for video re-injection using a multi-channel display interface. The device for video re-injection using a multi-channel display interface can be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.

[0040] Reference Figure 6 , Figure 6 Schematic diagram of the hardware structure of the device for performing video re-injection using multiple display interfaces involved in the embodiment of the present application. In the embodiment of the present application, the device for performing video re-injection using multiple display interfaces may include a processor, a memory, a communication interface, and a communication bus.

[0041] The communication bus may be of any type and is used to interconnect the processor, memory, and communication interface.

[0042] Communication interfaces include input / output (I / O) interfaces, physical interfaces, and logical interfaces. These interfaces interconnect components within a device using multiple display interfaces for video injection, as well as interfaces that connect the device to other devices (such as other computing devices or user devices). Physical interfaces can include Ethernet, fiber, or ATM interfaces; user devices can include displays and keyboards.

[0043] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0044] The processor may be a general-purpose processor that can invoke a program stored in memory for performing video re-injection using multiple display interfaces and execute the method for performing video re-injection using multiple display interfaces provided in the embodiments of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the program for performing video re-injection using multiple display interfaces is invoked can be found in the various embodiments of the method for performing video re-injection using multiple display interfaces of the present application and will not be further described here.

[0045] Those skilled in the art will understand that Figure 6 The hardware structure shown in the figure does not constitute a limitation to the present application and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0046] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit the "first", "second" and "third" to different types.

[0047] In the description of the embodiments of this application, the words "exemplary," "for example," or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary," "for example," or "for example" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary," "for example," or "for example" is intended to present the relevant concepts in a concrete manner.

[0048] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0049] In some processes described in the embodiments of the present application, multiple operations or steps are included that appear in a specific order. However, it should be understood that these operations or steps may not be performed in the order in which they appear in the embodiments of the present application or may be performed in parallel. The sequence numbers of the operations are only used to distinguish between different operations, and the sequence numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations or steps may be performed in sequence or in parallel, and these operations or steps may be combined.

[0050] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above and includes a number of instructions for enabling a terminal device to execute the methods described in each embodiment of this application.

[0051] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A system for video re-injection using multiple display interfaces, characterized in that: The system for video injection using a multi-channel display interface includes a video injection system and an autonomous driving system. The video injection system includes a main chip and multiple serializers corresponding to the main chip to convert the video stream data after color space conversion into protocol data of a long-distance transmission protocol and transmit it to the autonomous driving system.

2. The system for video re-injection using multiple display interfaces according to claim 1, wherein: The main chip includes a processor, a memory, multiple display units and multiple display interface controllers, and each display unit corresponds to at least one display interface controller; The display unit supports DMA and color space conversion. The display unit is used to obtain the video stream data stored in the memory and perform a color space conversion operation to convert the color space of the video stream data into the color space required by the autonomous driving system.

3. The system for video re-injection using multiple display interfaces according to claim 2, wherein: The serializer supports multi-channel data input, and a single serializer corresponds to multiple display interface controllers.

4. The system for video re-injection using multiple display interfaces according to claim 3, wherein: A plurality of deserializers are further provided between the video re-injection system and the autonomous driving system, wherein the number of the deserializers is the same as the number of the serializers and corresponds one-to-one with the serializers; The deserializer is used to convert data of a long-distance transmission protocol into target format data, and the deserializer and the serializer follow the same protocol.

5. The system for video re-injection using multiple display interfaces according to claim 4, wherein: The autonomous driving system is provided with multiple video input interfaces, and a single deserializer corresponds to at least one video input interface.

6. A method for video re-injection using a multi-channel display interface, implemented based on the system according to any one of claims 1 to 5, characterized in that: The method for performing video re-injection using multiple display interfaces includes: Based on the color space conversion operation, the video stream data in the BPC8 format is converted into the video stream data in the BPC10 format; Through the format conversion operation, the video stream data in the BPC10 format is converted into protocol data of the long-distance transmission protocol and transmitted to the autonomous driving system for processing.

7. The method for video re-injection using a multi-channel display interface according to claim 6, characterized in that: The color space conversion operation is based on converting the video stream data in the BPC8 format into the video stream data in the BPC10 format, specifically including: The display unit in the main chip obtains the video stream data in the BPC8 format stored in the memory in the main chip; Based on the color space conversion operation, the display unit converts the video stream data in the BPC8 format into the video stream data in the BPC10 format.

8. The method for video re-injection using a multi-channel display interface according to claim 7, characterized in that: The format conversion operation converts the video stream data in the BPC10 format into protocol data of a long-distance transmission protocol and transmits it to the autonomous driving system for processing, specifically including: The display interface controller in the main chip outputs the converted video stream data in BPC10 format to the display interface; The serializer of the video injection system converts the video stream data in the BPC10 format into protocol data of the long-distance transmission protocol, and transmits it to the autonomous driving system for processing after being processed by the deserializer.

9. The method for video re-injection using a multi-channel display interface according to claim 8, characterized in that: The serializer outputs data via a GMSL cable.

10. A device for video re-injection using a multi-channel display interface, characterized in that: The device for video re-annotation using multiple display interfaces includes a processor, a memory, and a program for video re-annotation using multiple display interfaces stored in the memory and executable by the processor, wherein when the program for video re-annotation using multiple display interfaces is executed by the processor, the steps of the method for video re-annotation using multiple display interfaces as described in any one of claims 6 to 9 are implemented.

Images