Systems and methods for asymmetric image segmenters with row mark memory
By combining row-marked memory and asymmetric image segmentation, the problems of design complexity and insufficient memory requirements of existing asymmetric image segmentation technologies are solved, thereby reducing memory requirements and system costs.
Patent Information
- Application Number
- CN202310310719.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-04-11
- Filing Date
- 2019-05-07
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2039-05-07
AI Technical Summary
Existing asymmetric image segmenters lack flexibility in design and memory requirements, necessitating vendor negotiation, which leads to increased complexity and cost.
By employing a combination of row-marked memory and asymmetric image segmenter, each row of the video image is marked by the row-marked memory to determine which rows are filler and which are valid data, thereby reducing memory requirements and allowing for independent design of the display's video clock frequency.
This allows for independent design of the display's video clock frequency while reducing memory requirements, thereby lowering system costs and complexity and increasing design flexibility.
Smart Images

Figure CN116347004B_ABST
Abstract
Description
[0001] This application is a divisional application of the application filed on May 7, 2019, with application number 201910374460.3 and entitled "System and Method for an Asymmetric Image Segmenter with Row Marking Memory". Cross-references to related applications
[0002] This patent application relates to jointly pending and jointly owned U.S. Provisional Patent Application No. 62 / 668,087, and claims priority interest under 35 USC § 119(e), entitled “SYSTEMS ANDMETHODS FOR ASYMMETRIC DUAL-VIEW SPLITTER WITH LOOK-UP TABLE,” named by inventors Yalcin Balcioglu, Arda Ka.mil Bafra, and Levent Yakay, and filed May 7, 2018, the patent document of which is incorporated herein by reference in its entirety and for all purposes.
[0003] A. Technical Field This disclosure generally relates to systems and methods for serializer / deserializer (SerDes) communication. More specifically, this disclosure relates to systems and methods for asymmetric image segmentation applications.
[0004] B. Background Technology Gigabit Multimedia Serial Link (GMSL) serializers and deserializers (SerDes) support the high bandwidth, complex interconnects, and data integrity requirements of a wide range of applications. For example, and not limitingly, applications may include the low power requirements of cameras and the widely varying bandwidths of sensor data aggregation. Using shielded twisted-pair (STP) or coaxial cable up to 15 meters long, GMSL serializers and deserializers can meet the stringent electromagnetic compatibility (EMC) requirements of the automotive and other industries. In some embodiments, applications may include future automotive infotainment and advanced driver assistance systems (ADAS).
[0005] Spread spectrum capability can be built into each serializer and deserializer IC to improve the electromagnetic interference (EMI) performance of the link without the need for an external spread spectrum clock. Interoperability between the serializers and deserializers within the series allows for the use of different interfaces at each end of the link. In addition to driving high-resolution center / rear-seat displays and dashboards, the GMSL SerDes system can also be used in megapixel camera systems.
[0006] In some embodiments, asymmetric image segmenters (e.g., asymmetric side-by-side dual-view segmenter engines) operate using algorithms to segment and extract individual video images from multiple images (e.g., dual-view images). These embodiments may result in requirements that the transmitter and receiver must be negotiated and pre-designed by multiple vendors, thus increasing complexity and memory requirements.
[0007] Therefore, what is needed is a system and method for implementing asymmetric image segmenters, which can be flexible and allow vendors to independently design their products while reducing memory requirements. Attached Figure Description
[0008] Reference will be made to embodiments of the invention, examples of which may be illustrated in the accompanying drawings. These drawings are intended to be illustrative and not restrictive. Although the invention has been described in general within the context of these embodiments, it should be understood that it is not intended to limit the scope of the invention to these specific embodiments.
[0009] Figure 1A The functionality of a serializer / deserializer (SerDes) according to various embodiments of this disclosure is demonstrated.
[0010] Figure 1B Two configurations for video multi-stream transmission according to various embodiments of this disclosure are described.
[0011] Figure 2A Superframes of video images for multi-stream video transmission according to various embodiments of this disclosure are depicted.
[0012] Figure 2B Vertical distribution image formatting of vertical images according to various embodiments of this disclosure is described.
[0013] Figure 3 Timing of an asymmetric image segmenter for multi-stream video image applications according to various embodiments of this disclosure is described.
[0014] Figure 4A and Figure 4B Reconstructed video images following asymmetric image segmentation processing according to various embodiments of this disclosure are depicted.
[0015] Figure 5 A configuration for video multistream transmission with internal or external memory integrated in a serializer is described according to an embodiment of the present invention.
[0016] Figure 6 A simplified block diagram of a computing device / information processing system according to embodiments of the present disclosure is depicted. Detailed Implementation
[0017] In the following description, specific details are set forth for purposes of explanation in order to provide an understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without these details. Furthermore, those skilled in the art will recognize that the embodiments of the invention described below can be implemented in various ways, such as processes, apparatuses, systems, devices, or methods, on tangible computer-readable media.
[0018] The components or modules shown in the figures illustrate exemplary embodiments of the invention and are intended to avoid obscuring the invention. It should also be understood that throughout this discussion, a component can be described as a separate functional unit that may include subunits; however, those skilled in the art will recognize that various components or portions thereof may be divided into separate components or may be integrated together, including within a single system or component. It should be noted that the functions or operations discussed herein can be implemented as components. Components can be implemented as software, hardware, or a combination thereof.
[0019] Furthermore, the connections between components or systems shown in the accompanying drawings are not intended to be limited to direct connections. However, data between these components may be modified, reformatted, or otherwise altered by intermediate components. Moreover, additional or fewer connections may be used. It should also be noted that the terms "coupled," "connected," or "communicatively coupled" should be understood to include direct connections, indirect connections via one or more intermediate devices, and wireless connections.
[0020] In this specification, references to "one embodiment," "preferred embodiment," "an embodiment," or "embodiments" mean that a specific feature, structure, characteristic, or function described in connection with the embodiment is included in at least one embodiment of the invention and may be included in more than one embodiment. Furthermore, the above phrases appearing in different places in this specification do not necessarily refer to the same embodiment or multiple embodiments.
[0021] Certain terms used in different places in this specification are for illustrative purposes and should not be construed as limiting. A service, function, or resource is not limited to a single service, function, or resource; the use of these terms may refer to a group of related services, functions, or resources, whether discrete or aggregated.
[0022] In addition, it should be noted that: (1) certain steps may be performed optionally; (2) the steps may not be limited to the specific order described herein; (3) certain steps may be performed in different orders; and (4) certain steps may be performed simultaneously.
[0023] Figure 1A The functionality of a serializer / deserializer (SerDes) 100 according to various embodiments of this disclosure is illustrated. The serializer / deserializer (SerDes) 100 can utilize multi-gigabit point-to-point connection technology. Effectively, the SerDes 100 is a pair of functional blocks that can be used for high-speed communication to compensate for limited inputs and outputs. These blocks can convert data between serial and parallel interfaces in each direction. In some embodiments, the SerDes 100 can provide data transmission via a single / differential line to minimize the number of I / O pins and interconnects. A parallel clock SerDes can be used to serialize the parallel bus inputs along with data addresses and control signals. The serialized stream can be transmitted along with a reference clock. In some embodiments, the clock can be provided via a phase-locked loop (PLL) oscillator.
[0024] In some embodiments, the multi-gigabit point-to-point connection technology is the Gigabit Multimedia Serial Link (GMSL) SerDes technology. GMSL technology can simultaneously transmit HD video, audio, control information, aggregated sensor data, and gigabit Ethernet over a single 15-meter coaxial cable or 10- to 15-meter shielded twisted-pair cable, all while meeting automotive electromagnetic compatibility (EMC) specifications. The use of coaxial and shielded twisted-pair cables reduces the complexity of the interconnect backbone, thereby supporting multi-stream video transmission at a lower system cost. Compared to Ethernet, GMSL technology can transmit images of millions of pixels without compression. Thus, GMSL can support machine vision technologies, which are crucial for safety features such as object and pedestrian detection. Other embodiments with data compression result in artifacts and lost information, which may limit the ability to support machine vision.
[0025] Each GMSL SerDes IC can have built-in spread spectrum capabilities, reducing electromagnetic interference (EMI) on the link. The IC's power coaxial architecture eliminates the need for additional power or ground lines, addressing vehicle weight challenges. A single microcontroller can program the serializer, deserializer, and all connected peripherals via an integrated bidirectional control channel. In one or more embodiments, this architecture eliminates the need for a remote-side microcontroller and its supporting components, simplifying design and reducing cost, size, and power consumption. Reduced system cost and shorter design time are further enhanced by the ability of GMSL to daisy-chain multiple camera modules. The technology's built-in diagnostic capabilities enhance data integrity.
[0026] Figure 1B Two configurations for video multi-stream transmission according to various embodiments of this disclosure are depicted. For example... Figure 1B As shown, GMSL technology allows for video aggregation and video segmentation. Video system 110 includes SoC 111, serializers (SER112, SER113), deserializers (DES116, DES117), CID 118, and IC 119. (CID = Central Instrument Display; IC = Dashboard) As illustrated, SER112 and SER113 receive video signals 131 and 132 separately from SoC 111. SER112 and SER113 process their corresponding video signals and generate video signals 114 and 115, respectively. Video signals 114 and 115 are coupled to DES116 and DES117, respectively, and generate CID 118 and IC 119, respectively. For video system 110, the video signals from SoC 111 are processed in parallel, which requires separate serializers and additional I / O interfaces.
[0027] Asymmetric image segmenter In some other embodiments, video system 120 utilizes GMSL technology, which allows for video aggregation and video segmentation. As shown, video system 120 includes SoC 121, serializer, SER 122, deserializer, DES 125 and DES 126, CID 127, and IC 128. Figure 1B SER122 receives multiple video signals 131 / 132 from SoC121. The multiple video signals 131 / 132 include the contents of video signals 131 and 132. In other words, the multiple video signals 131 / 132 comprise two video signals and can be referred to as a superframe image stream. SER122 includes an asymmetric image segmenter and generates GMSL123 and GMSL124. GMSL123 and GMSL124 are coupled to DES125 and DES126 and generate CID127 and IC128, respectively. Examples of asymmetric image segmenters at the input of SER122 may include, but are not limited to, asymmetric side-by-side dual-view segmenters. In some embodiments, multiple images in the input superframe can be formed side-by-side. As shown, utilizing GMSL technology can reduce infrastructure costs.
[0028] Many asymmetric image segmenters use PLLs to generate video clocks for individual displays. Using PLLs to generate video clocks for individual displays can significantly increase power and area consumption.
[0029] The number of displays in vehicles is constantly increasing. Traditionally, system designers need to associate a separate SOC video output port with each display. As the number of displays increases, this may mean requiring more SOCs or having to choose SOCs with more video output ports, thus increasing costs. If the SOC video ports can be used to drive multiple displays using a single virtual channel, a cheaper SOC with fewer output ports can be used in the system, reducing costs.
[0030] Many asymmetric image segmenters operate using algorithms to extract individual video images from multiple image inputs. This process may create a requirement where the transmitter and receiver must be negotiated and designed in advance by multiple vendors.
[0031] Figure 2A , Figure 2B , Figure 3 and Figure 4A as well as Figure 4B This demonstrates an asymmetric image segmenter with row tag memory that allows for efficient driving of multiple displays with different resolutions from the same SOC video output port. This method and apparatus can be implemented, for example, but not limited to, video system 120. In one or more embodiments, the superframe image stream may include image streams or image data from video, LIDAR (light detection and ranging), radar, or other sensors. The superframe image stream may include image streams supporting advanced driver assistance systems (ADAS) or automotive infotainment applications. In one or more embodiments, the row tag memory is a lookup table.
[0032] Figure 2A A superframe 200 for multi-stream video transmission according to various embodiments of this disclosure is depicted. Two video images may be combined side-by-side and top-aligned to form the video image superframe 200. The video image superframe 200 includes a combination of two video image displays 1202 and 2204 representing the original image at the source (i.e., SoC 121). These original images can then be transmitted to their respective displays. As shown, display 1202 has a greater height than display 2204. Therefore, the resolution of the video image on display 1202 is higher than that on display 2204. In some embodiments, display 1202 may be positioned to the upper left of display 2204 in the superframe 200. Therefore, display 2204 may be positioned to the upper right of display 1202 in the superframe 200. Figure 2A In this context, VT represents the vertical blanking amount, and HT represents the horizontal blanking amount.
[0033] Figure 2BA vertically distributed image formatting of video image 220 according to various embodiments of this disclosure is depicted. The image is formatted and sent across the SOC output to a serializer. In one embodiment, the serializer is SER122. Video images on display 1202 are vertically distributed line by line to form video image 222. Video images on display 2204 are vertically distributed line by line to form video image 224. To make video image 224 have the same height as video image 222, padding can be added to video image 224 via the SOC. Video image 220 represents the output of an SOC (e.g., but not limited to SoC121). And the output of the SOC is coupled to the input of the serializer.
[0034] Figure 3 Timing 300 of an asymmetric image segmenter for multi-stream video image applications according to various embodiments of this disclosure is described. Figure 3 The video processing in SER122 is demonstrated. Timer 300 shows three operation frames: frame 1, frame 2, and frame 3.
[0035] Timing 302 illustrates the serial frame transmission of Display 1 and Display 2. In frame 1, Display 1 experiences maximum buffer stress after transmitting its video image. In frame 1, Display 2 experiences maximum buffer stress after transmitting its video image. This is important because it allows system designers to understand the worst-case memory requirements for a specific selected resolution and to check whether the serializer memory is sufficient for that specific resolution.
[0036] Timing 304 represents the output of SoC121, which is the input of SER122. Video images from displays 1 and 2 are transmitted serially in frames 1, 2, and 3. Timing 304 also shows that for display 1, lines 1, 2, and 3 are written to buffers in frames 1, 2, and 3, respectively. For display 2, lines 1 and 2 are written to buffers in frames 1 and 2, respectively, while line 3 is not written to a buffer because it is padding. According to this sequence, data from the first three lines of displays 1 and 2 is processed and stored in the buffers of SER122. According to timing 304, padding includes padding for line 3 of display 1 and line 3 of display 1. This padding can be discarded based on instructions from the row marker memory.
[0037] Timing 306 indicates reading data from the buffer of SER122. Display 1, lines 1, 2, and 3 are read from the buffer, and display 1, lines 1, and 2 are read from the buffer. This process occurs during the time span of frames 1, 2, and 3. Timing 306 demonstrates the latency of reading from the buffer relative to writing to the buffer. Timing 306 also demonstrates that the video images of display 1 and display 2 are separated (split) and can be individually input to DES125 and DES126.
[0038] The line buffer in SER122 receives video image 222 and video image 224. Since video image 222 does not contain any padding, all lines of video image 222 can be written to the buffer. The line buffer then reads data from display 1 and generates a video clock associated with video image 222 to drive the input of display 2402.
[0039] Since the video image 224 includes padding, SER122 discards the padding when filling the buffer with data from the display 2 of the video image 224. The contents of the buffer are then read, and the generated video clock associated with the video image 224 drives the input to the display 2422. As discussed herein, the process of discarding padding can be facilitated via a row marker memory.
[0040] Figure 3 The terms included include: ○ HS = Horizontal Synchronization ○ VS = Vertical Synchronization ○ Tvbp = Vertical trailing edge time ○ Tvfp = Vertical Frontier Time ○ Tvact = Vertical effective time ○ Thbp = Horizontal trailing edge time ○ Thfp = Horizontal Front Time ○ Thact = Horizontal effective time ○ HAS = Horizontal Synchronization Effective Time ○ BL = Hidden Lines ○ B1, B2 = Blanking line time of monitor 1 and monitor 2 ○ T_V*_Fill = Time spent writing to the buffer ○ T_V*_Read = Time to read from the buffer Figure 4A and Figure 4BReconstructed video images 400 and 420, following asymmetric image segmentation processing according to various embodiments of this disclosure, are depicted respectively. These reconstructed images are input to DES125 and DES126, respectively. Reconstructed video image 400 includes the video image of display 1402, and reconstructed video image 420 includes the video image of display 2422. Video image 400 also corresponds to display 1202, and video image 420 corresponds to display 2204. The boundary around each video image represents a blanking region of the display.
[0041] The memory requirement in an asymmetric image segmenter can be defined by the difference between the input and output time difference for each individual pixel. After a pixel is received from the SOC, it must be read from the video receive buffer as early as possible. Since the display requires precise memory read timing, there is only one degree of freedom in the system when writing to the video receive buffer in the serializer. To ensure low memory requirements, the preceding points might suggest that pixels should be written to memory as late as possible but before they are read. This can be achieved by ensuring that data is not sent from the SOC to the serializer in bursts, but rather by extending the data by skipping rows in a timely manner to reduce burstiness. Comparing timing 304 and timing 306, for display line 1-1, the delay between writing to the buffer and reading from the buffer is B1. And comparing timing 304 and timing 306, for display line 2-1, the delay between writing to the buffer and reading from the buffer is B2.
[0042] The amount and arrangement of padding can determine the timing of writing and reading pixels in the serializer. Padding can occur in a variety of ways and partially determines the amount of memory required in the serializer. For example, this approach might require the largest possible amount of memory if all the data in the video image is transmitted first, followed by all the padding. The amount of memory required increases if the data arrives earlier. If a repetition pattern is used, such as Table 1 as described herein, the data for a vertical image is distributed vertically in time, thus the data is delayed and does not arrive too early. When padding is added, the data in the video image may be delayed, but in a smart way, i.e., received a little earlier than the data shown on the display. This approach can reduce memory requirements.
[0043] Because the vertical asymmetry of shorter displays creates more memory requirements in the splitter device, shorter images (e.g., display 2204) are vertically distributed to use blank video lines to match the height of the video (crosshair pattern) stream. In the superframe, the filled blank lines are transmitted from the video port of SoC 121, whose video clock frequency is faster than the video clocks of both individual displays 1402 and 2422.
[0044] When SER122 receives a superframe, it splits this image into two displays, each with unique display timing parameters and an associated display video clock frequency different from the superframe. For example, and not limitingly, the video clock of SoC121 could be 200 MHz, the video clock of display 1402 could be 148 MHz, and the video clock of display 2422 could be 67 MHz. After splitting, the individual video streams can be transmitted to a remote location using the same serial link or separate serial links.
[0045] Each display image is generated using a row-marker memory. Figure 5 A configuration for video multistream transmission 500 with internal or external memory having a serializer, according to an embodiment of the present invention, is depicted. As shown, SoC 521 outputs a video multistream transmission signal 531 to SER 522. SER 522 includes a row marker memory 530, which can be programmed to provide instructions for discarding padding for individual video frames. The outputs of SER 522 can be GMSL 523 and GMSL 524, which are the inputs to DES 525 and DES 526, respectively. DES 525 and DES 526 are coupled to CID 527 and IC 528, respectively.
[0046] The horizontal video line from the SOC input to the serializer reaches the asymmetric image segmenter block. The coordinates of video images 222 and 224 on superframe 200 are programmed into the firmware. This allows differentiation between video images 222 and 224 in the input data. Video image 222 can be directly written to the memory of display 1, and read from this memory to directly drive display 1402. Timing 304 illustrates this process.
[0047] Because video image 224 has vertically distributed and discontinuous rows, some video rows in the image are blank and should not be written to the memory of display 2. This result is achieved using a programmable row marker memory, which can be used to isolate blank rows from real video rows. After the serializer reads a row, the row marker memory can determine whether the row contains valid video data; otherwise, the serializer deletes the blank row, i.e., fills it in. Compared to algorithmic alternatives, row marker memory may consume more die area, but the data storage device / memory is flexible and allows users to employ any desired mechanism, thus allowing inherent compatibility with various SOC outputs. Video image 224 is read from memory to drive display 2.
[0048] To reduce the memory and associated die costs required by the asymmetric image segmenter, it may be crucial to place the video images 224 in a vertical row distribution so that they are written to memory just before being read. Users can use row marking memory to mark the valid display rows (square pattern 224) in the vertical row stream in any way they wish and conform to various available SOC outputs while keeping memory costs to a minimum.
[0049] Although the line marker memory has 2160 register bits for individual marking, there are no mode limitations on the effective video lines of display 2, and several pre-programmed line marker memory templates are also available to the user, such as those explained in Table 1:
[0050] Table 1 According to Table 1, if a 2:1 ratio is selected, the pattern 1010101010110.. can be used to select rows until the end of memory. If a 3:2 ratio is selected, the pattern 110110110 can be used to select rows until the end of memory, and so on. As mentioned above, designers can program custom patterns to mark which rows are not filled.
[0051] Designers can program the SOC and the serializer. The row mark memory can be user-programmed and used by a serializer such as the SER122. The serializer may include registers that control the operation of the row mark memory. Furthermore, the serializer can be programmed using a PC (internal integrated circuit). The PC is a synchronous, multi-master, multi-slave, packet-switched, single-ended, serial computer bus. Using the row mark memory, each line of a video image (i.e., display data) can be marked as padding to be discarded or display data to be displayed. The row mark memory can be based on the height ratio or repetition pattern of the video image in the superframe, as described in Table 1, or it can be customized by the user. The row mark memory is flexible and not constrained by any specific algorithm.
[0052] In some embodiments for multi-stream video, the superframe may have a constant amount of padding in each of its video frames. In other embodiments for multi-stream video, the superframe may have varying amounts of padding. The required amount of padding may vary based on the resolution difference between the video images in the superframe. The resolution difference may be represented by the height ratio of the two video images in the superframe.
[0053] In summary, in one or more embodiments, a system supporting multi-stream video transmission may include: 1) a system-on-a-chip (SoC) that generates a superframe of video images comprising a first video image and a second video image, wherein the second video image is shorter than the first video image and is vertically adjusted to the same height as the first video image by adding padding to selected display rows; 2) a serializer including an asymmetric image segmenter that segments the superframe of video images into the first video image and the second video image; and 3) a line tagging memory that tags each row of the second video image to determine which rows are padded and discarded, and which rows are data to be displayed. The user can determine whether the line tagging memory is based on the height ratio of the first video image to the second video image.
[0054] In one or more embodiments, a method for supporting multi-stream video transmission may include: 1) receiving multi-stream video including superframe video images, wherein each superframe video image includes a first video image and a second video image, and wherein the height of the first video image is greater than that of the second video image; 2) adjusting the vertical asymmetry of the second video image to the same height as the first video image by adding padding to the second video image; 3) segmenting the superframe video image into two separate video images using an asymmetric image segmentation engine; 4) using a line tagging memory to tag each line of the second video image to determine which lines are padded and discarded, and which lines are data to be displayed. The user can determine whether the line tagging memory is based on the height ratio of the first video image to the second video image. The line tagging memory may be a repeating pattern.
[0055] In one or more embodiments, a system supporting image multistream transmission includes a system-on-chip (SoC) that 1) generates a superframe of an image stream comprising multiple image streams, wherein a first image stream of the multiple image streams includes a first height, the first height being different from a second height of at least one other image in the multiple image streams; and 2) adjusts the first height to the second height by adding padding to one or more selected display rows in the first image stream. The system supporting image multistream transmission further includes: a serializer communicating with the SoC and including an asymmetric image segmenter engine configured to segment the superframe of the image stream into multiple separate image streams, the multiple separate image streams including the first image stream; and a data storage device communicating with the serializer and configured to mark each row of the first image stream to determine which rows include padding to be discarded and which rows include display data to be displayed.
[0056] In one or more embodiments, a system supporting multi-stream video transmission includes: a system-on-a-chip that generates frames of an image stream comprising a single video image and vertically adjusts the original video image to a user-configured video image height and vertical blanking parameters by adding padding to selected display rows; a serializer that includes an asymmetric image segmenter engine that retims the received video images to the original video images; and a row marker memory that marks each row of the received video images to determine which rows are filled and discarded, and which rows are data to be displayed.
[0057] System Implementation Examples In embodiments, aspects of this patent document may relate to or be implemented in an information processing system / computing system. For the purposes of this disclosure, a computing system may include any tool or set of tools operable for calculating, determining, classifying, processing, transmitting, receiving, retrieving, generating, routing, switching, storing, displaying, transmitting, highlighting, detecting, recording, reproducing, disposing of, or utilizing information, intelligence, or data of any form for business, scientific, control, or other purposes. For example, a computing system may be an element of a communication system, such as a GMSL serializer and deserializer capable of simultaneously transmitting multi-stream image data. A computing system may include random access memory (RAM), one or more processing resources such as a central processing unit (CPU) or hardware or software control logic, ROM, and / or other types of memory. Additional components of a computing system may include one or more network ports or wireless ports for communicating with external devices, and various input and output (I / O) devices such as a keyboard, mouse, touchscreen, and / or video display. A computing system may also include one or more buses operable for transmitting communication between various hardware components.
[0058] Figure 6 A simplified block diagram of a computing device / information processing system (or computing system) according to embodiments of this disclosure is depicted. It will be understood that the illustrated functions of system 600 can operate to support various embodiments of the information processing system, but it should be understood that the information processing system can be configured in different ways and include different components.
[0059] like Figure 6As shown, system 600 includes one or more central processing units (CPUs) 601, which provide computing resources and control the computer. CPU 601 may be implemented using a microprocessor or the like, and may also include one or more graphics processing units (GPUs) 617 and / or floating-point coprocessors for mathematical calculations. System 600 may also include system memory 602, which may take the form of random access memory (RAM), read-only memory (ROM), or both.
[0060] Multiple controllers and peripherals can also be provided, such as Figure 6 As shown. Input controller 603 represents an interface to various input devices 604, such as a keyboard, mouse, or stylus. An SoC controller 605 communicating with SoC device 606 may also be present. System 600 may further include a storage controller 607 for interface connection to one or more storage devices 608, each of which includes a storage medium such as flash memory, or an optical medium that can be used to record instructions for operating embodiments of systems, utilities, and applications that may include programs implementing various aspects of the present invention. The storage devices 608 may also be used to store processed data or data to be processed according to the present invention. System 600 may further include a display controller 609 for providing an interface to display device 611. Computing system 600 may further include a serializer controller 612 for communicating with serializer device 613. The communication controller 610 can interface with one or more communication devices 615, enabling the system 600 to connect to remote devices via any of a variety of networks or via any suitable electromagnetic carrier signal (including infrared signals), such as vehicular networks, the Internet, cloud resources (e.g., Ethernet cloud, Ethernet Fibre Channel (FCoE) / Data Center Bridge (DCB) cloud, etc.), local area networks (LAN), wide area networks (WAN), and storage area networks (SAN).
[0061] In the demonstrated system, all major system components can be connected to bus 616, which may represent more than one physical bus. However, the various system components may or may not be physically close to each other. For example, input data and / or output data can be remotely transmitted from one physical location to another. Furthermore, programs implementing various aspects of the invention can be accessed from a remote location (e.g., a server) via a network. Such data and / or programs can be delivered via any machine-readable medium of a variety of machine-readable media, including but not limited to: magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and holographic devices; magneto-optical media; and hardware devices specifically configured to store or store and execute program code, such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), flash memory devices, and ROM and RAM devices.
[0062] Embodiments of the invention can be encoded on one or more non-transitory computer-readable media using instructions that cause one or more processors or processing units to execute steps. It should be noted that the one or more non-transitory computer-readable media should include both volatile and non-volatile memory. It should be noted that alternative implementations are possible, including hardware implementations or software / hardware implementations. The functionality of a hardware implementation can be implemented using ASIC(s), programmable arrays, digital signal processing circuit systems, etc. Therefore, the term "apparatus" in any claim is intended to cover both software and hardware implementations. Similarly, the term "one or more computer-readable media" as used herein includes software and / or hardware or a combination thereof having a program of instructions embodied thereon. In consideration of these alternative implementations, it will be understood that the accompanying drawings and description provide functional information that those skilled in the art would need to write program code (i.e., software) and / or manufacture circuitry (i.e., hardware) to perform the desired processing.
[0063] It should be noted that embodiments of the present invention may further relate to computer products having a non-transitory tangible computer-readable medium having computer code thereon for performing various computer-implemented operations. The medium and computer code may be media and computer code specifically designed and constructed for the purposes of the present invention, or they may belong to a class well known or available to those skilled in the art. Examples of tangible computer-readable media include, but are not limited to: magnetic media, such as hard disks, floppy disks, and magnetic tapes; optical media, such as CD-ROMs and holographic devices; magneto-optical media; and hardware devices specifically configured to store or store and execute program code, such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), flash memory devices, and ROM and RAM devices. Examples of computer code include machine code generated by a compiler and files containing high-level code executed by a computer using an interpreter. Embodiments of the present invention may be implemented, in whole or in part, as machine-executable instructions that can reside in program modules executed by a processing device. Examples of program modules include libraries, programs, routines, objects, components, and data structures. In a distributed computing environment, program modules may be physically located in a local, remote, or both environment.
[0064] Those skilled in the art will recognize that the absence of a computing system or programming language is essential for the practice of this invention. They will also recognize that the various elements described above can be physically and / or functionally divided into sub-modules or combined together.
[0065] It will be understood by those skilled in the art that the foregoing examples and embodiments are exemplary and not limited to the scope of this disclosure. It is intended that all arrangements, enhancements, equivalents, combinations, and modifications thereof that will be apparent to those skilled in the art upon reading this specification and studying the accompanying drawings are included within the true spirit and scope of this disclosure. It should also be noted that elements in any of the claims may be arranged differently, thereby encompassing various relevance, configurations, and combinations.
Claims
1. A method for supporting multi-stream images, the method comprising: Receive a first video stream and a second video image stream, wherein the first video stream includes a plurality of first video frames having a first frame size, and the second video image stream includes a plurality of second video frames having a second frame size, wherein the second frame size is smaller than the first frame size; as well as A multi-stream video comprising multiple superframes is formed. Each superframe includes a first image corresponding to a first video frame and a second image corresponding to a second video frame. The first image and the second image are distributed row by row and arranged side by side in each superframe. The second image is formed by adding one or more padding rows to the second video frame, so that the first image and the second image have the same height. In each superframe, one or more coordinates of the first and second images are programmed as position information.
2. The method according to claim 1, wherein, The first video frame and the second video frame are uncompressed in each superframe.
3. The method according to claim 1, wherein, The multi-stream video supports data integrity features.
4. The method according to claim 1, wherein, The first video stream and the second video image stream are displayed on separate displays.
5. A method for multi-stream video transmission of images, the method comprising: Receive multi-stream video at a serializer, the multi-stream video comprising multiple superframes, wherein each superframe comprises a first image corresponding to a first video frame having a first video frame size and a second image corresponding to a second video frame having a second video frame size smaller than the first video frame size, the second image comprising one or more padding lines added to the second video frame such that the first image and the second image have the same height; The multi-stream video is divided into multiple streams using the asymmetric image segmenter in the serializer. These multiple streams include a first reconstructed video stream and a second reconstructed video stream. The first reconstructed video stream includes multiple reconstructed first video frames, and the second reconstructed video stream includes multiple reconstructed second video frames. The first reconstructed video stream is sent from the serializer to the first deserializer and the second reconstructed video stream is sent to the second deserializer for display.
6. The method according to claim 5, wherein, The serializer, the first deserializer, and the second deserializer are programmed by a microcontroller.
7. The method according to claim 5, wherein, The first image and the second image are distributed row by row and arranged side by side in each superframe.
8. A method for generating superframes, the method comprising: A first video stream and a second video image stream are obtained. The first video stream includes multiple first video frames with a first frame size, and the second video image stream includes multiple second video frames with a second frame size smaller than the first frame size. A multi-stream video comprising multiple superframes is formed, each superframe including: a first image corresponding to a first video frame; and a second image corresponding to a second video frame; The second image is formed by adding one or more padding rows to the first video frame, so that the first image and the second image have the same height. One or more coordinates of the first image and the second image on each superframe are programmed as position information. The first video frame and the second video frame are uncompressed in each superframe.
9. The method according to claim 8, wherein, The first image and the second image are distributed row by row and arranged side by side in each superframe.
10. The method according to claim 8, wherein, The multi-stream video supports data integrity features.
Citation Information
Patent Citations
Video coding rate adaptation to reduce packetization overhead
US20080101476A1
KR1018083270000B1
