Image processing device and image processing method

Abstract

An image processing device and an image processing method are provided. The image processing device is used to receive an input image data, the input image data includes a plurality of frames, and at least one frame of the plurality of frames contains at least one packet. The image processing device includes a receiving circuit, a storage circuit, a transmission circuit, and a control circuit. The receiving circuit receives the input image data. The storage circuit is used to store the at least one packet. The transmission circuit is used to output a current output image data. The control circuit is used to read at least one target packet from the storage circuit according to the current output image data, adjust the at least one target packet, and then output the at least one target packet through the transmission circuit.

Description

Technical Field

[0001] This invention relates to image processing, and more particularly to the adjustment of data packets in images. Prior Technology

[0002] Modern video interfaces (such as High Definition Multimedia Interface (HDMI) and DisplayPort (DP)) commonly employ data packet transmission mechanisms. Data packets are used to transmit audio-visual auxiliary data, including metadata and control information. This information is crucial for the correct display and processing of audio and video content, such as automatic audio and / or video synchronization, High Dynamic Range Imaging (HDR) video, and advanced color formats. Therefore, providing the correct data packets at the appropriate time is essential to ensure the correct display of image data. Summary of the Invention

[0003] In view of the shortcomings of the prior art, one object of the present invention is to provide an image processing apparatus and an image processing method to improve the shortcomings of the prior art.

[0004] One embodiment of the present invention provides an image processing apparatus for receiving input image data, the input image data comprising a plurality of frames, and at least one of the plurality of frames comprising at least one packet. The image processing apparatus includes: a receiving circuit, a storage circuit, a transmitting circuit, and a control circuit. The receiving circuit receives the input image data. The storage circuit is coupled to the receiving circuit for storing the at least one packet. The transmitting circuit is coupled to the receiving circuit for outputting current output image data. The control circuit is coupled to the storage circuit and the transmitting circuit for reading at least one target packet from the storage circuit according to the current output image data, adjusting the at least one target packet, and then outputting the at least one target packet through the transmitting circuit.

[0005] Another embodiment of the present invention provides an image processing method applied to an image processing apparatus. The image processing apparatus receives input image data, the input image data comprising a plurality of frames, and at least one of the plurality of frames comprising at least one packet. The image processing method includes: receiving the input image data; storing the at least one packet in a storage circuit; and reading at least one target packet from the storage circuit according to current output image data and adjusting the at least one target packet, and then outputting the at least one target packet.

[0006] The technical means embodied in the embodiments of the present invention can improve at least one of the disadvantages of the prior art. Therefore, the present invention can ensure the correct display of image data compared with the prior art.

[0007] The features, implementation, and effects of this invention are described in detail below with reference to the accompanying drawings. Simple Explanation of the Diagram

[0008] Figure 1 is a functional block diagram of one embodiment of the image processing apparatus of the present invention; Figure 2 is a flowchart of one embodiment of the image processing method of the present invention; Figure 3 is a schematic diagram of the coordinates of a frame; Figure 4A is a schematic diagram of one application example of the present invention; Figure 4B is a schematic diagram of another application example of the present invention; Figure 5 shows the sub-steps of step S230; Figure 6 is a schematic diagram of another application example of the present invention; Figure 7 shows the sub-steps of step S230; Figure 8 is a schematic diagram of another application example of the present invention; Figure 9 shows the sub-steps of step S230; Figure 10 is a functional block diagram of another embodiment of the image processing apparatus of the present invention; Figure 11 is a flowchart of another embodiment of the image processing method of the present invention; Figure 12 is a functional block diagram of another embodiment of the image processing apparatus of the present invention; Figure 13 is a flowchart of another embodiment of the image processing method of the present invention; and Figure 14 is a functional block diagram of another embodiment of the image processing apparatus of the present invention. Implementation

[0009] The technical terms used in the following description are based on the common terminology of this technical field. If this specification provides explanations or definitions for certain terms, the explanations or definitions in this specification shall prevail.

[0010] The disclosure of this invention includes an image processing apparatus and an image processing method. Since some components of the image processing apparatus of this invention may be known individually, details of known components will be omitted in the following description without affecting the full disclosure and implementability of the apparatus invention. Furthermore, some or all of the processes of the image processing method of this invention may be in the form of software and / or firmware, and may be executed by the image processing apparatus of this invention or its equivalents. Without affecting the full disclosure and implementability of the method invention, the following description of the method invention will focus on the steps rather than the hardware.

[0011] Please refer to Figure 1, which is a functional block diagram of one embodiment of the image processing apparatus of the present invention. The image processing apparatus 100 includes a receiving circuit 110, a transmitting circuit 120, a storage circuit 130, and a control circuit 140 coupled to each other. The storage circuit 130 may be a memory (e.g., Dynamic Random Access Memory (DRAM)) or a buffer circuit. The control circuit 140 may be a logic circuit.

[0012] Please refer to Figure 2, which is a flowchart of one embodiment of the image processing method 200 of the present invention. The image processing method 200 can be executed by the image processing apparatus 100. When the image processing method 200 is in the form of software or firmware, the control circuit 140 can be a circuit or electronic component with program execution capability, such as a central processing unit, microprocessor, microcontroller, microprocessor unit, digital signal processor (DSP), or equivalent circuit. The control circuit 140 implements the image processing method 200 by executing program code and / or program instructions stored in the storage circuit 130.

[0013] Image processing method 200 includes the following steps.

[0014] Step S210: The receiving circuit 110 receives the input image data Din. The input image data Din contains multiple frames (F_M, where M is the frame number, M=0, 1, 2, …), and each frame contains at least one data packet PKT_N (hereinafter referred to as a packet). N is the packet number, which can be represented as MP, where P is the packet number (P=0, 1, 2, …). For example, PKT_0-1 represents the first packet of the 0th frame. The control circuit 140 processes frames that require packet adjustment.

[0015] Step S215: The receiving circuit 110 determines whether frame F_M contains a packet. If yes, proceed to step S220; otherwise, proceed to step S225.

[0016] Step S220: The receiving circuit 110 stores the at least one packet PKT_N to the storage circuit 130, and stores the correspondence between the at least one packet PKT_N and the frame to the storage circuit 130. In some embodiments, the correspondence can be stored by storing the packet number. For example, the correspondence can be known from the packet number "MP" (i.e., the packet is the Pth packet of the Mth frame).

[0017] Step S225: The receiving circuit 110 records in the storage circuit 130 that frame F_M does not contain any packets.

[0018] Step S230: Control circuit 140 adjusts the target packet according to the current output image data Dout. Transmission circuit 120 outputs the current output image data Dout, which contains multiple frames, and each frame contains multiple pixels. In some embodiments, the current output image data Dout includes the frame being output by transmission circuit 120 (hereinafter referred to as the current frame), the frame number of the current frame, and the position of the pixel being output ((X,Y)) (hereinafter referred to as the current pixel position (X,Y)). For example, referring to Figure 3, the current output image data Dout may contain the (X1,Y1)th pixel of frame F_1 (frame number M=1), that is, the X1th pixel of column Y1.

[0019] The operation of adjusting the target packet includes, but is not limited to, reading the target packet PKT_K from the storage circuit 130 and then providing the target packet PKT_K to the transmission circuit 120 at an appropriate time. The control circuit 140 provides the target packet to the transmission circuit 120 at an appropriate time, which is equivalent to inserting the target packet into the current frame.

[0020] In other embodiments, the current output image data Dout includes the time of the image data (e.g., the current playback time at the current pixel position (X,Y)). Specifically, each frame corresponds to a specific playback time, and the control circuit 140 can calculate the current playback time corresponding to the current pixel position (X,Y) based on the resolution of the current frame and the current pixel position (X,Y).

[0021] Step S240: Transmission circuit 120 outputs the target packet PKT_K and the current frame.

[0022] Please refer to Figures 4A, 4B, and 5. Figures 4A and 4B are schematic diagrams of two application examples of the present invention, while Figure 5 shows the sub-steps of step S230. The application examples in Figures 4A and 4B can be related to frame rate conversion, for example, switching from a high frame rate to a low frame rate by discarding frame F_1. That is, originally, three frames, F_0, F_1, and F_2, were output between time point t0 and time point t3, but after conversion, only two frames, F_0 and F_2, are output in the same time period. In Figure 4A, frame F_0 originally (i.e., before frame rate conversion) contained packets PKT_0-0 and PKT_0-1, while in Figure 4B, frame F_0 originally did not contain any packets. It should be noted that since frame F_2 needs to refer to the packet of frame F_1 (including but not limited to packet PKT_1-0 and packet PKT_1-1), the packet of frame F_1 should be preserved after conversion. Otherwise, errors will occur when the backend display (not shown in the figure) displays frame F_2 (for example, the image quality will be degraded or it may even be unable to be displayed).

[0023] It should be noted that changes in frame resolution (i.e., changes in frame size, for example, from 4K to 1080P) can also cause situations similar to those shown in Figure 4A or Figure 4B.

[0024] Figure 5 corresponds to the application examples in Figures 4A and 4B, and includes the following steps.

[0025] Step S510: Control circuit 140 determines whether the next frame will be discarded. For example, referring to Figure 4A or Figure 4B, control circuit 140 determines, between time point t0 and time point t1, whether the next frame (i.e., frame F_1) of the current frame (i.e., frame F_0) will be discarded based on the conversion rate or frame resolution. If yes, then step S520 is executed; otherwise, step S530 is executed.

[0026] Step S520: The control circuit 140 uses the packets (PKT_1-0, PKT_1-1) of the next frame (frame F_1) as the target packet PKT_K. For example, referring to Figure 4A or Figure 4B, since the next frame after frame F_0 is frame F_1, and frame F_1 will be discarded, the target packet includes packets PKT_1-0 and PKT_1-1, and the control circuit 140 provides at least one target packet PKT_K to the transmission circuit 120. That is, after adjustment, the image processing device 100 outputs packets PKT_1-0 and PKT_1-1 in frame F_0, but does not output packets PKT_0-0 and PKT_0-1 (Figure 4A) nor does it output any packets (Figure 4B).

[0027] Step S530: Use the packet of the current frame as the target packet. For example, assuming the current frame is frame F_2, and the next frame of frame F_2 (not shown) will not be discarded, the target packet includes packets PKT_2-0 and PKT_2-1, and the control circuit 140 provides at least one target packet PKT_K to the transmission circuit 120. That is, the adjustment operation in step S230 includes retaining the at least one target packet. For another example, assuming frame F_2 does not contain any packets, and the next frame of frame F_2 (not shown) will not be discarded, since the information "frame F_2 does not contain any packets" is stored in the storage circuit 130 in step S225, the control circuit 140 will continue to maintain the state that the current frame (i.e., frame F_2) has no packets (i.e., the target packet is empty) in step S530. That is, the adjustment operation in step S230 includes maintaining the state that the current frame has no packets.

[0028] Please refer to Figures 6 and 7. Figure 6 is a schematic diagram of another application example of the present invention, while Figure 7 shows a sub-step of step S230. In the application example of Figure 6, the packets of the original frame F_0 (left column) (including but not limited to packets PKT_0-0 and PKT_0-1) are transmitted too late, so that the backend display (not shown) cannot process these packets in time. More specifically, the display needs to refer to the packets of frame F_0 when displaying frame F_1, and an error will occur when the display cannot process the packets of frame F_0 in time.

[0029] Figure 7 corresponds to the application example in Figure 6 and includes the following steps.

[0030] Step S710: Control circuit 140 determines whether the distance between the target packet and the lower boundary of the current frame is less than a preset value. If yes, then step S720 is executed; otherwise, step S730 is executed. For example, referring to Figure 6, the preset value can be Q column pixels (Q is a positive integer), or the transmission time corresponding to the Q column pixels (i.e., t1-t1', where time point t1 corresponds to the lower boundary of frame F_0 (i.e., the boundary between frame F_0 and frame F_1), and time point t1' corresponds to the top column of the Q column pixels). When the current frame is frame F_0 (the target packet contains PKT_0-0 and PKT_0-1), the result of step S710 is yes. When the current frame is frame F_1 (the target packet contains PKT_1-0 and PKT_1-1) or frame F_2 (the target packet contains PKT_2-0 and PKT_2-1), the result of step S710 is no. In some embodiments, the time difference between time point t1 and time point t1' is the time required for the display to process the target packet.

[0031] Step S720: The control circuit 140 advances the position or output time of the target packet, so that the distance between the advanced target packet and the lower boundary of the current frame is not less than a preset value. As shown in Figure 6 (right column), the at least one target packet is advanced to time point t1' or earlier, so the display has enough time to process the at least one target packet.

[0032] Step S730: Control circuit 140 maintains the current position or output time of the target packet. For example, referring to Figure 6, for frame F_1, the position or output time of packet PKT_1-0 and packet PKT_1-1 remains unchanged.

[0033] Please refer to Figures 8 and 9. Figure 8 is a schematic diagram of another application example of the present invention, while Figure 9 shows a sub-step of step S230. In the application example of Figure 8, the target packets of the current frame F_0 (including but not limited to PKT_0-0, PKT_0-2, and PKT_0-1) are in the wrong order (the correct order is: PKT_0-0→PKT_0-1→PKT_0-2), which will cause errors in the image displayed on the monitor. In some embodiments, packets PKT_0-0, PKT_0-1, and PKT_0-2 can be extended metadata packets (EMP) of HDMI.

[0034] Figure 9 corresponds to the application example in Figure 8 and includes the following steps.

[0035] Step S910: Control circuit 140 determines whether the order of the target packets is incorrect. If so, proceed to step S920; otherwise, proceed to step S930. Because the packets themselves contain order information (e.g., packet number), control circuit 140 can make the determination based on this information.

[0036] Step S920: Control circuit 140 changes the order (position) of the target packets. In the example of Figure 8, control circuit 140 swaps the positions of packets PKT_0-1 and PKT_0-2, so that the adjusted target packets present the correct order.

[0037] Step S930: Control circuit 140 maintains the position of the target packet.

[0038] In summary, the image processing apparatus 100 of the present invention can prevent errors from occurring when a multimedia device or system plays or displays images. In some embodiments, the image processing apparatus 100 may be a repeater or converter of a video interface.

[0039] Please refer to Figure 10, which is a functional block diagram of another embodiment of the image processing apparatus of the present invention. In addition to the receiving circuit 110, the transmitting circuit 120, the storage circuit 130, and the control circuit 140, the image processing apparatus 1000 further includes an encoding circuit 1010 and a decoding circuit 1020. The encoding circuit 1010 is coupled between the receiving circuit 110 and the storage circuit 130. The decoding circuit 1020 is coupled between the storage circuit 130 and the transmitting circuit 120, and is also coupled to the control circuit 140.

[0040] Figure 11 is a flowchart of another embodiment of the image processing method of the present invention. The image processing method 1100 of Figure 11 can be executed by the image processing apparatus 1000. In addition to steps S210 to S240, the flowchart of Figure 11 further includes the following steps.

[0041] Step S1110: Encoding circuit 1010 encodes at least one packet to reduce the amount of data in the packet. Step S1110 is performed before storing the packet (step S220). In some embodiments, encoding circuit 1010 encodes the packet content and its location information (e.g., see FIG. 3, relative to the origin (0,0) of a frame), and then stores the encoded packet in storage circuit 130. In other embodiments, encoding circuit 1010 compresses at least one packet, and then stores the compressed packet in storage circuit 130.

[0042] Step S1120: Decoding circuit 1020 decodes the at least one target packet PKT_K to reconstruct the target packet. More specifically, decoding circuit 1020 performs decoding operation in a decoding mode corresponding to the encoding mode of encoding circuit 1010, and then the decoded target packet is output (step S240).

[0043] Please refer to Figure 12, which is a functional block diagram of another embodiment of the image processing apparatus of the present invention. In addition to the receiving circuit 110, the transmitting circuit 120, the storage circuit 130, and the control circuit 140, the image processing apparatus 1200 further includes a packet selection circuit 1210. The packet selection circuit 1210 is coupled between the receiving circuit 110 and the storage circuit 130. In some embodiments, the packet selection circuit 1210 includes a packet parser.

[0044] Figure 13 is a flowchart of another embodiment of the image processing method of the present invention. The image processing method 1300 of Figure 13 can be executed by the image processing apparatus 1200. In addition to steps S210 to S240, the flowchart of Figure 13 further includes the following steps.

[0045] Step S1310: The packet selection circuit 1210 filters the at least one packet. Step S1310 is performed before storing the packet (step S220). More specifically, by filtering the packet, the image processing apparatus 1200 can process only the packets of interest (e.g., HDR-related packets), thereby saving storage circuit 130 and reducing processing time and / or power consumption of the image processing apparatus 1200.

[0046] Please refer to Figure 14, which is a functional block diagram of another embodiment of the image processing apparatus of the present invention. The image processing apparatus 1400 is a combination of the image processing apparatus 1000 and the image processing apparatus 1200. Those skilled in the art can understand the details of the image processing apparatus 1400 and its corresponding image processing method based on the description of Figures 10 to 13.

[0047] In summary, the image processing apparatus and image processing method of the present invention can adjust the data packets of image data when needed to provide the correct data packets at the appropriate time and avoid display errors.

[0048] Although the embodiments described above are illustrated using the cases shown in Figures 4A, 4B, 6, and 7 and 8 as examples, this is not intended to limit the invention. Those skilled in the art can appropriately apply the invention to other situations where adjustments to data packets are required, based on the disclosure of the invention.

[0049] Since those skilled in the art can understand the implementation details and variations of the method invention through the disclosure of the apparatus invention in this case, to avoid redundancy, repeated descriptions are omitted here without affecting the disclosure requirements and implementability of the method invention. Please note that the shapes, sizes, and proportions of the elements in the aforementioned figures are merely illustrative and are intended for those skilled in the art to understand the invention, and are not intended to limit the invention. Furthermore, in some embodiments, the order of the steps mentioned in the aforementioned flowchart may be adjusted according to actual operation, and they may even be performed simultaneously or partially simultaneously.

[0050] Although the embodiments of the present invention have been described above, these embodiments are not intended to limit the present invention. Those skilled in the art can make changes to the technical features of the present invention based on the explicit or implicit content of the present invention. All such changes may fall within the scope of patent protection sought by the present invention. In other words, the scope of patent protection of the present invention shall be determined by the scope of the patent application in this specification.

[0051] 100, 1000, 1200, 1400: Image processing device 110: Receiver circuit 120: Transmission circuit 130: Storage circuit 140: Control Circuit Din: Input image data Dout: Current output image data F_M,F_1,F_0,F_2: Frames M: Frame Number PKT_K: Target packet PKT_N,PKT_0-0,PKT_0-1,PKT_1-0,PKT_1-1,PKT_2-0,PKT_2-1,PKT_0-2: Packet (X,Y): Current pixel position 200, 1100, 1300: Image processing methods S210, S215, S220, S225, S230, S240, S510, S520, S530, S710, S720, S730, S910, S920, S930, S1110, S1120, S220 and S230, S1310, S220 to S240: Steps X, Y: Axes (0,0):origin (X1, Y1): Pixel position t0, t1, t1', t2, t3: Time points Q: Number of pixel columns 1010: Encoding Circuit 1020: Decoding Circuit 1210: Packet Selection Circuit

Claims

1. An image processing apparatus for receiving input image data, the input image data comprising a plurality of frames, and at least one frame of the plurality of frames comprising at least one packet, the image processing apparatus comprising: a receiving circuit for receiving the input image data; a storage circuit coupled to the receiving circuit for storing the at least one packet; a transmission circuit coupled to the receiving circuit for outputting current output image data; and a control circuit coupled to the storage circuit and the transmission circuit for reading at least one target packet from the storage circuit according to the current output image data, adjusting the at least one target packet, and then outputting the at least one target packet through the transmission circuit; wherein... The current output image data includes a current frame, and the control circuit changes the order of the at least one target packet when the order of the at least one target packet is incorrect.

2. The image processing apparatus as claimed in claim 1, wherein, The current output image data includes the frame number of the current frame and the current pixel position that the transmission circuit is outputting.

3. The image processing apparatus as claimed in claim 1, wherein, The current output video data includes a current playback time corresponding to the current pixel position of one of the current frames.

4. The image processing apparatus as described in claim 1, wherein, When the control circuit determines that the next frame of the current frame will be discarded, the control circuit reads at least one packet of the next frame as the at least one target packet.

5. The image processing apparatus as claimed in claim 1, wherein, When the control circuit determines that the distance between the at least one target packet and a boundary of the current frame is less than a preset value, the control circuit advances the position or output time of the at least one target packet, wherein the boundary is located between the current frame and the next frame after the current frame.

6. The image processing apparatus of claim 1 further includes: a packet selection circuit coupled to the receiving circuit for filtering the at least one packet.

7. The image processing apparatus as claimed in claim 1, wherein, The storage circuit further stores a one-to-one correspondence between the at least one packet and the at least one frame.

8. The image processing apparatus as claimed in claim 1, wherein, The input image data includes a target frame, and when the target frame does not contain any packets, the storage circuit records that the target frame does not contain any packets.

9. An image processing method applied to an image processing apparatus, the image processing apparatus receiving input image data, the input image data comprising a plurality of frames, and at least one of the plurality of frames comprising at least one packet, the image processing method comprising: receiving the input image data; storing the at least one packet in a storage circuit; reading at least one target packet from the storage circuit according to current output image data and adjusting the at least one target packet, and then outputting the at least one target packet, wherein... The current output image data includes a current frame; and when the order of the at least one target packet is incorrect, the order of the at least one target packet is changed.

Images