Method and image processing device

Abstract

A method for processing an image includes: receiving a complete image, a first modified image, and a second modified image, wherein the range of the first modified image and the range of the second modified image are respectively a part or all of the complete image; generating a mask corresponding to the complete image based on the first modified image and the second modified image, wherein the size of the mask corresponds to the first modified image and the second modified image; and generating a repaired image based on the mask and the complete image, wherein a part or all of the content of the repaired image is between the content of the first modified image and the content of the second modified image.

Description

[Technical Field]

[0001] This disclosure relates to an image processing technique, and more particularly to an image processing method and an image processing apparatus. [Previous Technology]

[0002] When a remote device transmits images, in order to save bandwidth, it usually only transmits changing images, and only transmits complete images at intervals. However, if a transmission problem occurs, causing one or more changing images to fail to arrive, ghosting may occur on the display. The ghosting problem can only be resolved after the next complete image arrives. Therefore, how to design a system to solve the above problem is an important issue in this field. [Summary of the Invention]

[0003] Embodiments of the present invention include a method for processing images. The method includes: receiving a complete image, a first modified image, and a second modified image, wherein the range of the first modified image and the range of the second modified image are respectively a part or all of the complete image; generating a mask corresponding to the complete image based on the first modified image and the second modified image, wherein the size of the mask corresponds to the first modified image and the second modified image; and generating a repaired image based on the mask and the complete image, wherein a part or all of the content of the repaired image is between the content of the first modified image and the content of the second modified image.

[0004] This embodiment of the invention includes an image processing apparatus. The image processing apparatus includes a processing module and a mask generation module. The processing module receives a complete image, a first modified image, and a second modified image; the mask generation module is coupled to the processing module and generates a mask based on the first modified image and the second modified image, the size of which corresponds to the size of the first modified image and the second modified image; the processing module generates a repaired image based on the mask and the complete image. The range of the first modified image and the range of the second modified image are respectively a part or all of the complete image. The content of a part or all of the repaired image is between the content of the first modified image and the content of the second modified image.

Implementation Method

[0006] In this document, when an element is referred to as a “connection” or “coupled,” it may mean an “electrical connection” or “electrical coupling.” “Connection” or “coupled” may also be used to indicate that two or more elements operate or interact with each other. Furthermore, although terms such as “first,” “second,” etc., are used herein to describe different elements, these terms are only used to distinguish elements or operations described using the same technical terms. Unless the context clearly indicates otherwise, these terms do not specifically refer to or imply order or sequence, nor are they intended to limit the scope of this application.

[0007] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this case pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning consistent with their meaning in the relevant technical context and in the context of this case, and shall not be interpreted as having an idealized or overly formal meaning unless expressly defined herein.

[0008] Figure 1 is a schematic diagram of a system 100 according to some embodiments of the present invention. As shown in Figure 1, the system 100 includes a remote device 110, an image processing device 120, and a display device 130. The image processing device 120 is coupled to the remote device 110 and the display device 130.

[0009] Please refer to Figure 2. The process of operating the remote device 110 will sequentially generate multiple frames, such as a complete frame F21 and frames F22~F24. The remote device 110 will sequentially transmit each complete frame to the image processing device 120, and transmit multiple variable frames corresponding to the subsequent complete frames and the position information of each variable frame to the image processing device 120. The variable frames are those that differ from the complete frames and / or variable frames of the previous moment in part or all of their size. Specifically, the variable frames can be the variable frames P21 and P22 shown in Figure 2. For example, the remote device 110 transmits the complete frame F21 to the image processing device 120 at the first moment. At the second moment after the first moment, the remote device 110 transmits the variable frame P21 to the image processing device 120. The variable frame P21 refers to the difference between the second moment's frame F22 and the first moment's complete frame F21. That is, at the second moment, the remote device 110 only transmits the changed block image in the complete frame F21 (i.e., the variable frame P21). At the third moment after the second moment, the remote device 110 transmits the changed image P22 to the image processing device 120. The changed image P22 refers to the block image (i.e., the changed image P22) that has changed from the complete image at the second moment (i.e., the complete image corresponding to the changed image P21) transmitted at the third moment, and so on.

[0010] The image processing apparatus 120 receives the image and corresponding position information transmitted by the remote device 110, processes the image and position information to generate a complete image and a repaired image, and transmits the complete image and the repaired image to the display device 130. Specifically, when the image processing apparatus 120 receives a complete image, it transmits the complete image to the display device 130. When the image processing apparatus 120 receives a modified image, it processes the modified image to generate a repaired image and transmits the repaired image to the display device 130. In some embodiments, the image processing apparatus 120 determines whether it is a complete image or a modified image based on the header of the image file transmitted by the remote device 110.

[0011] The display device 130 receives and displays the complete image and the repaired image from the image processing device 120, such as the complete image F21 and image F25 and the repaired image F27 shown in Figure 2.

[0012] As shown in Figure 1, in some embodiments, the image processing apparatus 120 includes a mask generation module 122 and a processing module 124. In various embodiments, the image processing apparatus 120 may include other modules different from the mask generation module 122 and the processing module 124, and the operations of the mask generation module 122 and the processing module 124 may be performed by the other modules, or the mask generation module 122 and the processing module 124 may be processed by a single processor. This is not intended to be limiting in this case, and the embodiment shown in Figure 1 is merely illustrative and not intended to limit the present invention. The above-mentioned modules may also be implemented in other systems, hardware, software, or combinations thereof.

[0013] In some embodiments, the mask generation module 122 generates a mask on the complete image received from the remote device 110, such as the mask M21 shown in Figure 2, and then transmits the image with the mask to the processing module 124.

[0014] In some embodiments, the processing module 124 receives the complete image with a mask and repairs the mask according to the area covered by the mask to generate a repaired block image, such as the repaired block image R21 shown in Figure 2. Then, the repaired block image R21 is combined with the complete image to generate a repaired image, which is then transmitted to the display device 130.

[0015] In some embodiments, the processing module 124 generates a repaired block image using generative image inpainting techniques, thereby restoring the image to a complete picture. Specifically, the processing module 124 receives a complete picture with a mask, such as picture F26 with mask M21 as shown in Figure 2, and generates a coarse result picture using a coarse network with spatially discounted reconstruction loss. Then, it generates a repaired block image using a refinement network with spatially discounted reconstruction loss and generative adversarial network (GAN) techniques. This invention is not limited to this; in some embodiments, the repaired block image located between the first and second modified pictures can be generated by taking the intermediate value between the first and second modified pictures.

[0016] Figure 2 further includes the screen displayed on the display device. As shown in Figure 2, in response to the image processing device 120 receiving the complete screen F21, the display device 130 displays the complete screen F21. In response to the image processing device 120 receiving the corresponding modified screen P21, the display device 130 displays screen F25. Before the image processing device 120 generates the repair block image R21, the display device 130 continues to display screen F25. Then, in response to the image processing device 120 generating the repair block image R21, the display device 130 displays the repaired screen F27 containing the repair block image R21. In response to the image processing device 120 receiving the modified screen P22, the display device 130 displays screen F24 containing the modified screen P22.

[0017] Figure 2 is a flowchart of method 200 corresponding to system 100 shown in Figure 1. The operation of method 200 is described below using components of system 100. It can be executed by other systems different from system 100, and this embodiment of the invention is not limited thereto. Method 200 includes a complete screen F21 and screens F22-F27, variable screens P21 and P22, and operations S21-S24. Operations S21-S24 are performed sequentially.

[0018] In some embodiments, the range of the changing screen P21 and the range of the changing screen P22 are a part or all of each of the complete screen F21 and screens F22 to F27. The content of a part or all of the complete screen F21 and screens F22 to F27 is between the content of the changing screen P21 and the content of the changing screen P22.

[0019] In some embodiments, the image F25 is a combination of the complete image F21 and the variable image P21. That is, after the processing module 124 obtains the complete image F21, in addition to outputting it, it will also temporarily store it in the storage. Then, when the variable image P21 is received, the processing module 124 will replace the corresponding position of the complete image F21 according to the position of the variable image P21 to generate the image F25. If there is a repair block image R21 generated in response to the mask, the processing module 124 will replace the image F25 according to the position of the repair block image R21 to generate the repaired image F27.

[0020] As shown in Figure 2, in the complete screen F21 and screens F22-F25 and F27, the object OB is located at different positions. Specifically, in the complete screen F21, the object OB is located at position L21. In screens F22 and F25, the object OB is located at position L22. In screen F23, the object OB is located at position L23. In screens F24 and F27, the object OB is located at position L24. In some embodiments, the object OB is a mouse cursor. The range of the changing screen P21 includes positions L21 and L22. The range of the changing screen P22 includes positions L23 and L24. The range of the mask M21 includes positions L22-L24. Positions L21-L24 correspond to position information. During operations S21-S24, the object OB moves from position L21, sequentially passing through positions L22 and L23 to position L24.

[0021] Referring to Figures 2 and 1, during operation S21, the remote device 110 generates a complete image F21 and transmits it to the image processing device 120. Then, the image processing device 120 determines that the complete image F21 is a complete image and transmits it to the display device 130. The display device 130 then displays the complete image F21.

[0022] During operation S22, the remote device 110 generates a frame F22 and transmits a changing frame P21, which differs from the complete frame F21, to the image processing device 120. The image processing device 120 then determines that the changing frame P21 is a changing frame and transmits it to the display device 130. The display device 130 then displays the changing frame P21 on the complete frame F21, i.e., the display device 130 displays frame F25. In some embodiments, frame F25 is identical to frame F22.

[0023] During operation S23, the remote device 110 generates screen F23, but does not generate a modified screen based on the differences between screen F23 and screen F22, either partially or entirely, and transmits it to the image processing device 120. At this time, the display device 130 still displays screen F25. In some embodiments, operation S23 corresponds to an abnormal operation of the system 100.

[0024] During operation S24, the remote device 110 generates screen F24 and transmits a changing screen P22, which differs from screen F23, to the image processing device 120. The image processing device 120 then determines that the changing screen P22 is a changing screen, and the mask generation module 122 generates a mask M21 on the complete screen F25, containing position information of the changing screens P21 and P22, i.e., screen F26. The processing module 124 then repairs the mask M21 based on screen F25 and the changing screen P22 to generate a repaired block image R21, and transmits the repaired block image R21 to the display device 130. The display device 130 then displays the repaired block image R21 on screen F25, i.e., the repaired screen F27. In some embodiments, the repaired screen F27 is identical to screen F23. In one embodiment, the image processing device 120 combines the repair block image R21 with the screen F25 to become the repair screen F27, and then transmits the repair screen F27 to the display device 130. In other words, after obtaining each complete screen and each modified screen, the image processing device 120 of the present invention processes them to become each repair screen and then transmits them to the display device 130 for display.

[0025] In some embodiments, the image processing device 120 determines whether to generate a mask based on whether it successfully received the changing image in the previous moment. Specifically, when the image processing device 120 receives the changing image in operation S23, the image processing device 120 does not generate a mask M21 and performs an operation similar to operation S22, transmitting the changing image received in operation S23 to the display device 130. Then, the display device 130 displays the changing image received in operation S23 on the screen F25, so that the screen F25 containing the changing image is consistent with the screen F23. When the image processing device 120 does not receive the changing image in operation S23, the image processing device 120 performs operation S24 to generate a mask M21.

[0026] Figure 3 is a schematic diagram 300 illustrating, according to some embodiments of the present invention, the change image transmitted by the remote device 110 and the mask M21 shown in Figures 1 and 2. As shown in Figure 3, schematic diagram 300 includes mask M31. In some embodiments, mask generation module 122 generates mask M31 according to schematic diagram 300.

[0027] The range of the mask M31 corresponds to the positions of different animated frames P31~P33 in the complete frame, as well as their width in the horizontal direction (e.g., the X direction) and height in the vertical direction (e.g., the Y direction). In some embodiments, the animated frames P31~P33 correspond to times t-2, t-1, and t, respectively, where time t-2 precedes time t-1 and time t-1 precedes time t. In some embodiments, each of the animated frames P31~P33 includes an object OB. In other words, the mask M31 includes at least the positions of the object OB at times t-2, t-1, and t. In some embodiments, the coordinates of the frame correspond to the X-coordinates and Y-coordinates of the frame located at the upper left corner. The X-coordinates increase horizontally from left to right, and the Y-coordinates increase vertically from top to bottom. The position L34, width, and height of the mask M31 correspond to the range of the mask M31.

[0028] Specifically, at time t-2, the changing screen P31 is located at position L31, and position L31 has coordinates (x t-2, y t-2), where coordinate x t-2 corresponds to the coordinates in the X direction, and coordinate y t-2 corresponds to the coordinates in the Y direction. The changing screen P31 has a width w t-2 in the horizontal direction and a height h t-2 in the vertical direction.

[0029] At time t-1, the changing screen P32 is located at position L32, and the changing screen P32 is located at coordinates (x t-1, y t-1), where coordinates x t-1 correspond to the coordinates in the X direction and coordinates y t-2 correspond to the coordinates in the Y direction.

[0030] At time t, the changing screen P33 is located at position L33, and the changing screen P33 is located at coordinates (xt, yt), where coordinate xt corresponds to the coordinates in the X direction and coordinate yt corresponds to the coordinates in the Y direction. The changing screen P33 has a width wt in the horizontal direction and a height ht in the vertical direction. In some embodiments, between times t-2 and t, the object OB moves from coordinates (xt-2, yt-2) through coordinates (xt-1, yt-1) to coordinates (xt, yt).

[0031] Referring to Figures 3 and 2, in some embodiments, time t-2 corresponds to operation S22. Time t-1 corresponds to operation S23. Time t corresponds to operation S24. Correspondingly, position L31 corresponds to position L22. Position L32 corresponds to position L23. Position L33 corresponds to position L24. Shifting screen P21 corresponds to shifting screen P31. Shifting screen P22 corresponds to shifting screen P33. The remote device 110 transmits shifting screens P31 and P33 to the image processing device 120. Mask M31 corresponds to mask M21. Mask M31 is located at position L34.

[0032] In some embodiments, the mask generation module 122 determines the position of the mask M31 and the width and height of the mask M31 in the X direction based on the positions of the changing images P31 and P33, the width of the changing image P31 in the X direction and the height in the Y direction, and the width of the changing image P33 in the X direction and the height in the Y direction.

[0033] Specifically, the mask generation module 122 determines the coordinates of the mask M31 in the X direction based on the smaller of the X-coordinates of the moving images P31 and P33. The mask generation module 122 further determines the coordinates of the mask M31 in the Y direction based on the smaller of the Y-coordinates of the moving images P31 and P33. The mask generation module 122 further determines the width of the mask M31 in the X direction based on the X-coordinates of the moving images P31 and P33 and the width of the moving image with the larger X-coordinate. The mask generation module 122 further determines the height of the mask M31 in the Y direction based on the Y-coordinates of the moving images P31 and P33 and the height of the moving image with the larger Y-coordinate.

[0034] For example, in the embodiment shown in Figure 3, the coordinates xt-2 of the changing image P31 in the X direction are greater than the coordinates xt of the changing image P33 in the X direction, and the coordinates yt of the changing image P33 in the Y direction are greater than the coordinates yt-2 of the changing image P31 in the Y direction. Correspondingly, the mask generation module 122 determines the position L34 of the mask M31 as coordinates (xt, yt-2) based on the coordinates (xt-2, yt-2) of position L31 and the coordinates (xt, yt) of position L33. Correspondingly, the mask generation module 122 determines that the mask M31 has a width wt-2+|xt-2-xt| in the X direction based on the width wt-2 of the changing screen P31 with coordinates xt-2, xt and the larger coordinate xt-2, and determines that the mask M31 has a height ht+|yt-2-yt| in the Y direction based on the height ht of the changing screen P33 with coordinates yt, yt-2 and the larger coordinate yt.

[0035] In some embodiments, the mask generation module 122 selects the smaller of coordinates xt-2 and xt by using the argument minimum, and selects the smaller of coordinates yt-2 and yt to determine the position L34 of the mask M31 as coordinates (xt, yt-2).

[0036] Referring to Figures 1 to 3, in some embodiments, the image processing device 120 generates a repair block image R21 based on screen F25 and the changing screens P21 and P22. The mask generation module 122 generates a mask M21 based on the changing screens P21 and P22. The processing module 124 generates the repair block image R21 based on screen F25 and mask M21.

[0037] Figure 4 is a flowchart illustrating a method 400 corresponding to the system 100 shown in Figure 1, according to some embodiments of this invention. The operation of method 400 will be described below using elements of system 100, but the embodiments of the present invention are not limited thereto. In various embodiments, method 400 may be performed by a system other than system 100. As shown in Figure 4, method 400 includes operations S41 to S46.

[0038] During operation S41, the image processing device 120 receives images from the remote device 110, such as the complete image F21, the changing images P21 and P22 shown in Figure 2.

[0039] In operation S42, the image processing device 120 determines whether the image received from the remote device 110 is a complete image. For example, the image processing device 120 determines that the complete image F21 is a complete image, and then executes operation S46. The image processing device 120 determines that the changing images P21 and P22 are changing images, and then executes operation S43.

[0040] During operation S43, the mask generation module 122 generates a mask based on the changing screens P21 and P22, such as the mask M21 in Figure 2.

[0041] In operation S44, the mask processing module 124 repairs the mask M21 based on the complete image F25 and the changed image P22 to generate the repair block image R21.

[0042] During operation S45, the masking module 124 transmits the repair block image R21 to the display device 130. The repair block image R21 is then overlaid on the screen F25.

[0043] During operation S46, the display device 130 displays the repair block image R21 and the complete screen F25, that is, the display device 130 displays the repair screen F27.

[0044] In some practices, when a remote device transmits an image, in order to save bandwidth, it usually only transmits the changing parts of the image, and only transmits the complete image at intervals. However, if a transmission problem occurs, causing one or more of the changing parts of the image to fail to arrive, ghosting may occur on the display. This problem can only be resolved when the next complete image arrives.

[0045] Compared to the above method, in this embodiment of the invention, the image processing device 120 generates a mask M21 by means of a mask generation module 122, and repairs the mask M21 by means of a processing module 124. In this way, the repaired image F27 displayed by the display device 130 is consistent with the image F24 of the remote device 110, thereby improving the image quality displayed by the display device 130.

[0046] Figure 5 is a schematic diagram of system 500 corresponding to system 100 shown in Figure 1, according to some embodiments of this case. Referring to Figures 5 and 1, system 500 is a variation of system 100. Image processing apparatus 520 is a variation of image processing apparatus 120. Processing module 524 is a variation of processing module 124. The labeling method in Figure 5 is similar to that in Figure 1. For the sake of brevity, the following discussion will focus on the differences between Figure 5 and Figure 1 rather than their similarities.

[0047] Compared to system 100, system 500 includes an image processing device 520 instead of image processing device 120. Image processing device 520 is coupled to remote device 110 and display device 130. Image processing device 520 includes a mask generation module 122, a processing module 524, a text cropping module 526, and a text and location storage module 528. Processing module 524 is coupled to mask generation module 122 and text and location storage module 528. Text and location storage module 528 is further coupled to text cropping module 526.

[0048] In operation, the text cropping module 526 uses a character recognition algorithm to identify the text portion of the complete image received from the remote device 110, extracts the text portion from the complete image as a text image, and stores the text image and its corresponding position in the text and position storage module 528. In some embodiments, the position of the text image includes the coordinates of the text image in the X direction, the coordinates in the Y direction, the width in the X direction, and the height in the Y direction.

[0049] The processing module 524 receives the masked image from the mask generation module 122, repairs the mask to generate a repaired image, extracts the corresponding text image from the text and position storage module 528 according to the position, height, and width of the mask, covers the position of the corresponding mask with the text image to generate a repaired image containing the text image, and then transmits the repaired image containing the text image to the display device 130. In some embodiments, the above module can be implemented by corresponding circuitry.

[0050] Figure 6 is a flowchart illustrating a method 600 corresponding to the system 500 shown in Figure 5, according to some embodiments of this invention. The operation of method 600 is described below using elements of system 500, but the embodiments of the present invention are not limited thereto. In various embodiments, method 600 can be executed by a system other than system 500. As shown in Figure 6, method 600 includes a complete screen F61, screens F62-F64, variable screens P61, P62, data D61, D62, and operations S61-S69. Operations S61-S69 are performed sequentially.

[0051] In some embodiments, the range of the changing screen P61 and the range of the changing screen P62 are respectively a part or all of the complete screen F61 and each of the screens F62 to F64. The content of the complete screen F61 and a part or all of the content of each of the screens F62 to F64 lies between the content of the changing screen P61 and the content of the changing screen P62. Screen F62 includes the changing screens P61 and P62. Screen F63 includes the mask M61. The repaired screen F64 includes the repaired block image R61.

[0052] Referring to Figures 1 to 6, during operation S61, the remote device 110 generates a complete image F61 and transmits it to the image processing device 520. Correspondingly, the image processing device 520 determines that the complete image F61 is a complete image, transmits it to the text cropping module 526 for processing, and then transmits it to the display device 130. The display device 130 then displays the complete image F61.

[0053] In operation S62, the text cropping module 526 identifies and extracts text frames W61 and W62 containing text portions from the complete screen F61, extracts the positions L61 and L62 corresponding to the text frames W61 and W62, and stores the text frames W61 and W62, and positions L61 and L62 as data D61. In the embodiment shown in Figure 6, the text frame W61 contains the text "Files_1". The text frame W62 contains the text "Doc-2". The position L61 has a position (x1, y1, w1, h1), indicating that the text frame W61 is located at the coordinates (x1, y1) of the complete screen F61, and has a width w1 and a height h1. Position L62 has a position (x 2, y 2, w 2, h 2), indicating that the text screen W62 is located at the coordinates (x 2, y 2) of the full screen F61, and has a width w 2 and a height h 2.

[0054] During operation S63, the text clipping module 526 stores the data D61 in the text and location storage module 528.

[0055] During operation S64, the mask generation module 122 receives screen F62, changing screen P61 and P62 sequentially from the remote device 110. In some embodiments, changing screen P61 corresponds to changing screen P21, and changing screen P62 corresponds to changing screen P22.

[0056] In operation S65, the mask generation module 122 generates a mask M61 based on the screen F62, the changing screen P61, and P62 in the manner described in schematic diagram 300, to generate screen F63. In the embodiment shown in Figure 6, the mask M61 is located at position L63, has a width wt1 in the X direction, and a height ht1 in the Y direction. Position L63 has (xt1, yt1), where coordinate xt1 corresponds to the coordinate in the X direction, and coordinate yt1 corresponds to the coordinate in the Y direction.

[0057] During operation S66, the processing module 524 repairs the mask M61 using generative image inpainting technology.

[0058] In operation S67, processing module 524 selects text images whose positions are partially or entirely within the area of ​​mask M61. Specifically, processing module 524 extracts data containing text positions within the area of ​​mask M61 from text and position storage module 528 based on the position L63 of mask M61, its width in the X direction, and its height in the Y direction. For example, in the embodiment shown in Figure 6, in the X direction, a portion or all of the range of coordinates x1 to x1+w1 lies between a portion or all of the range of coordinates xt1 to xt1+wt1, and in the Y direction, a portion or all of the range of coordinates y1 to y1+h1 lies between a portion or all of the range of coordinates yt1 to yt1+ht1. Correspondingly, processing module 524 extracts data D62 containing text images W61 and position L61 from text and position storage module 528.

[0059] During operation S68, the text and location storage module 528 provides data D62, including the text screen W61 and the location L61, to the processing module 524. In some embodiments, during operation S68, operation S65 is performed.

[0060] In operation S69, the processing module 524 repairs the mask M61 based on the text screen W61 and the position L61 to generate a repair block image R61, and then transmits the repair block image R61 to the display device 130. The display device 130 displays the repair block image R61 on the full screen F61, that is, the display device 130 displays the repair screen F64.

[0061] Figure 7 is a flowchart illustrating a method 700 corresponding to the system 500 shown in Figure 5, according to some embodiments of this invention. The operation of method 700 will be described below using the components of system 500, but the embodiments of the present invention are not limited thereto. In various embodiments, method 700 may be performed by a system other than system 500.

[0062] As shown in Figure 7, method 700 includes a complete image F71, images F72~F75, modified images P71, P72, repaired block images R71, R72, mask M71, and operations S71~S74. Mask M71 corresponds to mask M61. Operations S71~S74 are performed sequentially.

[0063] Please refer to Figures 1 to 7. During operation S71, the image processing device 520 receives the complete image F71, the variable image P71 and P72 sequentially from the remote device 110, generates the complete image F72, and then transmits the complete image F71, the variable image P71 and P72 to the mask generation module 122 for processing.

[0064] In operation S72, the mask generation module 122 generates a mask M71 based on the complete screen F71, the changing screen P71, and P72. The mask M71 is formed in the complete screen F71, and then the mask M71 and the complete screen F71 are transmitted to the processing module 524 for processing. In some embodiments, the mask generation module 122 generates the mask M71 according to the schematic diagram 300. The range of the mask M71 includes the range of the changing screen P71 and the range of the changing screen P72.

[0065] In operation S73, the processing module 524 repairs the mask M71 using generative image inpainting techniques to generate a repaired block image R71, wherein the range of the repaired block image R71 includes the range of the changed screen P71 and the range of the changed screen P72. In some embodiments, the text in the screen F74 may be different from the text in the original complete screen F71. For example, in the embodiment shown in Figure 7, the text in the original complete screen F71 is "Files_1", while the text in the repaired screen F74 is "Pllc5_1".

[0066] In operation S74, the processing module 524 extracts the text image and position (e.g., text image W61 and position L61) corresponding to the mask M71 from the text and position storage module 528 according to the range of the mask M71 (i.e., the range of the repair block image R71). Based on the text image and position of the corresponding mask M71, the processing module 524 repairs the mask M71 to generate the repair block image R72, and then transmits the repair block image R72 to the display device 130. The display device 130 displays the repair block image R72 on the complete screen F71, i.e., the display device 130 displays the repair screen F75. In the embodiment shown in Figure 7, the text in the repair screen F75 is "Files_1", which is consistent with the text in the original complete screen F71.

[0067] Figure 8 is a flowchart illustrating a method 800 corresponding to the system 500 shown in Figure 5, according to some embodiments of this invention. The operation of method 800 will be described below using elements of system 500, but the embodiments of the present invention are not limited thereto. In various embodiments, method 800 may be performed by a system other than system 500. As shown in Figure 8, method 800 includes operations S81 to S88.

[0068] During operation S81, the image processing device 520 receives images from the remote device 110, such as the complete image F61, the changing images P61 and P62 shown in Figure 6.

[0069] In operation S82, the image processing device 520 determines whether the image received from the remote device 110 is a complete image. For example, if the image processing device 520 determines that the complete image F61 is a complete image, it displays the complete image F61 on the display device 130, and then executes operation S83. The image processing device 520 determines that the changing images P61 and P62 are changing images, and then executes operation S84.

[0070] In operation S83, the text cropping module 526 extracts the text portion of the complete screen F61 and stores the text image and the position of the text in the text and position storage module 528, and then performs operation S86.

[0071] During operation S84, the mask generation module 122 generates mask M61 based on the changing screens P61 and P62.

[0072] During operation S85, the masking module 524 repairs the mask M61 based on the complete screen F61, the modified screen P61 and P62.

[0073] During operation S86, the mask processing module 524 uses the text image and text position of the storage module 528 and the mask M61 to repair the mask M61 to generate the repair block image R61.

[0074] During operation S87, the masking module 524 transmits the repair block image R61 to the display device 130. The repair block image R61 is then overlaid on the full screen F61.

[0075] During operation S88, the display device 130 displays the complete screen F61 and the repair block image R61, that is, the display device 130 displays the repair screen F64.

[0076] In some methods, when a remote device transmits an image, the processor does not extract the text portion of the image to memory. In the above methods, the text in the image is usually treated as an image for repair. However, treating the text in an image as an image for repair results in unclear text after repair.

[0077] Compared to the above approach, in this embodiment of the invention, the image processing device 520 extracts the text from the complete image using the text cropping module 526, stores the text and its position using the text and position storage module 528, and repairs the complete image of the display device 130 based on the stored text and its position using the processing module 524. As a result, the text displayed on the display device 130 is clear, thus improving the image quality.

[0078] Although the present disclosure has been disclosed above with reference to embodiments, it is not intended to limit the present disclosure. Anyone with ordinary knowledge in the art may make some modifications and refinements without departing from the spirit and scope of the present disclosure. Therefore, the scope of protection of the present disclosure shall be determined by the appended claims. [Simplified Explanation of the Diagram]

[0005] Figure 1 is a schematic diagram of a system according to some embodiments of this invention. Figure 2 is a flowchart of a method corresponding to the system shown in Figure 1. Figure 3 is a schematic diagram related to the changing screen and mask transmitted by the remote device, corresponding to Figures 1 and 2. Figure 4 is a flowchart of a method corresponding to the system shown in Figure 1. Figure 5 is a schematic diagram of a system according to some other embodiments of this invention. Figure 6 is a flowchart of a method corresponding to the system shown in Figure 5. Figure 7 is a flowchart of a method corresponding to the system shown in Figure 5. Figure 8 is a flowchart of a method corresponding to the system shown in Figure 5. [Biomaterial Storage]

[0080] Domestic storage information (please note in order of storage institution, date, and number): None. International storage information (please note in order of storage country, institution, date, and number): None.

Claims

1. A method for processing an image, comprising: receiving a complete image, a first variable image, and a second variable image, wherein the range of the first variable image and the range of the second variable image are respectively a part or all of the complete image; generating a mask corresponding to the complete image based on the first variable image and the second variable image, wherein the size of the mask corresponds to the first variable image and the second variable image; and generating a repaired image based on the mask and the complete image, wherein a part or all of the content of the repaired image is between the content of the first variable image and the content of the second variable image.

2. The method as described in claim 1 further comprises: generating a repair block image based on the mask; and setting the repair block image at a corresponding position in the complete image to generate the repaired image.

3. The method as described in claim 1 further comprises: identifying at least one character in the complete screen; obtaining at least one character image corresponding to the at least one character from the complete screen based on the position of the at least one character; and displaying the at least one character image on the repair screen based on the position of the at least one character.

4. The method described in claim 3 further comprises: storing the at least one text image and the at least one text position via a storage module; and providing the at least one text image and the at least one text position via the storage module when generating the repair image.

5. The method as described in claim 1 further comprises: generating the repaired screen based on a first changed screen position on the complete screen and a second changed screen position on the complete screen.

6. The method of claim 5 further comprises: in a first direction, selecting one of a first coordinate component of the first changed screen position and a second coordinate component of the second changed screen position as a first coordinate of a repair block image; and in a second direction, selecting one of a third coordinate component of the first changed screen position and a fourth coordinate component of the second changed screen position as a second coordinate of the repair block image, wherein the first direction is different from the second direction.

7. An image processing apparatus, comprising: a processing module for receiving a complete image, a first changing image, and a second changing image; and a mask generation module coupled to the processing module for generating a mask based on the first changing image and the second changing image, the size of the mask corresponding to the first changing image and the second changing image; wherein... The processing module generates a repaired image based on the mask and the complete image; wherein the range of the first modified image and the range of the second modified image are respectively a part or all of the complete image, and the content of the repaired image is between the content of the first modified image and the content of the second modified image.

8. The image processing apparatus of claim 7, wherein the processing module generates a repair block image based on the mask, and sets the repair block image at a position corresponding to the complete image to generate the repaired image.

9. The image processing apparatus as claimed in claim 8 further comprises: a text cropping module for identifying at least one character in the complete image and extracting a text image of the at least one character from the complete image based on the position of the at least one character; and a storage module for storing the text image of the at least one character and the position of the at least one character, and providing the text image of the at least one character and the position of the at least one character to the processing module.

10. The image processing apparatus of claim 9, wherein the processing module further determines the extent of the repair block image based on one of the width and height of the first changing image, one of the width and height of the second changing image, the position of the first changing image, and the position of the second changing image.

Images