Image super-resolution processing method and related device

Abstract

The present application relates to the technical field of image forming devices, and particularly relates to an image super-resolution processing method and related equipment. The method is applied to an image forming device, and comprises the following steps: obtaining a super-resolution configuration parameter, a job processing specification parameter, and a content type of an original document; wherein the super-resolution configuration parameter at least comprises an enhancement level; selecting a target model from a plurality of preset super-resolution models according to the job processing specification parameter and the content type; obtaining a scanned image of the original document; inputting the super-resolution configuration parameter and the scanned image into the target model to obtain a first image output by the target model; and performing fusion processing on the first image and the scanned image based on the enhancement level to generate a to-be-output image.

Description

Technical Field

[0001] This invention relates to the field of image forming apparatus technology, and in particular to an image super-resolution processing method and related equipment. Background Technology

[0002] In practical applications of image forming apparatuses, it is often necessary to enlarge or super-resolution images. Typical applications include copying smaller originals (such as A5 size) onto larger sheets of paper (such as A4 or A3), or printing images scanned at lower resolution (such as 300 dpi) at higher resolution (such as 600 dpi).

[0003] Currently, the industry commonly uses traditional interpolation algorithms and other image scaling algorithms to address this type of image enlargement requirement. The core principle is to estimate the value of new pixels based on the known values ​​and positions of pixels in the original image using a mathematical rule, thereby increasing the image's pixel size. However, these traditional methods have certain drawbacks: because the algorithm is essentially an estimation based on mathematical derivation rather than an intelligent understanding and reconstruction of image content, the enlarged image often exhibits significant loss of detail and texture blurring. While the output may meet the size requirements, visual clarity is significantly reduced, becoming smooth and lacking sharpness, failing to restore the true appearance of the original image and degrading the user experience. Summary of the Invention

[0004] Therefore, embodiments of the present invention provide an image super-resolution processing method and related equipment, which can improve the super-resolution processing effect.

[0005] In a first aspect, embodiments of the present invention provide a super-resolution processing method, applied to an image forming apparatus, comprising: Obtain super-resolution configuration parameters and job processing specification parameters, as well as the content type of the original document; wherein, the super-resolution configuration parameters include at least the enhancement level; Based on the job processing specifications and the content type, a target model is selected from multiple preset super-resolution models; Obtain a scanned image of the original document; The super-resolution configuration parameters and the scanned image are input into the target model to obtain the first image output by the target model. Based on the enhancement level, the first image and the scanned image are fused to generate an image to be output.

[0006] In one possible implementation, the content type includes one or more combinations of text, mixed text and graphics, patterns, and portraits.

[0007] In one possible implementation, the job processing specifications include original size, output size, scanning resolution, and output resolution. In one possible implementation, the super-resolution model has a fixed magnification factor and applicable type; the step of selecting a target model from multiple preset super-resolution models based on the job processing specification parameters and the content type includes: Determine the target magnification factor based on the aforementioned job processing specifications; The super-resolution model with the magnification factor equal to the target magnification factor and whose applicable type matches the content type among the multiple preset super-resolution models is determined as the target model.

[0008] In one possible implementation, the method prior to obtaining a scanned image of the original document further includes: The front-end scaling factor is determined based on the job processing specification parameters and the target magnification factor; The original document is scanned to obtain the original scanned image; The original scanned image is preprocessed according to the front-end scaling factor to obtain the scanned image.

[0009] In one possible implementation, the step of fusing the first image and the scanned image based on the enhancement level to generate an output image includes: The back-end scaling factor is determined based on the job processing specification parameters and the target magnification factor; The size of the first image is adjusted according to the back-end scaling factor; The scanned image is magnified based on a preset super-resolution algorithm to obtain a second image; the magnification factor of the preset super-resolution algorithm is determined by the target magnification factor. The second image and the resized first image are weighted and fused according to the enhancement level to obtain the image to be output.

[0010] In one possible implementation, the step of weighted fusion of the second image and the resized first image according to the enhancement level to obtain the output image includes: The weighting weights of the first image and the second image are determined based on the enhancement level; wherein, the larger the value of the enhancement level, the larger the weighting weight of the first image. The first image and the second image are fused according to the weighted weights to obtain the image to be output.

[0011] In one possible implementation, the magnification factor of the super-resolution algorithm is the product of the target magnification factor and the back-end scaling factor.

[0012] In one possible implementation, the image forming apparatus is provided with multiple computing units, and the step of inputting the super-resolution configuration parameters and the scanned image into the target model includes: The computational resource requirements are determined based on the size of the scanned image and the target model; Based on the computational resource requirements and the pre-obtained available computational resources of the image forming apparatus, it is determined whether the scanned image needs to be segmented into blocks; If it is determined that the scanned image needs to be segmented into blocks, then the scanned image is divided into multiple image blocks with overlapping areas; The image blocks are assigned to the plurality of processing units.

[0013] In one possible implementation, assigning the image patch to the plurality of processing units includes: The allocation ratio of each computing unit is determined based on its computing performance. The image blocks are allocated to the plurality of processing units according to the allocation ratio.

[0014] In a second aspect, embodiments of the present invention provide an image forming apparatus, comprising: The first acquisition module is used to acquire super-resolution configuration parameters and job processing specification parameters, as well as the content type of the original document; wherein, the super-resolution configuration parameters include at least the enhancement level; The model selection module is used to select a target model from multiple preset super-resolution models based on the job processing specification parameters and the content type. The second acquisition module is used to acquire a scanned image of the original document; The processing module is used to input the super-resolution configuration parameters and the scanned image into the target model to obtain the first image output by the target model; The generation module is used to fuse the first image and the scanned image based on the enhancement level to generate an image to be output.

[0015] Thirdly, embodiments of the present invention provide an electronic device, comprising: At least one processor; and At least one memory communicatively connected to the processor, wherein: The memory stores program instructions that can be executed by the processor, and the processor can execute the method described in the first aspect by calling the program instructions.

[0016] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer instructions that cause the computer to perform the method described in the first aspect.

[0017] In this embodiment of the invention, a suitable super-resolution model is selected by acquiring job processing specification parameters and super-resolution configuration parameters, thereby performing more suitable super-resolution processing on the scanned image. Furthermore, image fusion is performed on the first image output by the target model and the scanned image according to the enhancement level, increasing the clarity of the final output image without losing the original image details, thus improving the user experience. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart of an image super-resolution processing method provided in an embodiment of the present invention; Figure 2 A schematic diagram of a job configuration interface provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a super-resolution image enhancement interface provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a document classification model provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of a scaling factor configuration provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of a block slicing method provided in an embodiment of the present invention; Figure 7 A schematic diagram of a super-resolution processing architecture for scanned images provided in an embodiment of the present invention; Figure 8a This is a schematic diagram of a super-resolution processing result provided in an embodiment of the present invention; Figure 8b This is a schematic diagram of another super-resolution processing result provided by an embodiment of the present invention; Figure 8c This is a schematic diagram of another super-resolution processing result provided by an embodiment of the present invention; Figure 9 This is a schematic diagram of the structure of an image forming apparatus provided in an embodiment of the present invention; Figure 10This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0021] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0022] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0023] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0024] It should be understood that although terms such as first, second, third, etc., may be used to describe numbers in embodiments of the present invention, these numbers should not be limited to these terms. These terms are only used to distinguish numbers from each other. For example, without departing from the scope of embodiments of the present invention, a first number may also be referred to as a second number, and similarly, a second number may also be referred to as a first number.

[0025] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0026] To address the issues of poor performance in super-resolution processing of scanned images by image forming apparatuses, and the inability of the generated super-resolution images to accurately reproduce the original images, this invention provides an image super-resolution processing method that can improve the super-resolution processing performance of image forming apparatuses. Figure 1 This is a flowchart illustrating an image super-resolution processing method provided in an embodiment of the present invention. Figure 1As shown, the method is applied to an image forming apparatus, including: Step 101: Obtain the super-resolution configuration parameters, job processing specification parameters, and the content type of the original document. The super-resolution configuration parameters are manually set by the user. Default values ​​can be configured for the super-resolution configuration parameters. Users can change the super-resolution configuration parameters manually. The super-resolution configuration parameters include at least the enhancement level. The content type is either set by the user or automatically identified by the image forming apparatus.

[0027] The main specifications for job processing include original document size, output size (also known as print size or paper size), scanning resolution, and output resolution. Content types include text, mixed text and images, patterns, and portraits.

[0028] In some embodiments, when a user issues a scanning or copying job on the operation panel of the image forming apparatus or on a terminal device (such as a personal computer, tablet computer, mobile terminal, or other smart device) connected to the image forming apparatus, the user can set whether super-resolution processing is required for the current job, as well as set the job processing specifications and the content type of the original document. Figure 2 This is a schematic diagram of a job configuration interface provided in an embodiment of the present invention.

[0029] like Figure 2 As shown, the user sets the job type (including copying, scanning, printing, etc.) to copying. After performing operations on the operation panel of the image forming apparatus, the image forming apparatus displays... Figure 2 The configuration interface is shown below. The user sets the original document size to A5, the paper selection to A3, and enables super-resolution image enhancement. Figure 2 The paper selection option shown is the output size in the job processing specification parameters of the above embodiment.

[0030] Users also need to set some parameters when issuing jobs. Figure 2Parameters not shown in the diagram. For example, the user also needs to set the orientation of the original document and the paper orientation, including the long side and the short side. The original document orientation indicates the placement direction of the original document placed in the scanning area of ​​the image forming apparatus. The paper orientation indicates the placement direction of the paper in the paper tray. Another example is that the user also needs to set the scan / print resolution. For scanning jobs, the user can set the scan resolution (e.g., 300 dpi, 600 dpi, 1200 dpi, etc.). For copying jobs, the user needs to set the scan resolution and print resolution. The scan resolution can be set by the user or default to the maximum resolution of the scanning element of the image forming apparatus. The print resolution is set by the user according to their needs. Yet another example is that the user can also set the color type, including color, grayscale, black and white, etc.

[0031] When the user selects to enable super-resolution image enhancement, the super-resolution image enhancement configuration interface is displayed. Figure 3 This is a schematic diagram of a super-resolution image enhancement interface provided in an embodiment of the present invention. Figure 3 As shown, users can select the content type of the original document in the "Type" column. These include options such as Automatic, Text, Text + Image (i.e., mixed text and image content types in this application), Pattern, and Portrait. (Except for...) Figure 3 In addition to the options shown, users can also set super-resolution configuration parameters such as texture, sharpness, clarity, and visual quality to fine-tune the super-resolution processing.

[0032] In some embodiments, when the user is Figure 3 When the "Type" item is selected as "Automatic" in the interface shown, it means that the content type of the original document is automatically recognized by the image forming apparatus. For Figure 3 The enhancement level shown adjusts the effect of super-resolution processing, with a value ranging from 0 to 1. Specifically, the closer the enhancement level is to 0, the closer the final output image is to the result of traditional image scaling in existing technologies. The closer the enhancement level is to 1, the higher the quality of the final output image. However, excessively high enhancement levels may lead to more creative super-resolution processing results, generating new details adapted to the document content, which may deviate from the original document to some extent.

[0033] Step 102: Select the target model from multiple preset super-resolution models based on the job processing specifications and content type.

[0034] In some embodiments, the super-resolution model has a fixed magnification factor and applicable type. A super-resolution model for a specific applicable type provides better super-resolution performance for scanned images with a specific content type. For example, a super-resolution model applicable to images provides the best super-resolution performance for scanned images whose content type is images. The magnification factor of the super-resolution model may include 1×1. × And sizes such as 2×2. Because the multiples between the various standard paper types are... Or 2 (for example, the magnification factor of the width / height when enlarging an A5 paper size to an A4 paper size is ). Since the magnification factor (width / height) of an A5 paper size enlarged to an A3 paper size is 2, the fixed magnification factor for the super-resolution model is set to several specifications in Table 1-1. Table 1-1 shows the applicable types of some super-resolution models and their corresponding magnification factors.

[0035] Table 1-1 As shown in Table 1-1, different super-resolution models have different fixed magnification factors and applicable types. Therefore, when selecting a target model from multiple preset super-resolution models based on job processing specifications and content type, the target magnification factor can be determined according to job processing specifications such as original size, output size, scanning resolution, and output resolution. The target magnification factor is the total magnification ratio required for this super-resolution processing, calculated based on the above parameters. Its purpose is to be used for subsequent matching of target models with corresponding fixed magnification factors. For example, if the original size is A5, the output size is A4, and both the scanning resolution and output resolution are 600dpi × 600dpi, then the target magnification factor can be determined as follows: × .

[0036] Next, among several preset super-resolution models, the super-resolution model with the target magnification factor and whose applicable type matches the content type is determined as the target model. For example, the target magnification factor is... × If the content type is text, then the super-resolution model with the model name text_xq2xq2 in Table 1-1 can be selected as the target model.

[0037] The 1×1 super-resolution model is mainly used for super-resolution processing of images that are blurry in the original document. For example, if a user enlarges and copies a clear A5-sized original onto A4 paper using a traditional super-resolution method, and then uses the method provided in this embodiment of the invention to copy the blurry A4-sized copy onto A4 paper again, the clarity can be improved to a level close to that of the original A5 original.

[0038] Step 103: Obtain a scanned image of the original document.

[0039] The image forming apparatus scans the original document placed in the scanning area according to user settings, thereby obtaining a scanned image of the original document. Specifically, the front-end scaling factor can be determined based on job processing specifications such as original document size, output size, scanning resolution, and output resolution, as well as the target magnification factor. The front-end scaling factor is mainly used to adjust for situations where the width and resolution are asymmetrical. For example, if the scanning resolution is 600 dpi × 300 dpi, it needs to be adjusted to 600 dpi × 600 dpi according to the front-end scaling factor. Afterward, the original document is scanned again to obtain the raw scanned image. Finally, the size of the raw scanned image is adjusted according to the front-end scaling factor (i.e., preprocessing) to obtain the final scanned image. After obtaining the raw scanned image, it can also be determined whether image adjustment is needed based on the original document orientation and output orientation. Image adjustment can include image correction, image enhancement, and image rotation. During execution, the drive engine in the image forming apparatus sends job processing specification parameters and front-end scaling factors to the scanning engine, image sensor, processor, and peripheral components (such as contact image sensor (CIS), charge-coupled device (CCD), analog front end (AFE), field-programmable gate array (FPGA), etc.).

[0040] In some embodiments, if the user selects automatic content type recognition, the image forming apparatus needs to further recognize the scanned image of the original document after obtaining it, thereby determining the content type of the original document. Specifically, the scanned image can be preprocessed first. Preprocessing includes steps such as color conversion, rotation, cropping, and scaling to a single-channel, fixed-size image suitable for rapid content classification. Then, a pre-trained document classification model is used for recognition to determine the content type of the scanned image.

[0041] Figure 4 This is a schematic diagram of the structure of a document classification model provided in an embodiment of the present invention. Figure 4 As shown, the model consists of nine interconnected layers and can recognize ten content types, such as photos, image + text, text, and tables. The nine layers of the model, from top to bottom, are: first convolutional layer, first pooling layer, second convolutional layer, second pooling layer, third convolutional layer, third pooling layer, flattening layer, first fully connected layer, and second fully connected layer.

[0042] The first convolutional layer is denoted as Conv2D 32x(3,3). This is a two-dimensional convolutional layer with 32 filters and a 3×3 kernel (sampling size). The first pooling layer is denoted as MaxPooling2D(2,2). This is a two-dimensional max pooling layer with a pooling size of 2×2. The second convolutional layer is denoted as Conv2D 64x(3,3). This is a two-dimensional convolutional layer with 64 filters and a 3×3 kernel. The second pooling layer is denoted as MaxPooling2D(2,2). This is a two-dimensional max pooling layer with a pooling size of 2×2. The third convolutional layer is denoted as Conv2D 128x(3,3). This is a two-dimensional convolutional layer with 128 filters and a 3×3 kernel. The third pooling layer is denoted as MaxPooling2D(2,2). This is a two-dimensional max pooling layer with a pooling size of 2×2. Flatten layer. The first fully connected layer, denoted as Dense(128), is a fully connected layer with 128 units. The second fully connected layer, denoted as Dense(10), softmax, is a fully connected layer with 10 units, using the softmax activation function.

[0043] Figure 4 In the document classification model shown, the input is a single-channel grayscale image of size 181×256, and the output is one of 10 content types. The output data shapes of each layer of the model are as follows: First convolutional layer Conv2D 32x(3,3): 179, 254, 32. First pooling layer MaxPooling2D(2,2): 89, 127, 32. Second convolutional layer Conv2D 64x(3,3): 87, 125, 64. Second pooling layer MaxPooling2D(2,2): 43, 62, 64. Third convolutional layer Conv2D 128x(3,3): 41, 60, 128. Third pooling layer MaxPooling2D(2,2): 20, 30, 128. Flatten layer: 76800. First fully connected layer Dense(128): 128. The second fully connected layer is Dense(10), with softmax:10.

[0044] Step 104: Input the super-resolution configuration parameters and the scanned image into the target model to obtain the first image output by the target model.

[0045] The target model performs super-resolution processing on the scanned image based on the enhancement level and parameters such as texture, sharpness, clarity, and visual quality in the super-resolution configuration parameters, outputting a high-resolution first image. This first image is an optimized high-resolution image generated by the target model. Compared to traditional super-resolution methods, the first image is clearer in terms of detail restoration and visual quality, serving as the basis for generating the final output image.

[0046] Step 105: The first image and the scanned image are fused based on the enhancement level to generate an output image. After obtaining the output image, the image forming apparatus outputs the output image according to the job type. For example, if the job type is a photocopy job, the image forming apparatus prints the output image. Alternatively, if the job type is a scan job, the image forming apparatus outputs the output image to its own storage module for storage or outputs it to other terminals.

[0047] In some cases, there may be special dimensions in the output size and original size, such as letter paper size and SRA3 paper size. The magnification factor between these special paper sizes and common paper sizes such as A4 is not one of the common magnification factors. Therefore, in this case, it is necessary to fine-tune the first image output from the target model to achieve super-resolution magnification at any size. Specifically, this process includes: determining the back-end scaling factor based on the job processing specifications such as original size, output size, scanning resolution, and output resolution, as well as the target magnification factor. The back-end scaling factor is a supplementary scaling factor used to correct for scenarios where the inherent magnification factor of the target model differs from the total magnification ratio required by the actual job. Its function is to fine-tune the first image output from the target model to a precise resolution that perfectly matches the final output size.

[0048] The size of the first image is then adjusted according to the scaling factor in the backend. Next, the scanned image is magnified based on a preset super-resolution algorithm to obtain a second image. This second image is generated by directly magnifying the image using traditional super-resolution methods such as interpolation algorithms. This method can stably maintain the original structure and content, but the detail enhancement is limited. The magnification factor of the preset super-resolution algorithm is determined by the target magnification factor. Finally, the second image obtained through traditional super-resolution processing and the first image output by the target model are weighted and fused according to the enhancement level to obtain the final output image. The preset super-resolution algorithm is a conventional interpolation algorithm or other algorithm used by the image forming device, such as bicubic interpolation. By weightedly fusing the second image obtained through traditional super-resolution processing with the first image generated by the target model, a balance is achieved between the enhancement effect and the fidelity of the original content, thus obtaining a super-resolution effect that meets the user's expectations.

[0049] In one possible embodiment, a user needs to print a target image on A4 paper for clear display. However, during the actual printing process, it is found that the image output on A4 paper has low clarity and cannot meet the user's needs. Conversely, printing the target image on A5 paper results in higher clarity and better meets the user's requirements. In this case, the user can first print the target image on A5 paper and then place the printed A5 paper into an image forming apparatus to perform the image super-resolution processing method provided in this application. The original size is set to A5, and the output size is set to A4. This allows the image on the A5 paper to be adapted to the A4 size through super-resolution processing, thereby obtaining a clear output effect on a larger sheet of paper.

[0050] Alternatively, users can choose to directly print the target image on A4 paper, and then place the A4 paper with the target image printed on it into an image forming apparatus to perform the image super-resolution processing method provided in this application. The original size is set to A4, and the output size is also set to A4, so as to perform super-resolution processing on the A4 original at the same size and obtain A4 paper with improved target image clarity. Users can choose either of the above two processing methods according to the actual printing conditions to achieve the purpose of clearly displaying the target image on A4 paper.

[0051] Figure 5 This is a schematic diagram of a scaling factor configuration provided in an embodiment of the present invention.

[0052] like Figure 5 As shown, the target size is the same as the output size described above. The scanning sensor resolution is the same as the scanning resolution described above. The printing imaging resolution is the same as the output resolution described above.

[0053] When performing weighted fusion of the first and second images, the weighting weights of the first and second images can be determined based on the enhancement level. A higher enhancement level value results in a larger weighting weight for the first image and a smaller weighting weight for the second image. Then, the first and second images are fused according to these weighting weights to obtain the output image. For example, if the user sets the enhancement level to 0.8, the weighting weight of the first image is 0.8, and the weighting weight of the second image is 1 - 0.8 = 0.2. When the user sets the enhancement level to 1, the weighting weight of the first image is 1, and the weighting weight of the second image is 0; the resulting output image is the first image.

[0054] In the above embodiments, the first image needs to be scaled based on the back-end scaling factor before being merged with the second image. Therefore, the first image and the second image have the same size after being scaled by the back-end scaling factor. Both the preset super-resolution algorithm and the target model enlarge the scanned image. Therefore, the magnification factor of the preset super-resolution algorithm is the product of the target magnification factor and the back-end scaling factor. For example, if the inherent magnification factor of the target model is 2 and the back-end scaling factor is 0.94, then the magnification factor of the preset super-resolution algorithm is 2 × 0.94 = 1.88.

[0055] In some embodiments, the magnification factor of the preset super-resolution algorithm can also be set to be the same as the magnification factor of the target model. In this case, the first image and the second image are fine-tuned together based on the back-end magnification factor. That is, the first image and the second image are first scaled based on the back-end scaling factor before image fusion is performed.

[0056] Optionally, if the preset super-resolution algorithm uses the same magnification factor as the target model, the first image and the second image can be fused first, and then the fused image can be scaled using the back-end scaling factor to obtain a fused image that meets the output size.

[0057] In this embodiment, the scanned image is magnified using a super-resolution model and a traditional super-resolution algorithm, and then weighted and fused. The weights for weighted fusion are determined based on the enhancement level. The image forming apparatus can generate scanned and copied images with significantly higher clarity than traditional interpolation methods, especially when image magnification is required during the job. Furthermore, it allows users to adjust the image generation process according to the document content, improving the user experience. In some embodiments, a super-resolution model can be built using a generative adversarial neural network (GAN). Its architecture consists of three parts: a generator, a discriminator, and a perceptual loss. The generator converts low-resolution images into high-resolution images, the discriminator evaluates the images during training, and the perceptual loss is the gradient calculated by the generator and discriminator during training. These two models compete against each other during training, simultaneously completing the training process to achieve the goal of making the generator produce visual effects that more closely resemble real images.

[0058] Specifically, the generator can be composed of convolutional layers, ReLU activation layers, N residual network layers, and an upsampling layer (pixel recombination) connected in series. The generator takes a low-resolution image as input and outputs a high-resolution image. The discriminator is composed of multiple composite layers connected in series. Each composite layer consists of a convolutional layer (Conv), a batch normalization layer (BN), and a ReLU activation layer. The discriminator outputs a true / false judgment result. The generator's high-resolution output image and the real high-resolution image are input together into the perceptual loss function to obtain the gradient loss generated during training.

[0059] Based on the above training architecture, super-resolution models with different magnification coefficients and applicable types are obtained through model training. Specifically, training data is first collected. The training data corresponds to the content type, including plain text, plain images, mixed text and images, and original high-resolution scanned portrait images of various styles. Then, the training data is preprocessed, specifically by cropping the training data to the model training size (e.g., 256×256) to serve as real high-resolution images. The training data was then subjected to lossy compression and decompression, rotation, mirroring, and multiple scaling processes to obtain 1 / 1 and 1 / Low-resolution images with a 1 / 2 scaling factor are used. A training dataset is constructed based on the real high-resolution images and corresponding low-resolution images with the specified scaling factors from the training data. The initial model is then trained using the training dataset with a single scaling factor to obtain a general super-resolution model for the corresponding scaling factor. This general super-resolution model can perform super-resolution processing on scanned images of all content types at the specified scaling factor. Subsequently, the general super-resolution model is retrained using training data of a single content type from the training dataset to obtain a super-resolution model for the corresponding content type.

[0060] The above steps can yield a general super-resolution model with different fixed magnification factors applicable to all content types, as well as a super-resolution model with different fixed magnification factors applicable to different content types obtained after secondary training based on the general super-resolution model.

[0061] In some embodiments, the image forming apparatus may include different computing units, such as a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a Neural Processing Unit (NPU). Therefore, before inputting the scanned image into the target model for super-resolution processing, it is necessary to determine whether the current computing resources of the image forming apparatus can directly perform super-resolution processing on the entire scanned image. Specifically, the computing resource requirements are determined based on the size of the scanned image and the target model. Then, based on the computing resource requirements and the pre-obtained available computing resources of the image forming apparatus, it is determined whether the scanned image needs to be sliced ​​into blocks. If it is determined that slicing the scanned image is necessary, the scanned image is divided into multiple image blocks with overlapping regions.

[0062] The image blocks are then allocated to multiple processing units. When the computational resource demand exceeds the available computational resources, it indicates that the image forming apparatus cannot directly perform super-resolution processing on the entire scanned image. It is necessary to divide the image into blocks and then perform super-resolution processing on the resulting image blocks.

[0063] When allocating image blocks to various processing units, the allocation ratio for each processing unit can be determined based on its processing performance. Then, image blocks are allocated to multiple processing units according to this allocation ratio. This ensures that all parallel processing units complete their operations as simultaneously as possible, thereby improving the computational efficiency of super-resolution processing.

[0064] Figure 6 This is a schematic diagram of a block-slicing method provided in an embodiment of the present invention. For example... Figure 6 As shown, a scanned image with length *width* and height *height* is sliced ​​into several image blocks, with some overlap between adjacent blocks. Then, all image blocks are divided into N groups, each group containing *batch_size* image blocks. Finally, the N groups of image blocks are allocated to different processing units based on their processing performance. For example, if the GPU's processing performance is 5 times that of the CPU, the N groups of image blocks can be allocated to the GPU and CPU in a 5:1 ratio.

[0065] Subsequently, multiple computational units perform super-resolution processing using the target model, obtaining N×batch_size super-resolution output results. Finally, the N×batch_size super-resolution output results are concatenated, discarding overlapping portions between corresponding image blocks and only concatenating non-overlapping portions to obtain the final super-resolution result of the scanned image, i.e., the first image mentioned above.

[0066] Corresponding to the above-described scanning image super-resolution processing method, this embodiment of the invention provides a scanning image super-resolution processing architecture for executing the above-described scanning image super-resolution processing method.

[0067] Figure 7 This is a schematic diagram of a scanned image super-resolution processing architecture provided in an embodiment of the present invention. Figure 7 As shown, the processing architecture consists of a scanner, a print engine, a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), memory, storage, and a user interface.

[0068] The scanner is used to acquire images of the original document and transmits the acquired data to the internal driver of the image forming apparatus through the corresponding scanner interface. The print engine is used to print the image to be output onto imaging media such as paper. The print engine receives print commands from the internal driver of the image forming apparatus through the printer interface shown in the figure to execute the print output.

[0069] The Central Processing Unit (CPU), Graphics Processing Unit (GPU), and Neural Processing Unit (NPU) are the processing units that execute image super-resolution processing. Memory is used to store cached data during the image super-resolution processing. Storage is used to store firmware, image processing programs, parameters, model weight data, and other data.

[0070] A user interface is the module through which a user interacts with an image forming apparatus. For example... Figure 7 As shown, the user interface communicates via USB, Ethernet, Wi-Fi / BT, LCD panel, and other communication methods. User actions are translated into instructions and transmitted to the central processing unit (CPU).

[0071] In a specific example, the super-resolution processing flow in this embodiment of the invention for a scenario where a user copies an original document containing text content on A5 paper to A4 paper is as follows: The user sets the job processing specifications on the operation panel of the image forming device: copy, color, original size A5, paper selection A4, automatic scaling, super-resolution enhancement enabled, enhancement type: text, enhancement level 0.8. After the user clicks the "copy" button, the image forming device starts the job. First, the driver program of the image forming device detects that the original placed by the user in the scanning area is oriented along the long side of A5, while the paper in the paper tray is oriented along the short side of A4, requiring rotation. Therefore, the driver program determines the super-resolution parameters, front-end parameters, and back-end parameters to perform the job. The super-resolution parameters include: content type: text, scaling factor (i.e., target magnification factor): × Front-end parameters: A5 long side, color, scan resolution: 600dpi × 600dpi, front-end scaling factor: 1.0 × 1.0, rotation 90°. Back-end parameters: color, output resolution: 600dpi × 600dpi, back-end scaling factor: 1.0 × 1.0.

[0072] Then, the driver loads the "text, magnification factor" from memory. × The super-resolution model is sent to the processing unit, and the front-end and back-end parameters are set to the image processing module to begin image acquisition and processing. The driver program controls the scanner to acquire and process scanned images, which are then processed by the super-resolution model to obtain the first image. Simultaneously, the driver program uses a bicubic interpolation algorithm to interpolate and enlarge the scanned image to obtain the second image. Then, the driver program fuses the first image with a weighting of 0.8 and the second image with a weighting of 0.2 to obtain the output image. Finally, the driver program sends the output image to the printing module for image processing and printing control, ultimately printing the output image onto a short-side A4 sheet of paper. Figure 8a , Figure 8b as well as Figure 8c The super-resolution processing results for this specific example are shown. Figure 8a For the original high-resolution image, Figure 8b The second image is magnified using traditional bicubic interpolation. Figure 8c The image to be output is the result of super-resolution model processing and weighted fusion.

[0073] In summary, the super-resolution processing method for scanned images provided by the embodiments of the present invention enables the image forming apparatus to generate scanned and copied images with significantly higher clarity than those obtained through traditional interpolation methods, especially in scenarios where image magnification is required during the work process. Furthermore, it allows users to adjust the image generation process according to the document content, enabling the image forming apparatus to provide optimal image quality for users when dealing with various types of document content, thereby improving the user experience.

[0074] Corresponding to the above-described super-resolution processing method for scanned images, this embodiment of the invention provides an image forming apparatus. Figure 9 This is a schematic diagram of an image forming apparatus provided in an embodiment of the present invention. Figure 9 As shown, the image forming apparatus includes: a first acquisition module 901, a model selection module 902, a second acquisition module 903, a processing module 904, and a generation module 905.

[0075] The first acquisition module 901 is used to acquire super-resolution configuration parameters, job processing specification parameters, and the content type of the original document. Among them, the super-resolution configuration parameters include at least the enhancement level.

[0076] The model selection module 902 is used to select a target model from multiple preset super-resolution models based on the job processing specification parameters and content type.

[0077] The second acquisition module 903 is used to acquire scanned images of the original document.

[0078] The processing module 904 is used to input the super-resolution configuration parameters and the scanned image into the target model to obtain the first image output by the target model.

[0079] The generation module 905 is used to fuse the first image and the scanned image based on the enhancement level to generate the output image.

[0080] Figure 9 The image forming apparatus provided in the illustrated embodiment can be used to execute this specification. Figure 1 The implementation principle and technical effects of the method embodiment shown in -8 can be further referred to the relevant descriptions in the method embodiment.

[0081] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 10 As shown, the aforementioned electronic device may include at least one processor and at least one memory communicatively connected to the processor, wherein the memory stores program instructions executable by the processor, and the processor can execute this specification by calling the program instructions. Figure 1 The embodiment shown in Figure 8 provides an image super-resolution processing method.

[0082] like Figure 10 As shown, the electronic device is represented in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: one or more processors 1010, communication interface 1020 and memory 1030, and a communication bus 1040 connecting different system components (including memory 1030, communication interface 1020 and processor 1010).

[0083] The communication bus 1040 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MAC) buses, Enhanced ISA buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses.

[0084] Electronic devices typically include a variety of computer-readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, and removable and non-removable media.

[0085] Memory 1030 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. Memory 1030 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments described herein.

[0086] A program / utility having a set (at least one) of program modules may be stored in memory 1030. Such program modules include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The program modules typically perform the functions and / or methods described in the embodiments of this specification.

[0087] Processor 1010 executes various functional applications and data processing by running programs stored in memory 1030, such as implementing the functions described in this specification. Figure 1 The embodiment shown in Figure 8 provides an image super-resolution processing method.

[0088] This specification provides a computer program product, which includes a computer program that, when executed by a processor, performs the functions described in this specification. Figure 1 The embodiment shown in Figure 8 provides an image super-resolution processing method.

[0089] This specification provides a computer-readable storage medium storing computer instructions that cause a computer to execute this specification. Figure 1 The embodiment shown in Figure 8 provides an image super-resolution processing method.

[0090] The aforementioned computer-readable storage medium may be any combination of one or more computer-readable media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used by or in connection with an instruction execution system, apparatus, or device.

[0091] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0092] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0093] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this specification, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0094] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this specification includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which the embodiments of this specification pertain.

[0095] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0096] It should be noted that the devices involved in the embodiments of this specification may include, but are not limited to, personal computers (PCs), personal digital assistants (PDAs), wireless handheld devices, tablet computers, mobile phones, MP3 displays, MP4 displays, etc.

[0097] In the several embodiments provided in this specification, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms. Furthermore, the functional units in the various embodiments of this specification can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0098] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, a connector, or a network device, etc.) or a processor to execute some steps of the methods described in the various embodiments of this specification. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

[0100] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments and terminal embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

Claims

1. An image super-resolution processing method, characterized in that, Applied to an image forming apparatus, including: Obtain super-resolution configuration parameters and job processing specification parameters, as well as the content type of the original document; wherein, the super-resolution configuration parameters include at least the enhancement level; Based on the job processing specifications and the content type, a target model is selected from multiple preset super-resolution models; Obtain a scanned image of the original document; The super-resolution configuration parameters and the scanned image are input into the target model to obtain the first image output by the target model. Based on the enhancement level, the first image and the scanned image are fused to generate an image to be output.

2. The method according to claim 1, characterized in that, The content types include one or more combinations of text, mixed text and images, patterns, and portraits.

3. The method according to claim 1, characterized in that, The processing specifications include original size, output size, scanning resolution, and output resolution.

4. The method according to claim 1, characterized in that, The super-resolution model has a fixed magnification factor and applicable type; the selection of a target model from multiple preset super-resolution models based on the job processing specifications and the content type includes: Determine the target magnification factor based on the aforementioned job processing specifications; The super-resolution model with the magnification factor equal to the target magnification factor and whose applicable type matches the content type among the multiple preset super-resolution models is determined as the target model.

5. The method according to claim 4, characterized in that, The method prior to obtaining the scanned image of the original document further includes: The front-end scaling factor is determined based on the job processing specification parameters and the target magnification factor; The original document is scanned to obtain the original scanned image; The original scanned image is preprocessed according to the front-end scaling factor to obtain the scanned image.

6. The method according to claim 4, characterized in that, The step of fusing the first image and the scanned image based on the enhancement level to generate an output image includes: The back-end scaling factor is determined based on the job processing specification parameters and the target magnification factor; The size of the first image is adjusted according to the back-end scaling factor; The scanned image is magnified based on a preset super-resolution algorithm to obtain a second image; the magnification factor of the preset super-resolution algorithm is determined by the target magnification factor. The second image and the resized first image are weighted and fused according to the enhancement level to obtain the image to be output.

7. The method according to claim 6, characterized in that, The step of weighted fusing the second image and the resized first image according to the enhancement level to obtain the output image includes: The weighting weights of the first image and the second image are determined based on the enhancement level; wherein, the larger the value of the enhancement level, the larger the weighting weight of the first image. The first image and the second image are fused according to the weighted weights to obtain the image to be output.

8. The method according to claim 6, characterized in that, The magnification factor of the super-resolution algorithm is the product of the target magnification factor and the back-end scaling factor.

9. The method according to claim 1, characterized in that, The image forming apparatus is provided with multiple computing units, and the step of inputting the super-resolution configuration parameters and the scanned image into the target model includes: The computational resource requirements are determined based on the size of the scanned image and the target model; Based on the computational resource requirements and the pre-obtained available computational resources of the image forming apparatus, it is determined whether the scanned image needs to be segmented into blocks; If it is determined that the scanned image needs to be segmented into blocks, then the scanned image is divided into multiple image blocks with overlapping areas; The image blocks are assigned to the plurality of processing units.

10. The method according to claim 9, characterized in that, Assigning the image patch to the plurality of processing units includes: The allocation ratio of each computing unit is determined based on its computing performance. The image blocks are allocated to the plurality of processing units according to the allocation ratio.

11. An image forming apparatus, characterized in that, include: The first acquisition module is used to acquire super-resolution configuration parameters and job processing specification parameters, as well as the content type of the original document; wherein, the super-resolution configuration parameters include at least the enhancement level; The model selection module is used to select a target model from multiple preset super-resolution models based on the job processing specification parameters and the content type. The second acquisition module is used to acquire a scanned image of the original document; The processing module is used to input the super-resolution configuration parameters and the scanned image into the target model to obtain the first image output by the target model; The generation module is used to fuse the first image and the scanned image based on the enhancement level to generate an image to be output.

12. An electronic device, characterized in that, include: At least one processor; as well as At least one memory communicatively connected to the processor, wherein: The memory stores program instructions that can be executed by the processor, and the processor can execute the method according to any one of claims 1 to 10 by calling the program instructions.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause the computer to perform the method according to any one of claims 1 to 10.

Images