Image processing method and device based on deep learning, medium and program product

Abstract

The invention relates to the technical field of image processing, in particular to an image processing method and device based on deep learning, a medium and a program product. The image processing method comprises the following steps: providing an initial image, setting an extended epitaxial region on the outer contour of the initial image, and obtaining an intermediate image containing the initial image and the epitaxial region; the intermediate image is uniformly segmented to obtain a plurality of initial sub-images, each initial sub-image comprises a sub-image body and an edge band which expands outwards relative to the outer contour of the sub-image body, the width of the edge band is consistent with the width of the epitaxial region, and an overlapping region is arranged between the sub-image bodies in the adjacent initial sub-images; inputting all the initial sub-images into a deep learning model for image processing to obtain corresponding optimized sub-images; and splicing the sub-image bodies of all the optimized sub-images according to the segmentation positions of the initial sub-images to obtain an optimized image.

Description

[Technical field]

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, device, medium and program product based on deep learning. [Background technology]

[0002] With the continuous improvement of computer computing power, deep learning and graphics processing technologies have developed rapidly and have been widely used in fields such as autonomous driving, image resolution enhancement, and face recognition. With the popularity of artificial intelligence, chip designers have also begun to apply artificial intelligence products in the semiconductor field; however, in these applications, it is often necessary to process images of various sizes, including high-resolution large-size images. If these images are run directly, they may face huge computing and storage pressures.

[0003] Existing methods for processing high-resolution large-size images include acquiring images, setting overlapping areas, cropping sub-images and performing image processing on the sub-images, and splicing sub-images. Among them, image processing includes inputting sub-images into a deep learning model and obtaining the output results of the model. This processing method can not only retain the information of the existing resolution image, but also make full use of computer resources. However, it does not consider the edge effect of deep learning, which will result in that after the large-size image is processed, the information that can be extracted from the pixels close to the edge of the image is of low accuracy, thereby affecting the accuracy of calculating the pixel results of the large-size image.

[0004] In view of this, the image processing solution provided by the present invention improves the existing processing method to effectively reduce or even eliminate the edge effect of the image during deep learning. [Summary of the invention]

[0005] In order to solve the technical problem that the existing image processing methods do not consider the edge effect of deep learning, which affects the accuracy of the calculation of image pixel results, the present invention provides an image processing method, device, medium and program product based on deep learning.

[0006] The solution to the technical problem of the present invention is to provide an image processing method based on deep learning, comprising the following steps:

[0007] Providing an initial image, setting an extended extension area on the outer contour of the initial image, and obtaining an intermediate image including the initial image and the extended area;

[0008] The intermediate image is evenly divided to obtain a number of initial sub-images, wherein the initial sub-images include a sub-image body and an edge band extending outward from the outer contour of the sub-image body, wherein the width of the edge band is consistent with the width of the extension area, and an overlapping area is provided between the sub-image bodies in adjacent initial sub-images;

[0009] Input all initial sub-graphs into the deep learning model for image processing to obtain the corresponding optimized sub-graphs;

[0010] The sub-image bodies of all optimized sub-images are spliced ​​with reference to the segmentation positions of the initial sub-images to obtain an optimized image.

[0011] Preferably, uniformly dividing the intermediate image to obtain a plurality of initial sub-images comprises the following steps:

[0012] Based on the preset size of the initial sub-image input to the deep learning model, calculate the number of initial sub-images that can be cropped from the intermediate image;

[0013] When the calculation result shows that the intermediate image can be completely cropped into an integer number of initial sub-images, the intermediate image is evenly divided to obtain a number of initial sub-images;

[0014] Otherwise, an extended zero-padding area is first set on the outer contour of the intermediate image to obtain a zero-padding image including the initial image, the extended area and the zero-padding area, so that the intermediate image can be completely cropped into an integer number of initial sub-images, and then the zero-padding image is evenly divided to obtain several initial sub-images.

[0015] Preferably, the sub-image bodies of all optimized sub-images are spliced ​​with reference to the segmentation positions of the initial sub-images to obtain an optimized image, which includes the following steps:

[0016] Crop the edge band on the optimized sub-image and keep the sub-image body;

[0017] Based on a preset weight algorithm, the overlapping areas between the sub-graph bodies in adjacent optimized sub-graphs are fused to obtain the fused overlapping areas;

[0018] Referring to the segmentation position of the initial sub-graph, the fused overlapping area is spliced ​​to the sub-graph body of the corresponding adjacent optimized sub-graph;

[0019] The sub-image bodies containing overlapping areas in the optimized sub-images are spliced ​​to obtain an optimized image.

[0020] Preferably, fusing the overlapping regions between sub-graph bodies in adjacent optimized sub-graphs comprises the following steps:

[0021] Provide a first sub-image and a second sub-image adjacent to each other in the optimized sub-image, and obtain an overlapping area between the corresponding sub-image bodies and pixel points in the overlapping area;

[0022] According to the distance between the pixel point and the edge of the first sub-image, respectively obtain a first weight of the pixel point in the first sub-image and a second weight of the pixel point in the second sub-image;

[0023] Obtain a first pixel in the first sub-image and a second pixel in the second sub-image;

[0024] Based on the first pixel weighted by the first weight and the second pixel weighted by the second weight, a final pixel of the pixel point for fusing the overlapping area is calculated.

[0025] Preferably, the preset weight algorithm includes an adaptive weighted fusion algorithm, and the calculation formula I(x) of the final pixel obtained by using the adaptive weighted fusion algorithm is:

[0026]

[0027] Where x is the position of the pixel in the overlapping area, exp(0) is the first weight of the pixel in the first sub-image, exp(x) is the second weight of the pixel in the second sub-image, and I 1 (x) is the first pixel of the first sub-image, I 2 (x) is the second pixel in the second sub-image.

[0028] Preferably, the outer contour side length of the optimized sub-image is an integer multiple of the outer contour side length of the initial sub-image; the width of the edge band and the overlapping area in the optimized sub-image is an integer multiple of the width of the edge band and the overlapping area in the initial sub-image.

[0029] Preferably, when the intermediate image cannot be completely cropped into an integer number of initial sub-images, the extended zero-filling area is set on two adjacent sides of the outer contour of the intermediate image.

[0030] The present invention also provides a computer device, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned image processing method.

[0031] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned image processing method are implemented.

[0032] The present invention also provides a computer program product, comprising a computer program, which implements the steps of the above-mentioned image processing method when executed by a processor.

[0033] Compared with the prior art, the image processing method, device, medium and program product based on deep learning provided by the present invention have the following advantages:

[0034] 1. An embodiment of the present invention provides an image processing method based on deep learning. The method takes into account the edge effect of deep learning. Before processing the image, the edge of the image is expanded outward to reduce the edge effect, thereby improving the accuracy of the calculation of the image pixel result. Specifically, the image processing method first obtains an initial image, which is an image to be processed that needs to be processed. Before setting an overlapping area that is convenient for image segmentation and splicing, an expanded extension area is first set on the outer contour of the initial image to obtain an intermediate image containing the initial image and the extension area, and then the intermediate image is evenly divided to obtain an initial sub-image including a sub-image body and an edge band, which can facilitate subsequent parallel processing of multiple initial sub-images, wherein the edge band is expanded relative to the outer contour of the sub-image body and is consistent with the width of the extension area, which can further reduce the impact of the edge effect. The overlapping area serves as a hub between the sub-image bodies of the optimized sub-image in the splicing process, so as to facilitate the splicing of adjacent sub-image bodies.

[0035] It can be understood that by setting an extended area on the outer contour of the initial image and an edge band on the outer contour of the sub-image body, when the initial sub-image is input into the deep learning model for image processing, more contextual information can be carried on the sub-image body of the initial sub-image, so as to effectively reduce or even eliminate the edge effect of the image when performing deep learning tasks.

[0036] Furthermore, after performing image processing on all the initial sub-images and obtaining the corresponding optimized sub-images, the initial sub-image segmentation positions obtained by uniformly segmenting the intermediate image are obtained, and the sub-image bodies of all the optimized sub-images are spliced ​​with reference to these segmentation positions, wherein the edge bands on the optimized sub-images are not within the splicing range. That is, through this design, the accuracy of the information extracted from the edge pixels close to the edge points on the image can be improved, thereby improving the accuracy of the calculation of the image pixel results.

[0037] 2. In the image processing method provided in the embodiment of the present invention, before the intermediate image is evenly divided, it is necessary to obtain the preset size of the initial sub-image input into the deep learning model, and calculate the number of initial sub-images that can be cropped from the intermediate image based on the preset size to obtain the calculation result.

[0038] It should be noted that the numerical value of the calculation result can be an integer or a non-integer. When the calculation result is an integer, it indicates that the side length of the intermediate image can be evenly divided according to the size of the initial sub-image input into the deep learning model, and the side length of each initial sub-image obtained by segmentation is the same.

[0039] Furthermore, when the calculated result is not an integer, it indicates that the side length of the intermediate image cannot meet the requirements for cropping an integer number of initial sub-images. In this case, it is necessary to set an extended zero-padding area on the outer contour of the intermediate image. Through this design, preliminary preparations can be made for the step of inputting the initial sub-image into the deep learning model for image processing, thereby avoiding the mismatch between the side length of the initial sub-image and the model size of the deep learning model in the deep learning task, which affects the final output result of the image processing.

[0040] 3. In the image processing method provided in the embodiment of the present invention, after completing image processing on all initial sub-images obtained by evenly dividing an intermediate image to obtain corresponding optimized sub-images, it is necessary to determine the relative positions between all optimized sub-images with reference to the division positions during even division, so as to splice the sub-image bodies of all optimized sub-images to obtain the final optimized image.

[0041] It should be noted that when splicing the sub-image bodies of the optimized sub-images, it is necessary to first crop the edge bands contained in each optimized sub-image to effectively reduce or even eliminate the edge effect produced by the initial sub-image after it is input into the deep learning model for image processing; then, the fusion pixels of the overlapping areas between the sub-image bodies are determined by a preset weight algorithm. Before splicing the sub-image bodies containing overlapping areas in the optimized sub-images, it is necessary to first calculate the fusion pixels of the overlapping areas based on the preset weight algorithm to obtain the fused overlapping areas, and then splice the fused overlapping areas according to their positions on the initial sub-image. Through this design, the image transition between the optimized sub-images can be made smoother, further reducing the traces of splicing.

[0042] 4. In the image processing method provided in the embodiment of the present invention, after extracting the overlapping area between adjacent sub-image bodies and performing a deep learning model on the initial sub-image, an adaptive weighted fusion algorithm is used to perform weighted average pixel-level calculation on the overlapping area, and the pixels at each point in the overlapping area are calculated based on the weight to ensure smooth transition of subsequent optimized sub-images during splicing, so as to integrate and unify the pixel changes in the overlapping area between adjacent sub-image bodies, splice the fused overlapping area to the corresponding sub-image body, and then splice the sub-image body containing the overlapping area in the optimized sub-image into a complete optimized image. Through this design, the deep learning model can obtain better results when deployed, that is, the output result obtained after all the initial sub-images are input into the deep learning model for image processing.

[0043] It should be noted that the use of an adaptive weighted fusion algorithm can overcome the technical difficulties that the existing weighted average pixel-level processing calculation is too linear, and the image formed after splicing may have a block effect and the transition between pixels is not smooth enough. The calculation of the overlapping area in the embodiment of the present invention is relatively gentler, so that the overlapping areas between the sub-image bodies can achieve a smooth transition during the fusion process, thereby making the splicing between the sub-image bodies in the optimized sub-image smoother.

[0044] 5. The embodiment of the present invention further provides a computer device, which has the same beneficial effects as the above-mentioned image processing method, and will not be described in detail here.

[0045] 6. The embodiment of the present invention further provides a computer-readable storage medium, which has the same beneficial effects as the above-mentioned image processing method, and will not be described in detail here.

[0046] 7. The embodiment of the present invention further provides a computer program product, which has the same beneficial effects as the above-mentioned image processing method, and will not be described in detail here.

Brief Description of the Drawings

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0048] Figure 1 It is a flowchart of steps S1 to S4 in an image processing method based on deep learning provided in an embodiment of the present invention.

[0049] Figure 2 It is a schematic diagram of an intermediate image of an image processing method based on deep learning provided by an embodiment of the present invention and an initial sub-image obtained by segmenting the intermediate image.

[0050] Figure 3 It is a schematic diagram of a zero-filled image of an image processing method based on deep learning provided in an embodiment of the present invention.

[0051] Figure 4 It is a flowchart of steps S21 to S23 in a deep learning-based image processing method provided in an embodiment of the present invention.

[0052] Figure 5 It is a flowchart of steps S41 to S44 in a deep learning-based image processing method provided in an embodiment of the present invention.

[0053] Figure 6 It is a schematic diagram of a process of fusing overlapping areas between sub-image bodies in adjacent optimized sub-images in a deep learning-based image processing method provided by an embodiment of the present invention.

[0054] Figure 7 It is a flowchart of steps S421 to S424 in a deep learning-based image processing method provided in an embodiment of the present invention.

[0055] Figure 8 It is a schematic diagram of a framework of a computer device provided by an embodiment of the present invention.

[0056] Fig. 9 It is a schematic diagram of a framework of a computer-readable storage medium provided by an embodiment of the present invention.

[0057] Fig.10 It is a schematic diagram of the framework of a computer program product provided by an embodiment of the present invention.

[0058] Description of the accompanying drawings:

[0059] 1. Computer equipment; 11. Storage; 12. Processor;

[0060] 2. Computer-readable storage medium;

[0061] 3. Computer program products;

[0062] 100. Computer program. [Specific implementation method]

[0063] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and implementation examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0064] In the embodiments provided by the present invention, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0065] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. Those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required for the present invention.

[0066] In various embodiments of the present invention, it should be understood that the size of the serial numbers of the above-mentioned processes does not mean the necessary order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0067] The flow chart and block diagram in the accompanying drawings of the present invention illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which is determined based on the functions involved. It should be particularly noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs a specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0068] See also Figure 1 and Figure 2 , an embodiment of the present invention provides an image processing method based on deep learning, comprising the following steps:

[0069] S1: providing an initial image, setting an extended extension area on the outer contour of the initial image, and obtaining an intermediate image including the initial image and the extension area;

[0070] S2: Evenly divide the intermediate image to obtain a number of initial sub-images, wherein the initial sub-images include a sub-image body and an edge band extending outward from the outer contour of the sub-image body, wherein the width of the edge band is consistent with the width of the extension area, and an overlapping area is set between the sub-image bodies in adjacent initial sub-images;

[0071] S3: Input all initial sub-graphs into the deep learning model for image processing to obtain the corresponding optimized sub-graphs;

[0072] S4: splicing the sub-image bodies of all optimized sub-images with reference to the segmentation positions of the initial sub-images to obtain an optimized image.

[0073] For ease of understanding, the intermediate images involved in the embodiments of the present invention are as follows: Figure 2 As shown, the intermediate image includes an initial image and an extended area. After the intermediate image is evenly divided, an initial sub-image is obtained. The initial sub-image includes a sub-image body and an extended edge band, wherein the width of the edge band is consistent with the width of the extended area, that is, the initial sub-image divided at the edge of the intermediate image has an overlapping edge band with the extended area.

[0074] Understandably, the existing and most common method of image processing is to reduce the image resolution so that the image size matches the size of the deep learning model input, which is the process of downsampling. However, the low-resolution image obtained in this way often loses some key information, making the final output result less than ideal.

[0075] As an improvement, another existing image processing method is to first divide the image into sub-images and then input them into the deep learning model, and then splice the sub-images after obtaining the output results. This processing method can not only retain the information of the existing resolution image, but also make full use of computer resources. However, this method does not take into account the edge effect of deep learning. The information that can be extracted from the pixels close to the edge of the image is less accurate, resulting in insufficient information extraction in this area, affecting the accuracy of the calculation of the image pixel results.

[0076] In this regard, the deep learning-based image processing method provided in an embodiment of the present invention takes into account the edge effect of deep learning and overcomes the above-mentioned difficulties. Before processing the image, the edge of the image is expanded outward to reduce the edge effect.

[0077] Specifically, the image processing method first obtains an initial image, which is an image to be processed that needs to be processed; before setting an overlapping area that is convenient for image segmentation and splicing, first set an extended area on the outer contour of the initial image to obtain an intermediate image containing the initial image and the extended area, and then evenly divide the intermediate image to obtain an initial sub-image including a sub-image body and an edge band, wherein the edge band expands relative to the outer contour of the sub-image body and is consistent with the width of the extended area.

[0078] It should be noted that, in the image processing method provided by the embodiment of the present invention, the extended area on the intermediate image is formed by expanding the outer contour of the initial image, that is, the extended area is connected to the four edges of the initial image; at the same time, the initial sub-image formed by uniformly dividing the intermediate image includes a sub-image body located in the inner layer and an edge band in the outer layer, that is, the edge band is connected to the outer contours around the sub-image body. It can be understood that by first setting the extended extended area, the initial sub-image located in the outer circle can be convenient for forming a complete edge band.

[0079] Furthermore, by setting an extended area on the outer contour of the initial image and an edge band on the outer contour of the sub-image body, when the initial sub-image is input into the deep learning model for image processing, the edge effect of the image during deep learning tasks can be effectively reduced or even eliminated.

[0080] Among them, the edge effect is a characteristic of convolution. During the process of image processing or analysis, when filtering operations are performed on the edge of the image, part of the convolution kernel may be located outside the edge of the image. This phenomenon is called the edge effect. This effect may affect the processing of the edge of the image, resulting in unexpected effects on the edge of the processed image. In this regard, an embodiment of the present invention improves the edge effect by setting an edge band on the outer contour of the sub-image body.

[0081] After all the initial sub-images are processed and the corresponding optimized sub-images are obtained, the initial sub-image segmentation positions obtained by evenly segmenting the intermediate image are obtained, and the sub-image bodies of all the optimized sub-images are spliced ​​with reference to these segmentation positions. When segmenting, the edge band is not a segmentation line, but only contains the image information of the corresponding area on the adjacent initial sub-image. When splicing, the edge band on the optimized sub-image will be excluded, that is, the edge band is not within the splicing range. Through this design, the accuracy of the information extracted from the edge pixels on the image can be improved, thereby improving the accuracy of the calculation of the image pixel results.

[0082] The specific steps S1 to S4 of the image processing method will be described below.

[0083] First, in step S1, as the image to be processed that needs to be processed, the initial image can generally be provided manually, and in this field specifically refers to the chip design layout; the size of the chip design layout is very large, and in actual processing, the design layout is generally divided into several small areas (patches), but the size after being divided into small patches is still large, and the image size can reach 4096*4096 or 10000*10000 pixels.

[0084] In step S1, the intermediate image has an extension area connected to the four edges of the initial image compared to the initial image. The width of the extension area refers to the distance between the outer contour edge of the initial image and the outer contour edge of the extension area or the intermediate image.

[0085] In step S2, when the intermediate image is evenly divided, the sub-image body is used as the cutting boundary for division, and the edge band can overlap with the adjacent initial sub-image; for example, between two adjacent initial sub-images on the left and right, the edge band included in the initial sub-image on the right includes at least part of the sub-image body on the initial sub-image on the left.

[0086] In step S2, the adjacent initial sub-images obtained after uniform segmentation will each retain the same overlapping area, and the image information of the overlapping area will be stored in two adjacent initial sub-images respectively.

[0087] Furthermore, in step S2, after obtaining the intermediate image including the initial image and the extended area, the intermediate image is evenly segmented into multiple initial sub-images, and the side lengths of these initial sub-images are the same. In the initial sub-image whose segmentation position is at the edge, the edge band of the outer contour of the sub-image body includes the extended area of ​​the intermediate image before the uniform segmentation is performed, so there is no need to set up additional edge bands or extended areas, so that these initial sub-images can be input into the deep learning model for processing in step S3.

[0088] It should be noted that the width of the edge band refers to the distance between the outer contour edge of the sub-image body and the outer contour edge of the edge band; in the embodiment provided by the present invention, the width of the edge band is the same as the width of the extension area, wherein the distance that the edge band expands in all directions of the outer contour edge of the sub-image body is consistent, and the distance that the extension area expands in all directions of the outer contour edge of the initial image is also consistent.

[0089] In step S3, when all the initial sub-images are input into the deep learning model with the same input size for image processing, there may be a situation where the model size of the deep learning model does not match the input size. In this case, it is necessary to first scale the input size of the model by a certain multiple according to the same ratio, and then perform image processing on the initial sub-image by the deep learning model. The embodiment of the present invention sets the width of the edge band and the width of the extension area to the same value, so that after all the initial sub-images complete the image processing on the deep learning model to obtain the corresponding optimized sub-images, the sub-image bodies can be better spliced.

[0090] Among them, the deep learning model involved in step S3 is like a black box. After the initial sub-image is input into the deep learning model, some convolution or upsampling operations will be performed in the model. Among them, the convolution operation can effectively extract the local features of the image, and the upsampling operation can mainly amplify the original image or restore the details and size of the image. For example, if a blurred image with a resolution of 256*256 is input into the deep learning model, the model will output a 512*512 high-resolution image to achieve the effect of resolution enhancement.

[0091] In step S4, when the sub-image bodies of all the optimized sub-images are spliced ​​together, they need to be spliced ​​according to the division positions between the initial sub-images obtained by evenly dividing the intermediate image in step S2, wherein the sub-image bodies of the spliced ​​optimized sub-images can be spliced ​​one by one in the order of rows and columns, or all the optimized sub-images can be placed according to the division positions between the initial sub-images and these optimized sub-images can be spliced ​​together at the same time.

[0092] The steps of the image processing method provided in the embodiment of the present invention solve the technical problem that the existing image processing method does not consider the edge effect of deep learning, which affects the accuracy of the calculation of image pixel results.

[0093] See also Figure 3 and Figure 4 , evenly divide the intermediate image to obtain several initial sub-images, including the following steps:

[0094] S21: Based on a preset size of the initial sub-image input into the deep learning model, calculating the number of initial sub-images that can be cropped from the intermediate image;

[0095] S22: When the calculation result shows that the intermediate image can be completely cropped into an integer number of initial sub-images, the intermediate image is evenly divided to obtain a plurality of initial sub-images;

[0096] S23: Otherwise, an extended zero-padding area is first set on the outer contour of the intermediate image to obtain a zero-padding image including the initial image, the extended area and the zero-padding area, so that the intermediate image can be completely cropped into an integer number of initial sub-images, and then the zero-padding image is evenly divided to obtain a number of initial sub-images.

[0097] For ease of understanding, the zero-filled image involved in the embodiment of the present invention is as follows: Figure 3 As shown, when the calculation result is not an integer, the zero-filled image includes the initial image, the extended area, and the zero-filled area expanded relative to the outer contour of the extended area.

[0098] It is understandable that when the intermediate image is evenly divided into several initial sub-images, it is often not possible to crop to an integer number of initial sub-images. For example, when the intermediate image is evenly divided horizontally, the last initial sub-image will be incomplete. When the incomplete initial sub-image is input into the deep learning model for image processing, the different side lengths of the initial sub-images will cause the final model output result to be different from the expected output result.

[0099] In this regard, the image processing method provided in an embodiment of the present invention needs to obtain a preset size of the initial sub-image input into the deep learning model before evenly dividing the intermediate image, and calculate the number of initial sub-images that can be cropped from the intermediate image based on the preset size to obtain a calculation result.

[0100] It should be noted that the numerical value of the calculation result is the number of initial sub-images that can be cropped from the intermediate image. This value can be an integer or a non-integer. When the calculation result is an integer, that is, step S22, it indicates that the side length of the intermediate image can be evenly divided according to the input size of the deep learning model, and each initial sub-image obtained by segmentation has the same side length.

[0101] Furthermore, when the calculated result is not an integer, that is, step S23, indicating that the side length of the intermediate image cannot satisfy the requirement of cropping an integer number of initial sub-images, it is necessary to perform zero-padding operation on the edge of the intermediate image. Specifically, an extended zero-padding area is set on the outer contour of the intermediate image. Through this design, preliminary preparations can be made for the step of inputting the initial sub-image into the deep learning model for image processing, thereby avoiding the mismatch between the side length of the initial sub-image and the model size of the deep learning model in the deep learning task, which affects the final output result of the image processing.

[0102] In step S23, since the calculation result is that the intermediate image cannot be completely and evenly divided into several initial sub-images, it is necessary to further set a zero-padding area on the basis of the original intermediate image. The zero-padding area is formed by expanding the outer contour of the intermediate image. It can be understood that the zero-padding area is formed by expanding the outer contour of the extended area on the intermediate image, and the initial image, the extended area and the zero-padding area together form a zero-padding image.

[0103] See also Figure 2 and Figure 3 In step S21, the calculation formula for the number of initial sub-images that can be obtained by cropping the intermediate image is as follows:

[0104]

[0105] Among them, num_w and num_h are the initial sub-image numbers that can be cropped in the width and height directions of the intermediate image respectively; ceil means that when the calculation result is not an integer, the value of num_w or num_h is rounded up, that is, an extended zero-filling area is set on the outer contour of the intermediate image. img_w and img_h are the pixels of the initial image in the width and height directions respectively, in_w and in_h are the pixels of the initial sub-image in the width and height directions respectively, band is the pixels of the extended area and edge band in the width and height directions, and overlap is the pixels of the overlapping area in the width and height directions.

[0106] It should be noted that the numerator of the formula is the pixels of the intermediate image in the width and height directions, and the denominator of the formula is the pixels of the initial sub-image in the width and height directions that are subsequently input into the deep learning model for image processing.

[0107] See also Figure 5 , referring to the segmentation position of the initial sub-image, the sub-image bodies of all optimized sub-images are spliced ​​to obtain an optimized image, including the following steps:

[0108] S41: cutting the edge band on the optimized sub-image and retaining the sub-image body;

[0109] S42: Based on a preset weight algorithm, overlapping regions between sub-graph bodies in adjacent optimized sub-graphs are merged to obtain a merged overlapping region;

[0110] S43: referring to the segmentation position of the initial sub-graph, splicing the fused overlapping area to the sub-graph body of the corresponding adjacent optimized sub-graph;

[0111] S44: splicing the sub-image bodies containing overlapping areas in the optimized sub-images to obtain an optimized image.

[0112] For ease of understanding, in the embodiment of the present invention, the process of fusing the overlapping areas between the sub-graph bodies in adjacent optimized sub-graphs is as follows: Figure 6 As shown, Figure 6 The black squares and gray squares in represent the sub-image bodies between adjacent optimized sub-images, respectively. The overlapping area in the dotted box is located between the two adjacent sub-image bodies. After the adjacent optimized sub-images are divided, the blank area between the black wireframe and the black square and the blank area between the gray wireframe and the gray square are the edge bands that expand outward relative to the sub-image bodies on the adjacent optimized sub-images. That is, when the intermediate image is evenly divided to obtain the initial sub-images, the sub-image bodies in the adjacent initial sub-images are stored with the same overlapping area, and this overlapping area exists as the splicing hub between the sub-image bodies in the fusion and splicing process from step S42 to step S43.

[0113] It can be understood that after completing image processing on all initial sub-images obtained by uniformly dividing an intermediate image and obtaining the corresponding optimized sub-images, it is necessary to determine the relative positions between all optimized sub-images with reference to the division positions during uniform division, so as to splice the sub-image bodies of all optimized sub-images to obtain the final optimized image.

[0114] It should be noted that in the image processing method provided by the embodiment of the present invention, when the initial sub-image is input into the deep learning model for image processing, the edge effect brought about by the deep learning task can be effectively reduced or even eliminated by setting an outward-expanding edge band on the outer contour of the sub-image body. It can be seen that the edge band does not exist in the actual image to be processed, that is, it is not a part that needs to be spliced ​​between the optimized sub-images. When splicing the sub-image body of the optimized sub-image, it is necessary to first crop the edge band contained in each optimized sub-image.

[0115] Subsequently, the fused pixels of the overlapping areas between the sub-image bodies are determined by a preset weight algorithm. It can be understood that in adjacent optimized sub-images, the overlapping areas between the sub-image bodies may change during the image processing stage, thereby causing the pixels of the overlapping areas to change. Before splicing the sub-image bodies containing overlapping areas in the optimized sub-images, it is necessary to first calculate the fused pixels of the overlapping areas based on the preset weight algorithm to obtain the fused overlapping areas, and then splice the fused overlapping areas according to their positions on the initial sub-images.

[0116] In some embodiments, the sub-image bodies containing overlapping areas in the optimized sub-images are spliced. When a fused overlapping area is spliced ​​to the sub-image body, another sub-image body adjacent to the overlapping area is spliced ​​to the other end of the overlapping area, and so on to obtain an optimized image.

[0117] In some embodiments, the sub-image bodies containing overlapping areas in the optimized sub-image are spliced ​​together, or all the overlapping areas after fusion are first spliced ​​together to one of the sub-image bodies corresponding to these overlapping areas, and then the other sub-image bodies that have not been spliced ​​together are spliced ​​together with the corresponding overlapping areas by referring to the segmentation position of the initial sub-image to obtain an optimized image.

[0118] See also Figure 7 , based on a preset weight algorithm, the overlapping areas between subgraph bodies in adjacent optimized subgraphs are merged, including the following steps:

[0119] S421: providing a first sub-image and a second sub-image adjacent to each other in the optimized sub-image, and obtaining an overlapping area between the corresponding sub-image bodies and pixel points in the overlapping area;

[0120] S422: Obtaining a first weight of the pixel point in the first sub-image and a second weight of the pixel point in the second sub-image respectively according to the distance between the pixel point and the edge of the first sub-image;

[0121] S423: Acquire a first pixel of the first sub-image and a second pixel of the second sub-image;

[0122] S424: Calculate a final pixel of the pixel point for fusion of the overlapping area based on the first pixel weighted by the first weight and the second pixel weighted by the second weight.

[0123] It can be understood that in the image processing method provided in the embodiment of the present invention, a preset weight algorithm is first used to perform weighted averaging on the overlapping areas, the fused overlapping areas are spliced ​​onto the corresponding sub-image bodies, and then the sub-image bodies containing the overlapping areas in the optimized sub-images are spliced ​​into a complete optimized image.

[0124] It should be noted that the weight of the pixel in the overlapping area is determined according to the distance from the center of the overlapping area. The process from step S421 to step S424 will be described below through a calculation formula.

[0125] Specifically, let x be the position of the pixel point in the overlapping area in step S421; w 1 (x) and w 2 (x) are respectively the first weight of the pixel point in the first sub-image and the second weight of the pixel point in the second sub-image in step S422. The first weight and the second weight involved in the embodiment of the present invention are respectively the distance from the pixel point to the edge of the first sub-image and the distance to the edge of the second sub-image. The calculation formula I of the final pixel in step S424 is obtained. A (x) is:

[0126]

[0127] Among them, I 1 (x) and I 2 (x) are the first pixel of the first sub-image and the second pixel of the second sub-image in step S423, respectively. By using a preset weight algorithm to obtain a better result when deploying a deep learning model, the number of repeated calculations can be reduced and the utilization of computing resources can be optimized.

[0128] As an example, assuming that the length of the overlapping area is 10, the first sub-image overlaps the second sub-image from left to right, and the pixel point x is at a distance of 10 from the first sub-image, then the first weight of the first sub-image at the pixel point x is 10, the second weight of the second sub-image at the pixel point x is 0, and the pixel value at x is 1. A (x) = (10*I 1 (x)) / 10.

[0129] In some embodiments, the preset weight algorithm includes an adaptive weighted fusion algorithm.

[0130] The adaptive weighted fusion algorithm uses exponential weights for the first weight of the pixel in the first sub-image and the second weight of the pixel in the second sub-image.

[0131] It should be noted that when the calculation formula I is used A (x) When performing weighted averaging, the weight of the overlapping area is relatively linear, which may cause a block effect in the optimized image after the final stitching, and the pixel transformation from the first sub-image to the second sub-image is not smooth enough.

[0132] In this regard, the image processing method provided in the embodiment of the present invention overcomes the above difficulties. When obtaining the weight of the overlapping area after fusion, an adaptive weighted fusion algorithm is used on the overlapping area to obtain the calculation formula of the final pixel: B (x) is:

[0133]

[0134] Where x is the position of the pixel in the overlapping area, exp(0) is the first weight of the pixel in the first sub-image, exp(x) is the second weight of the pixel in the second sub-image, and I 1 (x) is the first pixel of the first sub-image, I 2 (x) is the second pixel of the pixel point in the second sub-image. It can be understood that I A (x) is the general solution of the calculation formula, I B (x) is I A A specific solution of (x).

[0135] As an example, assuming that the length of the overlapping area is 10, the first sub-image overlaps with the second sub-image from left to right, the pixel point is located at x and the distance from the second sub-image is 10, then the second weight of the second sub-image at the pixel point x is e 10 (or expressed as exp(10)), the first weight of the first sub-image at pixel x is e 0 =1 (i.e., exp(0)), the final pixel value at position x is: (exp(0)*I 1 (x)+exp(10)*I 2 (x)) / (exp(0)+exp(10)).

[0136] It can be understood that the calculation formula I A (x) and I B (x) differ only in the weight, I A The weight of pixel (x) at pixel x is linearly related to the distance between pixel x and the first sub-image and the second sub-image, while I B The adaptive weighted fusion algorithm of (x) is exponentially related to the distance between the pixel point x and the first sub-image and the second sub-image.

[0137] The adaptive weighted fusion algorithm provided in the embodiment of the present invention is relatively gentler, and can make the sub-image bodies in adjacent optimized sub-images transition smoothly during the fusion process, thereby making the joints between the sub-image bodies in the optimized sub-images smoother, reducing artifacts and discontinuities at the joints, and better preserving the details of the initial image, reducing visible defects at the joints on the optimized image.

[0138] In some embodiments, the outer contour side length of the optimized sub-image is an integer multiple of the outer contour side length of the initial sub-image.

[0139] It can be understood that if the pixels of the initial sub-image input into the deep learning model for image processing are defined as in_w and in_h in the width and height directions respectively, and the pixels of the output optimized sub-image are out_w and out_h in the width and height directions respectively, then out_w = in_w*N and out_h = in_h*N, and N is a positive integer.

[0140] Furthermore, the width of the edge band and the overlapping area in the optimized sub-image is an integer multiple of the width of the edge band and the overlapping area in the initial sub-image; through this design, the consistency at different zoom levels before and after image processing in the deep learning model can be guaranteed.

[0141] It can be understood that the initial sub-image obtained by uniformly segmenting the intermediate image is the basic unit for inputting into the deep learning model, and the width of its edge band must be less than For example, if the resolution of the initial sub-image is 256*256, the width of the edge band will be less than 128. In actual settings, the width of the edge band will be set smaller, generally around 32, and the width of the overlapping area will also be set very small, generally around 2.

[0142] It should be noted that the width of the edge band and the overlapping area can be customized according to the size limited by the initial sub-image input deep learning model and user preferences.

[0143] Among them, the width of the edge band can be set relatively larger, so that the sub-image body in the initial sub-image is located in the middle part as a whole, and the pixel result of the model output is finally more accurate, so as to effectively reduce or even eliminate the edge effect produced by the initial sub-image after being input into the deep learning model for image processing; and the width of the overlapping area can be set to a smaller value, with the successful implementation of all steps of the image processing method of the embodiment of the present invention as the standard. Through this setting, the resource occupation can be greatly reduced, and the time for inputting the deep learning model for image processing can be improved.

[0144] As an example, a picture with a resolution of 256*256 is input into the deep learning model, the width of its edge band is 32, and the width of the overlapping area is 2. If the picture is output as a picture with a resolution of 512*512 after image processing by the model, it is enlarged twice compared to the input (that is, N=2), then the corresponding edge band and overlapping area widths in the picture are also enlarged twice, which are 64 and 4 respectively.

[0145] In some embodiments, when the intermediate image cannot be completely cropped into an integer number of initial sub-images, the extended zero-padding area is set on two adjacent sides of the outer contour of the intermediate image.

[0146] It should be noted that when the intermediate image is evenly divided, it is relatively rare to be able to crop to an integer number of initial sub-images. In this case, it is necessary to set an extended zero-padding area on the intermediate contour; for example, when the intermediate image is evenly divided vertically, the last entire row of initial sub-images in the vertical direction may be incomplete. In this case, a zero-padding operation is required to add a zero-padding area to this missing area so that the side length of each initial sub-image is the same as the input size of the deep learning model, thereby avoiding the model size mismatch in the deep learning task and affecting the final output result of the image processing.

[0147] It can be understood that for the zero-padded image in the embodiment of the present invention, the zero-padded area is set at most on the outer contours of the adjacent two sides in the intermediate image or the extended area; similar to laying tiles throughout the room, generally we will start laying from one corner, such as the upper left corner, lower left corner, upper right corner or lower right corner. Through this design, the pressure on computer resources can be reduced and calculation is convenient.

[0148] As a preferred implementation scheme of the embodiment of the present invention, the extended zero-padding area is set on the outer contour of the right side and / or bottom side of the intermediate image. When the zero-padding area is only set on the outer contour of the right side or bottom side of the intermediate image, it indicates that the intermediate image can be completely cropped into an integer number of sub-images when evenly divided in one of the horizontal or vertical directions. Of course, in addition to being set on the outer contour of the right side and / or bottom side of the intermediate image, the zero-padding area can also be set on the outer contour of the left side and / or top side of the intermediate image, which is not limited here. Any modifications, equivalent replacements and improvements made within the principles of the present invention should be included in the protection scope of the present invention.

[0149] See also Figure 8 The embodiment of the present invention further provides a computer device 1, comprising a storage 11, a processor 12 and a computer program 100 stored in the storage 11, and the processor 12 executes the computer program 100 to implement the steps of the above-mentioned image processing method.

[0150] It can be understood that when the computer device 1 provided in the embodiment of the present invention is running, the image processing method based on deep learning described in the above embodiment can be implemented when the processor 12 executes the computer program 100, and the computer device 1 provided in the embodiment of the present invention has the same beneficial effects as the image processing method provided in the above embodiment.

[0151] In some embodiments, the computer device 1 is a computer device applied in the field of image processing technology, including but not limited to uniformly dividing the intermediate image to obtain several initial sub-images, inputting the initial sub-images into a deep learning model for image processing, and fusing the overlapping areas between the sub-image bodies in adjacent optimized sub-images. No further details are given here. In theory, the method steps involved in the technical scheme of the present invention can all be realized through the control of the computer device 1, and the relevant parameters involved can also be adjusted accordingly through the computer device 1, wherein the relevant parameters include but are not limited to the width of the edge band between the optimized sub-image and the initial sub-image and the width of the overlapping area between the optimized sub-image and the initial sub-image.

[0152] In some embodiments, the processor 12 provided in the embodiments of the present invention is a general-purpose processor, which is a microprocessor or any conventional processor, such as a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, which can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application.

[0153] In some embodiments, the method steps disclosed in the embodiments of the present application may be implemented as being executed by a hardware processor, or as being executed by a combination of hardware and software modules in the processor.

[0154] See also Fig. 9 The embodiment of the present invention further provides a computer-readable storage medium 2 on which a computer program 100 is stored. When the computer program 100 is executed by a processor, the steps of the above-mentioned image processing method are implemented.

[0155] It can be understood that the computer-readable storage medium 2 provided in the embodiment of the present invention stores a computer program 100, and the computer program 100 can be called by a processor to execute the deep learning-based image processing method described in the above embodiment.

[0156] It should be noted that the computer-readable storage medium 2 provided in the embodiment of the present invention has the same beneficial effects as the image processing method provided in the above embodiment, which will not be described in detail herein.

[0157] Specifically, the computer-readable storage medium 2 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, disk, optical disk, etc.

[0158] In some embodiments, the computer-readable storage medium 2 includes a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs, and modules.

[0159] Specifically, the computer-readable storage medium 2 provided in the embodiment of the present invention has storage space for computer programs 100 that execute any method steps in the above-mentioned image processing method, and these programs can be read from or written into one or more computer program products.

[0160] In some embodiments, the computer program 100 may be compressed in a suitable form.

[0161] See also Fig.10 The embodiment of the present invention further provides a computer program product 3, comprising a computer program 100, which implements the steps of the above-mentioned image processing method when executed by a processor.

[0162] It can be understood that the computer program product 3 provided in the embodiment of the present invention includes a computer program 100, and the computer program 100 can be called by a processor to execute the deep learning-based image processing method described in the above embodiment, which will not be repeated here.

[0163] The above is a detailed introduction to the deep learning-based image processing method, device, medium and program product disclosed in the embodiments of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea.

[0164] At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation of the present invention, and any modifications, equivalent substitutions and improvements made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An image processing method based on deep learning, characterized in that: The following steps are involved: Providing an initial image, setting an extended extension area on the outer contour of the initial image, and obtaining an intermediate image including the initial image and the extended area; The intermediate image is evenly divided to obtain a number of initial sub-images, wherein the initial sub-images include a sub-image body and an edge band extending outward from the outer contour of the sub-image body, wherein the width of the edge band is consistent with the width of the extension area, and an overlapping area is provided between the sub-image bodies in adjacent initial sub-images; Input all initial sub-graphs into the deep learning model for image processing to obtain the corresponding optimized sub-graphs; The sub-image bodies of all optimized sub-images are spliced ​​with reference to the segmentation positions of the initial sub-images to obtain an optimized image.

2. The image processing method according to claim 1, characterized in that: The intermediate image is evenly divided into several initial sub-images, including the following steps: Based on the preset size of the initial sub-image input to the deep learning model, calculate the number of initial sub-images that can be cropped from the intermediate image; When the calculation result shows that the intermediate image can be completely cropped into an integer number of initial sub-images, the intermediate image is evenly divided to obtain a number of initial sub-images; Otherwise, an extended zero-padding area is first set on the outer contour of the intermediate image to obtain a zero-padding image including the initial image, the extended area and the zero-padding area, so that the intermediate image can be completely cropped into an integer number of initial sub-images, and then the zero-padding image is evenly divided to obtain several initial sub-images.

3. The image processing method according to claim 2, characterized in that: Referring to the segmentation position of the initial sub-image, the sub-image bodies of all optimized sub-images are spliced ​​to obtain an optimized image, including the following steps: Crop the edge band on the optimized sub-image and keep the sub-image body; Based on a preset weight algorithm, the overlapping areas between the sub-graph bodies in adjacent optimized sub-graphs are merged to obtain the merged overlapping areas; Referring to the segmentation position of the initial sub-graph, the fused overlapping area is spliced ​​to the sub-graph body of the corresponding adjacent optimized sub-graph; The sub-image bodies containing overlapping areas in the optimized sub-images are spliced ​​to obtain an optimized image.

4. The image processing method according to claim 3, characterized in that: Based on a preset weight algorithm, the overlapping areas between subgraph bodies in adjacent optimized subgraphs are merged, including the following steps: Provide a first sub-image and a second sub-image adjacent to each other in the optimized sub-image, and obtain an overlapping area between the corresponding sub-image bodies and pixel points in the overlapping area; According to the distance between the pixel point and the edge of the first sub-image, respectively obtain a first weight of the pixel point in the first sub-image and a second weight of the pixel point in the second sub-image; Obtain a first pixel in the first sub-image and a second pixel in the second sub-image; Based on the first pixel weighted by the first weight and the second pixel weighted by the second weight, a final pixel of the pixel point for fusing the overlapping area is calculated.

5. The image processing method according to claim 4, characterized in that: The preset weight algorithm includes an adaptive weighted fusion algorithm. The calculation formula I(x) of the final pixel obtained by using the adaptive weighted fusion algorithm is: Among them, x is the position of the pixel point in the overlapping area, exp(0) is the first weight of the pixel point in the first sub-image, exp(x) is the second weight of the pixel point in the second sub-image, I1(x) is the first pixel of the pixel point in the first sub-image, and I2(x) is the second pixel of the pixel point in the second sub-image.

6. The image processing method according to claim 1, wherein: The outer contour length of the optimized sub-graph is an integer multiple of the outer contour length of the initial sub-graph; The width of the edge band and the overlapping area in the optimized sub-graph is an integer multiple of the width of the edge band and the overlapping area in the initial sub-graph.

7. The image processing method according to claim 2, wherein: When the intermediate image cannot be completely cropped into an integer number of initial sub-images, the extended zero-filling area is set on two adjacent sides of the outer contour of the intermediate image.

8. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the image processing method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the steps of the image processing method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the steps of the image processing method according to any one of claims 1 to 7.

Images