Image processing method, device, storage medium and equipment

Abstract

The disclosed embodiments disclose an image processing method, apparatus, storage medium and device. The method includes: obtaining edge image information in the expansion direction of the original image; selecting a target expansion method from at least two candidate expansion methods according to the edge image information; and processing the original image using the target expansion method to obtain a target image. By adopting the above technical solution, for the original image that needs to be expanded, a more suitable expansion method can be flexibly selected according to the edge image information in the expansion direction. Compared with the rigid use of a single expansion method, more targeted expansion processing can be achieved to obtain an expanded target image with a better expansion effect.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image processing technology, and in particular to image processing methods, devices, storage media, and equipment. Background Art

[0002] Images can contain rich content and information and are a common information carrier. Images come from a wide range of sources, such as being captured by an image acquisition device or drawn by mapping software. The use scenarios of images are also very diverse, and after the image is generated, it is often necessary to adjust it in some use scenarios to adapt to the personalized needs of the current use scenario, such as adapting to layout requirements or matching the display area size, etc.

[0003] At present, common adjustment methods include cropping and scaling, etc. However, cropping often causes part of the image information to be lost, while scaling often results in distorted images. In order to solve the problems caused by the above conventional adjustment methods, a solution for expanding the original image has emerged. For example, an expansion area can be added to the outside of the original image, and the expansion area can be filled with a solid color or gradient to obtain a larger image. However, the effect of the expanded image obtained by the existing expansion scheme is still not ideal and needs to be improved. Summary of the invention

[0004] The embodiments of the present disclosure provide an image processing method, an apparatus, a storage medium, and a device, which can optimize the existing image expansion processing solution.

[0005] In a first aspect, an embodiment of the present disclosure provides an image processing method, including:

[0006] Obtain edge image information in the expansion direction of the original image;

[0007] Selecting a target expansion mode from at least two candidate expansion modes according to the edge image information;

[0008] The target expansion method is used to process the original image to obtain a target image.

[0009] In a second aspect, an embodiment of the present disclosure provides an image processing device, including:

[0010] An edge image information acquisition module is used to acquire edge image information in an expansion direction of the original image;

[0011] An extension mode selection module, configured to select a target extension mode from at least two candidate extension modes according to the edge image information;

[0012] The image expansion processing module is used to process the original image using the target expansion method to obtain a target image.

[0013] In a third aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image processing method provided in the embodiment of the present disclosure.

[0014] In a fourth aspect, an embodiment of the present disclosure provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, an image processing method as provided in the embodiment of the present disclosure is implemented.

[0015] The image processing scheme provided in the embodiment of the present disclosure obtains edge image information in the expansion direction of the original image, selects a target expansion method from at least two candidate expansion methods according to the edge image information, and processes the original image using the target expansion method to obtain a target image. By adopting the above technical scheme, for the original image that needs to be expanded, a more suitable expansion method can be flexibly selected according to the edge image information in the expansion direction. Compared with the rigid use of a single expansion method, a more targeted expansion processing can be achieved, and an expanded target image with a better expansion effect can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0017] Figure 2 A schematic diagram of image expansion provided by an embodiment of the present disclosure;

[0018] Figure 3 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0019] Figure 4 A schematic diagram of a mask image generation process provided by an embodiment of the present disclosure;

[0020] Figure 5 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0021] Figure 6 A schematic diagram of an image processing process based on a generative adversarial neural network provided in an embodiment of the present disclosure;

[0022] Figure 7 A schematic diagram of an extended sequence provided for an embodiment of the present disclosure;

[0023] Figure 8 A schematic diagram of an image expansion process provided by an embodiment of the present disclosure;

[0024] Fig. 9A schematic diagram of image processing comparison provided by an embodiment of the present disclosure;

[0025] Fig.10 A structural block diagram of an image processing device provided by an embodiment of the present disclosure;

[0026] Fig.11 A structural block diagram of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0028] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0029] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0030] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0031] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0032] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0033] In the following embodiments, each embodiment provides both optional features and examples. The various features recorded in the embodiments can be combined to form multiple optional solutions. Each numbered embodiment should not be regarded as only one technical solution.

[0034] Figure 1 The present invention provides a flowchart of an image processing method according to an embodiment of the present invention. The method can be executed by an image processing device, wherein the device can be implemented by software and / or hardware and can generally be integrated into a computer device. The specific scenario to which the present invention is applicable is not limited, and can be an offline image editing scenario or an online image editing scenario. For example, an image that has been generated can be expanded to adapt to the current display scenario, such as to meet layout requirements or match the display area size, etc. Figure 1 As shown, the method includes:

[0035] Step 101: Obtain edge image information in an expansion direction of an original image.

[0036] The expansion in the disclosed embodiments can be understood as expanding the image size by expanding the image content, rather than expanding the image size by keeping the original image content unchanged, such as by zooming in. The original image can be understood as the image that needs to be expanded during this expansion process. The disclosed embodiments do not limit the source of the original image. For example, it can be an image captured by an image acquisition device such as a camera, an image drawn by a mapping software, an image stored locally on the device, an image obtained through the network, etc. The original image can also be an image that has undergone one or more expansion processes.

[0037] The shape of the original image is not specifically limited, and is generally a polygon, but can also be a circle or other irregular shapes. The expansion direction can be determined according to the current expansion requirements. The area where the image content added to the original image is located is called the expansion area, and the expansion direction is determined according to the relative position relationship between the expansion area and the original image. For ease of explanation, the following takes a more common rectangular image as an example. The rectangle has four boundaries, and each boundary can have a corresponding expansion area. Figure 2 A schematic diagram of image expansion provided by an embodiment of the present disclosure, such as Figure 2As shown, there is a corresponding extended area 202 at the right border of the original image 201, the left border of the extended area 202 coincides with the right border of the original image 201, and the vertical direction of the right border of the original image 201 and the direction pointing to the outside of the original image 201 is the extended direction 203 corresponding to the extended area 202, which can also be understood as a horizontal expansion to the right based on the original image 201. The image content in the extended area 202 is the newly added image content, which may be related to the image content in the original image 201, but is generally not exactly the same.

[0038] For example, the edge image information may include image information of the edge area of ​​the original image corresponding to the expansion direction. The image information may include pixel value, brightness or image content, etc., which is not specifically limited. The edge area can be determined based on the intersection of the original image and the expansion area, and can specifically be an image area near the boundary of the original image. The specific size of the edge area is not limited and can be determined based on the shape and size of the original image and the current expansion requirements. Figure 2 When the original image 201 is a rectangle, the edge region 204 may be a rectangle having the same height as the original image 201 , a smaller width than the original image 201 , and a right border that coincides with the right border of the original image 201 .

[0039] Step 102: Select a target expansion method from at least two candidate expansion methods according to the edge image information.

[0040] In the disclosed embodiment, multiple candidate extension methods may be preset, and the specific contents of the candidate extension methods are not limited, and may be set according to actual needs during specific implementation. The candidate extension methods may include, for example: filling the extension area with a preset image, filling the extension area with a solid color, filling the extension area with a gradient, copying the edge pixels of the original image to the extension area, inputting all or part of the original image into the neural network model to obtain an extended image, etc., and may also be any combination of the above methods.

[0041] Exemplarily, a target extension method is selected from at least two candidate extension methods according to edge image information, and the target extension method may be one or more, and is not specifically limited. Since edge image information can reflect the image condition at the junction of the original image and the extension area, for example, it can reflect the image content complexity, brightness or extension difficulty of the edge area, etc., therefore, the target extension method can be selected more specifically according to the edge image information, and the specific selection basis can be set according to actual conditions.

[0042] Step 103: Process the original image using the target expansion method to obtain a target image.

[0043] For example, after the target expansion method is selected in a targeted and reasonable manner according to the edge image information, the original image is processed using the target expansion method, and the obtained target image has a better expansion effect. The specific processing process may be determined by the selected target expansion method, which will not be described in detail here.

[0044] The image processing method provided in the embodiment of the present disclosure obtains edge image information in the expansion direction of the original image, selects a target expansion method from at least two candidate expansion methods according to the edge image information, and processes the original image using the target expansion method to obtain a target image. By adopting the above technical solution, for the original image that needs to be expanded, a more suitable expansion method can be flexibly selected according to the edge image information in the expansion direction. Compared with the rigid use of a single expansion method, a more targeted expansion processing can be achieved, and an expanded target image with a better expansion effect can be obtained.

[0045] In some embodiments, the at least two candidate extension modes include a first mode and a second mode, the first mode is implemented based on copying edge pixels, and the second mode is implemented based on a neural network model; the selecting a target extension mode from the at least two candidate extension modes according to the edge image information includes: determining the complexity of the image content of the edge area according to the edge image information; when the complexity is lower than or equal to a first preset complexity, determining the first mode as the target extension mode; when the complexity is higher than a second preset complexity, determining the second mode as the target extension mode, wherein the second preset complexity is higher than or equal to the first preset complexity. The advantage of such a setting is that the candidate extension modes can be set more reasonably and a suitable target extension mode can be selected for the current original image. It should be noted that, in addition to the above-mentioned first mode and second mode, the at least two candidate extension modes may also include other modes such as the third mode. When the second preset complexity is higher than the first preset complexity, if the complexity of the image content of the edge area is between the first preset complexity and the second preset complexity, the third mode or other modes may be used as the target extension mode. For ease of explanation, the following description is given by taking an example where at least two candidate extension methods include a first method and a second method, and the second preset complexity is equal to the first preset complexity, and they are collectively referred to as preset complexities.

[0046] Figure 3 A flowchart of an image processing method provided by an embodiment of the present disclosure is shown, which is optimized based on the above optional embodiments. Figure 3 As shown, the method may include:

[0047] Step 301: Obtain edge image information in the expansion direction of the original image.

[0048] Step 302: Determine the complexity of the image content in the edge area according to the edge image information.

[0049] Exemplarily, the complexity can be measured by a first indicator and a second indicator. The first indicator includes the mean square error of the pixel values ​​in the edge area. The larger the mean square error, the higher the corresponding complexity; the second indicator includes the gray value of the edge area. The larger the gray value, the lower the corresponding complexity. The advantage of this setting is that the complexity can be quantified more reasonably, so as to determine the target processing method more accurately.

[0050] Exemplarily, when the pixel value is represented by multiple channels, the mean square error of the pixel value may include the mean square error (also known as mean square error, MSE for short) of the values ​​of each channel. Taking the RGB format as an example, where R represents red, G represents green, and B represents blue, the mean square error of the pixel value can be expressed as the mean square error of the values ​​of each of these three channels, for example, it can be recorded as r_mse, g_mse and b_mse, where r represents the value of the red channel, g represents the value of the green channel, and b represents the value of the blue channel. In addition, the mean square error of the pixel value can also be calculated based on the mean square error of each channel, and the calculation method is such as averaging, etc., which is not specifically limited.

[0051] Exemplarily, the gray value of the edge area can be understood as a light color index, which can be recorded as grey_lev, and the specific calculation method can be set according to the actual situation. Taking the RGB format as an example, the calculation method can be grey_lev = r*0.299+g*0.587+b*0.114, where r represents the value of the red channel, g represents the value of the green channel, and b represents the value of the blue channel. Among them, the value of each channel can be the average value corresponding to all pixels. At this time, the gray value obtained can be directly used as the gray value of the edge area; the corresponding gray value can also be calculated according to each pixel, and then further calculations are performed on the gray values ​​of all pixels (such as averaging, etc.) to obtain the gray value of the edge area.

[0052] Step 303 , determine whether the complexity is lower than or equal to the preset complexity, if so, execute step 304 ; otherwise, execute step 305 .

[0053] Exemplarily, the representation method of the preset complexity can be the same as the complexity, and the thresholds corresponding to each indicator can be set to determine the complexity. Optionally, the threshold corresponding to the first indicator is the first preset complexity threshold, and the threshold corresponding to the second indicator is the second preset complexity threshold. When the first indicator is less than or equal to the first preset complexity threshold (when there are multiple channels, the mean square error corresponding to each channel can be less than or equal to the first preset complexity threshold) or the second indicator is greater than or equal to the second preset complexity threshold, it can be determined that the complexity is lower than or equal to the preset complexity. For example, the first preset complexity threshold is 40 and the second preset complexity threshold is 192. When the mean square error corresponding to the three RGB channels is less than or equal to 40, or the gray value of the edge area is greater than or equal to 192, it can be determined that the complexity is lower than or equal to the preset complexity, which can be specifically expressed as r_mse≤40 and g_mse≤40 and b_mse≤40, or grey_lev≥192.

[0054] Step 304: determine the first mode as the target expansion mode.

[0055] The first method is implemented based on copying edge pixels, wherein the edge pixels may include all pixels or part of pixels in the edge area. For a case where the image content is relatively simple, the first method is adopted to quickly generate a target image with a relatively ideal effect.

[0056] Step 305: determine the second mode as the target extension mode.

[0057] Among them, the second method is implemented based on a neural network model. The neural network model can have a strong learning and generation ability for image information, and is more suitable for expanding complex image content. The disclosed embodiment does not limit the specific type or structure of the neural network model. For example, it can be implemented based on a generative adversarial network (GAN), or based on a variational auto-encoder (VAE), based on a transformation model (Transformer), based on patch matching, and so on. After a lot of research, the inventor found that for a neural network model, if the image content is too simple, the effect of the target image generated in the end is not ideal, because a lot of additional image noise information is often generated.

[0058] Step 306: Process the original image in a target expansion manner to obtain a target image.

[0059] Exemplarily, when the target expansion mode is the first mode, the edge pixels can be copied and filled into the expansion area. For example, taking the expansion direction as horizontal to the right as an example, the pixels of a column on the rightmost edge can be copied, and then for each pixel, a row is filled horizontally. That is to say, for each row of pixels in the expansion area, the pixel values ​​are the same and equal to the pixel values ​​of the pixels on the boundary of the original image closest to the row. The effect after such processing intuitively feels like the original image has been stretched, which can also be called a stretching processing method. In addition, after the expansion area is filled by the copy-fill method, Gaussian filtering and other operations can also be performed on the entire expansion area to make the transition from the original image to the expansion area more natural. The effect after such processing intuitively feels like the original image has been stretched and gradually processed, which can also be called a gradual stretching processing method.

[0060] Exemplarily, when the target extension method is the second method, the original image or a part of the original image can be input into the target image generation network to obtain the target image, or a generated image containing the image content of the extended area can be obtained. Based on the generated image, the extended image can be intercepted and spliced ​​with the original image to obtain the target image.

[0061] The image processing method provided by the embodiment of the present disclosure determines the complexity of the image content in the edge area and then selects a suitable target expansion method. By combining the first method based on copying edge pixels and the second method based on a neural network model, the advantages of the two processing methods can be utilized, so that original images with different edge image complexities can all obtain better expansion effects.

[0062] In some embodiments, when the target extension method includes the second method, the target extension method is used to process the original image, including: intercepting an original sub-image from the original image according to the extension length corresponding to the extension direction; generating a mask image according to the original sub-image, wherein the size of the mask image is larger than the size of the original sub-image; inputting the original sub-image and the mask image into the target image generation network to obtain a generated image output by the target image generation network; intercepting an extended image from the generated image, and generating a target image based on the original image and the extended image. The advantage of this setting is that a part of the original image can be selected as the model input in a more targeted manner, so that a more accurate extension result can be obtained. It should be noted that the intercepted original sub-image may be a part of the original image, or it may be the original image itself, that is, the original sub-image overlaps with the original image.

[0063] For example, the extension length can be set according to actual extension requirements, such as Figure 2As shown, the extension length corresponding to the extension direction 203 may be the width of the extension area 202, that is, the length of the upper boundary and the lower boundary of the extension area 202. When the original sub-image is intercepted, the size of the original sub-image in the extension length direction may be determined according to the product of the extension length and the first preset ratio value, and the first preset ratio value may be freely set, for example, 2.5. Figure 4 A schematic diagram of a mask image generation process provided by an embodiment of the present disclosure is shown in FIG. Figure 4 As shown, an original sub-image 205 is intercepted from the original image 201. The size of the mask image can be set according to the characteristics of the model. Generally, a square mask image is more friendly to the neural network model. The size of the mask image can be set to a square. At this time, the side length of the mask image can be equal to the side length of the long side of the original sub-image, such as Figure 4 As shown, the mask image 206 is in the shape of a square, and the side length is equal to the side length of the long side of the original sub-image 205, that is, the height of the original image 201. The black area 207 in the mask image 206 corresponds to the portion of the original sub-image 205, which can also be called the black range or the original image range. The white area 208 in the mask image 206 corresponds to the portion of the extended area, which can be called the white range or the extended range.

[0064] After the target image generates the generated image output by the network, the extended image can be intercepted as needed. The size of the extended image can be the same as the size of the extended area, or it can be larger than the size of the extended area. If it is larger, it can form an overlapping area with the original image, which is more conducive to subsequent fusion processing. The overlapping area is formed because the boundary length of the extended image in the extension direction is greater than the required extension length. Therefore, in order to make the size of the target image consistent with the expected size, the extended image needs to overlap with part of the original image to form an overlapping area.

[0065] Exemplarily, the target image is generated according to the original image and the extended image, and specifically the original image and the extended image are spliced ​​or merged so that the obtained target image meets the current expected size.

[0066] In some embodiments, the target image generation network is obtained by training a preset network model, and the preset network model is implemented based on a generative adversarial network. The generative adversarial network has a strong learning and generation ability for image information, and can achieve better results for the expansion of complex images. In the process of training the preset network model, it can be divided into multiple training stages, that is, the training process of the preset network model involves multiple training stages, and the types of loss functions corresponding to each training stage increase successively from the first training stage to the last training stage. The advantage of this setting is that the model can be trained more efficiently.

[0067] Exemplarily, the first training stage includes a reconstruction loss function, the second training stage includes a reconstruction loss function, an adversarial loss function, and a perceptual loss function, and the third training stage includes a reconstruction loss function, an adversarial loss function, a perceptual loss function, and a structural information loss function. In the first training stage, i.e., the initial stage, only the reconstruction loss function is set so that the model can converge quickly to learn some coarse-grained information; in the second training stage, i.e., the mid-term stage, the adversarial loss function (adv_loss) and the perceptual loss function (perceptual loss) are added to optimize the details of the generated part; in the third training stage, the structural information loss function (structure similarity, abbreviated as SSIM) is added to improve the brightness and contrast of the generated effect.

[0068] In some embodiments, during the training process for the preset network model, the training samples can come from an open source image library or images collected according to actual needs. In order to improve the robustness of the expansion, that is, the expansion is not a fixed ratio, the ratio of the white range of the mask image in the mask image can be randomly set for the training samples (such as between 0.1 and 0.45).

[0069] In some embodiments, before the original sub-image and the mask image are input into the target image generation network, the method further includes: performing image style recognition on the original image to obtain the target image style; querying a preset correspondence relationship according to the target image style to obtain the target image generation network, wherein the preset correspondence relationship includes the correspondence between different image styles and image generation networks. The advantage of such a setting is that for images of different styles, image generation networks of corresponding styles can be trained separately, so that the generated image output by the target image generation network is more consistent with the style characteristics of the original image. Among them, the image style division method is not limited, for example, it can include natural scenery type, hand-painted style type, cartoon type, character type and building type, etc. Optionally, when performing image style recognition on the original image, an image style classification model can be used to complete it. The image style classification model can be, for example, a convolutional neural network (CNN) image classification model.

[0070] In some embodiments, if it is necessary to set up multiple image generation networks according to image styles, the initial and mid-term training samples may not be classified to improve training efficiency. In the later stage, the corresponding classified training samples are selected according to the image style categories to train the intermediate image generation network that has completed the mid-term training, so as to obtain the image generation network corresponding to the corresponding image style.

[0071] In some embodiments, generating a target image based on the original image and the extended image includes: performing pixel weighted splicing processing on the original image and the extended image to generate a target image; wherein, for each pixel position in the overlapping area of ​​the original image and the extended image, the weight of the corresponding first pixel is negatively correlated with the distance relative to the original image, and the weight of the corresponding second pixel is positively correlated with the distance relative to the original image, and the first pixel is derived from the original image, and the second pixel is derived from the extended image. In other words, the extended image is gradually fused with the original image, and the pixel weights of the seams of the original image and the extended image are mixed. The closer the part is to the original image, the higher the pixel weight of the original image is, and the closer the part is to the extended image, the higher the weight of the extended image is, thereby completing the gradual fusion. The advantage of such a setting is that the transition between the original image and the extended image can be made more natural, further improving the image quality and effect of the target image.

[0072] Figure 5 A flowchart of an image processing method provided by an embodiment of the present disclosure is shown, which is optimized based on the above optional embodiments. Figure 5 As shown, the method may include:

[0073] Step 501: Obtain edge image information in the expansion direction of the original image.

[0074] Exemplarily, before this step, the original image and the expansion target information corresponding to the original image may be obtained first, and the expansion target information may include, for example, the expansion direction, the size and position of the expansion area, etc. The expansion target information may be automatically set by the computer device according to the needs of the current scene, or may be set according to the received user input information. When the size of the expansion area is large, for example, much larger than the size of the original image, the expansion effect may be very poor, and at this time, a prompt message or an error message may be returned.

[0075] Exemplarily, for a rectangular original image, it is possible to first determine whether the length of the boundary parallel to the extension direction is greater than or equal to the length of the boundary perpendicular to the extension direction. If so, the subsequent process is allowed to proceed; if not, it is possible to continue to determine whether the length of the boundary parallel to the extension direction is greater than or equal to a second preset ratio value (less than 1, for example, 0.6) of the length of the boundary perpendicular to the extension direction. If so, the subsequent process is allowed to proceed; if not, the process can return an unprocessable information and end the process.

[0076] Exemplarily, the edge image information may be the pixel value of each pixel in the edge area of ​​a preset pixel width, and the preset pixel width may be, for example, a width of 3 pixels, such as 3 rows or 3 columns of pixels in the original image closest to the boundary of the extended area.

[0077] Step 502: Determine the mean square error and grayscale value of the pixel values ​​in the edge area according to the edge image information.

[0078] Step 503 , determine whether the pixel value mean square error is less than or equal to a preset mean square error threshold; if so, execute step 505 ; otherwise, execute step 504 .

[0079] It should be noted that the grayscale value may be determined first, and then the mean square error of the pixel value may be determined, and the specific order is not limited.

[0080] Step 504 , determine whether the grayscale value is greater than or equal to a preset grayscale value threshold. If so, execute step 505 ; otherwise, execute step 506 .

[0081] Optionally, if the determination results of both determination conditions in step 503 and step 504 are yes, step 505 may be executed; otherwise, step 506 may be executed.

[0082] Step 505: fill the extended area with edge pixels to obtain a target image.

[0083] For example, the pixels of one row or one column closest to the boundary of the extended area in the original image may be copied, and the extended area may be filled in units of columns or rows for each pixel value. After the extended area is filled, a Gaussian filter operation may be performed on the extended area.

[0084] In addition, the extended area filled here (the filled image can be called an extended image) can also be larger than the area actually required to be expanded. In this way, the original image and the extended image can have overlapping areas, and weighted stitching processing is performed on the overlapping areas to make the boundary transition more natural.

[0085] Exemplarily, the extended area is filled with edge pixels to obtain an extended image, and a target image is generated based on the original image and the extended image. Specifically, the original image and the extended image are subjected to pixel weighted splicing processing to generate the target image; wherein, for each pixel position in the overlapping area of ​​the original image and the extended image, the weight of the corresponding first pixel is negatively correlated with the distance relative to the original image, and the weight of the corresponding second pixel is positively correlated with the distance relative to the original image, the first pixel originates from the original image, and the second pixel originates from the extended image.

[0086] Step 506: extract an original sub-image from the original image according to the extension length corresponding to the extension direction.

[0087] Optionally, if the length of the boundary parallel to the extension direction is less than the length of the boundary perpendicular to the extension direction, and is greater than or equal to a second preset ratio value of the length of the boundary perpendicular to the extension direction, the interception step can also be omitted, that is, the original image is directly used as the original sub-image for subsequent operations. In addition, this step can also be performed before step 502 as an image preprocessing part.

[0088] Step 507: Generate a mask image according to the original sub-image.

[0089] The size of the mask image is larger than the size of the original sub-image.

[0090] Step 508: Perform image style recognition on the original image to obtain the target image style, and query the preset corresponding relationship according to the target image style to obtain the target image generation network.

[0091] Among them, the target image generation network is obtained by training a preset network model, and the preset network model is implemented based on a generative adversarial network.

[0092] Step 509: input the original sub-image and the mask image into the target image generation network to obtain the generated image output by the target image generation network.

[0093] Step 510: extract an extended image from the generated image, and generate a target image according to the original image and the extended image.

[0094] Figure 6 A schematic diagram of an image processing process based on a generative adversarial neural network provided in an embodiment of the present disclosure is shown in FIG. Figure 6 As shown, the input on the far left is the original sub-image and the mask image, the upper one is the original sub-image, and the lower one is the mask image. After being processed by the generative adversarial neural network, the output generated image is the same size as the mask image. The generated image is cropped as needed to obtain the extended image.

[0095] The image processing method provided by the embodiment of the present disclosure determines the mean square error and grayscale value of the pixel values ​​in the edge area according to the edge image information, and then selects a suitable target expansion method according to the mean square error and the grayscale value, adopts the edge pixel copying and filling method to process the original image with a simpler edge image, and adopts the generative adversarial network to process the original image with a more complex edge image. Before adopting the generative adversarial network for processing, it also selects a model with a more matching style through image style recognition, and can utilize the respective advantages of different processing methods, so that the original images with different edge image complexities can all obtain a better expansion effect.

[0096] In some embodiments, when there are multiple expansion directions, the method further includes: obtaining an expansion ratio corresponding to each of the multiple expansion directions, where the expansion ratio includes a ratio of an expansion length corresponding to an expansion direction to a side length of an original image corresponding to the expansion direction; determining a current expansion direction in ascending order of the expansion ratios, where starting from the second sorted expansion direction, the original image corresponding to the current expansion direction is the target image corresponding to the previous expansion direction. The advantage of this setting is that by using a sequential expansion method, it is ensured that more original image information can be referenced for each expansion, making the expanded part more coordinated with the original image.

[0097] Figure 7 FIG. is a schematic diagram of an expansion order provided by an embodiment of the present disclosure. As Figure 7 shown, assume that the initial original image is 0, and 0 is expanded up, down, left, and right. The expansion length in the upward expansion direction is the height of 1 in the figure, the expansion length in the downward expansion direction is the height of 4 in the figure, the expansion length in the leftward expansion direction is the width of 3 in the figure, and the expansion length in the rightward expansion direction is the width of 2 in the figure. The upward expansion ratio is the ratio k1 of the height of 1 to the height of 0, the downward expansion ratio is the ratio k4 of the height of 4 to the height of 0, the leftward expansion ratio is the ratio k3 of the height of 3 to the width of 0, and the rightward expansion ratio is the ratio k2 of the height of 2 to the width of 0. Assume k1 < k2 < k3 < k4, then the sorting of the expansion directions is upward, rightward, leftward, and downward, and the expansion is performed in this order. 1 is expanded on the basis of 0, that is, when processing the original image using the target expansion method, the original image corresponding to the upward expansion direction is 0, and the obtained target image is the image after splicing 0 and 1; 2 is expanded on the basis of 0 and 1, that is, when processing the original image using the target expansion method, the original image corresponding to the rightward expansion direction is the image after splicing 0 and 1; and so on, 3 is expanded on the basis of 0, 1, and 2; 4 is expanded on the basis of 0, 1, 2, and 3.

[0098] It should be noted that for a rectangular original image, if there are two expansion directions on the same straight line, such as horizontally left and horizontally right in terms of width, since the edges of the original images referred to are generally not relevant, the order of expansion does not need to be determined.

[0099] The following further illustrates with the example of left - right expansion in the width direction. The order of left - right expansion can be not limited. Figure 8 FIG. is a schematic diagram of an image expansion processing flow provided by an embodiment of the present disclosure. As Figure 8As shown, the image preprocessing module is input with the original image and the direction to be expanded as the width direction, and the expansion range in the two expansion directions, i.e., the expansion length, wherein the expansion length can be limited, for example, the expansion length <= 0.4*min(w, h). The image preprocessing module will first determine whether the original image meets the basic expansion conditions, and when the expansion conditions are met and cropping is required, the image preprocessing module crops the original sub-image, and then calculates the index of the edge image for measuring the content complexity. When the image content is relatively simple, the gradient stretching module processes it to obtain an extended image; when the image content is relatively complex, the style classification module first performs style classification, and then selects the image extension generation model with style matching to generate the extended image. Finally, the image splicing and fusion module performs gradient fusion on the extended part (i.e., the extended image) and the original image, and finally outputs the extended result image, i.e., the target image.

[0100] Fig. 9 A schematic diagram of image processing comparison provided by an embodiment of the present disclosure is shown in FIG. Fig. 9 The upper image is the original image, and the lower image is the target image. From the original image, it can be seen that the edge images in the left and right directions are relatively complex, and both can be expanded using the second method. When expanding, a target generation network corresponding to the natural scenery style can be selected to obtain a better expansion effect.

[0101] Fig.10 This is a structural block diagram of an image processing device provided in the fourth embodiment of the present disclosure. The device can be implemented by software and / or hardware, and can generally be integrated into a computer device, and can perform image processing by executing an image processing method. Fig.10 As shown, the device comprises:

[0102] The edge image information acquisition module 1001 is used to acquire edge image information in the expansion direction of the original image;

[0103] An extension mode selection module 1002, configured to select a target extension mode from at least two candidate extension modes according to the edge image information;

[0104] The image expansion processing module 1003 is used to process the original image using the target expansion method to obtain a target image.

[0105] The image processing device provided in the embodiment of the present disclosure obtains edge image information in the expansion direction of the original image, selects a target expansion method from at least two candidate expansion methods according to the edge image information, and processes the original image using the target expansion method to obtain a target image. By adopting the above technical solution, for the original image that needs to be expanded, a more suitable expansion method can be flexibly selected according to the edge image information in the expansion direction. Compared with the rigid use of a single expansion method, more targeted expansion processing can be achieved, and an expanded target image with better expansion effect can be obtained.

[0106] Optionally, the at least two candidate extension methods include a first method and a second method, the first method is implemented based on copying edge pixels, and the second method is implemented based on a neural network model;

[0107] The step of selecting a target extension mode from at least two candidate extension modes according to the edge image information includes:

[0108] determining the complexity of the image content in the edge area according to the edge image information;

[0109] When the complexity is lower than or equal to a first preset complexity, determining the first mode as a target expansion mode;

[0110] When the complexity is higher than a second preset complexity, the second mode is determined as a target extension mode, wherein the second preset complexity is higher than or equal to the first preset complexity.

[0111] Optionally, a first indicator and a second indicator are used to measure the complexity; the first indicator includes the mean square deviation of pixel values ​​in the edge area, and the larger the mean square deviation, the higher the corresponding complexity; the second indicator includes the grayscale value of the edge area, and the larger the grayscale value, the lower the corresponding complexity.

[0112] Optionally, when the target extension mode includes the second mode, the processing of the original image by using the target extension mode includes:

[0113] intercepting an original sub-image from the original image according to an extension length corresponding to the extension direction;

[0114] generating a mask image according to the original sub-image, wherein a size of the mask image is larger than a size of the original sub-image;

[0115] Inputting the original sub-image and the mask image into a target image generation network to obtain a generated image output by the target image generation network;

[0116] An extended image is intercepted from the generated image, and a target image is generated according to the original image and the extended image.

[0117] Optionally, the device further comprises:

[0118] A style recognition module, used for performing image style recognition on the original image to obtain a target image style before inputting the original sub-image and the mask image into a target image generation network;

[0119] The target image generation network acquisition module is used to query the preset corresponding relationship according to the target image style to obtain the target image generation network, wherein the preset corresponding relationship includes the corresponding relationship between different image styles and image generation networks.

[0120] Optionally, the target image generation network is obtained by training a preset network model, and the preset network model is implemented based on a generative adversarial network. The training process of the preset network model involves multiple training stages, and the types of loss functions corresponding to each training stage increase successively from the first training stage to the last training stage.

[0121] Optionally, the first training stage includes a reconstruction loss function, the second training stage includes a reconstruction loss function, an adversarial loss function, and a perceptual loss function, and the third training stage includes a reconstruction loss function, an adversarial loss function, a perceptual loss function, and a structural information loss function.

[0122] Optionally, generating a target image according to the original image and the extended image includes:

[0123] The original image and the extended image are subjected to pixel weighted stitching processing to generate a target image; wherein, for each pixel position in the overlapping area of ​​the original image and the extended image, the weight of the corresponding first pixel is negatively correlated with the distance relative to the original image, and the weight of the corresponding second pixel is positively correlated with the distance relative to the original image, the first pixel originates from the original image, and the second pixel originates from the extended image.

[0124] Optionally, the device further comprises:

[0125] An expansion ratio acquisition module, used for acquiring the expansion ratio corresponding to each of the multiple expansion directions when there are multiple expansion directions, wherein the expansion ratio includes the ratio of the expansion length corresponding to the expansion direction to the side length of the original image corresponding to the expansion direction;

[0126] The current expansion direction determination module is used to determine the current expansion direction in order from small to large according to the expansion ratio, wherein, starting from the second expansion direction, when the original image is processed using the target expansion method, the original image corresponding to the current expansion direction is the target image corresponding to the previous expansion direction.

[0127] Reference below Fig.11 , which shows a schematic diagram of the structure of a computer device 1100 suitable for implementing the embodiments of the present disclosure. The computer device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.11 The computer device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0128] like Fig.11 As shown, the computer device 1100 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage device 1108 to a random access memory (RAM) 1103. Various programs and data required for the operation of the computer device 1100 are also stored in the RAM 1103. The processing device 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0129] Typically, the following devices may be connected to the I / O interface 1105: input devices 1106 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 1107 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1108 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1109. The communication devices 1109 may allow the computer device 1100 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.11 The computer device 1100 is shown with various devices, but it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead.

[0130] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 1109, or installed from a storage device 1108, or installed from a ROM 1102. When the computer program is executed by the processing device 1101, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0131] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0132] The computer-readable medium may be included in the computer device, or may exist independently without being incorporated into the computer device.

[0133] The computer-readable medium carries one or more programs. When the one or more programs are executed by the computer device, the computer device: obtains edge image information in the expansion direction of the original image; selects a target expansion method from at least two candidate expansion methods based on the edge image information; and processes the original image using the target expansion method to obtain a target image.

[0134] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0135] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0136] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a module does not limit the module itself in some cases. For example, the edge image information acquisition module may also be described as a "module for acquiring edge image information in the extension direction of the original image".

[0137] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0138] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0139] According to one or more embodiments of the present disclosure, there is provided an image processing method, comprising:

[0140] Obtain edge image information in the expansion direction of the original image;

[0141] Selecting a target expansion mode from at least two candidate expansion modes according to the edge image information;

[0142] The target expansion method is used to process the original image to obtain a target image.

[0143] Furthermore, the at least two candidate extension methods include a first method and a second method, the first method is implemented based on copying edge pixels, and the second method is implemented based on a neural network model;

[0144] The step of selecting a target extension mode from at least two candidate extension modes according to the edge image information includes:

[0145] determining the complexity of the image content in the edge area according to the edge image information;

[0146] When the complexity is lower than or equal to a first preset complexity, determining the first mode as a target expansion mode;

[0147] When the complexity is higher than a second preset complexity, the second mode is determined as a target extension mode, wherein the second preset complexity is higher than or equal to the first preset complexity.

[0148] Furthermore, a first indicator and a second indicator are used to measure the complexity; the first indicator includes the mean square deviation of pixel values ​​in the edge area, and the larger the mean square deviation, the higher the corresponding complexity; the second indicator includes the grayscale value of the edge area, and the larger the grayscale value, the lower the corresponding complexity.

[0149] Further, when the target extension method includes the second method, the processing of the original image by using the target extension method includes:

[0150] intercepting an original sub-image from the original image according to an extension length corresponding to the extension direction;

[0151] generating a mask image according to the original sub-image, wherein a size of the mask image is larger than a size of the original sub-image;

[0152] Inputting the original sub-image and the mask image into a target image generation network to obtain a generated image output by the target image generation network;

[0153] An extended image is intercepted from the generated image, and a target image is generated according to the original image and the extended image.

[0154] Furthermore, before inputting the original sub-image and the mask image into the target image generation network, the method further includes:

[0155] Performing image style recognition on the original image to obtain a target image style;

[0156] A preset correspondence relationship is queried according to the target image style to obtain a target image generation network, wherein the preset correspondence relationship includes a correspondence relationship between different image styles and image generation networks.

[0157] Furthermore, the target image generation network is obtained by training a preset network model, and the preset network model is implemented based on a generative adversarial network. The training process of the preset network model involves multiple training stages. From the first training stage to the last training stage, the types of loss functions corresponding to each training stage increase successively.

[0158] Furthermore, the first training stage includes a reconstruction loss function, the second training stage includes a reconstruction loss function, an adversarial loss function, and a perceptual loss function, and the third training stage includes a reconstruction loss function, an adversarial loss function, a perceptual loss function, and a structural information loss function.

[0159] Furthermore, generating a target image according to the original image and the extended image includes:

[0160] The original image and the extended image are subjected to pixel weighted stitching processing to generate a target image; wherein, for each pixel position in the overlapping area of ​​the original image and the extended image, the weight of the corresponding first pixel is negatively correlated with the distance relative to the original image, and the weight of the corresponding second pixel is positively correlated with the distance relative to the original image, the first pixel originates from the original image, and the second pixel originates from the extended image.

[0161] Furthermore, when there are multiple expansion directions, the method further includes:

[0162] Acquire an expansion ratio corresponding to each expansion direction of the multiple expansion directions, wherein the expansion ratio includes a ratio of an expansion length corresponding to the expansion direction to a side length of the original image corresponding to the expansion direction;

[0163] The current expansion direction is determined in order from small to large according to the expansion ratio, wherein, starting from the second expansion direction, when the original image is processed by the target expansion method, the original image corresponding to the current expansion direction is the target image corresponding to the previous expansion direction.

[0164] According to one or more embodiments of the present disclosure, there is provided an image processing apparatus, including:

[0165] An edge image information acquisition module is used to acquire edge image information in an expansion direction of the original image;

[0166] An extension mode selection module, configured to select a target extension mode from at least two candidate extension modes according to the edge image information;

[0167] The image expansion processing module is used to process the original image using the target expansion method to obtain a target image.

[0168] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0169] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0170] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An image processing method, characterized in that: include: Obtain edge image information in the expansion direction of the original image; Selecting a target extension method from at least two candidate extension methods according to the edge image information; the at least two candidate extension methods include a first method and a second method, the first method is implemented based on copying edge pixels, and the second method is implemented based on a neural network model; Processing the original image using the target expansion method to obtain a target image; Wherein, selecting a target extension mode from at least two candidate extension modes according to the edge image information includes: Determine the complexity of the image content of the edge area according to the edge image information; wherein the complexity is measured by a first index and a second index; the first index includes the mean square error of the pixel values ​​in the edge area, and the second index includes the grayscale value of the edge area; When the complexity is lower than or equal to a preset complexity, determining the first mode as a target expansion mode; When the complexity is higher than the preset complexity, determining the second mode as a target expansion mode; When the mean square error is less than or equal to a first preset complexity threshold, or the gray value is greater than or equal to a second preset complexity threshold, it is determined that the complexity is lower than or equal to a preset complexity.

2. The method according to claim 1, characterized in that When the target extension mode includes the second mode, the processing of the original image by using the target extension mode includes: intercepting an original sub-image from the original image according to an extension length corresponding to the extension direction; generating a mask image according to the original sub-image, wherein a size of the mask image is larger than a size of the original sub-image; Inputting the original sub-image and the mask image into a target image generation network to obtain a generated image output by the target image generation network; An extended image is intercepted from the generated image, and a target image is generated according to the original image and the extended image.

3. The method according to claim 2, characterized in that Before inputting the original sub-image and the mask image into the target image generation network, the method further includes: Performing image style recognition on the original image to obtain a target image style; A preset correspondence relationship is queried according to the target image style to obtain a target image generation network, wherein the preset correspondence relationship includes a correspondence relationship between different image styles and image generation networks.

4. The method according to claim 2, characterized in that: The target image generation network is obtained by training a preset network model, and the preset network model is implemented based on a generative adversarial network. The training process of the preset network model involves multiple training stages. From the first training stage to the last training stage, the types of loss functions corresponding to each training stage increase successively.

5. The method according to claim 4, characterized in that The first training stage includes reconstruction loss function, the second training stage includes reconstruction loss function, adversarial loss function and perceptual loss function, and the third training stage includes reconstruction loss function, adversarial loss function, perceptual loss function and structural information loss function.

6. The method according to claim 2, characterized in that The step of generating a target image according to the original image and the extended image comprises: The original image and the extended image are subjected to pixel weighted stitching processing to generate a target image; wherein, for each pixel position in the overlapping area of ​​the original image and the extended image, the weight of the corresponding first pixel is negatively correlated with the distance relative to the original image, and the weight of the corresponding second pixel is positively correlated with the distance relative to the original image, the first pixel originates from the original image, and the second pixel originates from the extended image.

7. The method according to claim 1, characterized in that When there are multiple extension directions, the method further includes: Acquire an expansion ratio corresponding to each expansion direction of the multiple expansion directions, wherein the expansion ratio includes a ratio of an expansion length corresponding to the expansion direction to a side length of the original image corresponding to the expansion direction; The current expansion direction is determined in order from small to large according to the expansion ratio, wherein, starting from the second expansion direction, when the original image is processed by the target expansion method, the original image corresponding to the current expansion direction is the target image corresponding to the previous expansion direction.

8. An image processing device, characterized in that: include: An edge image information acquisition module is used to acquire edge image information in an expansion direction of the original image; an extension mode selection module, configured to select a target extension mode from at least two candidate extension modes according to the edge image information; the at least two candidate extension modes include a first mode and a second mode, the first mode is implemented based on copying edge pixels, and the second mode is implemented based on a neural network model; An image expansion processing module, used for processing the original image using the target expansion method to obtain a target image; Wherein, selecting a target extension mode from at least two candidate extension modes according to the edge image information includes: Determine the complexity of the image content of the edge area according to the edge image information; wherein the complexity is measured by a first index and a second index; the first index includes the mean square error of the pixel values ​​in the edge area, and the second index includes the grayscale value of the edge area; When the complexity is lower than or equal to a preset complexity, determining the first mode as a target expansion mode; When the complexity is higher than the preset complexity, determining the second mode as a target expansion mode; When the mean square error is less than or equal to a first preset complexity threshold, or the gray value is greater than or equal to a second preset complexity threshold, it is determined that the complexity is lower than or equal to a preset complexity.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images