Image processing method and device, electronic equipment and medium

Abstract

The invention relates to an image processing method and apparatus, an electronic device and a medium. The method comprises the steps of obtaining a to-be-processed image and corresponding other images; determining first negative samples, first positive samples, negative sample masks and positive sample masks in the to-be-processed image; on the basis of the position of each first negative sample, cutting each other image to obtain each second negative sample and each corresponding second positive sample; screening each first negative sample and each corresponding second negative sample to obtain a corresponding negative sample image, and fusing the first positive sample and each second positive sample to obtain a positive sample image; and based on each negative sample image, each negative sample mask, the positive sample image and the positive sample mask, performing color adjustment on each negative sample image through histogram matching, and eliminating a spliced edge between each negative sample image and the positive sample image after color adjustment to obtain a gray level image. According to the invention, efficient and automatic image color adjustment is realized, and the efficiency is improved.

Description

Technical Field

[0001] The present disclosure relates to technical fields such as image processing and satellite image processing. Specifically, the present disclosure relates to an image processing method, device, electronic device and medium. Background Art

[0002] As a training system, flight simulators have extremely high requirements for the authenticity and continuity of terrain images. In the prior art, terrain image resources are generally provided for flight simulators through satellite images. However, the existing satellite image resources have obvious splicing marks. This is because satellites shoot the ground at different times and seasons. Due to lighting conditions and vegetation changes, the images have large differences in color tone and content. Simply splicing these images will result in discontinuity of the terrain, affecting the pilot's judgment.

[0003] In the prior art, in order to solve the problem of obvious satellite image splicing traces, the following technical solutions are generally used for processing: 1. Manual PS processing method: Engineers use image processing software (Photoshop, PS for short) and other software to adjust the overall color of satellite images, and use tools such as layer fusion and stamps to repair splicing edges and areas with large color differences, making the image more coordinated and unified.

[0004] 2. Image selection and synthesis method: By comparing multiple satellite image sources, images with better quality in the same area are selected for comprehensive processing.

[0005] 3. Histogram matching method: Using histogram matching technology, the image histogram characteristics of the area to be processed are adjusted to match the histogram characteristics of the target area. Figure 1 to achieve color correction and uniformity.

[0006] However, the above method of the prior art has the following problems: 1. Although the manual post-processing method can improve the image effect, it is labor-intensive and inefficient, and it may also change the original terrain information.

[0007] 2. The comprehensive image selection method requires a lot of manual comparison and analysis work, which is labor-intensive and inefficient. It also cannot completely eliminate the color difference and splicing marks between images. Different film sources need to be purchased separately, which increases costs.

[0008] 3. The histogram matching method has high requirements for region selection and target histogram determination, and the processing effect also has certain limitations.

[0009] Therefore, how to efficiently eliminate the splicing traces of satellite images and maintain the accuracy of terrain information has become an urgent problem to be solved. Summary of the invention

[0010] The embodiments of the present disclosure provide an image processing method, device, electronic device and medium, which realize efficient and automatic image color adjustment of satellite images, solve the problem of obvious splicing marks of satellite images, and improve processing efficiency.

[0011] In one aspect, an embodiment of the present disclosure provides an image processing method, the method comprising: Acquire an image to be processed and at least one other image corresponding to the image to be processed, wherein any one of the other images is shot from a different source than the image to be processed; Determine at least one first negative sample in the image to be processed, and a negative sample mask corresponding to each of the first negative samples, determine a first positive sample in the image to be processed, and a positive sample mask corresponding to the first positive sample, wherein each of the first negative samples corresponds to a position, and the first positive sample is an image of the image to be processed excluding the area where each of the first negative samples is located; Based on the position of each of the first negative samples, crop each of the other images to obtain at least one second negative sample and at least one corresponding second positive sample; Based on the position of any first negative sample, the first negative sample and each of the second negative samples corresponding to the first negative sample are screened to determine an image that meets a preset condition, and a negative sample image corresponding to each of the first negative samples is obtained, and the first positive sample and each of the second positive samples are fused to obtain a positive sample image; Based on the negative sample image corresponding to each of the above-mentioned first negative samples, the negative sample mask corresponding to each of the above-mentioned first negative samples, the above-mentioned positive sample image and the above-mentioned positive sample mask, each of the above-mentioned negative sample images is color adjusted by histogram matching, and the splicing edges between each of the above-mentioned negative sample images and the above-mentioned positive sample image after color adjustment are eliminated to obtain a grayscale image.

[0012] In an optional embodiment, the method further includes: acquiring a target style image, and coloring the grayscale image based on the target style image to obtain a target image, wherein the target style image has the same image size as the grayscale image.

[0013] In an optional embodiment, the above-mentioned grayscale image is colorized based on the target style image to obtain the target image, including: cropping the above-mentioned target style image according to the target size to obtain S cropped target style images, wherein the above-mentioned S is a positive integer; cropping the above-mentioned grayscale image according to the above-mentioned target size to obtain S cropped grayscale images; for any cropped grayscale image, an image with the highest similarity to the cropped grayscale image is determined from the S cropped target style images as a similar style image; for any cropped grayscale image, a style transfer neural network is used to use the above-mentioned similar style image corresponding to the cropped grayscale image as the target color style, and style transfer is performed to obtain the corresponding colored image; the above-mentioned colored images corresponding to each of the above-mentioned cropped grayscale images are spliced ​​to obtain the spliced ​​target image.

[0014] In an optional embodiment, based on the position of any first negative sample, the first negative sample and the second negative samples corresponding to the first negative sample are screened to determine an image that meets a preset condition, and a negative sample image corresponding to each first negative sample is obtained, including: determining a color deviation value of the first positive sample to obtain a first color deviation value; based on the position of any first negative sample, determining the color deviation values ​​of the second negative samples corresponding to the position and the color deviation value of the first negative sample to obtain second color deviation values, determining a color deviation value among the second color deviation values ​​that is closest to the first color deviation value, and determining the negative sample corresponding to the color deviation value that is closest to the first color deviation value as the negative sample image corresponding to the first negative sample.

[0015] In an optional embodiment, the above-mentioned first positive sample and each of the above-mentioned second positive samples are fused to obtain a positive sample image, including: encoding the above-mentioned first positive sample to obtain an encoded first positive sample, encoding each of the above-mentioned second positive samples to obtain an encoded second positive sample; determining the color deviation value of the above-mentioned first positive sample to obtain a first color deviation value, determining the color deviation value of each of the above-mentioned encoded second positive samples to obtain each third color deviation value; based on the above-mentioned first color deviation value and each of the above-mentioned third color deviation values, determining the weights corresponding to the above-mentioned encoded first positive sample and each of the above-mentioned encoded second positive samples to obtain each target weight; based on the above-mentioned target weights, weightedly fusing the above-mentioned encoded first positive sample and each of the above-mentioned encoded second positive samples to obtain a fused positive sample, decoding the above-mentioned fused positive sample to obtain the above-mentioned positive sample image.

[0016] In an optional embodiment, based on the negative sample image corresponding to each of the above-mentioned first negative samples, the negative sample mask corresponding to each of the above-mentioned first negative samples, the positive sample image and the positive sample mask, the negative sample image corresponding to each of the above-mentioned first negative samples is color-adjusted by histogram matching, and the splicing edge between the negative sample image corresponding to each of the above-mentioned first negative samples and the positive sample image after color adjustment is eliminated to obtain a grayscale image, including: for each first negative sample, determining the multiplied image corresponding to the above-mentioned first negative sample according to the following operations: encoding the above-mentioned negative sample image based on the negative sample image corresponding to the above-mentioned first negative sample and the negative sample mask corresponding to the above-mentioned first negative sample , obtain an encoded negative sample image, encode the positive sample image based on the positive sample image and the positive sample mask to obtain an encoded positive sample image; perform histogram matching on the encoded negative sample image according to the positive sample histogram corresponding to the encoded positive sample image to obtain a matching result, and decode the matching result to obtain a decoded image; perform weighted calculation processing on the decoded image to obtain a weighted image; multiply the decoded image with the weighted image to obtain a multiplied image corresponding to the first negative sample; add the multiplied image corresponding to each first negative sample to the positive sample image to obtain a fused image; perform grayscale conversion on the fused image to obtain the grayscale image.

[0017] In an optional embodiment, the above-mentioned cropping of each of the other images based on the position of each of the above-mentioned first negative samples to obtain at least one second negative sample and a corresponding at least one second positive sample includes: based on the position corresponding to any first negative sample, cropping the area corresponding to the position in each of the above-mentioned other images to obtain at least one of the above-mentioned second negative samples corresponding to the position; based on the position corresponding to each of the above-mentioned first negative samples, cropping the area in each of the above-mentioned other images except the area corresponding to each position to obtain at least one of the above-mentioned second positive samples.

[0018] In an optional embodiment, the above-mentioned image to be processed is a satellite image to be processed, and the image size of the satellite image to be processed is an image size that is not supported by the image processing device, and the above-mentioned other images are satellite images taken from other satellites, wherein the above-mentioned other satellite is a satellite different from the satellite that took the above-mentioned satellite image to be processed, and the above-mentioned image processing device is a device used to process the satellite image to be processed.

[0019] In one aspect, an embodiment of the present disclosure provides an image processing device, the device comprising: An acquisition module, used for acquiring an image to be processed and at least one other image corresponding to the image to be processed, wherein any one of the other images is shot from a different source than the image to be processed; A determination module, used to determine at least one first negative sample in the image to be processed, and a negative sample mask corresponding to each of the first negative samples, determine a first positive sample in the image to be processed, and a positive sample mask corresponding to the first positive sample, wherein each of the first negative samples corresponds to a position, and the first positive sample is an image of the image to be processed excluding the area where each of the first negative samples is located; A first processing module, configured to crop each of the other images based on the position of each of the first negative samples to obtain at least one second negative sample and at least one corresponding second positive sample; A second processing module is used to screen the first negative sample and each of the second negative samples corresponding to the first negative sample based on the position of any first negative sample, determine an image that meets the preset conditions, obtain a negative sample image corresponding to each of the first negative samples, and fuse the first positive sample with each of the second positive samples to obtain a positive sample image; The third processing module is used to perform color adjustment on each of the negative sample images corresponding to the first negative samples, the negative sample masks corresponding to the first negative samples, the positive sample images and the positive sample masks through histogram matching, and eliminate the splicing edges between each of the negative sample images and the positive sample images after color adjustment to obtain a grayscale image.

[0020] In an optional embodiment, the device further includes: a fourth processing module, configured to obtain a target style image, and colorize the grayscale image based on the target style image to obtain a target image, wherein the target style image has the same image size as the grayscale image.

[0021] In an optional embodiment, the fourth processing module is specifically used to: crop the target style image according to the target size to obtain S cropped target style images, where S is a positive integer; crop the grayscale image according to the target size to obtain S cropped grayscale images; for any cropped grayscale image, determine the image with the highest similarity to the cropped grayscale image from the S cropped target style images as a similar style image; for any cropped grayscale image, use a style transfer neural network to use the similar style image corresponding to the cropped grayscale image as the target color style, perform style transfer, and obtain the corresponding colored image; splice the colored images corresponding to each of the cropped grayscale images to obtain the spliced ​​target image.

[0022] In an optional embodiment, the second processing module is specifically used to: determine the color deviation value of the first positive sample to obtain a first color deviation value; based on the position of any first negative sample, determine the color deviation values ​​of the second negative samples corresponding to the position and the color deviation value of the first negative sample to obtain the second color deviation values, determine the color deviation value among the second color deviation values ​​that is closest to the first color deviation value, and determine the negative sample corresponding to the color deviation value closest to the first color deviation value as the negative sample image corresponding to the first negative sample.

[0023] In an optional embodiment, the second processing module is specifically used to: encode the first positive sample to obtain the encoded first positive sample, encode each of the second positive samples to obtain the encoded second positive sample; determine the color deviation value of the first positive sample to obtain the first color deviation value, determine the color deviation values ​​of each of the encoded second positive samples to obtain each third color deviation value; based on the first color deviation value and each of the third color deviation values, determine the weights corresponding to the encoded first positive sample and each of the encoded second positive samples to obtain each target weight; based on each of the target weights, perform weighted fusion of the encoded first positive sample and each of the encoded second positive samples to obtain a fused positive sample, and decode the fused positive sample to obtain the positive sample image.

[0024] In an optional embodiment, the third processing module is specifically used for: for each first negative sample, determining the multiplied image corresponding to the first negative sample according to the following operations: encoding the negative sample image based on the negative sample image corresponding to the first negative sample and the negative sample mask corresponding to the first negative sample to obtain the encoded negative sample image, encoding the positive sample image based on the positive sample image and the positive sample mask to obtain the encoded positive sample image; performing histogram matching on the encoded negative sample image according to the positive sample histogram corresponding to the encoded positive sample image to obtain a matching result, and decoding the matching result to obtain a decoded image; performing weighted calculation processing on the decoded image to obtain a weighted image; multiplying the decoded image with the weighted image to obtain the multiplied image corresponding to the first negative sample; adding the multiplied image corresponding to each first negative sample to the positive sample image to obtain a fused image; performing grayscale conversion on the fused image to obtain the grayscale image.

[0025] In an optional embodiment, the first processing module is specifically used for: based on the position of each of the first negative samples, cropping each of the other images to obtain at least one second negative sample and the corresponding at least one second positive sample, including: based on the position corresponding to any one of the first negative samples, cropping the area corresponding to the position in each of the other images to obtain at least one second negative sample corresponding to the position; based on the position corresponding to each of the first negative samples, cropping the area in each of the other images except the area corresponding to each position to obtain at least one second positive sample.

[0026] In an optional embodiment, the above-mentioned image to be processed is a satellite image to be processed, and the image size of the satellite image to be processed is an image size that is not supported by the image processing device, and the above-mentioned other images are satellite images taken from other satellites, wherein the above-mentioned other satellite is a satellite different from the satellite that took the above-mentioned satellite image to be processed, and the above-mentioned image processing device is a device used to process the satellite image to be processed.

[0027] On the one hand, an embodiment of the present disclosure provides an electronic device, which includes a processor and a memory, which are connected to each other; the memory is used to store computer programs; the processor is configured to execute a method provided by any possible implementation of the above-mentioned image processing method when calling the above-mentioned computer program.

[0028] On the one hand, an embodiment of the present disclosure provides a computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement a method provided by any possible implementation of the above-mentioned image processing method.

[0029] On the one hand, the embodiments of the present disclosure provide a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided by any possible implementation of the above-mentioned image processing method.

[0030] The technical solution provided by the present application has the following beneficial effects: obtaining an image to be processed and at least one other image from a different shooting source from the image to be processed; determining a negative sample area in the image to be processed that needs to be color adjusted to obtain at least one first negative sample, and determining a negative sample mask corresponding to each first negative sample, determining a first positive sample excluding the image area where each first negative sample is located, and determining a positive sample mask of the first positive sample; then, based on the position of each first negative sample, cropping each other image from other sources to obtain each second negative sample and each second positive sample; then, the first negative samples, Each second negative sample is screened, and an image that meets the preset conditions is selected, and the image is used as the negative sample image of the position area. In the same way, the negative sample images corresponding to each first negative sample can be obtained, and the first positive sample and each second positive sample are fused to obtain a positive sample image; based on any negative sample image, the negative sample mask corresponding to the negative sample image, the positive sample image and the positive sample mask, the color of the negative sample image is adjusted by histogram matching. After the color adjustment of each negative sample image is completed, the splicing edge between each negative sample image and the positive sample image is eliminated to obtain a grayscale image. In the above manner, when the image to be processed is a satellite image, the splicing traces of the satellite image to be processed can be eliminated, and the accuracy of the terrain information can be maintained to meet the needs of flight simulation training. In the prior art, although the method of using manual post-PS processing can improve the image effect, the workload is large and the original terrain information may also be changed. In the embodiment of the present disclosure, through this automated satellite image processing method, not only can the problem of poor image quality be solved, but also the processing efficiency can be greatly improved, providing a more realistic and reliable terrain environment for flight simulation training. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for describing the embodiments of the present disclosure are briefly introduced below.

[0032] Figure 1 A flowchart of an image processing method provided by an embodiment of the present disclosure; Figure 2 A schematic diagram of an image to be processed with different illumination in a regular area provided by an embodiment of the present disclosure; Figure 3 A schematic diagram of an image to be processed at different times in an irregular area provided by an embodiment of the present disclosure; Figure 4 A schematic diagram of an image to be processed in which the illumination of the entire image is inconsistent with that of other images provided by an embodiment of the present disclosure; Figure 5A schematic diagram of an image to be processed of cloud shadows in an irregular area provided by an embodiment of the present disclosure; Figure 6 A schematic diagram of an image to be processed in which an original image of an irregular region has been fused according to an embodiment of the present disclosure; Figure 7 A schematic diagram of a negative sample mask provided by an embodiment of the present disclosure; Figure 8 A schematic diagram of an image corresponding to a first negative sample provided in an embodiment of the present disclosure; Fig. 9 A schematic diagram of a positive sample mask provided by an embodiment of the present disclosure; Fig.10 A schematic diagram of an image corresponding to a first positive sample provided in an embodiment of the present disclosure; Fig.11 A schematic diagram of a grayscale image provided by an embodiment of the present disclosure; Fig.12 A schematic diagram of a target image provided by an embodiment of the present disclosure; Fig.13 A schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure; Fig.14 A schematic diagram of the structure of an electronic device for an image processing method provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The embodiments of the present disclosure are described below in conjunction with the drawings in the present disclosure. It should be understood that the implementation methods described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions of the embodiments of the present disclosure.

[0034] It will be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "above", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present disclosure refer to that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude the implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the technical field. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or may refer to the one element and the other element establishing a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein indicates at least one of the items defined by the term, for example, "A and / or B" or "A, B" indicates implementation as "A", or implementation as "B", or implementation as "A and B".

[0035] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0036] See also Figure 1 , Figure 1 is a flowchart of an image processing method provided by an embodiment of the present disclosure, such as Figure 1 As shown, the image processing method provided by the present disclosure includes the following steps: Step S101, obtaining an image to be processed and at least one other image corresponding to the image to be processed, wherein any one of the other images is shot from a different source than the image to be processed; Step S102, determining at least one first negative sample in the image to be processed, and a negative sample mask corresponding to each of the first negative samples, determining a first positive sample in the image to be processed, and a positive sample mask corresponding to the first positive sample, wherein each of the first negative samples corresponds to a position, and the first positive sample is an image of the image to be processed excluding the area where each of the first negative samples is located; Step S103, based on the position of each of the first negative samples, cropping each of the other images to obtain at least one second negative sample and at least one corresponding second positive sample; Step S104, based on the position of any first negative sample, the first negative sample and each of the second negative samples corresponding to the first negative sample are screened to determine an image that meets a preset condition, and a negative sample image corresponding to each of the first negative samples is obtained, and the first positive sample and each of the second positive samples are fused to obtain a positive sample image; Step S105, based on the negative sample image corresponding to each of the above-mentioned first negative samples, the negative sample mask corresponding to each of the above-mentioned first negative samples, the above-mentioned positive sample image and the above-mentioned positive sample mask, color adjustment is performed on each of the above-mentioned negative sample images through histogram matching, and the splicing edges between each of the above-mentioned negative sample images and the above-mentioned positive sample images after color adjustment are eliminated to obtain a grayscale image.

[0037] Optionally, the image processing method in the embodiment of the present disclosure may be executed by an image processing device, wherein the image processing device may be a server, which may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server or server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The image processing device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a laptop computer, a digital broadcast receiver, a MID (Mobile Internet Devices), a PDA (Personal Digital Assistant), a desktop computer, a vehicle-mounted terminal (such as a vehicle-mounted navigation terminal), a smart speaker, a smart watch, etc.

[0038] The network used by the above-mentioned image processing method when performing image processing may include but is not limited to: a wired network, a wireless network, wherein the wired network includes: a local area network, a metropolitan area network and a wide area network, and the wireless network includes: Bluetooth, Wi-Fi and other networks that realize wireless communication.

[0039] Optionally, in an embodiment of the present disclosure, the above-mentioned image processing method specifically includes the following steps: An image to be processed and at least one other image corresponding to the image to be processed is obtained, wherein, in an optional embodiment, the image to be processed is a satellite image to be processed, and the image size of the satellite image to be processed is an image size that is not supported by the image processing device, and the other image is a satellite image taken from other satellites, wherein the other satellite is a satellite different from the satellite that took the satellite image to be processed, and the image processing device is a device for processing the satellite image to be processed.

[0040] Specifically, the image to be processed is a large-scale image and a satellite image. The large scale can be understood as a situation where the resolution of the satellite image to be processed is very large, such as an image with a size of 16384*16384. The image size of the above-mentioned satellite image to be processed is an image size that is not supported by the image processing device, which can be understood as: due to the limitations of the GPU video memory and computing power of the image processing device, the image processing device cannot support the processing of the large-scale satellite image to be processed. The image to be processed and each other image can be understood as satellite images from multiple sources, and multiple sources refer to images taken by two or more different satellites at the same geographical location. It can be understood that the embodiments of the present disclosure are applicable to situations where there are satellite images with multiple sources and color adjustment is required.

[0041] Optionally, the image to be processed in the embodiment of the present disclosure can be understood as an image that needs to be color adjusted, which can be classified into the following types: regular area with different illumination ( Figure 2 ), irregular areas at different times ( Figure 3 ), the lighting of the whole map is inconsistent with other maps ( Figure 4 )、Irregular area cloud shadows( Figure 5 )、The original image of the irregular area has been fused ( Figure 6 ).

[0042] in, Figure 2 A schematic diagram of an image to be processed with different illumination in a regular area provided by an embodiment of the present disclosure, such as Figure 2 As shown, Figure 2 There are two regular areas in the image. The two areas are taken by satellites in different lighting environments in a short period of time (such as a few days) and then stitched together. The content of this image is only different in the main color, and the content is not much different. Among them, the regular area refers to a simple concave and convex polygon formed by a small number of points.

[0043] Figure 3 A schematic diagram of an image to be processed at different times in an irregular area provided by an embodiment of the present disclosure, such as Figure 3 As shown, Figure 3 It is taken by satellite in similar lighting environment but at a long time (such as more than a few weeks), and the area after stitching is irregular. Regular area means that a small number of points can form a simple concave and convex polygon. Among them, irregular area means the area that cannot be selected by a simple polygon.

[0044] Figure 4 A schematic diagram of an image to be processed in which the illumination of the entire image is inconsistent with that of other images provided in an embodiment of the present disclosure, such as Figure 4 As shown, Figure 4 The entire image is taken by the satellite under non-ideal lighting conditions, seasons, time and other conditions. There is no obvious difference in color tone in the image, and the color tone of the entire image is unified, so the color of the entire image needs to be adjusted.

[0045] Figure 5 A schematic diagram of an image to be processed of cloud shadows in an irregular area provided by an embodiment of the present disclosure, such as Figure 5 As shown, Figure 5 There is a thin cloud below the satellite when taking the photo, which appears as shadowed or whitish irregular areas on the satellite image.

[0046] Figure 6 A schematic diagram of an image to be processed in which an original image of an irregular region has been fused is provided in an embodiment of the present disclosure, such as Figure 6 As shown, Figure 6 In the above cases, the images have been modified and adjusted by others when they were acquired.

[0047] Optionally, after obtaining the image to be processed in the above manner, it is necessary to determine the first negative sample in the image to be processed, and the image to be processed may include at least one first negative sample, wherein the negative sample can be understood as the area to be processed in the image to be processed (also referred to as the image area to be corrected), and the area to be processed is a specific geographical area in the image to be processed or an area with defective parts in the image. The positive sample is an area with a target color style. After determining each first negative sample, the other areas except each first negative sample area are determined as the first positive sample of the image to be processed. And the negative sample masks corresponding to each first negative sample are determined respectively, and each negative sample mask is obtained, and the positive sample mask corresponding to the first positive sample is determined.

[0048] Optionally, the embodiment of the present disclosure provides a method for determining a region to be processed (i.e., an image region to be corrected), wherein the region to be processed is selected mainly by manual selection, and the selection process is specifically as follows: Step S1, the image to be processed (i.e., the satellite image to be processed) used in the embodiment of the present disclosure is used as the original image, with a resolution of 16384*16384 (but not limited to this size), and the original image is recorded as Figure A. An original image contains positive and negative samples, which must contain positive samples, and the number of negative samples is at least one. After the image processing device reads Figure A, it reduces the size of Figure A, wherein the reduced size is 16384 / 2048=8 times, that is, the size of Figure A is reduced to 2048*2048 (but not limited to this size), and the Figure A reduced to 2048*2048 is recorded as Figure B and displayed. After compression, Figure B still includes positive and negative samples, and the number is the same as that of Figure A. Since the resolution of Figure A is too large, not all pixels are displayed on the display screen, so the rough selection-fine selection method is adopted in the embodiment of the present disclosure to obtain more accurate boundary coordinates of negative samples.

[0049] Step S2, select a point on the boundary area of ​​the negative sample on Figure B, the coordinates of the point are (xb, yb), which is regarded as the coarse selected coordinates. Accordingly, in Figure A, the image with coordinates in the interval (xb*8-100~xb*8+100, yb*8-100 ~yb*8+100) is intercepted and displayed, and the intercepted image is recorded as Figure C.

[0050] Step S3, select a point on the graph C, the coordinates of the point are (xc, yc), then the coordinates of the point restored to the original image or on the original image are (xb*8+xc-100, yb*8+yc-100), and the coordinates of the point are the boundary coordinates of the selected negative sample selected after two selections.

[0051] Step S4, repeating steps S2 and S3 until a negative sample area is selected.

[0052] Step S5: if there are multiple negative sample areas in an original image, then repeat steps S2, S3, and S4 to finally obtain at least one polygon, and process the at least one polygon to obtain the corresponding at least one first negative sample.

[0053] Among them, based on any polygon, the specific process of obtaining the corresponding first negative sample is as follows: after completing the above manual selection area, all points selected for all negative samples, that is, all points on the boundary area of ​​the selected negative samples in Figure A, constitute a polygon, and the image area corresponding to the polygon is a first negative sample. The inside of the first negative sample is marked as 255 and the outside is marked as 0, and a negative sample mask can be obtained, such as Figure 7 As shown, the white part is Figure 7 The interior of the polygon is the mask part. The image corresponding to the original image of the internal area of ​​the polygon is retained to obtain the graphic area of ​​the first negative sample. The image corresponding to the first negative sample is as follows: Figure 8 shown.

[0054] According to the above method, all the first negative samples in the original image and the negative sample masks corresponding to each first negative sample can be obtained, and then the sum of all the negative sample masks is negated to obtain the positive sample mask, such as Fig. 9 As shown in , the white part is the positive sample mask. By cropping the image to be processed according to the positive sample mask, the first positive sample can be obtained, as shown in Fig.10 As shown, Fig.10 The part with the image is the first positive sample.

[0055] After determining the first negative samples and the first positive samples, the other images may be cropped based on the positions of the first negative samples to obtain the second negative samples and the corresponding second positive samples. In an optional embodiment, the above-mentioned cropping of the other images based on the positions of the first negative samples to obtain at least one second negative sample and the corresponding at least one second positive sample includes: Based on the position corresponding to any one of the first negative samples, the region corresponding to the position in each of the other images is cropped to obtain at least one of the second negative samples corresponding to the position; Based on the positions corresponding to the first negative samples, the regions in the other images except the regions corresponding to the positions are cropped to obtain at least one second positive sample.

[0056] The following is an example to illustrate the cutting process: Assume that there are three first negative samples in the image to be processed, namely, negative sample 1, negative sample 2, and negative sample 3, wherein negative sample 1 is at position 1 of the original image, negative sample 2 is at position 2 of the original image, and negative sample 3 is at position 3 of the original image. In addition to the original image, there are three sources, namely, source 1, source 2, and source 3. Based on position 1, the images of source 1, source 2, and source 3 at the positions corresponding to position 1 are cropped out to obtain three negative samples related to source 1, source 2, and source 3 and position 1. These three negative samples are the three second negative samples obtained based on position 1. Similarly, three second negative samples related to source 1, source 2, and source 3 and position 2, as well as three second negative samples related to source 1, source 2, and source 3 and position 3 can be obtained. Finally, 9 second negative samples can be obtained.

[0057] Then, taking source 1 as an example, after clipping source 1 based on position 1, position 2, and position 3, three second negative samples can be obtained, and the remaining part after clipping source 1 is used as a second positive sample. Similarly, the second positive samples corresponding to source 2 and source 3 can be obtained, and finally three second positive samples can be obtained.

[0058] Optionally, the following operations are performed on each second negative sample and each second positive sample obtained by the above method: based on the position of any first negative sample, the first negative sample and each second negative sample at the same position are screened, and a negative sample with the color style closest to the first positive sample is screened out as the negative sample image at the position. In this way, the negative sample images corresponding to each first negative sample can be obtained. In addition, the first positive sample and each second negative sample are fused to obtain a positive sample image.

[0059] Then, the following operations are performed on each negative sample image: based on the negative sample image, the negative sample mask of the first negative sample corresponding to the negative sample image, the positive sample image and the positive sample mask, the negative sample image is color adjusted.

[0060] After all negative sample images have completed the color adjustment step, the splicing edges between each negative sample image and the positive sample image after color adjustment are eliminated to obtain a grayscale image. Fig.11 shown.

[0061] In an optional embodiment, the above method further includes: A target style image is obtained, and the grayscale image is colored based on the target style image to obtain a target image, wherein the target style image has the same image size as the grayscale image.

[0062] Among them, the selection criteria of the above-mentioned target style image can be determined according to actual needs, which is the ideal color style of the satellite image selected manually. For example, the color style of the target style image can be mostly green forest, with some towns and some loess. This standard is related to the geographical location of the processing. For example, if the satellite image taken is a desert area, a yellow satellite image of a larger area can be selected as the target style image. It can be understood that the embodiment of the present disclosure does not limit the specific color style of the target style image.

[0063] Then, the grayscale image can be colored based on the target style image through the style transfer neural network to obtain the target image. The specific style transfer process is described in detail below.

[0064] Through the embodiment of the present disclosure, the area to be processed on the image to be processed is first selected by manual selection, and targeted processing can be performed on a specific geographical area or image defects to improve the pertinence of the method; then the selected area is screened, and the advantages of multiple sources can be integrated; the overall color is adjusted using the histogram matching technology, which can effectively eliminate the color difference between images and achieve a unified tone effect. By eliminating the splicing edge, the natural transition of the edge is achieved, and the obvious splicing traces are eliminated. This step can greatly improve the visual effect of the image; finally, the style transfer neural network is used to color the target style image, and the style transfer is achieved; the present disclosure makes full use of the advantages of deep learning to achieve the automation and intelligence of image processing. At the same time, combined with steps such as manual selection and color adjustment, the limitations that may be caused by a single algorithm are effectively avoided, and the quality and efficiency of image processing are greatly improved. In addition, the embodiment of the present disclosure can perform well in different processing tasks such as different illumination in regular areas, different time in irregular areas, inconsistent illumination of the whole image with other images, cloud shadows in irregular areas, and fusion of different original images in irregular areas. The processing results are unified in style and meet the actual use needs.

[0065] In an optional embodiment, coloring the grayscale image based on the target style image to obtain the target image includes: Crop the target style image according to the target size to obtain S cropped target style images, where S is a positive integer; Cropping the grayscale image according to the target size to obtain S cropped grayscale images; For any cropped grayscale image, determine the image with the highest similarity to the cropped grayscale image from the S cropped target style images as the similar style image; For any cropped grayscale image, use the style transfer neural network to transfer the above-mentioned similar style image corresponding to the cropped grayscale image as the target color style to obtain the corresponding colored image; The colored images corresponding to the cropped grayscale images are spliced ​​to obtain the spliced ​​target image.

[0066] Optionally, a grayscale image can be obtained through the above description. Due to the performance limitation of the image processing device, the grayscale image can be segmented, and each segment obtained after segmentation is used to match the part of the style image (i.e., the target style image) with the highest similarity to it, and the style image with the highest similarity is used for style transfer. The following is an example to illustrate the specific process of style transfer, which is as follows: Step 1, image similarity matching: The acquired target style image, i.e., the target style image, is cropped, wherein the image size of the target style image is 16384*16384. It can be understood that the disclosed embodiment does not limit the image size of the target style image. The target style image is cropped into 256 images with an image size of 1024*1024, denoted as Gm (G1...G256), wherein the 256 cropped images correspond to the S cropped target style images in the foregoing text, and the S is determined by the image size of the target style image and the image size after cropping, which is not limited in the disclosed embodiment. The content image to be colored (i.e., the grayscale image to be colored) is cropped, where the image size of the content image to be colored is 16384*16384, and it is cropped into images with an image size of 1024*1024, a total of 256 images, denoted as Hn (H1...H256), where the 256 cropped images correspond to the S cropped grayscale images in the previous text.

[0067] Then, use the Siamese Network to calculate the similarity between each segmented content image Hn and all images in G1 to G256. Take H1 as an example, calculate the similarity between H1 and each image in G1 to G256, select the image with the highest similarity between H1 and G1 to G256, and record the corresponding relationship. Assuming that G6 is the image with the highest similarity to H1, then G6 is the similar style image corresponding to H1 (also called similar style image).

[0068] Step 2, style transfer: According to the previous part, each segmented content image Hn to be colored has a corresponding most similar similar style image Gm. For any Hn, take the Gm corresponding to the Hn as the target style image, and take the Hn as the content image to be colored, use the style transfer neural network to perform style transfer, and obtain the colored image In. Optionally, in practical applications, the style transfer neural network can also be optimized by an optimization algorithm (Limited-memory BFGS, LBFGS for short), where LBFGS is an optimizer for the style transfer neural network. After processing the 256 segmented content images Hn to be colored, 256 colored images In (I1...I256) can be obtained. I1...I256 are spliced ​​to obtain the target image, where the target image is as follows. Fig.12 shown.

[0069] Through the embodiments of the present disclosure, style transfer is performed on grayscale images through proximity matching and style transfer. This method of achieving the purpose of overall style transfer by local style transfer enables each content image to be colored after segmentation to be matched with the style image most similar to it for style transfer, thereby improving the accuracy of style transfer and the overall color visual effect of the target image.

[0070] In an optional embodiment, fusing the first positive sample and each of the second positive samples to obtain a positive sample image includes: Encoding the first positive sample to obtain an encoded first positive sample, and encoding each of the second positive samples to obtain an encoded second positive sample; Determine the color deviation value of the first positive sample to obtain a first color deviation value, determine the color deviation value of each of the encoded second positive samples to obtain each third color deviation value; Based on the first color deviation value and each of the third color deviation values, determining the weights corresponding to the encoded first positive sample and each of the encoded second positive samples, to obtain each target weight; Based on the above target weights, the above encoded first positive samples and the above encoded second positive samples are weightedly fused to obtain fused positive samples, and the above fused positive samples are decoded to obtain the above positive sample images.

[0071] Optionally, the following is a method for determining a positive sample image provided in an embodiment of the present disclosure in combination with an example, and the specific steps are as follows: Assume that there are n sources and m negative sample areas, where n is the number of images including the original image (i.e., the image to be processed), and m is the number of negative sample areas contained in the original image (the same as the number of the first negative samples in the image to be processed, the cropped positive samples P1...Pn (including the above first positive samples and each second positive sample), the negative samples are N11...Nnm (including the above first negative samples and each second negative sample), where n is a positive integer and m is a positive integer. The sample images (i.e., positive samples P1...Pn and negative samples N11...Nnm) are encoded according to their corresponding masks, and then multi-source fusion is performed. The specific process is as follows: Encode the positive samples P1...Pn respectively (the encoding and decoding methods can refer to the encoding and decoding methods of positive sample images and negative sample images in the following text), and obtain the encoded positive samples … , calculate the encoded positive sample … The color deviation values ​​corresponding to each other are obtained to obtain color deviation values ​​d1...dn, wherein each color deviation value d1...dn includes the color deviation value of the first positive sample, and the color deviation value corresponding to the first positive sample in each color deviation value d1...dn is recorded as d0 (that is, the above-mentioned first color deviation value), and the color deviation values ​​in each color deviation value d1...dn except d0 (that is, except the first color deviation value) are used as the above-mentioned third color deviation values. … The weights are calculated according to the weighted fusion formula, and weighted fusion is performed to obtain the fused positive sample, which is recorded as .

[0072] The weight of weighted fusion is the following formula (1): Formula (1) in, It is a picture The weight of The larger the value of is, the smaller the weight of weighted fusion is. Corresponding to the above target weights.

[0073] image Calculate according to the following formula (2): Formula (2)

[0074] Then to the Decode and obtain a positive sample image, which is recorded as P0.

[0075] It should be noted that the color deviation value ‌ refers to the difference between the formula calculated by the computer and the target standard. In image processing, color deviation refers to the difference between the color of the image and the original hue, which is usually caused by improper exposure time, film, flash, etc. The calculation of the color deviation value usually involves color comparison and difference calculation. In computer calculation, the color deviation value refers to the difference between the formula calculated by the computer under a certain light source and the target standard. The smaller the value, the higher the accuracy. Among them, the calculation of the color deviation value is a prior art and will not be described in detail in the embodiments of the present disclosure.

[0076] Through the embodiments of the present disclosure, the second positive samples from various film sources and the first positive samples of the image to be processed can be effectively integrated, thereby achieving comprehensive use of image information and improving the utilization rate of image information in the image processing process.

[0077] In an optional embodiment, based on the position of any first negative sample, the first negative sample and each of the second negative samples corresponding to the first negative sample are screened to determine an image that meets a preset condition, and a negative sample image corresponding to each of the first negative samples is obtained, including: Determine the color deviation value of the first positive sample to obtain a first color deviation value; Based on the position of any first negative sample, the color deviation values ​​of the second negative samples corresponding to the position and the color deviation value of the first negative sample are determined to obtain the second color deviation values, and the color deviation value closest to the first color deviation value among the second color deviation values ​​is determined, and the negative sample corresponding to the color deviation value closest to the first color deviation value is determined as the negative sample image corresponding to the first negative sample.

[0078] The preset condition may be whether the color deviation value of the negative sample corresponding to the negative sample image is closest to the first color deviation value. If so, the negative sample closest to the first color deviation value may be determined as a negative sample image. In order to more clearly understand how to determine the negative sample image, see the following description for details.

[0079] Optionally, the method for determining a positive sample image is described above. In combination with the method for determining a positive sample image, an embodiment of the present disclosure provides a method for determining a negative sample image. The specific steps are as follows: For a certain negative sample area j, 1<=j<=m, calculate the color deviation values ​​d1j…dnj of N1j…Nnj, and calculate the color deviation value d0j of the first negative sample corresponding to the negative sample area j in the original image, select the negative sample closest to the d0 value (the color deviation value of the first positive sample in the image to be processed, that is, the above-mentioned first color deviation value) from d0j, d1j…dnj, and record the negative sample as N0j. N0j is the negative sample image corresponding to the above-mentioned negative sample area j, and is also the negative sample image corresponding to the first negative sample corresponding to the negative sample area j.

[0080] The following is an example to illustrate the specific process: Assume that the number of film sources n is 4, the negative sample area m is 3, and the negative sample area j, 1<=j<=m, the original image (i.e., the image to be processed) includes three negative sample areas, namely negative sample area 1, negative sample area 2, and negative sample area 3. It can be understood that the three negative sample areas correspond to the three first negative samples of the original image (i.e., the image to be processed).

[0081] For negative sample area 1, that is, when j=1, calculate the color deviation values ​​of each film source corresponding to the negative sample area, that is, calculate the color deviation values ​​corresponding to N11, N21, N31, and N41 corresponding to negative sample area 1, and obtain d11, d21, d31, and d41. Calculate the color deviation value d01 of the first negative sample in the original image corresponding to the negative sample area 1, and select the value closest to the d0 value from d11, d21, d31, d41, and d01. Assume it is d21, then the negative sample N21 corresponding to d21 is determined as the negative sample image corresponding to the negative sample area 1.

[0082] For the negative sample area 2, that is, when j=2, the color deviation values ​​of each film source corresponding to the negative sample area are calculated, that is, the color deviation values ​​corresponding to N12, N22, N32, and N42 corresponding to the negative sample area 2 are calculated to obtain d12, d22, d32, and d42. The color deviation value d02 of the first negative sample in the original image corresponding to the negative sample area 2 is calculated, and the value closest to the d0 value is selected from d12, d22, d32, d42, and d02. Assuming it is d32, the negative sample N32 corresponding to d32 is determined as the negative sample image corresponding to the negative sample area 2.

[0083] For the negative sample area 3, that is, when j=3, the color deviation values ​​of each film source corresponding to the negative sample area are calculated, that is, the color deviation values ​​corresponding to N13, N23, N33, and N43 corresponding to the negative sample area 3 are calculated, and d13, d23, d33, and d43 are obtained. The color deviation value d03 of the first negative sample in the original image corresponding to the negative sample area 3 is calculated, and the value closest to the d0 value is selected from d13, d23, d33, d43, and d03. Assuming it is d13, the negative sample N13 corresponding to d13 is determined as the negative sample image corresponding to the negative sample area 3.

[0084] Finally, three negative sample areas corresponding to the three negative sample areas in the original image are obtained.

[0085] Through the embodiments of the present disclosure, the color deviation value can be calculated to select the negative sample closest to the first positive sample in color from the negative samples at each position, thereby paving the way for subsequent adjustment of the color of the image to be processed and improving the accuracy of image color adjustment.

[0086] In an optional embodiment, based on the negative sample image corresponding to each of the first negative samples, the negative sample mask corresponding to each of the first negative samples, the positive sample image and the positive sample mask, color adjustment is performed on the negative sample image corresponding to each of the first negative samples by histogram matching, and the splicing edge between the negative sample image corresponding to each of the first negative samples after color adjustment and the positive sample image is eliminated to obtain a grayscale image, including: For each first negative sample, the multiplied image corresponding to the first negative sample is determined by the following operation: Based on the negative sample image corresponding to the first negative sample and the negative sample mask corresponding to the first negative sample, the negative sample image is encoded to obtain an encoded negative sample image, and based on the positive sample image and the positive sample mask, the positive sample image is encoded to obtain an encoded positive sample image; According to the positive sample histogram corresponding to the encoded positive sample image, histogram matching is performed on the encoded negative sample image to obtain a matching result, and the matching result is decoded to obtain a decoded image; Performing weighted calculation processing on the above decoded image to obtain a weighted image; Multiplying the decoded image by the weight image to obtain a multiplied image corresponding to the first negative sample; Add the multiplied image corresponding to each first negative sample to the positive sample image to obtain a fused image; The fused image is subjected to grayscale conversion to obtain the grayscale image.

[0087] Optionally, the embodiment of the present disclosure provides a method for determining a grayscale image, which can use a histogram matching and an iterative color fusion method to determine the grayscale image. The specific process is as follows: Since the positive sample image and each negative sample image are irregular areas, and there are large black areas in the positive sample image and each negative sample image, the effect of directly performing histogram matching is very poor. In the embodiment of the present disclosure, the positive sample image and each negative sample image are encoded according to their respective corresponding masks, and then histogram matching is performed, and finally the result is decoded to generate a normal image. The specific steps are as follows: Among them, in step 1 to step 4, any negative sample image and positive sample image are used as examples for explanation. It can be understood that each negative sample image needs to perform the following steps 1 to step 4 with the positive sample image.

[0088] Step 1: Create an empty array. The positive and negative sample images and the positive and negative sample masks have the same size, i.e., 16384*16384. For any negative sample image, traverse the negative sample image and the negative sample mask corresponding to the negative sample image at the same time. When the pixel value of the traversed negative sample mask is 255, add the pixel at the corresponding position to the array. When the pixel value of the traversed negative sample mask is 0, skip it. The array finally generated is the result of the conversion of the negative sample image. After the above processing, the negative sample image is converted to obtain the converted negative sample image (i.e., the encoded negative sample image).

[0089] Create a new empty array. For the positive sample image, traverse the positive sample image and the positive sample mask corresponding to the positive sample image at the same time. When the pixel value of the traversed positive sample mask is 255, add the pixel at the corresponding position to the array. When the pixel value of the traversed positive sample mask is 0, skip it. The final generated array is the result of the conversion of the positive sample image. After the above processing, the positive sample image is converted to obtain the converted positive sample image (that is, the encoded positive sample image).

[0090] Step 2: Perform histogram matching on any encoded negative sample image according to the positive sample histogram corresponding to the encoded positive sample image to obtain a matching result, and decode the matching result. The method for calculating the histogram is an existing mature technology and will not be described in detail.

[0091] The decoding method is as follows: create a completely black image C, traverse the negative sample mask corresponding to the above-encoded negative sample image, and when the mask value of a certain position (xd, yd) of the negative sample mask is 255, take out the first pixel of the array to be decoded (i.e. the array corresponding to the matching result) and put it into the position (xd, yd) corresponding to the image C. When a mask value of 255 is found, a pixel is taken out of the array to be decoded, and then the negative sample mask is traversed until all negative sample masks are traversed, decoding is completed, and the decoded image C is obtained, where the decoded image C is the image storing the result after decoding.

[0092] Step 3, create a new image D with all pixel values ​​1 and size 16384*16384. Calculate the brightness ratio of each pixel at the boundary of the decoded image C to the corresponding positive sample image. The specific method is as follows: Assume that the pixel coordinates of a point on the decoded image C are (xe, ye), first calculate the average value of all non-zero positive sample pixels within the center side length of 100 pixels of the pixel at the point (referring to the average value of all pixels in the positive sample area with a center side length of 100 at the point), recorded as AverP, calculate the average value of all non-zero negative sample pixels within the center side length of 100 pixels of the pixel at the point (referring to the average value of all pixels in the negative sample area with a center side length of 100 at the point), recorded as AverN. The brightness ratio is R = AverP / AverN, then the point value of the image D coordinate (xe, ye) is assigned to R. Among them, if the positive sample area is not included within the 100-pixel range of the point, then AverP=0, R=0, and all cases of R=0 are changed to R=1, that is, the brightness is not processed. Through the above method, the weighted image D after assigning values ​​to the image D can be obtained.

[0093] Step 4, traverse the weighted image D. If the calculated brightness ratio (non-1) exists in the 3*3 area near a certain pixel of the weighted image D, then the brightness ratio of the point is the average of the brightness ratios (non-1) in the nearby 3*3 area, otherwise it is 1. Repeat step 4 1000 times, and finally a grayscale brightness weight transition band with a width of 1000 can be obtained at the adjacent contact boundary of the positive sample image and the negative sample image. After multiplying this weighted transition band with the negative sample image, the grayscale difference between the adjacent contact boundaries of the positive sample image and the negative sample image can be largely eliminated, so as to achieve smooth image transition. This step can be selected according to the situation. If the difference is too small, it can be omitted. If the difference is too large, this step is adopted. The size of the difference can be specified in advance according to the requirements.

[0094] Step 5: For any decoded image C, multiply the decoded image C with the weight image D to obtain the multiplied image corresponding to the decoded image C, and then add the multiplied image to the positive sample image to obtain the image to be added. Assuming there are m negative sample areas, the number of negative sample images is m, and there are m corresponding decoded images C. Each decoded image C is multiplied with its corresponding weight image D to obtain m images to be added, and then the sum of all the m images to be added is added to the positive sample image to obtain image E, which is the above-mentioned fused image.

[0095] Step 6: Convert image E into a grayscale image to obtain image F, which is the grayscale image mentioned above.

[0096] Through the embodiments of the present disclosure, histogram matching can effectively eliminate color differences between images and achieve a unified tone effect. The iterative color fusion method is used to eliminate the stitching edges. By applying gradient color weights to the stitching edges, a natural transition is achieved and obvious stitching marks are eliminated, which can greatly improve the visual effect of the image.

[0097] The present disclosure provides an image processing device, such as Fig.13 As shown, the image processing device 10 may include: an acquisition module 101, a determination module 102, a first processing module 103, a second processing module 104, and a third processing module 105, wherein: An acquisition module 101 is used to acquire an image to be processed and at least one other image corresponding to the image to be processed, wherein any one of the other images is shot from a different source than the image to be processed; The determination module 102 is used to determine at least one first negative sample in the image to be processed, and a negative sample mask corresponding to each of the first negative samples, determine a first positive sample in the image to be processed, and a positive sample mask corresponding to the first positive sample, wherein each of the first negative samples corresponds to a position, and the first positive sample is an image of the image to be processed excluding the area where each of the first negative samples is located; A first processing module 103 is used to crop each of the other images based on the position of each of the first negative samples to obtain at least one second negative sample and at least one corresponding second positive sample; The second processing module 104 is used to screen the first negative sample and each of the second negative samples corresponding to the first negative sample based on the position of any first negative sample, determine an image that meets the preset conditions, obtain a negative sample image corresponding to each of the first negative samples, and fuse the first positive sample with each of the second positive samples to obtain a positive sample image; The third processing module 105 is used to perform color adjustment on each of the negative sample images through histogram matching based on the negative sample images corresponding to each of the first negative samples, the negative sample masks corresponding to each of the first negative samples, the positive sample images and the positive sample masks, and eliminate the splicing edges between each of the negative sample images and the positive sample images after color adjustment to obtain a grayscale image.

[0098] In an optional embodiment, the device further includes: a fourth processing module, configured to obtain a target style image, and colorize the grayscale image based on the target style image to obtain a target image, wherein the target style image has the same image size as the grayscale image.

[0099] In an optional embodiment, the fourth processing module is specifically used to: crop the target style image according to the target size to obtain S cropped target style images, where S is a positive integer; crop the grayscale image according to the target size to obtain S cropped grayscale images; for any cropped grayscale image, determine the image with the highest similarity to the cropped grayscale image from the S cropped target style images as a similar style image; for any cropped grayscale image, use a style transfer neural network to use the similar style image corresponding to the cropped grayscale image as the target color style, perform style transfer, and obtain the corresponding colored image; splice the colored images corresponding to each of the cropped grayscale images to obtain the spliced ​​target image.

[0100] In an optional embodiment, the second processing module 104 is specifically used to: determine the color deviation value of the first positive sample to obtain a first color deviation value; based on the position of any first negative sample, determine the color deviation values ​​of the second negative samples corresponding to the position and the color deviation value of the first negative sample to obtain the second color deviation values, determine the color deviation value closest to the first color deviation value among the second color deviation values, and determine the negative sample corresponding to the color deviation value closest to the first color deviation value as the negative sample image corresponding to the first negative sample.

[0101] In an optional embodiment, the second processing module 104 is specifically used to: encode the first positive sample to obtain the encoded first positive sample, encode each of the second positive samples to obtain the encoded second positive sample; determine the color deviation value of the first positive sample to obtain the first color deviation value, determine the color deviation value of each of the encoded second positive samples to obtain each third color deviation value; based on the first color deviation value and each of the third color deviation values, determine the weights corresponding to the encoded first positive sample and each of the encoded second positive samples to obtain each target weight; based on each of the target weights, perform weighted fusion on the encoded first positive sample and each of the encoded second positive samples to obtain a fused positive sample, and decode the fused positive sample to obtain the positive sample image.

[0102] In an optional embodiment, the third processing module 105 is specifically used for: for each first negative sample, determining the multiplied image corresponding to the first negative sample according to the following operations: encoding the negative sample image based on the negative sample image corresponding to the first negative sample and the negative sample mask corresponding to the first negative sample to obtain the encoded negative sample image, encoding the positive sample image based on the positive sample image and the positive sample mask to obtain the encoded positive sample image; performing histogram matching on the encoded negative sample image according to the positive sample histogram corresponding to the encoded positive sample image to obtain a matching result, and decoding the matching result to obtain a decoded image; performing weighted calculation processing on the decoded image to obtain a weighted image; multiplying the decoded image with the weighted image to obtain the multiplied image corresponding to the first negative sample; adding the multiplied image corresponding to each first negative sample to the positive sample image to obtain a fused image; performing grayscale conversion on the fused image to obtain the grayscale image.

[0103] In an optional embodiment, the first processing module 103 is specifically used for: based on the position of each of the first negative samples, cropping each of the other images to obtain at least one second negative sample and the corresponding at least one second positive sample, including: based on the position corresponding to any one of the first negative samples, cropping the area corresponding to the position in each of the other images to obtain at least one of the second negative samples corresponding to the position; based on the position corresponding to each of the first negative samples, cropping the area in each of the other images except the area corresponding to each position to obtain at least one of the second positive samples.

[0104] Through the disclosed embodiments, when the image to be processed is a satellite image, the splicing traces of the satellite image to be processed can be eliminated, and the accuracy of the terrain information can be maintained, meeting the needs of flight simulation training. In the prior art, although the method of manual post-PS processing can improve the image effect, the workload is large and the original terrain information may also be changed. In the disclosed embodiments, through this automated satellite image processing method, not only can the problem of poor image quality be solved, but the processing efficiency can also be greatly improved, providing a more realistic and reliable terrain environment for flight simulation training.

[0105] The device of the embodiment of the present disclosure can execute the method provided by the embodiment of the present disclosure, and its implementation principle is similar and has corresponding technical effects. The actions performed by each module in the device of each embodiment of the present disclosure correspond to the steps in the method of each embodiment of the present disclosure. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.

[0106] An embodiment of the present disclosure provides an electronic device (computer device / equipment / system), including a memory, a processor, and a computer program stored in the memory, and the processor executes the above computer program to implement the steps of the method provided in any optional embodiment of the present disclosure.

[0107] In an alternative embodiment, an electronic device is provided, such as Fig.14 As shown, Fig.14 The electronic device 1000 shown includes: a processor 1001 and a memory 1003. The processor 1001 and the memory 1003 are connected, such as through a bus 1002. Optionally, the electronic device 1000 may also include a transceiver 1004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 1004 is not limited to one, and the structure of the electronic device 1000 does not constitute a limitation on the embodiments of the present disclosure.

[0108] Processor 1001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 1001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0109] The bus 1002 may include a path to transmit information between the above components. The bus 1002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 1002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0110] The memory 1003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage medium, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0111] The memory 1003 is used to store the computer program for executing the embodiment of the present disclosure, and the execution is controlled by the processor 1001. The processor 1001 is used to execute the computer program stored in the memory 1003 to implement the steps shown in the above method embodiment.

[0112] An embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0113] The embodiments of the present disclosure also provide a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiments when executed by a processor.

[0114] It should be understood that, although the flowchart of the embodiment of the present disclosure indicates each operation step by arrows, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present disclosure, the implementation steps in each flowchart can be executed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times. In scenarios with different execution times, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present disclosure does not limit this.

[0115] The above is only an optional implementation method for some implementation scenarios of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the scheme of the present disclosure, other similar implementation methods based on the technical ideas of the present disclosure are also within the protection scope of the embodiments of the present disclosure.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an image to be processed and at least one other image corresponding to the image to be processed, wherein any one of the other images has a different shooting source from the image to be processed; Determine at least one first negative sample in the image to be processed, and a negative sample mask corresponding to each of the first negative samples, determine a first positive sample in the image to be processed, and a positive sample mask corresponding to the first positive sample, wherein each of the first negative samples corresponds to a position, and the first positive sample is an image of the image to be processed excluding the area where each of the first negative samples is located; Based on the position of each of the first negative samples, crop each of the other images to obtain at least one second negative sample and at least one corresponding second positive sample; Based on the position of any first negative sample, the first negative sample and each of the second negative samples corresponding to the first negative sample are screened to determine an image that meets a preset condition, and a negative sample image corresponding to each of the first negative samples is obtained, and the first positive sample and each of the second positive samples are fused to obtain a positive sample image; Based on the negative sample image corresponding to each of the first negative samples, the negative sample mask corresponding to each of the first negative samples, the positive sample image and the positive sample mask, color adjustment is performed on each of the negative sample images through histogram matching, and the splicing edges between each of the negative sample images and the positive sample image after color adjustment are eliminated to obtain a grayscale image.

2. The method according to claim 1, characterized in that The method further comprises: A target style image is obtained, and the grayscale image is colored based on the target style image to obtain a target image, wherein the target style image has the same image size as the grayscale image.

3. The method according to claim 2, characterized in that The step of coloring the grayscale image based on the target style image to obtain the target image includes: Cropping the target style image according to the target size to obtain S cropped target style images, where S is a positive integer; Cropping the grayscale image according to the target size to obtain S cropped grayscale images; For any cropped grayscale image, determine an image with the highest similarity to the cropped grayscale image from the S cropped target style images as a similar style image; For any cropped grayscale image, use a style transfer neural network to transfer the similar style image corresponding to the cropped grayscale image as the target color style to obtain a corresponding colored image; The colored images corresponding to the cropped grayscale images are spliced ​​to obtain the spliced ​​target image.

4. The method according to claim 1, characterized in that: Based on the position of any first negative sample, the first negative sample and each of the second negative samples corresponding to the first negative sample are screened to determine an image that meets a preset condition, and to obtain a negative sample image corresponding to each of the first negative samples, including: Determine the color deviation value of the first positive sample to obtain a first color deviation value; Based on the position of any first negative sample, the color deviation values ​​of the second negative samples corresponding to the position and the color deviation value of the first negative sample are determined to obtain the second color deviation values, and the color deviation value closest to the first color deviation value is determined among the second color deviation values, and the negative sample corresponding to the color deviation value closest to the first color deviation value is determined as the negative sample image corresponding to the first negative sample.

5. The method according to claim 1, characterized in that The fusing the first positive sample and each of the second positive samples to obtain a positive sample image includes: Encoding the first positive sample to obtain an encoded first positive sample, and encoding each of the second positive samples to obtain an encoded second positive sample; Determine the color deviation value of the first positive sample to obtain a first color deviation value, determine the color deviation value of each encoded second positive sample to obtain each third color deviation value; Based on the first color deviation value and each of the third color deviation values, determine the weights corresponding to the encoded first positive sample and each of the encoded second positive samples to obtain each target weight; Based on each of the target weights, the encoded first positive samples and each of the encoded second positive samples are weightedly fused to obtain a fused positive sample, and the fused positive sample is decoded to obtain the positive sample image.

6. The method according to claim 1, characterized in that The method of color-adjusting the negative sample image corresponding to each of the first negative samples, the negative sample mask corresponding to each of the first negative samples, the positive sample image and the positive sample mask by histogram matching, and eliminating the splicing edge between the negative sample image corresponding to each of the first negative samples and the positive sample image after the color adjustment to obtain a grayscale image includes: For each first negative sample, the multiplied image corresponding to the first negative sample is determined by the following operation: Based on the negative sample image corresponding to the first negative sample and the negative sample mask corresponding to the first negative sample, the negative sample image is encoded to obtain an encoded negative sample image, and based on the positive sample image and the positive sample mask, the positive sample image is encoded to obtain an encoded positive sample image; According to the positive sample histogram corresponding to the encoded positive sample image, histogram matching is performed on the encoded negative sample image to obtain a matching result, and the matching result is decoded to obtain a decoded image; Performing weighted calculation processing on the decoded image to obtain a weighted image; Multiplying the decoded image by the weight image to obtain a multiplied image corresponding to the first negative sample; Adding the multiplied image corresponding to each first negative sample to the positive sample image to obtain a fused image; The fused image is subjected to grayscale conversion to obtain the grayscale image.

7. The method according to claim 1, characterized in that The step of cropping each of the other images based on the position of each of the first negative samples to obtain at least one second negative sample and at least one corresponding second positive sample includes: Based on the position corresponding to any one of the first negative samples, cropping the area corresponding to the position in each of the other images to obtain at least one of the second negative samples corresponding to the position; Based on the positions corresponding to the first negative samples, regions in the other images except those corresponding to the positions are cropped to obtain at least one of the second positive samples.

8. An image processing device, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be processed and at least one other image corresponding to the image to be processed, wherein any one of the other images is shot from a different source than the image to be processed; A determination module, configured to determine at least one first negative sample in the image to be processed, and a negative sample mask corresponding to each of the first negative samples, and to determine a first positive sample in the image to be processed, and a positive sample mask corresponding to the first positive sample, wherein each of the first negative samples corresponds to a position, and the first positive sample is an image of the image to be processed excluding the area where each of the first negative samples is located; A first processing module, configured to crop each of the other images based on the position of each of the first negative samples to obtain at least one second negative sample and at least one corresponding second positive sample; A second processing module is used to screen the first negative sample and each of the second negative samples corresponding to the first negative sample based on the position of any first negative sample, determine an image that meets a preset condition, obtain a negative sample image corresponding to each of the first negative samples, and fuse the first positive sample with each of the second positive samples to obtain a positive sample image; The third processing module is used to perform color adjustment on each of the negative sample images based on the negative sample images corresponding to each of the first negative samples, the negative sample masks corresponding to each of the first negative samples, the positive sample images and the positive sample masks through histogram matching, and eliminate the splicing edges between each of the negative sample images and the positive sample images after color adjustment to obtain a grayscale image.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images