Image processing method, image processing device, storage medium and electronic equipment

Abstract

The invention relates to the technical field of image processing, in particular to an image processing method and device, a computer readable storage medium and electronic equipment. The method comprises the following steps: acquiring an original image, and determining a ghosting area in the original image; generating a first ghosting vector of the ghosting region, and matching the first ghosting vector with a second ghosting vector in a preset ghosting database; and according to a matching result, performing ghosting repair on the ghosting region to obtain a target image. The image quality can be improved.

Description

Background Art

[0002] Ghosting is also called glare ghosting. When shooting against the light, the phenomenon of whitish photos and halo caused by strong light is called "glare". When sunlight or point light source enters the lens, after multiple reflections, a clear bright spot is formed at the relative position of the light source, like a ghost, which is called a ghost.

[0003] A camera lens is made up of multiple lenses, which are generally made of materials such as glass or plastic. When strong light enters the lens, the light reflected from the surface of each lens will be reflected multiple times inside the lens and the camera, which will produce "glare".

[0004] In the related art, hardware coating is used to reduce ghosting, but this method is not effective under strong light, and the ghosting is still very obvious, affecting the image quality.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The purpose of the present disclosure is to provide an image processing method, an image processing device, a computer-readable medium and an electronic device, thereby improving ghost images in an image at least to a certain extent and improving image quality.

[0007] According to a first aspect of the present disclosure, there is provided an image processing method, comprising: acquiring an original image, and determining a ghost area in the original image; generating a first ghost vector for the ghost area, and matching the first ghost vector with a second ghost vector in a preset ghost database; and based on the matching result, performing ghost repair on the ghost area to obtain a target image.

[0008] According to a second aspect of the present disclosure, an image processing device is provided, including: a ghost area identification module, configured to acquire an original image and determine a ghost area in the original image; a database matching module, configured to generate a first ghost vector for the ghost area and match the first ghost vector with a second ghost vector in a preset ghost database; and a ghost repair module, configured to perform ghost repair on the ghost area according to the matching result to obtain a target image.

[0009] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image processing method described in the first aspect is implemented.

[0010] According to a fourth aspect of the present disclosure, an electronic device is provided, characterized in that it includes: one or more processors; and a memory for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the image processing method described in the first aspect above.

[0011] According to a fifth aspect of the present disclosure, there is provided a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the steps of the image processing method as described in the first aspect.

[0012] The technical solution disclosed in this disclosure has the following beneficial effects:

[0013] In the present disclosure, by matching a first ghost vector corresponding to a ghost area in an original image with a second ghost vector in a preset ghost database, the ghost area can be repaired according to the matching result to improve the image quality.

[0014] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0016] Figure 1 A ghost image in an exemplary embodiment of the present disclosure is shown;

[0017] Figure 2 A schematic diagram showing a system architecture to which exemplary embodiments of the present disclosure may be applied;

[0018] Figure 3 A schematic diagram showing a flow chart of an image processing method in an exemplary embodiment of the present disclosure;

[0019] Figure 4 A schematic flow chart showing a method for determining a ghost area in an exemplary embodiment of the present disclosure;

[0020] Figure 5 A schematic flow chart showing a method for determining a candidate ghost area in an exemplary embodiment of the present disclosure;

[0021] Figure 6 A schematic flow chart showing another method for determining a ghost area in an exemplary embodiment of the present disclosure;

[0022] Figure 7 A flow chart showing a method for performing ghost image repair according to a matching result in an exemplary embodiment of the present disclosure;

[0023] Figure 8 A flow chart showing a method for repairing a ghost area according to a reference image quality parameter in an exemplary embodiment of the present disclosure;

[0024] Fig. 9 A schematic diagram showing a flow chart of a ghost image repair method in an exemplary embodiment of the present disclosure;

[0025] Fig.10 A framework diagram showing a ghost restoration system in an exemplary embodiment of the present disclosure;

[0026] Fig.11 A schematic diagram showing the composition of an image processing device in an exemplary embodiment of the present disclosure;

[0027] Fig.12 A schematic diagram showing an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0029] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0030] Ghosting is also called glare ghosting. When shooting against the light, the phenomenon of whitish photos and halo caused by strong light is called "glare". When sunlight or point light source enters the lens, after multiple reflections, a clear bright spot is formed at the relative position of the light source, like a ghost, which is called ghosting. It is a normal physical optical phenomenon.

[0031] For example, Figure 1Ghost 11 and Ghost 12 in the game are based on Figure 1 The ghost image generated by the light source 13 in FIG.

[0032] In a related technology, the occurrence of ghosting is reduced by adding a coating to the lens, but it can only reduce but not eliminate it. In some strong light source environments, ghosting will still occur.

[0033] Another related technology uses an illustration recognition scheme to perform ghost image restoration, but this method has poor restoration authenticity.

[0034] In view of the above problems, an exemplary embodiment of the present disclosure provides an image processing method.

[0035] Next, combine Figure 2 The system architecture of the operating environment of this exemplary embodiment is exemplarily described.

[0036] Figure 2 A schematic diagram of a system architecture is shown, and the system architecture 200 may include a terminal 210 and a server 220. The terminal 210 may be a terminal device such as a smart phone, a tablet computer, a desktop computer, a laptop computer, an intelligent wearable electronic device (such as a smart watch), and the server 220 generally refers to a background system that provides related services of the image processing method in this exemplary embodiment. The server 220 may be a cloud server. The terminal 210 and the server 220 may be connected via a wired or wireless communication link to perform data exchange.

[0037] In an exemplary implementation, the image processing method of the present disclosure may be performed by the terminal 210. For example, the camera module in the terminal 210 captures the original image, and when it is determined that there is a ghost area in the original image, a first ghost vector of the ghost area is generated, and the first ghost vector is matched with a second ghost vector in a preset ghost database pre-stored in the terminal 210, so that the ghost area is ghost-repaired according to the matching result to obtain a target image.

[0038] In another exemplary implementation, the image processing method of the present disclosure may also be performed by the server 220. For example, the server 220 obtains an original image and determines whether there is a ghost area in the original image, generates a first ghost vector of the ghost area when it is determined that there is a ghost area in the original image, matches the first ghost vector with a second ghost vector in a preset ghost database pre-stored in the terminal 210, and performs ghost repair on the ghost area according to the matching result to obtain a target image.

[0039] In another exemplary embodiment, the image processing method in the exemplary embodiment can also be performed jointly by the terminal 210 and the server 220. For example, the camera module in the terminal 210 captures the original image, and when it is determined that there is a ghost area in the original image, a first ghost vector of the ghost area is generated, and the first ghost vector is sent to the server 220. The server 220 matches the received first ghost vector with a second ghost vector in a preset ghost database pre-stored in the server 220, and performs ghost repair on the ghost area according to the matching result to obtain a target image, and sends the target image to the terminal 210 to display the target image in the terminal 210.

[0040] As can be seen from the above, the execution subject of the image processing method in this exemplary embodiment can be the above-mentioned terminal 210, or the above-mentioned server 220, or the above-mentioned terminal 210 and server 220 can serve as the execution subject together, and this disclosure does not limit this.

[0041] Combine the following Figure 3 An image processing method in this exemplary embodiment is described. Figure 3 An exemplary process of the image processing method is shown, which may include:

[0042] Step S310, acquiring an original image and determining a ghost area in the original image;

[0043] Step S320, generating a first ghost vector of the ghost area, and matching the first ghost vector with a second ghost vector in a preset ghost database;

[0044] Step S330: Perform ghost repair on the ghost area according to the matching result to obtain a target image.

[0045] Based on the above method, by matching the first ghost vector corresponding to the ghost area in the original image with the second ghost vector in the preset ghost database, the ghost area can be repaired according to the matching result to improve the image quality.

[0046] Below Figure 3 The steps shown are explained in detail.

[0047] In step S310, an original image is acquired, and a ghost area in the original image is determined.

[0048] In an exemplary embodiment, the original image can be understood as an image of a RAW format file, that is, a RAW image, which is in an unprocessed and natural state. That is, the RAW file directly recorded by the camera is the original data of the light signal captured by the sensor converted into a digital signal.

[0049] The original image may include a RAW image directly captured by a camera in real time, or may include a RAW image stored locally. That is, the present disclosure may perform ghost image repair processing on the original image captured by the camera in real time, or may perform ghost image repair processing on the RAW image stored in the terminal or server, and this exemplary embodiment does not specifically limit this.

[0050] For example, Figure 4 A schematic flow chart of a method for determining a ghost area in an exemplary embodiment of the present disclosure is shown. Figure 4 , the method may include steps S410 to S420.

[0051] in:

[0052] In step S410, an original image is acquired, and ghost detection is performed on the original image to determine a candidate ghost area in the original image.

[0053] Exemplary, combined Figure 5 The specific implementation of step S410 is further described. Figure 5 The method for determining a candidate ghost region shown may include steps S510 to S530. In which:

[0054] In step S510, an original image is acquired, and a YUV image corresponding to the original image in a YUV color space is determined.

[0055] Among them, YUV is a color encoding mode, in which Y represents brightness (Luminance), that is, grayscale value, and UV represents chrominance (Chrominance) and concentration (Chroma), which is used to describe the color and saturation of the image and is used to specify the color of the pixel.

[0056] For example, the original image may be encoded to determine a YUV image corresponding to the original image in the YUV color space.

[0057] In step S520, the point corresponding to the center of the light source in the original image in the YUV image is taken as the center point, and sampling is performed in the Y channel of the YUV image according to a preset radius.

[0058] For example, the light source area in the original image can be identified, and then the point corresponding to the center of the light source area in the YUV image can be determined. With the point as the center of the circle, sampling is performed in the Y channel of the YUV image according to a preset radius to obtain the grayscale value of the point and the sampling point.

[0059] In step S530, candidate ghost regions in the original image are determined according to regions corresponding to sampling points having grayscale values ​​greater than the center point.

[0060] For example, after obtaining the grayscale value of the sampling point, the sampling point whose grayscale value is greater than the center point can be determined, and then the number of sampling points in each grayscale value or each grayscale value range greater than the center point is counted, and the sampling points whose number of sampling points is less than the first preset value or the grayscale value range are excluded, and the candidate ghost area in the original image is determined according to the area corresponding to the remaining sampling points. The neighborhood grayscale variance can also be calculated, and the sampling points whose variance is greater than the second preset value are excluded, and the candidate ghost area in the original image is determined according to the area corresponding to the remaining sampling points. The sampling points whose number of sampling points is less than the first preset value or the grayscale value range and the sampling points whose variance is greater than the second preset value can also be excluded at the same time, and the candidate ghost area in the original image is determined according to the area corresponding to the remaining sampling points.

[0061] In an exemplary embodiment, multiple different preset radii can be determined in advance. Based on the point corresponding to the center of the light source area in the YUV image as the center of the circle, neighborhood sampling is performed with different preset radii, and then the grayscale values ​​of the sampling points obtained with different preset radii are calculated. For each preset radius, the candidate ghost area determined by the preset radius can be obtained based on the area corresponding to the sampling points whose grayscale values ​​are greater than the grayscale value of the center of the light source. The candidate ghost areas determined by different preset radii are superimposed, that is, the union is taken, to obtain the final candidate ghost area.

[0062] Through the above steps S510 to S530, the candidate ghost areas in the original image can be quickly determined.

[0063] Continue to refer Figure 4 In step S420, feature detection is performed on the candidate ghost area, and when the candidate ghost area meets the preset ghost feature, the candidate ghost area is determined as a ghost area.

[0064] In an exemplary embodiment, the preset ghost features include one or more features related to ghosting, such as the existence of a light source, the existence of a fogging phenomenon, the central symmetry of the positions of the ghost area and the light source, the ghost area being obviously green, and the ghost area completely restoring the shape of the light source.

[0065] For example, through the above steps S510 to S530, a suspicious ghost area in the original image can be determined, and feature detection is performed on the suspicious ghost area. When the suspicious ghost area meets the preset ghost features, the suspicious ghost area is determined as the final ghost area.

[0066] Through the above-mentioned steps S410 to S420, the candidate ghost area can be first determined by ghost detection, and then the characteristics of the candidate ghost area are matched with the preset ghost characteristics, and the candidate ghost area that meets the preset ghost characteristics is determined as the final ghost area, which can improve the efficiency and accuracy of ghost detection.

[0067] For example, Figure 6 FIG. 1 is a flow chart showing another method for determining a ghost area in an exemplary embodiment of the present disclosure. Figure 6 , the method may include steps S610 to S680. Wherein: in step S610, an original image is acquired; in step S620, ghost detection is performed on the original image; in step S630, it is determined whether the overlap of the detected ghost area is greater than a preset threshold, if so, it is transferred to step S640, otherwise, it is transferred to step S690; in step S640, the ghost area is determined as a candidate ghost area; in step S650, feature detection is performed on the candidate ghost area; in step S660, it is determined whether the feature of the candidate ghost area matches the preset ghost feature, if it matches, it is transferred to step S670, otherwise it is transferred to step S680; in step 670, the candidate ghost area is determined as a ghost area; in step S680, the candidate ghost area is determined as a non-ghost area; in step S690, the ghost area is determined as a non-ghost area.

[0068] Among them, the ghost detection method may include: (1) extracting the Y channel of the YUV image, and according to the preset input radius, taking the center of the light source as the center of the circle on the Y channel map, sampling at equal intervals within a neighborhood circle of the preset input radius to obtain sampling points; (2) calculating the number of points whose grayscale values ​​are greater than the center point, and the variance of the neighborhood grayscale, excluding points with too few points and too large variance, and determining the corresponding ghost area under the preset input radius based on the remaining points; (3) superimposing the detection results of different preset input radii to obtain the ghost area.

[0069] As mentioned above, the CG (cover glass) ghost features (i.e., preset ghost features) may include one or more of the following: the presence of a light source in the image, fog in the ghost area, the central symmetry of the positions of the ghost and the light source, the ghost being obviously green, and the ghost completely restoring the shape of the light source.

[0070] For example, after obtaining the original image, ghost detection can be performed first, and the ghost area with an overlap degree of more than 70% (the preset threshold can be customized according to needs) can be determined as a candidate ghost area. Then, feature detection is performed on the candidate ghost area to confirm whether the candidate ghost area matches the preset ghost feature. If the match is successful, the candidate ghost area is determined as a ghost area.

[0071] Continue to refer Figure 3 In step S320, a first ghost vector of the ghost area is generated, and the first ghost vector is matched with a second ghost vector in a preset ghost database.

[0072] For example, an affine transformation may be performed on the features of the ghost region to generate the first vector of the ghost region. Of course, the first vector of the ghost region may also be generated in other ways, which are not particularly limited in this exemplary embodiment.

[0073] In an exemplary embodiment, the preset ghost database may include the association relationship between the second ghost vectors of different ghost areas collected in advance and the image quality parameters corresponding to each second ghost vector. In other words, the preset ghost database pre-stores the second ghost vectors of different ghost areas and the image quality parameters corresponding to the second ghost vectors.

[0074] The image quality parameter corresponding to each second ghost vector may include one or more of image brightness, image clarity, and image authenticity. The image quality parameter corresponding to each second ghost vector may be determined based on the image quality parameter corresponding to the repaired area obtained after performing ghost repair on the ghost area corresponding to the second ghost vector.

[0075] Exemplarily, a specific implementation of matching the first ghost vector and the second ghost vector may include: calculating the vector distance between the first ghost vector and the second ghost vector, and determining whether the first ghost vector and the second ghost vector match according to the vector distance. If the vector distance is greater than a vector distance threshold, it is determined that the first ghost vector and the second ghost vector fail to match, and the first ghost vector and the second ghost vector do not match; if the vector distance is less than or equal to the vector distance threshold and the vector distance is the smallest of all vector distances that are less than or equal to the vector distance threshold, it is determined that the first ghost vector and the second ghost vector match successfully.

[0076] For example, the vector distance between the first ghost vector and each second ghost vector in the preset ghost database can be calculated. If all the vector distances are greater than the vector distance threshold, the first ghost vector and the second ghost vector in the preset ghost database fail to match; if there is a second ghost vector whose vector distance is less than or equal to the vector distance threshold, the first ghost vector and the second ghost vector in the preset ghost database match successfully. At the same time, all vector distances that satisfy the vector distance less than or equal to the vector distance threshold are sorted in ascending order, and the second ghost vector indicated by the vector distance sorted first is selected as the second ghost vector that successfully matches the first ghost vector.

[0077] Of course, the similarity between the first ghost vector and the second ghost vector may also be measured through other implementations to match the first ghost vector and the second ghost vector, and this exemplary implementation does not specifically limit this.

[0078] Continue to refer Figure 3 In step S330, based on the matching result, the ghost area is repaired to obtain a target image.

[0079] Exemplarily, a specific implementation of step S330 may include: selecting a target ghost repair method from a plurality of ghost repair methods according to the matching result, and performing ghost repair on the ghost area based on the target ghost repair method to obtain a target image.

[0080] Among them, multiple ghost repair methods include ghost repair based on multi-camera images captured by the camera module of the terminal device in multi-camera mode, ghost repair based on the second ghost vector, and ghost repair based on the fusion result of the second ghost vector and multi-camera images.

[0081] For example, in the event of a matching failure, the target ghost repair method includes performing ghost repair based on multi-camera images captured by the camera module of the terminal device in multi-camera mode; in the event of a matching success, the target ghost repair method includes performing ghost repair based on a second ghost vector that has successfully matched, or performing ghost repair based on a fusion result of the second ghost vector and the multi-camera image.

[0082] For example, Figure 7 A flowchart of a method for performing ghost image repair according to a matching result in an exemplary embodiment of the present disclosure is shown. Figure 7 , the method may include steps S710 to S750.

[0083] In step S710, it is determined whether the first ghost vector and the second ghost vector are matched successfully. If they are matched successfully, the process goes to step S720; otherwise, the process goes to step S740.

[0084] As mentioned above, the vector distance between the first ghost vector and the second ghost vector can be calculated, and based on the size relationship between the vector distance and the vector distance threshold, it is determined whether there is a second ghost vector that successfully matches the first ghost vector in the preset ghost database, which will not be repeated here.

[0085] In step S720, a reference ghost vector of the first ghost vector is determined according to the successfully matched second ghost vector.

[0086] As mentioned above, in the case where there are multiple second ghost vectors in the preset ghost database that satisfy a vector distance less than a vector distance threshold, the vector distances corresponding to the multiple second ghost vectors can be sorted in ascending order, and the second ghost vector indicated by the vector distance of the first sort (i.e., the smallest vector distance) is selected and used as the reference ghost vector of the first ghost vector. In the case where there is only one second ghost vector in the preset ghost database that satisfies a vector distance less than a vector distance threshold, the second ghost vector can be directly determined as the reference ghost vector of the first ghost vector.

[0087] In step S730, the ghost area is repaired based on the reference image quality parameter corresponding to the reference ghost vector to obtain a target image.

[0088] In an exemplary embodiment, the original image includes an image captured by a camera module of the terminal device, based on which, Figure 8 A flowchart of a method for repairing a ghost area according to a reference image quality parameter in an exemplary embodiment of the present disclosure is shown. Figure 8 , the method may include steps S810 to S820. Wherein:

[0089] In step S810, a multi-camera image captured by the camera module in a multi-camera mode is obtained, and a target area corresponding to the ghost area is determined in the multi-camera image.

[0090] For example, in response to the presence of a ghost area in the original image captured by the camera module, the multi-camera mode of the camera module is activated to obtain a multi-camera image captured in the multi-camera mode. In other words, when it is determined that the original image captured by the camera module has a ghost area, the multi-camera mode of the camera module can be automatically activated to capture based on the multi-camera mode to obtain a multi-camera image. Then, the target area corresponding to the ghost area in the original image in the multi-camera image is determined.

[0091] In step S820, the image quality of the target area is adjusted according to the reference image quality parameter corresponding to the reference ghost vector, so as to obtain a target image based on the multi-camera images after the target area adjustment.

[0092] For example, the reference image quality parameter corresponding to the reference ghost vector and the multi-camera image obtained based on the multi-camera mode may be fused to obtain the target image.

[0093] Among them, an exemplary fusion method may include: after determining a reference ghost vector based on a successfully matched second ghost vector, determining a reference image quality parameter associated with the reference ghost vector in a preset ghost database, and then based on the reference image quality parameter, adjusting the image quality of a target area in a multi-camera image to obtain an image quality of the target area that meets the reference image quality parameter, thereby obtaining a target image based on the adjusted multi-camera image of the target area.

[0094] Exemplarily, a specific implementation of step S820 may include: performing white balance adjustment on the target area so that the image quality of the target area satisfies the reference image quality parameter, thereby obtaining the target image.

[0095] Taking the reference image quality parameters including reference image brightness, reference image clarity, and reference image authenticity as an example, the white balance adjustment can be performed on the target area so that the image brightness of the target area is consistent with the reference image brightness, the image clarity of the target area is consistent with the reference image clarity, and the image authenticity of the target area is consistent with the reference image authenticity.

[0096] So that the image brightness of the target area is consistent with the reference image brightness. When the current image brightness of the target area is within the range indicated by the reference image brightness or the difference with the reference image brightness is less than a preset value, it can be determined that the image brightness of the target area is consistent with the reference image brightness.

[0097] In another exemplary embodiment, when the matching is successful, the image quality of the ghost area in the original image can be adjusted directly according to the reference image quality parameter to obtain the target image.

[0098] Exemplarily, white balance adjustment can be performed on the ghost area in the original image, and the target image can be obtained when the ghost area meets the reference image quality parameters; wherein the reference image quality parameters include one or more of reference image brightness, reference image clarity, and reference image realism.

[0099] The fact that the ghost area satisfies the reference image quality parameter can be understood as the image quality of the ghost area being consistent with the reference image quality indicated by the reference image quality parameter.

[0100] In step S740, multi-camera images captured by the camera module in the multi-camera mode are obtained.

[0101] In step S750, a target image is obtained based on the multiple images.

[0102] For example, when the first ghost vector and the second ghost vector fail to match, that is, there is no second ghost vector in the preset ghost database that successfully matches the first ghost vector, that is, the second ghost vector cannot be used to repair the ghost area. Then the final target image can be determined directly based on the multi-camera image. Since the multi-camera image is taken by multiple cameras based on multiple angles, different cameras have different photosensitivity and ghost angles, and can accurately construct a 3D scene to avoid the existence of ghost areas in the multi-camera images taken, thereby obtaining a target image without ghost areas to repair the ghost areas in the original image.

[0103] Through the above-mentioned steps S710 to S750, the original image can be ghosted by using the preset ghost database and multi-camera synthesis. The two methods complement each other to improve the efficiency and accuracy of ghost repair.

[0104] Exemplarily, when the first ghost vector and the second ghost vector fail to match, the image quality parameters of the target area corresponding to the ghost area in the multi-camera image can also be determined, and the image quality parameters include one or more of image brightness, image clarity, and image realism; the first ghost vector corresponding to the ghost area and the image quality parameters of the target area are associated to obtain a candidate association relationship; and the preset ghost database is updated according to the candidate association relationship.

[0105] For example, in the event of a matching failure, the image quality parameters of the target area corresponding to the ghost area in the multi-camera images can be obtained, and then a correlation relationship between the first ghost vector of the ghost area and the image quality parameters of the target area can be generated, and the correlation relationship can be added to a preset ghost database to update the preset ghost database, thereby supplementing the ghost vectors and their corresponding image quality parameters in the preset ghost database, thereby continuously and automatically enriching the preset ghost database.

[0106] Exemplarily, another specific implementation of step S330 may include: detecting edge points of the ghost area; and performing ghost repair on other pixel points in the ghost area except the edge points according to the matching result to obtain the target image.

[0107] In an exemplary embodiment, the edge exclusion method in Harris corner point detection can be used to detect edge points of the ghost area. Of course, other edge detection methods can also be used to detect edge points of the ghost area, and this exemplary embodiment does not make special limitations on this.

[0108] Among them, Harris corner point detection can include the following process: first construct the M matrix, Among them, I x and Iy They are the Sobel gradients in the x-direction and the y-direction respectively; then the average M matrix in the 3*3 neighborhood is calculated; based on the average M matrix, the corner point response R is calculated, R=Det(M)-k*Tr(M)*Tr(M), where Det(M) is the determinant of the matrix M, Tr(M) is the trace of the matrix M, and k is the Harris coefficient; finally, the points where R is less than zero are determined as edge points.

[0109] For example, after the edge points of the ghost area are determined, when the ghost area is repaired, the edge points are not considered for repair, that is, ghost repair is performed on other pixel points outside the edge points of the ghost area, thereby obtaining a target image.

[0110] Considering that there are obvious edge points around the ghost image, performing ghost repair on these edge points will lead to obvious errors, because these edge points do not necessarily belong to the ghost image area. If ghost repair is performed on them, the edge points will be distorted. Therefore, in the present disclosure, edge point detection and exclusion are performed on the ghost image area, and ghost repair is not performed on the edge points, thereby further ensuring the accuracy and repair effect of ghost repair.

[0111] For example, Fig. 9 A schematic diagram showing a flow chart of a ghost image repair method in an exemplary embodiment of the present disclosure is shown. Fig. 9 , the method may include steps S901 to S912. Wherein: in step S901, the original image is obtained; in step S902, it is determined whether there is a ghost area in the original image, if yes, then go to step S903 and step S906 at the same time, otherwise, go to step S912; in step S903, a first ghost vector corresponding to the ghost area is generated; in step S904, it is determined whether the first ghost vector and the second ghost vector in the preset ghost database are matched successfully, if successful, then go to step S905, otherwise go to step S907; in step S905, the ghost vector is matched with the second ghost vector in the preset ghost database with reference to the second ghost vector that successfully matched In step S906, the multi-camera mode is turned on to establish a three-dimensional image based on the multi-camera mode; in step S907, the ghost area is repaired according to the three-dimensional image to obtain a second repaired image; in step S908, the first repaired image and the second repaired image are merged to obtain a candidate target image; in step S909, spot detection is performed on the candidate target image; in step S910, the detected spots are processed to obtain a target image; in step S911, the target image is output; in step S912, the original image is output.

[0112] In an exemplary application scenario, a user uses a terminal device, such as a camera module in a mobile phone, to take a photo. After the camera module in the mobile phone captures the original image, it can determine whether there is a ghost area in the original image through the above-mentioned ghosting detection and preset ghost feature matching method. If it is determined that there is no ghost area in the original image, the original image is directly output to display the original image in the graphical user interface of the terminal device; if it is determined that there is a ghost area in the original image, a first ghost vector of the ghost area can be generated to match the first ghost vector with a second ghost vector in a preset ghost database, and the multi-camera mode can be automatically turned on immediately or the user is reminded to turn on the multi-camera mode, so as to use the multi-camera mode to shoot a 3D image, thereby based on the 3D image A second repaired image without ghosts is restored. When the first ghost vector and the second ghost vector match successfully, the ghost area in the original image is repaired with reference to the image quality parameter corresponding to the successfully matched second ghost vector, so that the image quality of the ghost area satisfies the image quality parameter corresponding to the successfully matched second ghost vector, thereby obtaining a first repaired image, and then the first repaired image and the second repaired image are fused to obtain a candidate target image, and then DOG spot detection is performed on the candidate target image, and the detected spots are eliminated, such as replacing the pixels at the spot with the average pixels in the spot neighborhood to eliminate the spots, thereby obtaining a final target image, and displaying the target image in a graphical user interface of the terminal device.

[0113] In the case that the first ghost vector and the second ghost vector fail to match, spot detection is directly performed on the second repaired image, and the detected spots are eliminated to obtain the final target image, and the target image is displayed in the graphical user interface of the terminal device.

[0114] Among them, an exemplary implementation of fusing the first repaired image and the second repaired image to obtain a candidate target image may include: determining a target area corresponding to the ghost area in the second repaired image, and adjusting the image quality of the target area in the second repaired image based on the image quality of the repaired ghost area in the first repaired image, so that the image quality of the target area is consistent with the image quality of the repaired ghost area in the first repaired image, thereby obtaining the candidate target image based on the second repaired image after the target area is adjusted.

[0115] In the present disclosure, ghost images are repaired based on a preset ghost database and multi-camera synthesis. When it is determined that a ghost exists, a reference ghost vector is determined from the preset ghost database for ghost repair and the multi-camera mode is turned on. Images are captured simultaneously by multiple cameras. Due to the different photosensitivity and shooting angles of different cameras, 3D scenes can be constructed more accurately, and real images can be restored to avoid ghosts in the captured images. The reference ghost vector determined in the preset ghost database and the first ghost vector of the ghost area have a high similarity, which can more accurately describe the real image brightness, real image clarity and real image authenticity of the ghost area. The combination of the two can ensure the authenticity of the image while eliminating ghosts, improve the accuracy of ghost repair, and thus improve image quality.

[0116] Fig.10 FIG. 1 is a schematic diagram showing a ghost image repair system in an exemplary embodiment of the present disclosure. Fig.10 The ghost repair system includes a ghost detection module 101, a ghost alignment module 102 and a ghost repair module 103.

[0117] Among them, the ghost detection module 101 is used to identify whether there is a ghost area in the original image. It mainly realizes accurate ghost judgment through the coincidence detection algorithm and the ghost key feature recognition and matching method to determine whether there is a ghost area in the original image.

[0118] The ghost alignment module 102 is used to perform affine transformation on the features of the ghost area when it is determined that there is a ghost area in the original image, use a measurable model, such as a first ghost vector, to describe the ghost area, and match the first ghost vector with a second ghost vector in a preset ghost database, and provide a ghost repair method for the subsequent ghost repair module 103 based on the matching result.

[0119] For example, it can calculate the vector distance between the first ghost vector and the second ghost vector, and when the vector distance between the first ghost vector and the second ghost vector is less than or equal to a threshold, determine that the first ghost vector and the second ghost vector match successfully. When the first ghost vector and the second ghost vector match successfully, the image quality parameter corresponding to the second ghost vector can be used to repair the ghost area to obtain a ghost repaired image; when the vector distance between the first ghost vector and the second ghost vector is greater than the threshold, it is determined that the first ghost vector and the second ghost vector fail to match, and the first ghost vector is marked as to be added, and based on the ghost pre-training model (i.e., the multi-camera mode shooting model), a multi-camera image shot in the multi-camera mode is obtained, and the final output ghost repair image is obtained according to the multi-camera image. At the same time, when the first ghost vector is marked as to be added, the image quality parameter value of the target area corresponding to the ghost area in the multi-camera image is obtained, the first ghost vector and the image quality parameter value are associated, and their association relationship is added to the preset ghost database.

[0120] In the present disclosure, the ghost area is repaired based on the preset ghost database and the multi-camera mode. The multi-camera mode can avoid ghosts in the captured images based on different angles, and the preset ghost database can determine the real brightness, real clarity and image realism of the current ghost based on similar ghosts. The combination of the two can improve the accuracy of ghost area repair. At the same time, the ghost can be eliminated without coating the lens, which can save hardware costs.

[0121] At the same time, in the present disclosure, ghost detection and ghost feature matching are combined to identify the ghost area, which improves the detection accuracy and detection efficiency of the ghost area.

[0122] Furthermore, the present invention performs edge point detection on the ghost area, so that ghost repair is not performed on the edge points, avoiding the problem of repair distortion caused by ghost repair on pixel points that do not belong to the ghost area, further improving the ghost repair effect, and ensuring the quality of ghost repair.

[0123] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0124] For further reference, Fig.11 As shown, in the implementation of this example, an image processing device 1100 is also provided, and the image processing device 1100 may include: a ghost region identification module 1110, a database matching module 1120 and a ghost repair module 1130. Among them:

[0125] The ghost region identification module 1110 is configured to obtain an original image and determine a ghost region in the original image;

[0126] The database matching module 1120 is configured to generate a first ghost vector of the ghost area, and match the first ghost vector with a second ghost vector in a preset ghost database;

[0127] The ghost repair module 1130 is configured to perform ghost repair on the ghost area according to the matching result to obtain a target image.

[0128] In an exemplary embodiment, based on the aforementioned embodiment, the ghost repair module 1130 can be specifically configured as follows: in the case of a successful match, determine a reference ghost vector of the first ghost vector based on the successfully matched second ghost vector; based on a reference image quality parameter corresponding to the reference ghost vector, repair the ghost area to obtain a target image.

[0129] In an exemplary embodiment, based on the aforementioned embodiment, the original image includes an image taken by a camera module of a terminal device, and the ghost area is repaired based on a reference image quality parameter corresponding to the reference ghost vector to obtain a target image, including: obtaining multi-camera images taken by the camera module in a multi-camera mode, and determining a target area corresponding to the ghost area in the multi-camera images; and adjusting the image quality of the target area according to the reference image quality parameter corresponding to the reference ghost vector to obtain the target image based on the multi-camera images after the target area adjustment.

[0130] In an exemplary embodiment, based on the aforementioned embodiment, based on the reference image quality parameters corresponding to the reference ghost vector, the ghost area is repaired to obtain a target image, including: performing white balance adjustment on the ghost area, and obtaining the target image when the ghost area meets the reference image quality parameters; wherein the reference image quality parameters include one or more of reference image brightness, reference image clarity, and reference image realism.

[0131] In an exemplary embodiment, based on the aforementioned embodiment, the original image includes an image taken by a camera module of a terminal device. Based on this, the ghost repair module 1130 can be specifically configured as follows: in case of a matching failure, obtaining multi-camera images taken by the camera module in a multi-camera mode; and obtaining a target image based on the multi-camera images.

[0132] In an exemplary embodiment, based on the aforementioned embodiment, the device also includes a preset ghost database update module, which can be specifically configured as follows: in the event of a matching failure, determining the image quality parameters of the target area corresponding to the ghost area in the multi-camera image, the image quality parameters including one or more of image brightness, image clarity, and image realism; associating the first ghost vector corresponding to the ghost area with the image quality parameters of the target area to obtain a candidate association relationship; and updating the preset ghost database based on the candidate association relationship.

[0133] In an exemplary embodiment, based on the aforementioned embodiments, the ghost area identification module 1110 can be specifically configured to: acquire an original image, perform ghosting detection on the original image to determine a candidate ghost area in the original image; perform feature detection on the candidate ghost area, and if the candidate ghost area meets a preset ghost feature, determine the candidate ghost area as a ghost area.

[0134] In an exemplary embodiment, based on the aforementioned embodiment, the original image is acquired and ghost detection is performed on the original image to determine the candidate ghost area in the original image, including: acquiring the original image, determining the YUV image corresponding to the original image in the YUV color space; taking the point corresponding to the center of the light source in the original image in the YUV image as the center point, sampling in the Y channel of the YUV image according to a preset radius; determining the candidate ghost area in the original image based on the area corresponding to the sampling point whose grayscale value is greater than the center point.

[0135] In an exemplary implementation, based on the aforementioned embodiments, the ghost repair module 1130 may be specifically configured to: detect edge points of the ghost area; and based on the matching results, perform ghost repair on other pixel points in the ghost area except the edge points to obtain a target image.

[0136] The specific details of each module in the above device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0137] The exemplary embodiments of the present disclosure also provide an electronic device for executing the above-mentioned image processing method, and the electronic device may be the above-mentioned terminal 210. Generally, the electronic device may include a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the above-mentioned image processing method by executing the executable instructions.

[0138] Below Fig.12The mobile terminal 1200 in FIG. 1 is taken as an example to exemplify the structure of the electronic device. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Fig.12 The construction in can also be applied to fixed type equipment.

[0139] like Fig.12 As shown, the mobile terminal 1200 may specifically include: a processor 1201, a memory 1202, a bus 1203, a mobile communication module 1204, an antenna 1, a wireless communication module 1205, an antenna 2, a display screen 1206, a camera module 1207, an audio module 1208, a power module 1209 and a sensor module 1210.

[0140] The processor 1201 may include one or more processing units, for example, the processor 1210 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc. For example, a GPU may be used to process the original image based on the image processing method in the present disclosure, thereby obtaining a target image after ghost repair processing.

[0141] The processor 1201 may be connected to the memory 1202 or other components via a bus 1203 .

[0142] The memory 1202 may be used to store computer executable program codes, which include instructions. The processor 1201 executes various functional applications and data processing of the mobile terminal 1200 by running the instructions stored in the memory 1202. The memory 1202 may also store application data, such as images, videos and other files. The memory 1202 may also store the original image and the target image obtained by performing ghost repair processing on the original image using the image processing method in the present disclosure. The memory 1202 may also store a preset ghost database.

[0143] The communication function of the mobile terminal 1200 can be implemented by the mobile communication module 1204, antenna 1, wireless communication module 1205, antenna 2, a modulation and demodulation processor, and a baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 1204 can provide 2G, 3G, 4G, 5G and other mobile communication solutions applied to the mobile terminal 1200. The wireless communication module 1205 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near field communication applied to the mobile terminal 1200.

[0144] The display screen 1206 is used to implement display functions, such as displaying a user interface and displaying a target image obtained after ghost image repair processing according to the image processing method of the present disclosure. The camera module 1207 is used to implement shooting functions, such as shooting original images, etc. The audio module 1208 is used to implement audio functions, such as playing audio, collecting voice, etc. The power module 1209 is used to implement power management functions, such as charging the battery, powering the device, monitoring the battery status, etc. The sensor module 1210 may include a depth sensor 12101, a speed sensor 12102, a gyroscope sensor 12103, an air pressure sensor 12104, etc., to implement corresponding sensing detection functions.

[0145] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0146] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network, and / or installed based on a removable medium. When the computer program is executed by a central processing unit (CPU), various functions defined in the method and apparatus of the present application are executed.

[0147] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method of the present specification is stored. In some possible implementations, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of the present specification, for example, it can execute Figure 3 Any one or more steps in .

[0148] The computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0149] In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0150] In addition, program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0151] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0152] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Acquire an original image, and determine a ghost area in the original image; Generate a first ghost vector of the ghost area, and match the first ghost vector with a second ghost vector in a preset ghost database; According to the matching result, the ghost image is repaired in the ghost image area to obtain the target image.

2. The image processing method according to claim 1, characterized in that: The step of performing ghost repair on the ghost area according to the matching result to obtain a target image includes: In the case of successful matching, determining a reference ghost vector of the first ghost vector according to the successfully matched second ghost vector; Based on the reference image quality parameter corresponding to the reference ghost vector, the ghost area is repaired to obtain a target image.

3. The image processing method according to claim 2, characterized in that: The original image includes an image captured by a camera module of a terminal device, and the repairing of the ghost area based on a reference image quality parameter corresponding to the reference ghost vector to obtain a target image includes: Acquire multiple images taken by the camera module in a multi-camera mode, and determine a target area corresponding to the ghost area in the multiple images; The image quality of the target area is adjusted according to the reference image quality parameter corresponding to the reference ghost vector, so as to obtain a target image based on the multi-camera images after the target area adjustment.

4. The image processing method according to claim 2, characterized in that: Repairing the ghost region based on a reference image quality parameter corresponding to the reference ghost vector to obtain a target image includes: Performing white balance adjustment on the ghost area, and obtaining a target image when the ghost area meets the reference image quality parameter; The reference image quality parameters include one or more of reference image brightness, reference image clarity, and reference image authenticity.

5. The image processing method according to claim 1, characterized in that: The original image includes an image captured by a camera module of a terminal device, and the ghost repairing of the ghost area according to the matching result to obtain a target image includes: In case of matching failure, obtaining multi-camera images taken by the camera module in the multi-camera mode; A target image is obtained based on the multiple images.

6. The image processing method according to claim 5, characterized in that: In the event of a matching failure, the method further comprises: Determine an image quality parameter of a target area corresponding to the ghost area in the multi-camera image, wherein the image quality parameter includes one or more of image brightness, image clarity, and image authenticity; Associating a first ghost vector corresponding to the ghost area with an image quality parameter of the target area to obtain a candidate association relationship; The preset ghost database is updated according to the candidate association relationship.

7. The image processing method according to claim 1, characterized in that: The acquiring of the original image and determining the ghost area in the original image comprises: Acquire an original image, and perform ghost detection on the original image to determine a candidate ghost area in the original image; Perform feature detection on the candidate ghost area, and if the candidate ghost area meets a preset ghost feature, determine the candidate ghost area as a ghost area.

8. The image processing method according to claim 7, characterized in that: The acquiring of the original image and performing ghost detection on the original image to determine a candidate ghost area in the original image includes: Acquire an original image, and determine a YUV image corresponding to the original image in a YUV color space; The point corresponding to the center of the light source in the original image in the YUV image is taken as the center point, and sampling is performed in the Y channel of the YUV image according to the preset radius; Based on the regions corresponding to the sampling points whose grayscale values ​​are greater than the center point, the candidate ghost regions in the original image are determined.

9. The image processing method according to claim 1, characterized in that: The step of performing ghost repair on the ghost area according to the matching result to obtain a target image includes: Detecting edge points of the ghost area; According to the matching result, ghost repair is performed on other pixel points in the ghost area except the edge points to obtain a target image.

10. An image processing device, characterized in that: include: A ghost region identification module is configured to obtain an original image and determine a ghost region in the original image; A database matching module is configured to generate a first ghost vector of the ghost area, and match the first ghost vector with a second ghost vector in a preset ghost database; The ghost repair module is configured to perform ghost repair on the ghost area according to the matching result to obtain a target image.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

12. An electronic device, characterized in that: include: one or more processors; as well as A memory for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the method according to any one of claims 1 to 9.

Images