Image processing method and device, electronic equipment and computer readable storage medium

Abstract

The invention relates to an image processing method and device, electronic equipment, a storage medium and a computer program product. The method comprises the following steps: performing layout analysis on a first image to obtain a layout area; the layout area comprises a first text area and a first privacy area, and the first privacy area has privacy information; performing character recognition on the first image to obtain a second text area; determining a second privacy area based on the first text area and the second text area; the second privacy area has privacy information; and performing fuzzy processing on the first privacy area and the second privacy area to obtain a second image. By adopting the method, the fuzzy processing efficiency of the privacy information in the image can be improved.

Description

Technical Field

[0001] The present application relates to the field of imaging technology, and in particular to an image processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] With the popularity of mobile devices such as smartphones, people may inadvertently leak private information in pictures when chatting or sharing information in other ways. Leakage of one's own private information may lead to harassment, fraud, spam calls and text messages, impersonation of business, intimidation and fraud, etc., resulting in damage to personal reputation and financial loss. If you inadvertently leak someone else's private information, it may also constitute a violation of the law and you need to bear the corresponding legal responsibility.

[0003] The solution to this problem is usually for users to manually erase private information when sharing information, which results in low image processing efficiency. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, device, electronic device, computer-readable storage medium, and computer program product, which can improve image processing efficiency.

[0005] In a first aspect, the present application provides an image processing method. The method comprises:

[0006] Performing layout analysis on the first image to obtain a layout area; the layout area includes a first text area and a first privacy area, and the first privacy area has privacy information;

[0007] Performing character recognition on the first image to obtain a second text area;

[0008] Based on the first text area and the second text area, a second privacy area is determined; the second privacy area has privacy information;

[0009] The first privacy area and the second privacy area are blurred to obtain a second image.

[0010] In a second aspect, the present application also provides an image processing device. The device comprises:

[0011] A layout analysis module, used for performing layout analysis on the first image to obtain a layout area; the layout area includes a first text area and a first privacy area, and the first privacy area has privacy information;

[0012] A text recognition module, used for performing character recognition on the first image to obtain a second text area;

[0013] The text recognition module is further used to determine a second privacy area based on the first text area and the second text area; the second privacy area has privacy information;

[0014] The fuzzy processing module is used to perform fuzzy processing on the first privacy area and the second privacy area to obtain a second image.

[0015] In a third aspect, the present application further provides an electronic device. The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0016] Performing layout analysis on the first image to obtain a layout area; the layout area includes a first text area and a first privacy area, and the first privacy area has privacy information;

[0017] Performing character recognition on the first image to obtain a second text area;

[0018] Based on the first text area and the second text area, a second privacy area is determined; the second privacy area has privacy information;

[0019] The first privacy area and the second privacy area are blurred to obtain a second image.

[0020] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0021] Performing layout analysis on the first image to obtain a layout area; the layout area includes a first text area and a first privacy area, and the first privacy area has privacy information;

[0022] Performing character recognition on the first image to obtain a second text area;

[0023] Based on the first text area and the second text area, a second privacy area is determined; the second privacy area has privacy information;

[0024] The first privacy area and the second privacy area are blurred to obtain a second image.

[0025] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0026] Performing layout analysis on the first image to obtain a layout area; the layout area includes a first text area and a first privacy area, and the first privacy area has privacy information;

[0027] Performing character recognition on the first image to obtain a second text area;

[0028] Based on the first text area and the second text area, a second privacy area is determined; the second privacy area has privacy information;

[0029] The first privacy area and the second privacy area are blurred to obtain a second image.

[0030] The above-mentioned image processing method, device, electronic device, computer-readable storage medium and computer program product perform layout analysis on the first image to obtain a layout area, which includes a first text area and a first privacy area, and the first privacy area has privacy information; perform character recognition on the first image to obtain a second text area; then, based on the first text area and the second text area, it is possible to accurately determine the second privacy area in which the text in the first image has privacy information, and the second privacy area also has privacy information; therefore, the first privacy area and the second privacy area are blurred to quickly obtain the second image, avoiding the problem of slow efficiency caused by manual coding of the image, thereby improving the efficiency of blurring the privacy information in the image, while also avoiding the problem of privacy information leakage caused by manual coding, thereby improving the accuracy of blurring the privacy information in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 is a flowchart of an image processing method in one embodiment;

[0033] Figure 2 is a schematic diagram of image segmentation in one embodiment;

[0034] Figure 3 A schematic diagram of a result of matching a first text region and a second text region in one embodiment;

[0035] Figure 4 is a schematic diagram of performing layout analysis on a first image in one embodiment;

[0036] Figure 5 is a flowchart of an image processing method in another embodiment;

[0037] Figure 6 is a structural block diagram of an image processing device in one embodiment;

[0038] Figure 7 FIG. 4 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0040] In one embodiment, an image processing method is provided. This embodiment is illustrated by applying the method to an electronic device. The electronic device may be a terminal or a server. It is understandable that the method may also be applied to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. The terminal may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, smart cars, etc. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, etc. The server may be implemented as an independent server or a server cluster consisting of multiple servers.

[0041] In one embodiment, Figure 1 As shown, an image processing method is provided, which is applied to an electronic device, and the image processing method comprises the following steps:

[0042] Step S102, performing layout analysis on the first image to obtain a layout area; the layout area includes a first text area and a first privacy area, and the first privacy area includes privacy information.

[0043] The first image may be a screenshot, a captured photo, an animated photo, or a synthesized image.

[0044] Layout analysis refers to a process of analyzing a first image to divide the first image into different regions, wherein each region represents a picture, a paragraph, or a text line.

[0045] The first text area is an area containing text obtained by layout analysis. The first text area may contain private information, may not contain private information, or may contain both private information and non-private information.

[0046] The first privacy area is a non-text area in the first image that contains privacy information. The privacy information includes but is not limited to faces, identification codes, various account passwords or fingerprints.

[0047] Optionally, in response to the privacy blurring operation on the first image, the electronic device performs layout analysis on the first image to obtain a layout area. In response to the triggering operation on the first image, the electronic device displays various operation elements for the first image, wherein the various operation elements include the privacy blurring operation element; in response to the triggering operation on the privacy blurring operation element, i.e., the privacy blurring operation on the first image, the layout analysis is performed on the first image.

[0048] Among them, each operation element may include a privacy blurring operation element, and may also include at least one of an editing operation element or a downloading operation element, etc. Optionally, the operation element is displayed in the form of a virtual button or a virtual control.

[0049] Optionally, the electronic device uses a preset layout analysis algorithm to perform layout analysis on the first image to obtain a layout area.

[0050] Optionally, the electronic device performs layout analysis on the first image to obtain a layout area of ​​a preset category. The preset category can be set as needed, and the preset category can include but is not limited to avatars, nicknames, pictures, QR codes, faces, text areas, voices, red envelopes, etc. The layout analysis can be modeled as a detection task.

[0051] Optionally, the electronic device determines the first privacy area contained in the publication area while performing layout analysis on the first image. Optionally, the electronic device can determine the first privacy information of the target category and non-text from the layout area of ​​the preset category. The target category can be set as needed. Optionally, the electronic device detects the first privacy information from the layout area of ​​the preset category using a preset privacy information detection algorithm. In other embodiments, the first privacy area can also be determined in other ways, which are not limited here.

[0052] Step S104: performing character recognition on the first image to obtain a second text region.

[0053] The second text region is a region containing text obtained by character recognition.

[0054] Optionally, the electronic device performs optical character recognition (OCR) on the first image to obtain a second text region. The second text region may contain private information, or may not contain private information, or may contain both private information and non-private information.

[0055] Optionally, the electronic device inputs the first image into an OCR model, and performs optical character recognition on the first image through the OCR model to obtain a second text area. The purpose of the OCR model is to detect and recognize text information in the first image, and the coordinates of the second text area (text box) and the text recognition result string can be output simultaneously, as well as the single coordinates of each character.

[0056] Step S106: determining a second privacy area based on the first text area and the second text area; the second privacy area has privacy information.

[0057] The second privacy is an area with privacy information in the text area of ​​the first image.

[0058] Optionally, the electronic device matches the first text area and the second text area to determine a second privacy area.

[0059] Optionally, the electronic device performs intent recognition on the first text area and the second text area distribution, determines a first intent corresponding to the first text area and a second intent corresponding to the second text area, and determines a second privacy area having privacy information from the first intention and the second intention.

[0060] Optionally, the electronic device may also determine the second private information having the private information in other ways, which are not limited here.

[0061] Step S108: blurring the first privacy area and the second privacy area to obtain a second image.

[0062] Optionally, the electronic device codes the first privacy area and the second privacy area to obtain a second image.

[0063] Optionally, in response to a sharing operation on the first image, the electronic device performs layout analysis on the first image to obtain a layout area; after obtaining the second image, the second image is shared.

[0064] The above-mentioned image processing method performs layout analysis on the first image to obtain a layout area, which includes a first text area and a first privacy area, and the first privacy area has privacy information; performs character recognition on the first image to obtain a second text area; then, based on the first text area and the second text area, it is possible to accurately determine the second privacy area in which the text in the first image has privacy information, and the second privacy area also has privacy information; therefore, the first privacy area and the second privacy area are blurred to quickly obtain the second image, avoiding the problem of slow efficiency caused by manual coding of the image, thereby improving the efficiency of blurring the privacy information in the image, while also avoiding the problem of privacy information leakage caused by manual coding, thereby improving the accuracy of blurring the privacy information in the image.

[0065] Furthermore, the above-mentioned image processing method can prevent the user's private information from being leaked inadvertently. It only requires the input of the first image, has no additional requirements for developers and users, and can complete the detection and identification of any known private information.

[0066] In one embodiment, before performing layout analysis on the first image or character recognition on the first image, the electronic device may further include: if the size of the original image exceeds a preset size threshold, segmenting the original image to obtain at least two first images.

[0067] The preset size threshold can be set as needed. The size of the original image may include at least one of the length, width, area or aspect ratio of the original image.

[0068] like Figure 2 As shown, if the aspect ratio of the original image exceeds the preset size threshold, directly resizing the original image may cause serious deformation of the image or loss of important information, resulting in failure to detect the layout area or character recognition errors.

[0069] Therefore, when the aspect ratio of the input original image is greater than the preset size threshold, the original image is electronically segmented to obtain at least two first images; after obtaining the second image corresponding to each first image, the second images are merged to obtain the target image.

[0070] Among them, the method for the electronic device to segment the original image includes but is not limited to: the width of each first image remains unchanged from the original image, the aspect ratio of each first image is a preset ratio threshold, and the original image is segmented into multiple first images of the same size, and each first image may have overlapping parts.

[0071] In one embodiment, based on the first text area and the second text area, determining the second privacy area includes: matching the first text area and the second text area to determine the third text area; performing named entity recognition on the third text area to determine the second privacy area from the third text area.

[0072] The third text area is the final area containing text obtained by matching the first text area and the second text area. The third text area may have private information, or may not have private information, or may have both private information and non-private information. The third text area carries text character information, which is obtained by OCR, and the text character information may include the coordinates of each character, etc.

[0073] Optionally, the electronic device performs named entity recognition (NER) on a preset field in the third text area to determine the second privacy area from the third text area. The preset field is a field range for performing named entity recognition, such as name, ID number, license plate number, etc.

[0074] The NER algorithm uses a long string of descriptive text as input to analyze the private information contained in the text in the third text area. The private information includes but is not limited to name, age, ID number, phone number, company name, etc. Figure 3 As shown, the electronic device performs layout analysis on the first image to obtain the first text area 302, performs character recognition on the first image to obtain the second text area 304, and the second text area 304 also includes the coordinate box of each character; the electronic device matches the first text area with the second text area to determine the third text area, performs named entity recognition on the third text area, and the characters in the recognized third text area include "Contact, Zhang San". The output content should be a dictionary, indicating that this string contains 1 private information, {"Contact": "Zhang San"}. Among them, the dictionary is a data format, and like a list and an array, the dictionary has key and value, the contact is the key, and Zhang San is the value.

[0075] In one embodiment, matching the first text region and the second text region to determine the third text region includes: traversing each second text region, matching the second text region with each first text region, and determining the correspondence result between the second text region and the first text region; traversing each first text region, and determining the third text region based on the correspondence result between the first text region and the second text region.

[0076] The corresponding result between the second text area and the first text area may include that the number of second text areas corresponding to the first text area is greater than or equal to 1, and at least one corresponding result in which the first text area does not have a corresponding second text area.

[0077] Optionally, matching the second text region with each first text region to determine a correspondence result between the second text region and the first text region includes: matching the second text region with each first text region to determine an intersection-and-union ratio between the second text region and each first text region; and determining a correspondence result between the second text region and the first text region based on the intersection-and-union ratio between the second text region and each first text region.

[0078] Among them, the intersection over union (IoU) is the ratio of the intersection to the union, which is used to evaluate the area overlap between two geometric figures. The larger the IoU between the second text region and the first text region, the higher the area overlap between the second text region and the first text region, and the greater the probability of establishing a corresponding relationship between the second text region and the first text region.

[0079] Optionally, after calculating the IoU between the second text area and each of the first text areas, the electronic device compares the IoU with a first IoU threshold, and determines the correspondence between the second text area and the first text area based on the comparison result. The first IoU threshold can be set as needed and is not limited here.

[0080] Optionally, if the intersection-and-union ratio between the second text region and the first text region is greater than a first intersection-and-union ratio threshold, it indicates that the second text region and the first text region match; if the intersection-and-union ratio between the second text region and the first text region is less than or equal to the first intersection-and-union ratio threshold, it indicates that the second text region and the first text region do not match.

[0081] Optionally, based on the intersection-and-union ratio between the second text area and each first text area, determining the correspondence result between the second text area and the first text area includes: if the number of first text areas matching the second text area is greater than 2, determining the first text area corresponding to the second text area from each matching first text area; if the number of first text areas matching the second text area is 1, determining the matching first text area as the first text area corresponding to the second text area; if the second text area does not have a matching first text area, determining the second text area as the first text area corresponding to itself.

[0082] Optionally, if the number of first text regions matching the second text region is greater than 2, the first text region corresponding to the second text region is determined based on an intersection-over-union ratio between the second text region and each of the matching first text regions.

[0083] Optionally, the electronic device may determine the first text region with the maximum intersection-and-union ratio corresponding to the second text region from the intersection-and-union ratios between the second text region and each of the matched first text regions. It is understandable that the maximum intersection-and-union ratio between the second text region and the first text region indicates that the area overlap between the second text region and the first text region is the highest, and the first text region is most likely to correspond to the second text region.

[0084] In other optional embodiments, the electronic device may also determine the first text region with the next largest intersection-and-union ratio corresponding to the second text region, and may also randomly determine the first text region corresponding to the second text region from the matching first text regions.

[0085] Optionally, if the number of first text areas that match the second text area is one, the matching first text area is directly determined as the first text area corresponding to the second text area; if there is no matching first text area in the second text area, it means that the layout analysis missed detecting the layout area corresponding to the position of the second text area, then a new first text area is added at the position corresponding to the second text area, and the first text area is determined as the first text area corresponding to the second text area.

[0086] It can be understood that the first text area obtained by layout analysis determines the coordinate position of the first text area, while the second text area obtained by character recognition can not only obtain the coordinate position of the second text area, but also simultaneously identify the text recognition result string of the second text area and the single coordinates of each character in the string. Therefore, the second text area obtained by character recognition is more accurate than the first text area obtained by layout analysis. When the number of first text areas matching the second text area is greater than 2, a first text area is selected from the matching first text areas to correspond to the second text area. When there is no matching first text area in the second text area, it means that the layout analysis missed the detection of the second text area. Then, a first text area corresponding to the second text area is added, which can improve the accuracy of text area matching and more accurately detect the final third text area, thereby more accurately performing image processing.

[0087] Optionally, based on the correspondence result between the first text area and the second text area, determining the third text area includes: if the number of second text areas corresponding to the first text area is greater than or equal to 1, splicing the second text areas corresponding to the first text area to obtain the third text area; if the first text area does not have a corresponding second text area, deleting the first text area.

[0088] It is understandable that after traversing each second text area, the correspondence between the second text area and the first text area is determined, each second text area corresponds to one or more first text areas, and for each first text area, there may be one, more or zero second text areas.

[0089] If the corresponding result indicates that the number of second text areas corresponding to the first text area is equal to 1, it means that the first text area corresponds to the second text area, and the first text area and the second text area are basically the same, and the first text area or the second text area can be used as the third text area; if the corresponding result indicates that the number of second text areas corresponding to the first text area is greater than 1, it means that the first text area includes multiple second text areas, and then each second text area is spliced ​​according to the position of each second text area in the first image to obtain the third text area; if there is no corresponding second text area for the first text area, it means that the first text area obtained by the layout analysis may be incorrect, and then the first text area is deleted. Among them, the electronic device can splice each second text area according to the position of each second text area in the first image, from top to bottom and from left to right, and then string the characters in each second text area to obtain the third text area.

[0090] In this embodiment, the electronic device traverses each second text area, matches the second text area with each first text area, determines the correspondence result between the second text area and the first text area, traverses each first text area, and determines the third text area based on the correspondence result between the first text area and the second text area, so that the area that finally contains the text can be determined more accurately. Further, when traversing each first text area, the third text area can be accurately spliced ​​or the first text area can be deleted based on the correspondence result between the first text area and the second text area, so as to more accurately screen out the final third text area.

[0091] In one embodiment, a layout analysis is performed on the first image to obtain a layout area, including: performing a layout analysis on the first image to obtain candidate areas for each preset category; each candidate area carries the confidence of the preset category to which the candidate area belongs; for each preset category, based on the confidence of each candidate area of ​​the preset category, selecting a publication layout area from each candidate area.

[0092] The preset categories may include but are not limited to avatars, nicknames, pictures, QR codes, faces, text areas, voices, red envelopes, etc. The candidate areas of the preset categories may contain private information or not, or may contain both private information and non-private information.

[0093] Optionally, the electronic device performs a series of feature spectrum extraction on the first image according to the preset parameters and structure of the layout analysis model, analyzes the extracted feature spectra, and outputs one or more multidimensional arrays, each multidimensional array representing a candidate area, and each multidimensional array carries elements of a preset category and elements of confidence of the preset category.

[0094] Exemplarily, the multidimensional array output by the layout analysis model can be a 6-dimensional array, 6-dimensional array [x, y, w, h, c, s], where x represents the horizontal coordinate of the upper left corner of the candidate area, y represents the vertical coordinate of the upper left corner of the candidate area, w represents the area width of the candidate area, h represents the area height of the candidate area, c represents the preset category of the candidate area, and s represents the confidence level of the preset category to which the candidate area belongs (value range 0 to 1).

[0095] Multidimensional arrays can also use other size elements, such as elements representing the upper left corner coordinates, width elements, and height elements, or elements representing the upper left corner coordinates and lower right corner coordinates, or elements representing the center point coordinates and width and height elements. There is no limitation here.

[0096] Optionally, before performing layout analysis on the first image, the electronic device may also preprocess the first image to obtain a preprocessed image, and then perform layout analysis on the preprocessed image. The preprocessing includes but is not limited to normalizing the size of the first image (resize, padding), normalizing pixel values ​​(normalizing values ​​to -1 to 1), etc.

[0097] Optionally, for each preset category, the electronic device deletes identical candidate areas based on the confidence of each candidate area of ​​the preset category to obtain a layout area.

[0098] Optionally, for each preset category, the electronic device deletes candidate areas with confidence levels lower than a preset confidence threshold to obtain a layout area.

[0099] It is understandable that if after the layout analysis of the first image, there are candidate areas of different preset categories that partially overlap, then candidate areas of each preset category are determined separately, and each candidate area may overlap. Exemplarily, there is a candidate area of ​​the photo category in the first image, and there is also a QR code in the photo, that is, there is also a candidate area of ​​the QR code category in the candidate area of ​​the photo category.

[0100] In this embodiment, the electronic device performs layout analysis on the first image to obtain candidate areas of each preset category. For each preset category, the publication area can be accurately selected from each candidate area based on the confidence level of each candidate area of ​​the preset category.

[0101] In one embodiment, optionally, a layout analysis is performed on the first image to obtain candidate areas for each preset category, including: performing recognition analysis on the first image to obtain a first-level area for each first-level category; each first-level area carries the confidence of the first-level category to which the first-level area belongs; for each first-level category, if the confidence of the first-level category to which the first-level area belongs is greater than a confidence threshold, then a classification analysis is performed on the first-level area to obtain a second-level category of the first-level area, and the first-level area of ​​the second-level category is used as a candidate area; the second-level category is a subcategory of the first-level category, and the second-level category belongs to the preset category.

[0102] Among them, the first-level category is the parent category, and the second-level category is the subcategory of the first-level category. Figure 4 As shown, the electronic device can pre-model the layout analysis task as a layout analysis model, find the target of interest in the first image, that is, the candidate area, through the layout analysis model, and determine the category and coordinate position of the candidate area. The confidence threshold can be set as needed and is not limited here.

[0103] It is understandable that the layout analysis model can identify targets of different appearances, shapes and postures, but in some scenarios, the difference between targets and non-targets can only be distinguished based on location or surrounding information. For example, in a chat screenshot of a social application, there are user avatars and pictures of avatars sent by users in the chat box. It is easy to identify user avatars of the avatar category and pictures of avatars of the image category as the same category. Therefore, in order to solve this problem, the electronic device adds a classification network after the layout analysis model to obtain a new layout analysis module.

[0104] Optionally, the electronic device recognizes and analyzes the first image and inputs it into a new layout analysis model, and recognizes and analyzes the first image through the new layout analysis module to obtain a first-level area for each first-level category, and each first-level area carries the confidence of the first-level category to which the first-level area belongs; for each first-level category, if the confidence of the first-level category to which the first-level area belongs is greater than the confidence threshold, it means that the first-level category to which the first-level area belongs is more credible, and then the first-level area of ​​the first-level category continues to be input into the classification network through the new layout analysis module, and the first-level area carries the first-level category and coordinate position to which it belongs, and the first-level area is classified and analyzed through the classification network to obtain the second-level category of the first-level area, and the first-level area of ​​the second-level category is used as a candidate area.

[0105] Optionally, for each first-level category, if the confidence of the first-level category to which the first-level region belongs is less than or equal to the confidence threshold, it means that the confidence of the first-level category to which the first-level region belongs is small, and the first-level region is discarded.

[0106] In this embodiment, the electronic device recognizes and analyzes the first image to obtain the first-level area of ​​each first-level category. For each first-level area, if the confidence of the first-level category to which the first-level area belongs is higher, it means that the first level is more credible, and the first-level area is further classified and analyzed to determine a more accurate second-level category, and the first-level area of ​​the second-level category is used as a candidate area.

[0107] In one embodiment, based on the confidence of each candidate area of ​​a preset category, a publication page area is selected from each candidate area, including: matching every two candidate areas of the preset category to determine the intersection-and-union ratio between every two candidate areas; if the intersection-and-union ratio between the two candidate areas is greater than a second intersection-and-union ratio threshold, selecting the candidate area with higher confidence from the two candidate areas as the layout area; if the intersection-and-union ratio between the two candidate areas is less than or equal to the second intersection-and-union ratio threshold, both candidate areas are used as layout areas.

[0108] The second intersection-over-union ratio threshold can be set as needed.

[0109] Optionally, for each preset category, the electronic device calculates the intersection-and-union ratio between each two candidate areas. If the intersection-and-union ratio is greater than a second intersection-and-union ratio threshold, indicating that the two candidate areas are the same, the candidate area with higher confidence is selected from the two candidate areas as the layout area, and the candidate area with lower confidence is deleted; if the intersection-and-union ratio between the two candidate areas is less than or equal to the second intersection-and-union ratio threshold, indicating that the two candidate areas are different, both candidate areas are used as layout areas.

[0110] In this embodiment, the electronic device can filter out the same candidate areas in the same preset category through the intersection-and-union ratio between every two candidate areas, so as to obtain a more accurate layout area.

[0111] In one embodiment, Figure 5As shown, the electronic device obtains an input original image, determines whether image segmentation is required based on the size and aspect ratio of the original image, and directly enters target detection if not required. If required, the original image is segmented to obtain at least two first images; target detection is performed on the first image, that is, the first image is subjected to a layout analysis algorithm to detect the publishing area, and the layout area includes a first text area and a first privacy area; optical character recognition is performed on the first image, and the result of the optical character recognition is matched with the layout analysis result to obtain a third text area, each of which includes text information and text coordinates; the text information in the third text area is input into the NER module for named entity recognition to obtain the text privacy information contained in the third text area; the area to be coded and the location of the text privacy information are selected from the results of the image layout analysis and the results of the named entity recognition; if each first image is obtained by segmentation, each first image is integrated together; if the first image is not segmented, that is, the first image is an original image, the result is directly returned to the coding system for coding.

[0112] In one embodiment, an image processing method is also provided, which is applied to an electronic device, and the image processing method comprises the following steps:

[0113] Step A1, performing recognition analysis on the first image to obtain a primary region of each primary category; each primary region carries the confidence of the primary category to which the primary region belongs.

[0114] Step A2: for each first-level category, if the confidence of the first-level category to which the first-level area belongs is greater than the confidence threshold, the first-level area is classified and analyzed to obtain the second-level category of the first-level area, and the first-level area of ​​the second-level category is used as the candidate area; the second-level category is a subcategory of the first-level category, and the second-level category belongs to the preset category. Each candidate area carries the confidence of the preset category to which the candidate area belongs.

[0115] Step A3: Match every two candidate regions of the preset category to determine the intersection-over-union ratio between every two candidate regions.

[0116] Step A3: if the IoU between the two candidate regions is greater than a second IoU threshold, then selecting a candidate region with a higher confidence level from the two candidate regions as the layout region;

[0117] Step A3: if the IoU between the two candidate regions is less than or equal to the second IoU threshold, both candidate regions are used as layout regions; the layout region includes a first text region and a first privacy region, and the first privacy region has privacy information.

[0118] Step A4: performing character recognition on the first image to obtain a second text area.

[0119] Step A5, traverse each second text region, match the second text region with each first text region, and determine the intersection-and-union ratio between the second text region and each first text region; if the intersection-and-union ratio between the second text region and the first text region is greater than the first intersection-and-union ratio threshold, it indicates that the second text region and the first text region match; if the intersection-and-union ratio between the second text region and the first text region is less than or equal to the first intersection-and-union ratio threshold, it indicates that the second text region and the first text region do not match.

[0120] Step A6, if the number of first text areas that match the second text area is greater than 2, then determine the first text area corresponding to the second text area from the matching first text areas; if the number of first text areas that match the second text area is 1, then determine the matching first text area as the first text area corresponding to the second text area; if there is no matching first text area in the second text area, then take the second text area as the first text area corresponding to itself.

[0121] Step A7, traverse each first text area, if the number of second text areas corresponding to the first text area is greater than or equal to 1, then splice the second text areas corresponding to the first text area to obtain a third text area; if the first text area does not have a corresponding second text area, then delete the first text area.

[0122] Step A8, performing named entity recognition on the third text region, and determining a second privacy region from the third text region; the second privacy region has privacy information;

[0123] Step A9: blurring the first privacy area and the second privacy area to obtain a second image.

[0124] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0125] Based on the same inventive concept, the embodiment of the present application also provides an image processing device for implementing the above-mentioned image processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations on the image processing method above, and will not be repeated here.

[0126] In one embodiment, Figure 6 As shown, an image processing device is provided, including: a layout analysis module 602, a text recognition module 604 and a fuzzy processing module 606, wherein:

[0127] The layout analysis module 602 is used to perform layout analysis on the first image to obtain a layout area; the layout area includes a first text area and a first privacy area, and the first privacy area has privacy information.

[0128] A text recognition module 604 is used to perform character recognition on the first image to obtain a second text area;

[0129] The text recognition module 604 is further configured to determine a second privacy area based on the first text area and the second text area; the second privacy area has privacy information.

[0130] The blur processing module 606 is used to perform blur processing on the first privacy area and the second privacy area to obtain a second image.

[0131] The above-mentioned image processing device performs layout analysis on the first image to obtain a layout area, which includes a first text area and a first privacy area, and the first privacy area has privacy information; performs character recognition on the first image to obtain a second text area; then, based on the first text area and the second text area, it is possible to accurately determine the second privacy area in which the text in the first image has privacy information, and the second privacy area also has privacy information; therefore, the first privacy area and the second privacy area are blurred to quickly obtain the second image, avoiding the problem of slow efficiency caused by manual coding of the image, thereby improving the efficiency of blurring the privacy information in the image, while also avoiding the problem of privacy information leakage caused by manual coding, thereby improving the accuracy of blurring the privacy information in the image.

[0132] In one embodiment, the text recognition module 604 is further used to match the first text region with the second text region to determine a third text region; perform named entity recognition on the third text region to determine a second privacy region from the third text region.

[0133] In one embodiment, the text recognition module 604 is further used to traverse each second text region, match the second text region with each first text region, and determine the correspondence result between the second text region and the first text region; traverse each first text region, and determine the third text region based on the correspondence result between the first text region and the second text region.

[0134] In one embodiment, the text recognition module 604 is further used to match the second text region with each first text region, determine the intersection-and-union ratio between the second text region and each first text region; and determine the correspondence result between the second text region and the first text region based on the intersection-and-union ratio between the second text region and each first text region.

[0135] In one embodiment, if the IoU between the second text region and the first text region is greater than a first IoU threshold, it indicates that the second text region and the first text region match; if the IoU between the second text region and the first text region is less than or equal to the first IoU threshold, it indicates that the second text region and the first text region do not match; the above-mentioned text recognition module 604 is also used to determine the first text region corresponding to the second text region from the matched first text regions if the number of first text regions matching the second text region is greater than 2; if the number of first text regions matching the second text region is 1, determine the matched first text region as the first text region corresponding to the second text region; if the second text region does not have a matched first text region, use the second text region as the first text region corresponding to itself.

[0136] In one embodiment, the text recognition module 604 is also used to, if the number of second text areas corresponding to the first text area is greater than or equal to 1, splice the second text areas corresponding to the first text area to obtain a third text area; if the first text area does not have a corresponding second text area, delete the first text area.

[0137] In one embodiment, the above-mentioned layout analysis module 602 is also used to perform layout analysis on the first image to obtain candidate areas for each preset category; each candidate area carries the confidence of the preset category to which the candidate area belongs; for each preset category, based on the confidence of each candidate area of ​​the preset category, a publication layout area is selected from each candidate area.

[0138] In one embodiment, the layout analysis module 602 is also used to identify and analyze the first image to obtain a first-level region of each first-level category; each first-level region carries the confidence of the first-level category to which the first-level region belongs; for each first-level category, if the confidence of the first-level category to which the first-level region belongs is greater than the confidence threshold, the first-level region is classified and analyzed to obtain a second-level category of the first-level region, and the first-level region of the second-level category is used as a candidate region; the second-level category is a subcategory of the first-level category, and the second-level category belongs to a preset category.

[0139] In one embodiment, the above-mentioned layout analysis module 602 is also used to match every two candidate areas of a preset category to determine the intersection-and-union ratio between every two candidate areas; if the intersection-and-union ratio between the two candidate areas is greater than a second intersection-and-union ratio threshold, then the candidate area with higher confidence is selected from the two candidate areas as the layout area; if the intersection-and-union ratio between the two candidate areas is less than or equal to the second intersection-and-union ratio threshold, then both candidate areas are used as layout areas.

[0140] Each module in the above-mentioned image processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in an electronic device in the form of hardware, or can be stored in a memory in an electronic device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0141] In one embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown. The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and an external device. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image processing method is implemented. The display unit of the electronic device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the electronic device casing, or an external keyboard, touchpad or mouse.

[0142] Those skilled in the art will understand that Figure 7 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0143] The embodiment of the present application further provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, enable the processors to perform the steps of the image processing method.

[0144] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the image processing method.

[0145] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0146] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0147] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0148] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: include: Performing layout analysis on the first image to obtain a layout area; the layout area includes a first text area and a first privacy area, and the first privacy area has privacy information; Performing character recognition on the first image to obtain a second text area; Based on the first text area and the second text area, a second privacy area is determined; the second privacy area has privacy information; The first privacy area and the second privacy area are blurred to obtain a second image.

2. The method according to claim 1, characterized in that The determining a second privacy area based on the first text area and the second text area includes: Matching the first text area with the second text area to determine a third text area; Perform named entity recognition on the third text region, and determine a second privacy region from the third text region.

3. The method according to claim 2, characterized in that The matching the first text area and the second text area to determine a third text area includes: Traversing each of the second text regions, matching the second text regions with each of the first text regions, and determining a correspondence result between the second text regions and the first text regions; Each first text region is traversed, and a third text region is determined based on the correspondence result between the first text region and the second text region.

4. The method according to claim 3, characterized in that The matching of the second text region with each first text region to determine the corresponding result between the second text region and the first text region includes: Matching the second text region with each first text region, and determining an intersection-over-union ratio between the second text region and each first text region; Based on the intersection-over-union ratio between the second text region and each first text region, a correspondence result between the second text region and the first text region is determined.

5. The method according to claim 4, characterized in that If the intersection-and-union ratio between the second text region and the first text region is greater than a first intersection-and-union ratio threshold, it indicates that the second text region matches the first text region; If the intersection-and-union ratio between the second text region and the first text region is less than or equal to a first intersection-and-union ratio threshold, it indicates that the second text region and the first text region do not match; The determining the correspondence result between the second text region and the first text region based on the intersection-over-union ratio between the second text region and each first text region includes: If the number of first text regions matching the second text region is greater than 2, determining the first text region corresponding to the second text region from among the matching first text regions; If the number of first text regions matching the second text region is 1, determining the matching first text region as the first text region corresponding to the second text region; If there is no matching first text region for the second text region, the second text region is used as the first text region corresponding to itself.

6. The method according to claim 3, characterized in that The determining of the third text region based on the correspondence result between the first text region and the second text region includes: If the number of second text areas corresponding to the first text area is greater than or equal to 1, concatenating the second text areas corresponding to the first text area to obtain a third text area; If the first text area does not have a corresponding second text area, the first text area is deleted.

7. The method according to any one of claims 1 to 6, characterized in that: The step of performing layout analysis on the first image to obtain a layout area includes: Performing layout analysis on the first image to obtain candidate regions of each preset category; each candidate region carries the confidence of the preset category to which the candidate region belongs; For each preset category, a publication area is selected from each candidate area based on the confidence of each candidate area of ​​the preset category.

8. The method according to claim 7, characterized in that The performing layout analysis on the first image to obtain candidate regions of each preset category includes: Performing recognition analysis on the first image to obtain a first-level region of each first-level category; each first-level region carries the confidence of the first-level category to which the first-level region belongs; For each first-level category, if the confidence of the first-level category to which the first-level area belongs is greater than the confidence threshold, the first-level area is classified and analyzed to obtain the second-level category of the first-level area, and the first-level area of ​​the second-level category is used as a candidate area; the second-level category is a subcategory of the first-level category, and the second-level category belongs to a preset category.

9. The method according to claim 7, characterized in that: The step of selecting a publication area from each candidate area based on the confidence of each candidate area of ​​the preset category includes: Matching every two candidate regions of the preset category to determine an intersection-over-union ratio between every two candidate regions; If the intersection-and-union ratio between the two candidate regions is greater than a second intersection-and-union ratio threshold, selecting a candidate region with a higher confidence level from the two candidate regions as the layout region; If the IoU between the two candidate regions is less than or equal to the second IoU threshold, both candidate regions are taken as layout regions.

10. An image processing device, characterized in that: include: A layout analysis module, used for performing layout analysis on the first image to obtain a layout area; the layout area includes a first text area and a first privacy area, and the first privacy area has privacy information; A text recognition module, used for performing character recognition on the first image to obtain a second text area; The text recognition module is further used to determine a second privacy area based on the first text area and the second text area; the second privacy area has privacy information; The fuzzy processing module is used to perform fuzzy processing on the first privacy area and the second privacy area to obtain a second image.

11. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by the processor, the processor is caused to perform the steps of the image processing method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images