Image processing method and apparatus, and electronic device
By detecting the target area in the image and performing privacy processing, the problem of image privacy information automation in the prior art is solved, the function of automatically hiding private information is realized, and the user operation efficiency is improved.
Patent Information
- Application Number
- PCT/CN2024/094279
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-26
- Filing Date
- 2024-05-20
- Publication Date
- 2024-12-05
AI Technical Summary
The prior art cannot automatically locate and code private information in images with one click, resulting in users needing to manually crop or smear, which is time-consuming and labor-intensive.
By detecting the target area in the image, including text and image areas, determining the privacy area and performing privacy processing, an image processing method that automatically hides privacy information is realized.
Save user operation time, improve operation efficiency, and realize automated privacy information hiding.
Smart Images

Figure CN2024094279_05122024_PF_FP_ABST
Abstract
Description
Image processing method, device and electronic equipment
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on May 26, 2023, with application number 202310609506.1 and titled “Image processing method, device and electronic device”, the entire contents of which are incorporated by reference into this application. Technical Field
[0003] The present application belongs to the field of image processing technology, and specifically relates to an image processing method, device and electronic equipment. Background Art
[0004] Currently, after users capture an image (such as a screenshot), they often share it. However, because images contain a lot of user privacy information, for information security reasons, this privacy information needs to be hidden before sharing. However, existing technologies cannot automatically locate and decrypt private information in images with one click. Users are required to manually use image cropping tools to cut out private areas, or manually use image smearing tools to mosaic each area designed to be private, which is time-consuming and laborious.
[0005] Summary of the Invention
[0006] The purpose of the embodiments of the present application is to provide an image processing method, device, and electronic device that can solve the problem that the existing technology of encoding private information in images is time-consuming and labor-intensive.
[0007] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:
[0008] acquiring a first image;
[0009] Detecting a target area in the first image; wherein the target area includes: at least one text area and at least one image area;
[0010] determining a privacy area according to the target area;
[0011] Perform privacy processing on the privacy area to obtain a processed second image.
[0012] In a second aspect, an embodiment of the present application provides an image processing device, comprising:
[0013] An acquisition module, configured to acquire a first image;
[0014] A detection module, configured to detect a target area in the first image; wherein the target area includes: at least one text area and at least one image area;
[0015] A first determining module, configured to determine a privacy area according to the target area;
[0016] The processing module is used to perform privacy processing on the privacy area to obtain a processed second image.
[0017] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.
[0018] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.
[0019] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method described in the first aspect.
[0020] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the method described in the first aspect.
[0021] In a seventh aspect, an embodiment of the present application provides an electronic device, which is configured to execute the method described in the first aspect.
[0022] In an embodiment of the present application, an image processing method includes: acquiring a first image; detecting a target area in the first image; wherein the target area includes at least one text area and at least one image area; determining a privacy area based on the target area; and performing privacy processing on the privacy area to obtain a processed second image. This method can help users automatically process privacy information in images, saving user operation time and improving user operation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] FIG1 is a flow chart of an image processing method according to an embodiment of the present application;
[0024] FIG2 is a schematic diagram of a network model according to an embodiment of the present application;
[0025] FIG3 is a schematic structural diagram of the SPPF according to an embodiment of the present application;
[0026] FIG4 is a schematic structural diagram of CSP1_x according to an embodiment of the present application;
[0027] FIG5 is a schematic structural diagram of a Res unit according to an embodiment of the present application;
[0028] FIG6 is a schematic diagram of a multi-branch structure of three convolution kernels of different sizes according to an embodiment of the present application;
[0029] FIG7 is a schematic diagram of a detection head according to an embodiment of the present application;
[0030] FIG8 is a schematic diagram of another detection head according to an embodiment of the present application;
[0031] FIG9 is a schematic diagram of a first image according to an embodiment of the present application;
[0032] FIG10 is a second schematic diagram of the first image according to an embodiment of the present application;
[0033] FIG11 is a structural block diagram of an image processing apparatus according to an embodiment of the present application;
[0034] FIG12 is a schematic structural diagram of an electronic device according to an embodiment of the present application;
[0035] FIG13 is a schematic diagram of the hardware structure of the electronic device according to an embodiment of the present application. Specific embodiments
[0036] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.
[0037] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein. Furthermore, the objects distinguished by "first," "second," and the like generally refer to a class and do not limit the number of objects. For example, the first object can be one or more. Furthermore, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the connected objects are in an "or" relationship.
[0038] The image processing provided by the embodiments of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.
[0039] Referring to FIG1 , an embodiment of the present application provides an image processing method, including:
[0040] Step 101, acquiring a first image;
[0041] Optionally, the first image includes a screenshot image (such as a screenshot of a chat interface, a screenshot of a user details page, etc.), a saved image, a photo, etc.
[0042] Optionally, the style of the first image includes but is not limited to: a single-screen image style, a dual-screen image style, and a long image style.
[0043] Step 102: Detect a target area in the first image; wherein the target area includes: at least one text area and at least one image area;
[0044] For example, the text area includes at least one of the following: user information area (such as basic personal information, device information, behavior information, location information, etc.), chat content area, chat member nickname area, group name area, and time information area; the image area includes at least one of the following: portrait area and object image area. For example, the image area for a screenshot of the chat interface includes: avatar area, chat emoticon area, and chat picture area.
[0045] Step 103, determining a privacy area according to the target area;
[0046] Exemplarily, for a chat interface image, the privacy areas include: a chat member nickname area, an avatar area, and a group name area.
[0047] Step 104: Perform privacy processing on the privacy area to obtain a processed second image.
[0048] Exemplarily, privacy processing includes: mosaic processing, preset image overlay processing, hiding processing, erasing processing, etc.
[0049] In this embodiment, after acquiring a first image, the first image is detected to obtain a target area in the first image; a privacy area is determined based on the target area; and privacy processing is performed on the privacy area. This helps users automatically hide private information in the first image, saving user operation time and improving user operation efficiency.
[0050] In one embodiment, in step 102, detecting the target area in the first image includes:
[0051] Extracting feature maps of the top layer, middle layer, and bottom layer of the first image through the backbone network of the network model;
[0052] Splicing the feature maps through a feature fusion network of the network model to obtain an enhanced feature map;
[0053] The enhanced feature map is input into the head prediction network of the network model to obtain the target area; wherein the head prediction network includes two detection heads; one detection head is used to predict the at least one image area, and the other detection head is used to predict the at least one text area, and the detection box output by each detection head includes the following three feature information: the category of the detection box is text or image, the position of the detection box and the confidence of the detection box.
[0054] As shown in Figure 2, the network model consists of three parts: the backbone network (Backbone), the feature fusion network (Neck), and the head prediction network (Head). Specifically:
[0055] First, the backbone network, Backbone, is primarily used to extract features of different sizes from the first image. The network architecture uses the CSPdarknet53 architecture, with an input size of 640*640*3 and outputs feature vectors of three different sizes: 64*80*80, 128*40*40, and 256*20*20. The network parameters are the same as those for yolov5n. The CBL network unit in Backbone is a basic module commonly used in convolutional neural networks. CBL consists of three parts: a convolutional layer (Conv), batch normalization (BN), and a rectified linear unit (ReLU). The SPPF in Backbone is a module that is further improved upon the SPP (Spatial Pyramid Pooling) module. The specific structure of SPPF is shown in Figure 3. Max Pool refers to maximum pooling. CSP1_1 is used to extract multi-scale features at the top layer, CSP1_2 is used to extract multi-scale features at the bottom layer, and CSP1_3 is used to extract multi-scale features at the middle layer. As shown in FIG4 , there is a structural diagram of CSP1_x (CSP1_1, CSP1_2, CSP1_3); the specific structure of the residual block Res unit is shown in FIG5 .
[0056] Secondly, for the feature fusion network, the Neck module fuses the multi-scale features of the top, middle and bottom layers extracted by the backbone to obtain higher-level features.
[0057] Third, regarding the head module, the current common object detection model YOLO outputs the regression box and confidence score in the same detection head through 1*1 convolution. However, in this application's solution, we decouple head image and text detection, and decouple the classification (cls) and regression (reg, conf) in each detection head, proposing a network model that can detect both heads and text simultaneously.
[0058] Specifically, the head module in this application includes two detection blocks: DetBlock1 and DetBlock2. Each block is applied to the three features of different sizes output by the Neck network.
[0059] As shown in Figure 7, in the first detection head DetBlock1_x, 1*1 convolution is first used for dimensionality reduction, then two 3*3 convolutions are connected in parallel to decouple the classification and regression task features, and finally 1*1 convolution is connected on the three branches to reduce the dimensionality to the output dimension. This is suitable for detecting relatively square avatar frames.
[0060] As shown in Figure 8, the second detection head DetBlock2_x is designed in this application for long strips of text lines. First, 1*1 convolution is used for dimensionality reduction, and then a two-branch convolution structure is used to decouple the classification and regression task features. Because the text lines are multi-dimensional strips of different lengths, each branch uses a 1*5 convolution and a 1*3 convolution structure and stacks them together to extract features. Finally, each branch uses 1*1 convolution to reduce the dimensionality to the output dimension.
[0061] For example, the network layer parameter configuration of the Head module is shown in Table 1 below.
[0062] Table 1
[0063] In an optional embodiment, the detection frame output by each detection head includes the following three feature information: the category of the detection frame, which is text or image, the position of the detection frame, and the confidence level of the detection frame.
[0064] As shown in Figures 7 and 8, the size of cls is 1*C*H*W, indicating the category to which each detection box belongs; the size of reg is 1*4*H*W, indicating the position of the detection box (i.e., the position in the screenshot); the size of conf is 1*1*H*W, indicating the confidence of the detection box; where C is the number of categories, which is 2 in this application (i.e., text and image); H*W indicates the output size, such as 20*20; 4 contains the detection box position information (x, y, w, h), (x, y) indicates the coordinates of the center of the box, and (w, h) indicates the width and height of the detection box.
[0065] It's important to note that, as shown in Figure 2, after the Neck module outputs features for each size, it is connected to two detection heads: one detection head (DetBlock1_x) outputs an image detection box (img box), and one detection head (DetBlock2_x) outputs a text detection box (tex box). It's important to note that each detection box corresponds to a text region or image region.
[0066] It should be noted that the targets to be detected in the first image include two types: avatars and texts. Avatars are mostly square or circular in shape, with rich and continuous regional content, while text lines are mostly horizontal long strips of varying lengths, with discontinuous content. Therefore, the characteristics of these two types of targets are very different. If the existing general target detection model is directly used to detect text lines, the effect is difficult to meet the requirements, and there are few existing target detection models that can directly detect these two types of targets at the same time. In response to this situation, in the above embodiment of the present application, the target detection of text and avatar categories is decoupled, and a network model that can detect avatars and text at the same time is proposed.
[0067] Optionally, in the post-processing part of the network model, NMS (non-maximum suppression) can be used in the derivation stage to filter the candidate boxes to obtain the final detection results.
[0068] In an optional embodiment, the feature fusion network includes an FPN structure and a PAN structure connected in sequence; the cross-stage local network CspNet in the FPN structure and the PAN structure is replaced by a multi-branch structure (MutilBranchBlock), each multi-branch structure includes three convolution kernels of different sizes; the feature fusion network of the network model splices the feature map to obtain an enhanced feature map, including:
[0069] As shown in Figure 2, the top-layer feature map extracted by the backbone network (i.e., the output of CSP1_1 in Figure 2) is input to the first network unit CBL of the FPN structure (i.e., CBL1 in Figure 2), and the middle-layer feature map extracted by the backbone network (i.e., the output of CSP1_3 in Figure 2) is input to the first multi-branch structure (MutilBranchBlock1) of the FPN structure, and the output of the first multi-branch structure is used as the input of the second CBL (i.e., CBL2 in Figure 2);
[0070] After upsampling (i.e., Upsampling) the feature map output by the first CBL, it is concatenated (i.e., concat) with the underlying feature map extracted by the backbone network (i.e., the output of CSP1_2 in Figure 2) and the feature map output by the second CBL, and the concatenated feature map is used as the input of the second multi-branch structure (MutilBranchBlock2) in the PAN structure;
[0071] The output of the second multi-branch structure is used as the input of the third CBL (i.e., CBL3 in FIG. 2 ) in the PAN structure, the output of the second CBL is used as the input of the third multi-branch structure (MutilBranchBlock3) in the PAN structure, and the output of the third multi-branch structure is used as the input of the fourth CBL (i.e., CBL4 in FIG. 2 ) in the PAN structure;
[0072] After downsampling the feature map output by the third CBL, it is spliced with the top-level feature map extracted by the backbone network and the feature map output by the fourth CBL. The spliced feature map is used as the input of the fourth multi-branch structure (MutilBranchBlock4), and the enhanced feature map is output by the fourth multi-branch structure.
[0073] Specifically, as shown in FIG2 , the output ends of the second multi-branch structure, the third multi-branch structure and the fourth multi-branch structure are each connected to two detection heads, one detection head is used to predict the at least one image area, and the other detection head is used to predict the at least one text area.
[0074] In this embodiment, the feature fusion network uses the FPN+PAN structure used in yolo5, and makes the following two improvements:
[0075] Improvement 1: A multi-branch structure (MutilBranchBlock) was designed to replace the CspNet in yolo5 to build the Neck module. As shown in Figures 2 and 6, the Block uses a multi-branch structure with three convolution kernels of different sizes (such as 3*3, 1*3, and 3*1), which can extract richer target features from different horizontal and vertical directions.
[0076] Improvement 2: Added feature fusion between the bottom and top layers of the FPN+PAN architecture. As shown in Figure 2, in the FPN architecture, the input to the second multi-branch architecture is the result of stacking (feature concatenation) the outputs of CSP1_2 and CBL2. The output of CBL1 is upsampled and added to the input of the second multi-branch architecture. Similarly, in the PAN architecture, the features output by CBL3 are downsampled and added to the input of the fourth multi-branch architecture. By adding network connections between the bottom and top layers, features of different sizes can be better fused, resulting in better detection results.
[0077] For example, the network layer parameters in the Neck module may be as shown in Table 2 below.
[0078] Table 2
[0079] In an optional embodiment, the method further includes: using at least one of the following loss functions to perform model training to obtain a network model:
[0080] Loss function 1: Binary cross entropy BCE loss function for classification.
[0081] In the embodiment of the present application, the classification loss uses BCE_Loss. The network model output contains the category information of each detection box (text detection box or avatar detection box), as well as the probability values of belonging to each category of text or avatar. The maximum probability is taken as the category of the corresponding detection box. Through the term loss function, the detection box output by the initial model will be more accurately classified into the corresponding category, reducing the possibility of false detection.
[0082] Loss function 2: Focal loss function used to calculate confidence;
[0083] In the embodiment of the present application, the confidence loss uses Focal_loss. The confidence value corresponding to each detection box in the output of the network model indicates the degree of confidence that it is a true target box. During the model optimization process, the detection box that contains more target areas (calculated by IOU) has a better confidence, and the confidence of the detection box that does not contain the target is 0. During the model inference process, each output detection box has a corresponding confidence. In the post-processing, the detection box with a confidence value less than the set threshold will be deleted, thereby reducing the number of false detection boxes and improving the effect of target detection.
[0084] Loss function three: CIoU loss function used to constrain the position of the image detection box;
[0085] In the embodiment of the present application, CIoU is used to constrain the position of the avatar detection frame. This loss causes the detection frame output by the network model to have a higher area overlap with the real detection frame, making the position of the detection frame more accurate.
[0086] Loss function 4: TIoU loss function used to constrain the position of the text detection box.
[0087] In the embodiment of the present application, TIoU is a loss function designed specifically for long strips of text lines. The characteristic of text is that even if a small area is missing, it may still be subjectively recognized by people. Only coding part of the text area cannot activate the privacy protection function. Therefore, for the text line detection result, in addition to requiring that the detection box output by the network model and the GT (Ground Truth, referring to the box as the standard answer) have a high overlapping area, it is also required that the text box detected by the network model has as many real texts as possible that can be recognized by the optical character recognition (OCR) algorithm. In this way, the privacy information of the text can be more completely protected after privacy coding. In this application, by adding the TIOU loss function, the position of the text detection box output by the network model covering the text can be made more accurate.
[0088] Optionally, in the embodiment of the present application, the calculation formula of the TIoU loss function is: TIou=ln inter(char pred , chargt ) / union(char pred , char gt );
[0089] Among them, inter refers to intersection, union refers to union, char_pred refers to the characters detected by OCR in the predicted text box, and har_gt refers to the characters detected by OCR in the true value box; pred represents the result of network prediction, and gt represents the manually labeled true value. Here, the true value box can be used as the detection box of the standard answer.
[0090] In an optional embodiment, each detection head in the head prediction network includes: TIoU loss function, CIoU loss function, Focal loss function, and BCE loss function.
[0091] For example, the calculation formula of the overall loss function is:
[0092] It should be pointed out that the above overall function calculation formula is only an example. When calculating the overall loss function corresponding to each detection head, different weights can also be assigned to each loss function. Among them, for the TIoU loss function and CIoU loss function, the smaller the calculated loss value, the better, while for the Focal loss function and BCE loss function, the larger the calculated loss value, the better. In order to allow the calculation company of the overall loss function that combines the four loss functions to accurately measure the deviation between the model prediction results and the actual results, in the above overall loss function formula, CIoU (reg pred , reg gt ) and TIoU(reg pred , reg gt ) is preceded by a minus sign, BCEloss(cls pred ,cls gt ) and Focalloss(conf pred ,conf gt ) is preceded by a plus sign.
[0093] During the network model training process, the input size is 640*640*3, and the number of epochs is set to 150. During inference, the first image is first scaled to 640*640 and then fed into the network model. After inference, the network model generates the head detection frame and text detection frame. The coordinates of these detection frames are then mapped back to the original input first image. In other words, the location of the head detection frame corresponds to the image area within the target region, and the location of the text detection frame corresponds to the text area within the target region.
[0094] In an optional embodiment, in step 103, determining the privacy area according to the target area includes:
[0095] dividing at least one image region in the target area into a first set located to the left of the middle vertical line and a second set located to the right of the middle vertical line according to the middle vertical line of the first image;
[0096] determining a first head portrait area set located on the left from the first set according to the horizontal coordinates of the image areas in the first set;
[0097] determining a second head portrait area set located on the right from the second set according to the horizontal coordinates of the image areas in the second set;
[0098] The privacy area is determined according to the first avatar area set and the second avatar area set.
[0099] In a specific implementation, when the first image is a chat interface image, after the network model detects at least one image region in the first image, to identify the avatar region on the left side of the chat interface and the avatar region on the right side of the chat interface, this embodiment uses the center vertical line of the first image to divide the at least one image region output by the network model into a first set located to the left of the center vertical line and a second set located to the right of the center vertical line. In other words, the avatars on the left side of the chat interface are located in the first set, and the avatars on the right side of the chat interface are located in the second set.
[0100] Furthermore, for the chat interface image, since the horizontal coordinate range of the avatar area located on the left side of the chat interface screenshot is the same, the horizontal coordinate range of the avatar area located on the right side of the chat interface screenshot is also the same, therefore, for the image areas in the first set and the second set, the present application projects all the image areas in the first set onto the horizontal line to obtain the first overlapping area with the largest horizontal coordinate overlap; for each image area in the first set, if the projected image area includes the first overlapping area, then the image area is judged to be an avatar area, and finally the first avatar area set is obtained; similarly, all the image areas in the second set are projected onto the horizontal line to obtain the second overlapping area with the largest horizontal coordinate overlap; for each image area in the second set, if the projected image area includes the second overlapping area, then the image area is judged to be an avatar area, and finally the second avatar area set is obtained.
[0101] In the above embodiment, the obtained first avatar area set and second avatar area set can be determined as the user avatar part in the privacy area, the avatar area can be further screened based on the first avatar area set and the second avatar area set, and the user nickname can be detected based on the first avatar area set and the second avatar area set.
[0102] In an optional embodiment, for further screening and detection of user avatars, determining the privacy area based on the first avatar area set and the second avatar area set includes:
[0103] Using a sliding window, slide within the target longitudinal region of the first image according to a preset step size to obtain N pixels within the sliding window, where N is a positive integer; wherein the target longitudinal region corresponding to the first profile picture region set is a first longitudinal region located on the left, and the target longitudinal region corresponding to the second profile picture region set is a second longitudinal region located on the left;
[0104] determining whether the image within the sliding window matches the first region based on three-channel color differences between the N pixels and pixels at corresponding positions within the first region, wherein the first region is any one of the first profile picture region set and the second profile picture region set;
[0105] When the image in the sliding window matches the first area, traversing the head portrait areas in the first head portrait area set and the second head portrait area set, and determining an overlap ratio between each head portrait area and the image area in the sliding window;
[0106] When the overlap ratio between the avatar area in the first avatar area set and the second avatar area set and the image area in the sliding window is less than a second preset value, the image area in the sliding window, the first avatar area set and the second avatar area set are determined as the user avatar part in the privacy area.
[0107] As shown in Figure 9, for the avatar area detection on the left side of the chat interface, a sliding window c with the same size as the avatar frame is used to slide from top to bottom in the first longitudinal area b1 according to a preset step size. For each step, N pixels in the sliding window c are randomly obtained, and the RGB three-channel color difference between the N pixels and any avatar area A in the first avatar area set is calculated; similarly, for the avatar area detection on the right side of the chat interface, a sliding window c with the same size as the avatar frame is used to slide from top to bottom in the second longitudinal area b2 according to a preset step size. For each step, N pixels in the sliding window c are randomly obtained, and the RGB three-channel color difference between the N pixels and any avatar area A in the second avatar area set is calculated.
[0108] Specifically, the calculation formula for the color difference of the three RGB channels is as follows:
[0109] D(A, c) = ∑ i∈A,j∈c |r i -r j |+|b i-b j |+|g i -g j |; Among them, the relative position of pixel i in the avatar area A is the same as the relative position of pixel j in the sliding window c; the three RGB channels are R (red), G (green), and B (blue).
[0110] Furthermore, taking the first longitudinal region b1 as an example, if the average color difference between all pixels j and pixel i extracted from the sliding window c is less than a first preset threshold (e.g., 20), then the two are determined to be matched. If the sliding window c matches the avatar region A, the overlap ratio (IOU) between the sliding window c and each avatar region C in the avatar region set is calculated. If the IOU is less than a second preset threshold (e.g., 0.1), then the sliding window is determined to be matched. c The area within the sliding window c also belongs to the avatar area and should be added to the first avatar area set, and the area within the sliding window c and the avatar area A are recorded as the avatars of the same contact; conversely, if the IOU is greater than or equal to the second preset threshold, the avatar area A and the avatar area C are recorded as the avatars of the same contact and do not need to be added to the first avatar area set.
[0111] Regarding nickname detection, in an optional embodiment, after determining the image area within the sliding window, the first profile picture area set, and the second profile picture area set as the user profile picture portion in the privacy area, the method further includes:
[0112] Determining the nickname area of the chat group member from at least one text area in the target area according to a positional relationship between the user avatar area and the nickname area of the chat group member;
[0113] The nickname area of the chat group member is determined as the nickname part in the privacy area.
[0114] In this embodiment, for nickname detection, it is known in advance based on the APP's historical statistical information or historical probability that a nickname can only appear next to an avatar and have a fixed positional relationship with it. Therefore, this embodiment of the application determines the nickname area of the chat group member from at least one text area in the target area based on the positional relationship between the user's avatar area and the nickname area of the chat group member.
[0115] In the specific implementation, for a chat interface image, first, remember the coordinates of the upper left corner of the avatar area as (head left , head up ), the width and height of the avatar area are head_w and head_h respectively, and the coordinates of the upper left corner of the text area are (text left ,text up), the lower right corner coordinates are (text right ,text bottom ).
[0116] Secondly, according to the APP type to which the chat interface screenshot belongs, the corresponding historical information is obtained as follows:
[0117] (1) The minimum distance l_min between the avatar frame (each avatar area corresponds to an avatar frame) and the nickname frame (each text area corresponds to a nickname frame); wherein the avatar frame is the avatar detection frame, and the nickname frame is the nickname detection frame.
[0118] (2) The relative position between the upper boundary of the nickname frame and the upper boundary of the avatar frame is: the upper boundary coordinate of the nickname frame text up In [head up +a*head_h,head up +b*head_h] coordinates, such as a=-0.25, b=0.5.
[0119] (3) The relative position of the lower boundary of the nickname frame and the upper boundary of the avatar frame is: the lower boundary coordinate of the nickname frame is text bottom In [head up +c*head_h,head up +d*head_h] coordinates, such as c=0.25, d=0.75.
[0120] Again, for an avatar on the left side of the chat interface screenshot, a text box (i.e., text area) is considered to be its corresponding nickname area if it meets the following conditions:
[0121] Similarly, for the avatar in the screenshot on the right side of the chat interface, a full text box (i.e., text area) is considered to be its corresponding nickname area and must meet the following conditions at the same time:
[0122] It should be noted that for each avatar, if there are multiple text boxes that meet the conditions, the text box closest to it is selected as the corresponding nickname position. After screening in this step, each avatar corresponds to at most one nickname box.
[0123] In an optional embodiment, after obtaining the avatar area set, for multiple avatar frames belonging to the same contact, the left and right horizontal coordinates and vertical coordinates of the nickname frames corresponding to the avatar frames are aligned.
[0124] Specifically, let nick1 be the nickname of head1 (avatar 1), nick2 be the nickname of head2 (avatar 2), (left, up) be the coordinates of the upper left corner of the nickname box, and (right, bottom) be the coordinates of the lower right corner of the nickname box. The result to be aligned is shown in the following formula:
[0125] It should be noted that if undetected nickname regions are detected during the nickname frame alignment, they will be completed accordingly. After the alignment operation, the detection accuracy and recall of nickname regions can be significantly improved.
[0126] In an optional embodiment, in step 103, determining the privacy area according to the target area includes:
[0127] Selecting a candidate region from at least one text region in the target region, wherein the candidate region is a text region located above a dividing line of the first image;
[0128] Determining a group name area of the chat group from the candidate area according to the vertical line in the first image and historical information of the application to which the first image belongs;
[0129] The group name area of the chat group is determined as the group name part in the privacy area.
[0130] In a specific implementation, when the first image is a screenshot of a chat interface, historical information about the application can be obtained based on the type of application to which the first image belongs; and the target boundary line can be obtained based on the historical information about the application. For example, as shown in Figure 9, for existing chat software, after obtaining multiple avatar areas in the first image, the target avatar area located closest to the top of the multiple avatar areas can be selected from the multiple avatar areas; and the straight line containing the upper boundary of the target avatar area is used as the target boundary line a.
[0131] Furthermore, since the group name area of the chat group is generally located on the left and in the center of the chat interface. Therefore, it is possible to determine whether the group name is in the center or on the left based on the historical information of the application. Furthermore, the group name area of the chat group is determined from the alternative area based on the middle vertical line of the first image and the historical information of the application to which the first image belongs. For example, if the group name is located in the center, the text area covering the coordinates of the middle vertical line is selected from the alternative area as the group name alternative; if the group name is located on the left, the text area covering 1 / 4 of the screenshot width is selected from the alternative area as the group name alternative. Finally, based on statistical historical information, it can be seen that the font of the group name is larger than the font of other information (such as time) in the top area, so the text area with the largest text box height can be selected from the group name alternatives as the group name area of the chat group.
[0132] In an optional embodiment, the image processing method further includes:
[0133] When acquiring the first image, correspondingly acquiring the name of the application to which the first image belongs;
[0134] The application name is written into the attribute information of the first image.
[0135] In a specific implementation, when acquiring the first image, the name of the application to which the chat interface belongs is acquired at the same time, and the name of the application is written into the attribute information of the first image. In this way, the name of the application can be obtained based on the attribute information of the first image, thereby obtaining the historical information of the application.
[0136] In an optional embodiment, before detecting the target area in the first image, the method further includes:
[0137] Determine the image type of the first image according to the attribute information of the first image; the image type includes: single-screen image style, dual-screen image style, and long image style;
[0138] When it is determined that the picture format of the first image is the dual-screen image format or the long image format, the first image is processed into the single-screen image format.
[0139] For example, when the first image is a screenshot of a chat interface, different applications display different styles. For example, some chat interfaces directly tile a single interface across two screens, while others display the chat list or address book list on the left screen and the chat interface on the right screen. Therefore, when cropping the first image in a dual-screen image format, the cropping method can be determined based on the application type.
[0140] Exemplarily, for the first image in a long image style, as shown in FIG10 , a single-screen picture can be extracted by sliding in the first image through a longitudinal sliding window d with a step size of 200 and a height of 1000, that is, the image area within each longitudinal sliding window d is regarded as a single-screen picture. In this way, a first image in a long image style can be processed into multiple single-screen pictures.
[0141] For example, for a long image in a wide screen, the dual-screen image cropping logic is first executed, and then the vertical sliding window cropping logic is executed on the cropped result. After this processing, the input first image can be processed into multiple single-screen images.
[0142] In specific implementation, the type of application can be determined based on the attribute information of the first image; based on the type of application, it is determined whether to crop the first image; if the first image is not to be cropped, the multi_window value is marked as false; if the first image needs to be cropped into the left screen area and the right screen area according to the middle line, the multi_window value is marked as true.
[0143] In the above embodiment, before inputting the first image into the network model, a better detection effect can be obtained by processing the first image into a single-screen image style.
[0144] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, the image processing device provided in the embodiment of the present application is described by taking the image processing method executed by the image processing device as an example.
[0145] Referring to FIG. 11 , an embodiment of the present application provides an image processing apparatus 1100, including:
[0146] A first acquisition module 1101 is configured to acquire a first image;
[0147] A detection module 1102 is configured to detect a target area in the first image; wherein the target area includes: at least one text area and at least one image area;
[0148] A first determining module 1103 is configured to determine a privacy area according to the target area;
[0149] The processing module 1104 is configured to perform privacy processing on the privacy area to obtain a processed second image.
[0150] Optionally, the detection module 1102 includes:
[0151] A first detection submodule, configured to extract feature maps of the top layer, middle layer, and bottom layer of the first image through a backbone network of a network model;
[0152] A second detection submodule is configured to concatenate the feature maps through a feature fusion network of the network model to obtain an enhanced feature map;
[0153] The third detection submodule is used to input the enhanced feature map into the head prediction network of the network model to obtain the target area; wherein the head prediction network includes two detection heads; one detection head is used to predict the at least one image area, and the other detection head is used to predict the at least one text area, and the detection box output by each detection head includes the following three feature information: the category of the detection box is text or image, the position of the detection box and the confidence of the detection box.
[0154] Optionally, the feature fusion network includes an FPN structure and a PAN structure connected in sequence; the cross-stage local network CspNet in the FPN structure and the PAN structure is replaced by a multi-branch structure with three convolution kernels of different sizes;
[0155] The second detection submodule is specifically configured to:
[0156] Input the top-layer feature map extracted by the backbone network into the first network unit CBL of the FPN structure, input the middle-layer feature map extracted by the backbone network into the first multi-branch structure of the FPN structure, and use the output of the first multi-branch structure as the input of the second CBL;
[0157] After upsampling the feature map output by the first CBL, the feature map is concatenated with the bottom-layer feature map extracted by the backbone network and the feature map output by the second CBL, and the concatenated feature map is used as the input of the second multi-branch structure in the PAN structure;
[0158] using the output of the second multi-branch structure as the input of the third CBL in the PAN structure, using the output of the second CBL as the input of the third multi-branch structure in the PAN structure, and using the output of the third multi-branch structure as the input of the fourth CBL in the PAN structure;
[0159] After downsampling the feature map output by the third CBL, it is spliced with the feature map of the top layer extracted by the backbone network and the feature map output by the fourth CBL, and the spliced feature map is used as the input of the fourth multi-branch structure, and the enhanced feature map is output by the fourth multi-branch structure.
[0160] Optionally, the apparatus 1100 further includes:
[0161] When training the network model, the TIoU loss function is used to constrain the position of the text detection box;
[0162] Among them, the calculation formula of the TIoU loss function is:
[0163] TIou=ln inter(char pred , char gt ) / union(char pred , char gt );
[0164] Among them, inter refers to intersection, union refers to union, char_pred refers to the characters detected by OCR in the predicted text box, and har_gt refers to the characters detected by OCR in the true value box.
[0165] Optionally, the first determining module 1103 includes:
[0166] a first determining submodule, configured to divide, according to a central vertical line of the first image, at least one image region in the target area into a first set located to the left of the central vertical line and a second set located to the right of the central vertical line;
[0167] a second determining submodule, configured to determine a first head portrait area set located on the left from the first set according to the horizontal coordinates of the image areas in the first set;
[0168] a third determining submodule, configured to determine a second head portrait area set located on the right from the second set according to the horizontal coordinates of the image areas in the second set;
[0169] The fourth determining submodule is configured to determine the privacy area according to the first profile picture area set and the second profile picture area set.
[0170] Optionally, the fourth determining submodule includes:
[0171] A first determining unit is configured to use a sliding window to slide within a target longitudinal region of the first image according to a preset step size, and obtain N pixels within the sliding window, where N is a positive integer; wherein the target longitudinal region corresponding to the first profile picture region set is the first longitudinal region located on the left, and the target longitudinal region corresponding to the second profile picture region set is the second longitudinal region located on the left;
[0172] a second determining unit, configured to determine whether the image within the sliding window matches the first region based on a three-channel color difference between the N pixels and pixels at corresponding positions within the first region; wherein the first region is any one of the first profile picture region set and the second profile picture region set;
[0173] a third determining unit, configured to, when the image in the sliding window matches the first area, traverse the head portrait areas in the first head portrait area set and the second head portrait area set, and determine an overlap ratio between each head portrait area and the image area in the sliding window;
[0174] The fourth determination unit is used to determine the image area in the sliding window, the first avatar area set and the second avatar area set as the user avatar part in the privacy area when the overlapping ratio between the avatar area and the image area in the sliding window in the first avatar area set and the second avatar area set is less than a second preset value.
[0175] Optionally, the device further comprises:
[0176] a fifth determining unit, configured to determine the nickname area of the chat group member from at least one text area in the target area according to a positional relationship between the user avatar area and the nickname area of the chat group member;
[0177] The sixth determining unit is configured to determine the nickname area of the chat group member as the nickname part in the privacy area.
[0178] Optionally, the first determining module 1103 includes:
[0179] a fifth determining submodule, configured to select a candidate region from at least one text region in the target region, the candidate region being a text region located above a dividing line of the first image;
[0180] a sixth determining submodule, configured to determine a group name area of the chat group from the candidate areas based on the vertical line in the first image and historical information of an application to which the first image belongs;
[0181] The seventh determining submodule is used to determine the group name area of the chat group as the group name part in the privacy area.
[0182] Optionally, the apparatus 1100 further includes:
[0183] A second acquisition module, configured to acquire a name of an application to which the first image belongs when acquiring the first image;
[0184] The third acquisition module is configured to write the application name into the attribute information of the first image.
[0185] Optionally, the apparatus 1100 includes:
[0186] a second determining module, configured to determine a picture type of the first image according to attribute information of the first image; the picture types include: a single-screen image style, a dual-screen image style, and a long image style;
[0187] The third determining module is configured to process the first image into the single-screen image style when it is determined that the picture style of the first image is the dual-screen image style or the long image style.
[0188] In an embodiment of the present application, a first image is acquired; a target area is detected in the first image; the target area includes at least one text area and at least one image area; a privacy area is determined based on the target area; and privacy processing is performed on the privacy area to obtain a processed second image. This allows users to hide private information with one click, saving time and facilitating operations.
[0189] The image processing device in the embodiment of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not specifically limit it.
[0190] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.
[0191] The image processing device provided in the embodiment of the present application can implement each process implemented in the method embodiments of Figures 1 to 10. To avoid repetition, they will not be described here.
[0192] Optionally, as shown in Figure 12, an embodiment of the present application also provides an electronic device 1200, including a processor 1201 and a memory 1202, and the memory 1202 stores a program or instruction that can be run on the processor 1201. When the program or instruction is executed by the processor 1201, the various steps of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0193] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.
[0194] FIG13 is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.
[0195] The electronic device 1300 includes but is not limited to components such as a radio frequency unit 1301 , a network module 1302 , an audio output unit 1303 , an input unit 1304 , a sensor 1305 , a display unit 1306 , a user input unit 1307 , an interface unit 1308 , a memory 1309 , and a processor 1310 .
[0196] Those skilled in the art will appreciate that the electronic device 1300 may further include a power source (e.g., a battery) to power various components. The power source may be logically connected to the processor 1310 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The electronic device structure shown in FIG13 does not limit the electronic device. The electronic device may include more or fewer components than shown, or may combine certain components, or have different component arrangements, which will not be described in detail here.
[0197] Among them, the processor 1310 is used to obtain a first image; detect a target area in the first image; wherein the target area includes: at least one text area and at least one image area; determine a privacy area based on the target area; perform privacy processing on the privacy area to obtain a processed second image.
[0198] Optionally, the processor 1310 is specifically used to extract feature maps of the top layer, middle layer and bottom layer of the first image through the backbone network of the network model; splice the feature maps through the feature fusion network of the network model to obtain an enhanced feature map; input the enhanced feature map into the head prediction network of the network model to obtain the target area; wherein the head prediction network includes two detection heads; one detection head is used to predict the at least one image area, and the other detection head is used to predict the at least one text area, and the detection box output by each detection head includes the following three feature information: the category of the detection box is text or image, the position of the detection box and the confidence of the detection box.
[0199] Optionally, the feature fusion network includes an FPN structure and a PAN structure connected in sequence; the cross-stage local network CspNet in the FPN structure and the PAN structure is replaced with a multi-branch structure with three convolution kernels of different sizes; the processor 1310 is specifically configured to:
[0200] Input the top-layer feature map extracted by the backbone network into the first network unit CBL of the FPN structure, input the middle-layer feature map extracted by the backbone network into the first multi-branch structure of the FPN structure, and use the output of the first multi-branch structure as the input of the second CBL;
[0201] After upsampling the feature map output by the first CBL, the feature map is concatenated with the bottom-layer feature map extracted by the backbone network and the feature map output by the second CBL, and the concatenated feature map is used as the input of the second multi-branch structure in the PAN structure;
[0202] using the output of the second multi-branch structure as the input of the third CBL in the PAN structure, using the output of the second CBL as the input of the third multi-branch structure in the PAN structure, and using the output of the third multi-branch structure as the input of the fourth CBL in the PAN structure;
[0203] After downsampling the feature map output by the third CBL, it is spliced with the feature map of the top layer extracted by the backbone network and the feature map output by the fourth CBL, and the spliced feature map is used as the input of the fourth multi-branch structure, and the enhanced feature map is output by the fourth multi-branch structure.
[0204] Optionally, the processor 1310 is specifically configured to:
[0205] dividing at least one image region in the target area into a first set located to the left of the middle vertical line and a second set located to the right of the middle vertical line according to the middle vertical line of the first image;
[0206] determining a first head portrait area set located on the left from the first set according to the horizontal coordinates of the image areas in the first set;
[0207] determining a second head portrait area set located on the right from the second set according to the horizontal coordinates of the image areas in the second set;
[0208] The privacy area is determined according to the first avatar area set and the second avatar area set.
[0209] Optionally, the processor 1310 is specifically configured to:
[0210] Using a sliding window, slide within the target longitudinal region of the first image according to a preset step size to obtain N pixels within the sliding window, where N is a positive integer; wherein the target longitudinal region corresponding to the first profile picture region set is a first longitudinal region located on the left, and the target longitudinal region corresponding to the second profile picture region set is a second longitudinal region located on the left;
[0211] determining whether the image within the sliding window matches the first region based on three-channel color differences between the N pixels and pixels at corresponding positions within the first region, wherein the first region is any one of the first profile picture region set and the second profile picture region set;
[0212] When the image in the sliding window matches the first area, traversing the head portrait areas in the first head portrait area set and the second head portrait area set, and determining an overlap ratio between each head portrait area and the image area in the sliding window;
[0213] When the overlap ratio between the avatar area in the first avatar area set and the second avatar area set and the image area in the sliding window is less than a second preset value, the image area in the sliding window, the first avatar area set and the second avatar area set are determined as the user avatar part in the privacy area.
[0214] Optionally, the processor 1310 is further configured to:
[0215] Determining the nickname area of the chat group member from at least one text area in the target area according to a positional relationship between the user avatar area and the nickname area of the chat group member;
[0216] The nickname area of the chat group member is determined as the nickname part in the privacy area.
[0217] Optionally, the processor 1310 is specifically configured to:
[0218] Selecting a candidate region from at least one text region in the target region, wherein the candidate region is a text region located above a dividing line of the first image;
[0219] Determining a group name area of the chat group from the candidate area according to the vertical line in the first image and historical information of the application to which the first image belongs;
[0220] The group name area of the chat group is determined as the group name part in the privacy area.
[0221] Optionally, the processor 1310 is specifically configured to:
[0222] When acquiring the first image, correspondingly acquiring the name of the application to which the first image belongs;
[0223] The application name is written into the attribute information of the first image.
[0224] Optionally, the processor 1310 is further configured to:
[0225] Determine the image type of the first image according to the attribute information of the first image; the image type includes: single-screen image style, dual-screen image style, and long image style;
[0226] When it is determined that the picture style of the first image is the dual-screen image style or the long image style, the first image is processed into the single-screen image style.
[0227] In an embodiment of the present application, an electronic device acquires a first image; detects a target area in the first image; wherein the target area includes at least one text area and at least one image area; determines a privacy area based on the target area; and performs privacy processing on the privacy area to obtain a processed second image. This allows users to hide private information with one click, saving time and facilitating operations.
[0228] It should be understood that in an embodiment of the present application, the input unit 1304 may include a graphics processing unit (GPU) 13041 and a microphone 13042, and the graphics processor 13041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1306 may include a display panel 13061, and the display panel 13061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1307 includes a touch panel 13071 and at least one of the other input devices 13072. The touch panel 13071 is also called a touch screen. The touch panel 13071 may include two parts: a touch detection device and a touch controller. Other input devices 13072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.
[0229] The memory 1309 can be used to store software programs and various data. The memory 1309 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1309 may include a volatile memory or a non-volatile memory, or the memory 1309 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 1309 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.
[0230] Processor 1310 may include one or more processing units. Optionally, processor 1310 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 1310.
[0231] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0232] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0233] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0234] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0235] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0236] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0237] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0238] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.
Claims
1. A method for processing an image, comprising: acquiring a first image; Detecting a target area in the first image; wherein the target area includes: at least one text area and at least one image area; Determining a privacy area according to the target area; Perform privacy processing on the privacy area to obtain a processed second image.
2. The image processing method according to claim 1, wherein: The detecting the target area in the first image includes: Extracting the top, middle and bottom feature maps of the first image through the backbone network of the network model; The feature maps are spliced through a feature fusion network of the network model to obtain an enhanced feature map; The enhanced feature map is input into the head prediction network of the network model to obtain the target area; wherein the head prediction network includes two detection heads; one detection head is used to predict the at least one image area, and the other detection head is used to predict the at least one text area, and the detection frame output by each detection head includes the following three feature information: the category of the detection frame is text or image, the position of the detection frame and the confidence of the detection frame.
3. The image processing method according to claim 2, wherein: The feature fusion network includes an FPN structure and a PAN structure connected in sequence; the cross-stage local network CspNet in the FPN structure and the PAN structure is replaced by a multi-branch structure, and each of the multi-branch structures includes three convolution kernels of different sizes; The step of splicing the feature maps through the feature fusion network of the network model to obtain an enhanced feature map includes: Input the feature map of the top layer extracted by the backbone network into the first network unit CBL of the FPN structure, input the feature map of the middle layer extracted by the backbone network into the first multi-branch structure of the FPN structure, and use the output of the first multi-branch structure as the input of the second network unit CBL; After upsampling the feature map output by the first network unit CBL, splicing it with the bottom feature map extracted by the backbone network and the feature map output by the second network unit CBL, and using the spliced feature map as the input of the second multi-branch structure in the PAN structure; Using the output of the second multi-branch structure as the input of the third network unit CBL in the PAN structure, using the output of the second network unit CBL as the input of the third multi-branch structure in the PAN structure, and using the output of the third multi-branch structure as the input of the fourth network unit CBL in the PAN structure; After downsampling the feature map output by the third network unit CBL, it is spliced with the feature map of the top layer extracted by the backbone network and the feature map output by the fourth network unit CBL, and the spliced feature map is used as the input of the fourth multi-branch structure, and the enhanced feature map is output by the fourth multi-branch structure.
4. The image processing method according to claim 1, wherein: The step of determining the privacy area according to the target area includes: According to a middle vertical line of the first image, dividing at least one image area in the target area into a first set located on the left side of the middle vertical line and a second set located on the right side of the middle vertical line; Determining a first head portrait area set located on the left from the first set according to the horizontal coordinates of the image areas in the first set; Determining a second head portrait area set located on the right from the second set according to the horizontal coordinates of the image areas in the second set; The privacy area is determined according to the first avatar area set and the second avatar area set.
5. The image processing method according to claim 4, wherein: The determining the privacy area according to the first head portrait area set and the second head portrait area set includes: Using a sliding window, slide in the target longitudinal area of the first image according to a preset step length to obtain N pixel points in the sliding window; N is a positive integer; wherein the target longitudinal area corresponding to the first head portrait area set is a first longitudinal area located on the left, and the target longitudinal area corresponding to the second head portrait area set is a second longitudinal area located on the left; Determining whether the image in the sliding window matches the first area according to the three-channel color differences between the N pixels and the pixels at corresponding positions in the first area; wherein the first area is any one of the first avatar area set and the second avatar area set; When the image in the sliding window matches the first area, traversing the avatar areas in the first avatar area set and the second avatar area set to determine an overlap ratio between each avatar area and the image area in the sliding window; When the overlap ratio between the avatar area in the first avatar area set and the second avatar area set and the image area in the sliding window is less than a second preset value, the image area in the sliding window, the first avatar area set and the second avatar area set are determined as the user avatar part in the privacy area.
6. The image processing method according to claim 5, wherein: After determining the image area in the sliding window, the first avatar area set, and the second avatar area set as the user avatar part in the privacy area, the method further includes: Determining the nickname area of the chat group member from at least one text area in the target area according to the positional relationship between the user avatar area and the nickname area of the chat group member; The nickname area of the chat group member is determined as the nickname part in the privacy area.
7. The image processing method according to claim 1, wherein: The step of determining the privacy area according to the target area includes: Selecting a candidate region from at least one text region in the target region, the candidate region being a text region located above a boundary line of the first image; Determining a group name area of a chat group from the candidate areas according to the vertical line in the first image and historical information of an application to which the first image belongs; The group name area of the chat group is determined as the group name part in the privacy area.
8. The image processing method according to claim 1, wherein: The method further comprises: When acquiring the first image, correspondingly acquiring the name of the application to which the first image belongs; The application name is written into the attribute information of the first image.
9. The image processing method according to claim 8, wherein: Before detecting the target area in the first image, the method further includes: Determine the image type of the first image according to the attribute information of the first image; the image type includes: single-screen image style, dual-screen image style and long image style; When it is determined that the picture style of the first image is the dual-screen image style or the long image style, the picture type of the first image is processed into the single-screen image style.
10. An image processing device, comprising: A first acquisition module, used to acquire a first image; A detection module, configured to detect a target area in the first image; wherein the target area includes: at least one text area and at least one image area; A first determining module, configured to determine a privacy area according to the target area; The processing module is used to perform privacy processing on the privacy area to obtain a processed second image.
11. The image processing device according to claim 10, wherein: The detection module includes: A first detection submodule, used for extracting feature maps of the top layer, middle layer and bottom layer of the first image through a backbone network of a network model; A second detection submodule is used to splice the feature map through a feature fusion network of the network model to obtain an enhanced feature map; The third detection submodule is used to input the enhanced feature map into the head prediction network of the network model to obtain the target area; wherein the head prediction network includes two detection heads; one detection head is used to predict the at least one image area, and the other detection head is used to predict the at least one text area, and the detection frame output by each detection head includes the following three feature information: the category of the detection frame is text or image, the position of the detection frame and the confidence of the detection frame.
12. The image processing device according to claim 11, wherein: The feature fusion network includes an FPN structure and a PAN structure connected in sequence; the cross-stage local network CspNet in the FPN structure and the PAN structure is replaced by a multi-branch structure with three convolution kernels of different sizes; The second detection submodule is specifically used for: Input the feature map of the top layer extracted by the backbone network into the first network unit CBL of the FPN structure, input the feature map of the middle layer extracted by the backbone network into the first multi-branch structure of the FPN structure, and use the output of the first multi-branch structure as the input of the second network unit CBL; After upsampling the feature map output by the first network unit CBL, splicing it with the bottom feature map extracted by the backbone network and the feature map output by the second network unit CBL, and using the spliced feature map as the input of the second multi-branch structure in the PAN structure; Using the output of the second multi-branch structure as the input of the third network unit CBL in the PAN structure, using the output of the second network unit CBL as the input of the third multi-branch structure in the PAN structure, and using the output of the third multi-branch structure as the input of the fourth network unit CBL in the PAN structure; After downsampling the feature map output by the third network unit CBL, it is spliced with the feature map of the top layer extracted by the backbone network and the feature map output by the fourth network unit CBL, and the spliced feature map is used as the input of the fourth multi-branch structure, and the enhanced feature map is output by the fourth multi-branch structure.
13. The image processing device according to claim 10, wherein: The first determining module includes: A first determination submodule, configured to divide at least one image region in the target region into a first set located on the left side of the middle vertical line and a second set located on the right side of the middle vertical line according to the middle vertical line of the first image; A second determining submodule, configured to determine a first head portrait area set located on the left side from the first set according to the horizontal coordinates of the image areas in the first set; A third determining submodule, configured to determine a second head portrait area set located on the right side from the second set according to the horizontal coordinates of the image areas in the second set; The fourth determining submodule is used to determine the privacy area according to the first avatar area set and the second avatar area set.
14. The image processing device according to claim 13, wherein: The fourth determining submodule includes: The first determining unit is configured to use a sliding window to slide in a target longitudinal region of the first image according to a preset step length to obtain N pixel points in the sliding window; N is a positive integer; wherein the target longitudinal region corresponding to the first head portrait region set is a first longitudinal region located on the left, and the target longitudinal region corresponding to the second head portrait region set is a second longitudinal region located on the left; A second determining unit is used to determine whether the image in the sliding window matches the first area according to a three-channel color difference between the N pixels and the pixels at corresponding positions in the first area; wherein the first area is any one of the first avatar area set and the second avatar area set; a third determining unit, configured to, when the image in the sliding window matches the first area, traverse the avatar areas in the first avatar area set and the second avatar area set, and determine an overlap ratio between each avatar area and the image area in the sliding window; The fourth determination unit is used to determine the image area in the sliding window, the first avatar area set and the second avatar area set as the user avatar part in the privacy area when the overlap ratio between the avatar area and the image area in the sliding window in the first avatar area set and the second avatar area set is less than a second preset value.
15. The image processing device according to claim 14, wherein: The device also includes: a fifth determining unit, configured to determine the nickname area of the chat group member from at least one text area in the target area according to a positional relationship between the user avatar area and the nickname area of the chat group member; The sixth determining unit is used to determine the nickname area of the chat group member as the nickname part in the privacy area.
16. The image processing device according to claim 10, wherein: The first determining module includes: a fifth determination submodule, configured to select a candidate region from at least one text region in the target region, wherein the candidate region is a text region located above a boundary line of the first image; a sixth determination submodule, configured to determine a group name area of the chat group from the candidate areas according to the vertical line in the first image and historical information of an application to which the first image belongs; The seventh determination submodule is used to determine the group name area of the chat group as the group name part in the privacy area.
17. The image processing device according to claim 10, wherein: The device further comprises: A second acquisition module, configured to acquire a name of an application to which the first image belongs when acquiring the first image; The third acquisition module is used to write the application name into the attribute information of the first image.
18. The image processing device according to claim 17, wherein: The device comprises: A second determination module, configured to determine a picture type of the first image according to the attribute information of the first image; the picture type includes: a single-screen image style, a dual-screen image style, and a long image style; The third determining module is used to process the first image into the single-screen image style when it is determined that the picture style of the first image is the dual-screen image style or the long image style.
19. An electronic device comprising a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 9 are implemented.
20. A readable storage medium storing a program or instruction, wherein the program or instruction, when executed by a processor, implements the steps of the image processing method according to any one of claims 1 to 9.
21. A chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the image processing method according to any one of claims 1 to 9.
22. A computer program product, wherein the computer program product is executed by at least one processor to implement the steps of the image processing method according to any one of claims 1 to 9.
23. An electronic device, configured to execute the steps of the image processing method according to any one of claims 1 to 9.
Citation Information
Patent Citations
Image processing method and device
CN108629200A
Screenshot processing method and mobile terminal
CN110262735A
Target identification method and device, electronic equipment and medium
CN115170819A
Image processing method and device and electronic equipment
CN116703705A
Authentication control apparatus, authentication system, authentication control method, and storage medium
US20230141541A1
