Image processing method and electronic device

Abstract

The application provides an image processing method, characterized in that the method comprises: inputting an object to be audited into a content audit neural network model to obtain a plurality of first feature maps output by a plurality of convolution layers; performing image upsampling and image splicing on the plurality of first feature maps respectively to obtain a plurality of second feature maps; performing image fusion on a class activation mapping diagram corresponding to the plurality of second feature maps and the object to be audited to generate a heat map image or a heat map image frame corresponding to the plurality of second feature maps; and shielding illegal content in the object to be audited by using the heat map image or the heat map image frame. The application also simultaneously provides an electronic device.

Description

Technical Field

[0001] This invention relates to image processing technology, specifically to an image processing method and an electronic device. Background Technology

[0002] Currently, a large number of videos on the edge of legality have appeared on the business side. For videos with a small amount of illegal content, the machine will directly block them after review and identification. As a result, the entire video uploaded by the user will be blocked and cannot be viewed, which has led to user complaints. Summary of the Invention

[0003] To address the existing technical problems, the technical solution of this application is implemented as follows:

[0004] According to one aspect of this application, an image processing method is provided, characterized in that the method includes:

[0005] The object to be reviewed is input into the content review neural network model, resulting in multiple first feature maps output by multiple convolutional layers;

[0006] The multiple first feature maps are upsampled and stitched together to obtain multiple second feature maps;

[0007] The class activation map corresponding to the multiple second feature maps is fused with the object to be reviewed to generate a heat map image or heat map image frame corresponding to the multiple second feature maps.

[0008] The heatmap image or the heatmap image frame is used to block the illegal content in the object to be reviewed.

[0009] In the above scheme, the step of using the heatmap image or the heatmap image frame to block the illegal content in the object to be reviewed includes:

[0010] The response region in the heatmap image or the heatmap image frame is occluded to obtain an occluded image; wherein the response region represents the illegal content;

[0011] Replace the corresponding target image or target image frame in the object to be reviewed with the occluded image.

[0012] In the above scheme, before performing image fusion of the class activation maps corresponding to the multiple second feature maps with the object to be reviewed, the method further includes:

[0013] The multiple second feature maps are input into a global mean pooling layer for image stitching to obtain a third feature map;

[0014] The third feature map is input into a fully connected layer for category classification to obtain the category score of the third feature map;

[0015] If the category score meets the activation condition, gradient-weighted class activation mapping is performed on the category score to generate class activation mapping maps corresponding to the multiple second feature maps.

[0016] The method in the above scheme further includes:

[0017] Compare the category score with a score threshold;

[0018] If the comparison result indicates that the category score is greater than or equal to the score threshold, it is determined that the category score satisfies the activation condition.

[0019] In the above scheme, the step of occluding the response region in the heatmap image to obtain an occluded image includes at least one of the following methods:

[0020] The response regions in the heatmap image are pixelated to obtain a mosaic image;

[0021] The response region in the heatmap image is overexposed to obtain an overexposed image;

[0022] The response areas in the heatmap image are overlaid to obtain the overlaid image.

[0023] In the above scheme, the step of performing mosaic processing on the response region in the heatmap image to obtain a mosaic image includes:

[0024] Convert the heatmaps corresponding to the multiple second feature maps into mask images;

[0025] The pixel points of the mask image are subjected to mean blurring to obtain the mosaic image.

[0026] The method in the above scheme further includes:

[0027] The heatmap images are numbered.

[0028] Save the heatmap image marked with the specified serial number.

[0029] The method in the above scheme further includes:

[0030] Construct a sample dataset, which consists of several images with known target categories;

[0031] A convolutional neural network is constructed, and the convolutional neural network is trained using the sample dataset to obtain the content moderation neural network model.

[0032] According to another aspect of this application, an electronic device is provided, characterized in that the electronic device comprises:

[0033] The input unit is used to input the object to be reviewed into the content review neural network model to obtain multiple first feature maps output by multiple convolutional layers;

[0034] The stitching unit is used to perform image upsampling and image stitching on the multiple first feature maps respectively to obtain multiple second feature maps;

[0035] The fusion unit is used to perform image fusion between the class activation map corresponding to the multiple second feature maps and the object to be reviewed, and generate a heat map image or heat map image frame corresponding to the multiple second feature maps.

[0036] The shielding unit is used to shield the illegal content in the object to be reviewed using the heat map image or the heat map image frame.

[0037] According to a third aspect of this application, an electronic device is provided, characterized in that the electronic device comprises: a processor and a memory for storing a computer program capable of running on the processor.

[0038] When the processor runs the computer program, it executes any one of the method steps of any of the above-described image processing methods.

[0039] According to a fourth aspect of this application, a computer-readable storage medium is provided, characterized in that it is used to store computer instructions, which, when executed by a processor, perform any one of the method steps of any of the above-described image processing methods.

[0040] The image processing method and electronic device provided in this invention upsample the feature maps output from different channels and then stitch them together during the generation of a heatmap image or heatmap image frame. Then, the class activation map corresponding to the stitched feature map is fused with the original image to generate a heatmap image or heatmap image frame. This heatmap image or heatmap image frame is used to block illegal content in the object to be reviewed. Thus, by using this multi-scale fusion network structure for gradient-weighted class activation mapping, a more accurate class activation map can be generated. Therefore, when multiple identical targets exist in an image, this class activation map can be used to accurately identify multiple identical targets in an image. Attached Figure Description

[0041] Figure 1 This is a schematic diagram illustrating the flow of the image processing method in this application. Figure 1 ;

[0042] Figure 2This is a schematic diagram of the multi-scale fusion network structure and Grad-CAM in this application;

[0043] Figure 3 This is a schematic diagram illustrating the flow of the image processing method in this application. Figure 2 ;

[0044] Figure 4 This is a schematic diagram of the structural composition of the electronic device in this application. Figure 1 ;

[0045] Figure 5 This is a comparative schematic diagram of Grad-CAM in this application;

[0046] Figure 6 This is a schematic diagram of the structural composition of the electronic device in this application. Figure 2 . Detailed Implementation

[0047] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0048] Figure 1 This is a schematic diagram illustrating the flow of the image processing method in this application. Figure 1 ;like Figure 1 As shown, the method includes:

[0049] Step 101: Input the object to be reviewed into the content review neural network model to obtain multiple first feature maps output by multiple convolutional layers;

[0050] Step 102: Upsample and stitch the multiple first feature maps to obtain multiple second feature maps;

[0051] Step 103: Perform image fusion between the class activation mapping maps corresponding to the multiple second feature maps and the object to be reviewed to generate a heatmap image or heatmap image frame corresponding to the multiple second feature maps;

[0052] Step 104: Use the heatmap image or the heatmap image frame to block the illegal content in the object to be reviewed.

[0053] In this application, the method can be applied to various electronic devices capable of reviewing videos or images, such as computers, servers, media devices, etc. The electronic device has a content review neural network model capable of identifying the content of the object to be reviewed. By inputting the object to be reviewed into the content review neural network model, a category score for the object can be obtained. Based on this category score, it can be determined whether there is any illegal content in the object to be reviewed, and if illegal content is determined to exist, the illegal content is blocked.

[0054] Here, when constructing the content moderation neural network model, the electronic device can first construct a sample dataset, which can be several images of known target categories; then construct a convolutional neural network, and use the sample dataset to train the convolutional neural network, thereby obtaining the content moderation neural network model.

[0055] In one implementation of this application, the object to be reviewed can be a video. When the object to be reviewed is a video, the electronic device can perform frame extraction on the video to be reviewed using keyframes to obtain image frames. Then, the image frames are input into the content review neural network model to obtain the category score of the image frames. The category score is then compared with a score threshold to obtain a comparison result. If the comparison result indicates that the category score is greater than or equal to the score threshold, it is determined that the video to be reviewed has not passed the system review; if the comparison result indicates that the category score is less than the score threshold, it is determined that the video to be reviewed has passed the system review.

[0056] Here, the electronic device can also perform frame extraction on the video to be reviewed according to a preset interval to obtain image frames. For example, if the preset interval is 1 second, then the video to be reviewed will be extracted once every 1 second. Here, the video to be reviewed can have multiple image frames within 1 second.

[0057] In another implementation of this application, the object to be reviewed can be an image. When the object to be reviewed is an image, the electronic device can directly input the image to be reviewed into the content review neural network model. The content review neural network model can obtain the category score of the image to be reviewed, and then compare the category score with a score threshold to obtain a comparison result. If the comparison result indicates that the category score is greater than or equal to the score threshold, it is determined that the image to be reviewed has not passed the system review; if the comparison result indicates that the category score is less than the score threshold, it is determined that the image to be reviewed has passed the system review.

[0058] Here, the failure of a video or image to pass the system review indicates that the video or image contains illegal content. This illegal content refers to inappropriate information that is not suitable for dissemination online, including but not limited to pornography, sensitive content, horror, crime, erotic content, and gore.

[0059] In this application, the content moderation neural network model can be a U-net network structure model, which can be well applied to image segmentation. When the electronic device obtains the category score of the image frame through the content moderation neural network model, it can first input the object to be moderated into the content moderation neural network model. The content moderation neural network model can obtain multiple first feature maps of different scales output by multiple convolutional layers. Then, the multiple first feature maps of different scales are upsampled and image-stitched to obtain multiple second feature maps of different scales. Then, the multiple second feature maps of different scales are input into a global mean pooling layer for image-stitching to obtain a third feature map. Then, the third feature map is input into a fully connected layer for category classification to obtain the category score of the third feature map (i.e., the category score of the image frame).

[0060] In this application, after obtaining the category score of the feature map, if the category score meets the activation condition, the electronic device performs gradient-weighted class activation mapping on the category score to generate the class activation mapping map corresponding to the multiple second feature maps.

[0061] Here, the electronic device can compare the category score with a score threshold to obtain a comparison result. If the comparison result indicates that the category score is greater than or equal to the score threshold, it is determined that the category score meets the activation condition, indicating that there is illegal content in the object to be reviewed and it has not passed the system review. If the comparison result indicates that the category score is less than the score threshold, it is determined that the category score does not meet the activation condition, indicating that there is no illegal content in the object to be reviewed and it has passed the system review.

[0062] Here, when the category score meets the activation condition, the electronic device can also perform gradient-weighted class activation mapping on the category score to generate class activation mapping maps corresponding to the multiple second feature maps. Then, the class activation mapping maps corresponding to the multiple second feature maps are fused with the corresponding image frames in the object to be reviewed to generate heatmap images or heatmap image frames corresponding to the multiple second feature maps.

[0063] In this application, the feature maps at various scales are fused with the original image through the U-net network structure, and gradient-weighted class activation mapping is performed to generate a more accurate class activation map. Using this class activation map, multiple identical targets in an image can be accurately identified and located.

[0064] In this application, when the electronic device uses the heatmap image or the heatmap image frame to block the illegal content in the object to be reviewed, it can also occlude the response area in the heatmap image or the heatmap image frame to obtain an occluded image; wherein, the response area represents the illegal content in the object to be reviewed; and then the occluded image is used to replace the corresponding target image or target image frame in the object to be reviewed, so that the object to be reviewed meets the review requirements and is suitable for dissemination on the network.

[0065] Here, when the electronic device occludes the response region in the heatmap image to obtain an occluded image, it can do so by at least any one or a combination of the following methods:

[0066] The response areas in the heatmap image are pixelated to obtain a mosaic image;

[0067] The response region in the heatmap image is overexposed to obtain an overexposed image;

[0068] The response regions in the heatmap image are overlaid to obtain the overlaid image.

[0069] Here, when the electronic device performs mosaic processing on the response area in the heatmap image to obtain a mosaic image, it can first convert the heatmaps corresponding to the multiple second feature maps into a mask image; then, it can perform mean blur processing on the pixels of the mask image to obtain the mosaic image.

[0070] Here, when the electronic device performs overexposure processing on the response area in the heatmap image to obtain an overexposed image, it can first convert the heatmaps corresponding to the multiple second feature maps into a mask image; then, it can adjust the exposure parameter of the response area of ​​the heatmap corresponding to the mask image to the maximum to obtain the overexposed image.

[0071] Here, when the electronic device performs image overlay processing on the response area in the heat map image to obtain the overlaid image, it can overlay the preset image onto the response area in the heat map image to obtain the overlaid image.

[0072] Thus, by using the class activation map to occlude (e.g., mosaic) the area of ​​the class response, this application can provide the reason why the object to be reviewed fails the system review.

[0073] Figure 2 The diagrams shown are schematics of the multi-scale fusion network structure and Grad-CAM in this application. Figure 2 As shown,

[0074] The original image frame 201 (which can be understood as the image frame obtained after the video to be reviewed is extracted by the review system) is input into the content review neural network model. The computational unit (basebone) 202 in the content review neural network model produces the output results of three convolutional layers, which are called feature maps f1, f2, and f3, respectively. The feature maps f1, f2, and f3 have different scales, such as 28*28, 14*14, and 7*7, respectively. Then, feature map f3 is upsampled and concatenated with feature map f2 to obtain feature map f4, as shown in the following formula (I):

[0075] f4=concat(f2,upsampling(f3)) (1);

[0076] Then, feature map f4 is upsampled and concatenated with feature map f1 to obtain feature map f5, as shown in the following equation:

[0077] f5=concat(f1,upsampling(f4)) (2);

[0078] Finally, the obtained feature maps f3, f4, and f5 are input into the global average pooling layer (avgpool) 203 for mean calculation to obtain three arrays. Then, the three arrays are merged by the concat unit (concat) 204 (i.e., image stitching of feature maps f3, f4, and f5). The stitched feature maps f3, f4, and f5 are then input into the fully connected layer (Fc_layers) 205 for class classification, and the class score Y is output. c .

[0079] Where Y represents the score, c represents the category, and the score Y for category c is... c This can be represented as a linear combination of the global mean pooling of the last three feature maps f3, f4, and f5:

[0080]

[0081] Where k represents the feature icon number, such as f3, f4, f5. This represents the class activation map generated by backpropagation, where i and j represent the row and column lengths of the feature map, Z is the number of pixels in the effect map (i*j), and w represents the linear weight from the feature map to the class, calculated as follows:

[0082]

[0083] Among them, Y c It is a class activation map graph The differentiability of Y is why Gradient-weighted Class Activation Mapping (Grad-CAM) allows class activation mapping (CAM) to generalize to any convolutional neural network (CNN) model. Since this application performs global mean pooling on the feature maps after the last three convolutional layers and then concatenates them before inputting them into the fully connected layer, Y... c Three class activation maps A can be generated. These three class activation maps A are then upsampled to the original image frame (i.e., the object to be reviewed) of size 201, resulting in class activation maps A1, A2, and A3. Heatmaps of the response regions of class activation maps A1, A2, and A3 are then generated, and the maximum response value in each class activation map is taken to obtain the final A. final As shown below:

[0084] A final =max(A1,A2,A3) (V);

[0085] Here, the response area corresponding to the maximum response value is the violation area. Then, the heatmap of the response areas of the class activation map A1, A2, and A3 is converted into a mask map 206. The mask map 206 is then overlaid on the original image frame 201 to obtain a heatmap image frame 207. Then, the pixels of the response area in the heatmap image frame 207 are subjected to a 3x3 mean blur operation to obtain a mosaic image frame. This mosaic image frame is used to replace the corresponding original image frame 201 in the video to be reviewed.

[0086] In this application, when the electronic device obtains multiple heat map images or heat map image frames corresponding to the second feature maps, it can also mark the heat map images or heat map image frames with serial numbers; and save the heat map images or heat map image frames marked with serial numbers.

[0087] Here, when the electronic device marks the heat map image or heat map image frame with a serial number, it can mark the heat map image or heat map image frame with a serial number using Arabic numerals.

[0088] For example, if there are 5 heatmap image frames (i.e., violation image frames) in the original video, these 5 heatmap image frames can be marked sequentially according to Arabic numerals, that is, the marked heatmap image frames are: heatmap image frame 1, heatmap image frame 2, heatmap image frame 3, heatmap image frame 4, and heatmap image frame 5.

[0089] Here, when the electronic device marks the heatmap image or heatmap image frame with a sequence number, it can also mark the heatmap image or heatmap image frame with a sequence number according to the image frame sequence of the heatmap image frame in the video to be reviewed.

[0090] For example, if the original video contains 5 heatmap image frames (i.e., violation image frames), located at frames 5, 6, 15, 16, and 20, where frame 5 corresponds to 10 minutes and 10 seconds in the original video, frame 6 to 10 minutes and 20 seconds, frame 15 to 20 minutes and 10 seconds, frame 16 to 20 minutes and 20 seconds, and frame 20 to 15 minutes and 10 seconds, then these heatmap image frames are marked according to their corresponding times in the original video. That is, the marked heatmap image frames are: frame 5 (10 minutes and 10 seconds), frame 6 (10 minutes and 20 seconds), frame 15 (20 minutes and 10 seconds), frame 16 (20 minutes and 20 seconds), and frame 20 (15 minutes and 10 seconds).

[0091] In this application, the electronic device uses a class activation map to generate heatmap image frames (or heatmap keyframes), and saves the set of heatmap keyframes as the review content of the video to be reviewed, so that manual review can quickly grasp the key to the content violation (heatmap), which has the technical effect of improving the efficiency of content review and saving a lot of manual review costs.

[0092] Figure 3 This is a schematic diagram illustrating the flow of the image processing method in this application. Figure 2 ,like Figure 3 As shown,

[0093] Step 301: Input the video to be reviewed into the review system in the electronic device;

[0094] Step 302: The review system performs frame extraction on the video to be reviewed to obtain image frame F;

[0095] Step 303: Input the image frame F into the pre-trained content moderation neural network model;

[0096] Here, after the content moderation neural network model calculates the content frame F, it can output the category score corresponding to the content frame F;

[0097] Step 304: Determine whether the image frame F has passed system review;

[0098] Here, the review system can compare the category score with the score threshold. If the comparison result indicates that the category score is greater than or equal to the score threshold, the image frame F (or the video to be reviewed) is determined not to pass the system review, and step 306 is executed; if the comparison result indicates that the category score is less than the score threshold, the image frame F (or the video to be reviewed) is determined to pass the system review, and step 305 is executed.

[0099] Step 305: No processing operations are performed on the image frame F;

[0100] Here, if image F passes the system review, the next image frame in the video to be reviewed can be extracted, and the relevant operation steps for image frame F can be repeated.

[0101] Step 306: Perform Grad-CAM operation on the category score of image frame F;

[0102] Here, by performing a Grad-CAM operation on the class score of image frame F, the gradient of the corresponding region of image frame F can be visualized;

[0103] Step 307: Generate a heatmap of the response region of the image frame F.

[0104] Here, because Grad-CAM can visualize the score response of an image region without human review and without human attention, it can be used to better understand CNN convolutional neural networks and model decisions.

[0105] Step 308: The generated heatmap is fused with image frame F to generate a heatmap image frame;

[0106] Here, the response area in the heatmap image frame is the reason why the image frame F fails the system review, as indicated by the content review neural network model.

[0107] Step 309: Mosaic processing is performed on the response area in the heatmap image frame to obtain a mosaic image frame;

[0108] Here, the heatmap can be converted into a mask layer, and the original image pixels in the mask can be subjected to a 3x3 mean blur operation to obtain a mosaic image frame; then the image frame F in the video to be reviewed (i.e., the original video) can be replaced with the mosaic image frame.

[0109] Step 310: Save the heatmap image frame of image frame F.

[0110] The image processing method provided in this application incorporates a U-net network structure into a multi-scale fusion network structure, improving target localization. During Grad-CAM gradient feedback, it can better target local areas, resulting in better heatmap generation. It can also effectively locate multiple targets within a single image, improving the accuracy of target area recognition and localization. Furthermore, by automatically mosaicking inappropriate content in the video to be reviewed, the safety and health of video content on the internet can be ensured, promptly curbing opportunistic individuals from spreading harmful information online and causing negative impacts, while simultaneously ensuring the storage of good information. Using a heatmap keyframe sequence synthesized by Grad-CAM to replace the original video as the review content reduces video review time and labor costs, improving manual review efficiency. Saving the heatmap image frames as the result of the video's manual review allows reviewers to determine whether the video violates regulations simply by examining the image sequence composed of these heatmap image frames. Replacing the original video with these images for manual review reduces the burden and cost of manual review, accelerating its efficiency.

[0111] Figure 4 This is a schematic diagram of the structural composition of the electronic device in this application. Figure 1 ,like Figure 4 As shown, the electronic device includes:

[0112] Input unit 401 is used to input the object to be reviewed into the content review neural network model to obtain multiple first feature maps output by multiple convolutional layers;

[0113] The stitching unit 402 is used to perform image upsampling and image stitching on the multiple first feature maps respectively to obtain multiple second feature maps;

[0114] The fusion unit 403 is used to perform image fusion between the class activation map corresponding to the multiple second feature maps and the object to be reviewed, and generate a heat map image or heat map image frame corresponding to the multiple second feature maps.

[0115] The shielding unit 404 is used to shield the illegal content in the object to be reviewed using the heat map image or the heat map image frame.

[0116] In this application, the electronic device can be any type of electronic device with media content review capabilities, including but not limited to computers, servers, and multimedia processing devices.

[0117] In a preferred embodiment, the shielding unit 404 includes a processing unit 405 and a replacement unit 406;

[0118] The processing unit 405 is used to occlude the response region in the heatmap image or the heatmap image frame to obtain an occluded image; wherein the response region represents the illegal content.

[0119] Replacement unit 406 replaces the corresponding target image or target image frame in the object to be reviewed with the occluded image.

[0120] In a preferred embodiment, the electronic device further includes: a computing unit 407;

[0121] Specifically, the stitching unit 402 is also used to input the multiple second feature maps into the global mean pooling layer for image stitching to obtain a third feature map;

[0122] The input unit 401 is further configured to input the third feature map into a fully connected layer for category classification, thereby obtaining a category score for the third feature map;

[0123] The calculation unit 407 is used to perform gradient-weighted class activation mapping on the class scores if the class scores meet the activation conditions, and generate class activation mapping maps corresponding to the multiple second feature maps.

[0124] In a preferred embodiment, the electronic device further includes a comparison unit 408 and a determination unit 409;

[0125] The comparison unit 408 is used to compare the category score with a score threshold.

[0126] The determining unit 409 is used to determine that the category score satisfies the activation condition if the comparison result indicates that the category score is greater than or equal to the score threshold.

[0127] In a preferred embodiment, the processing unit 405 is specifically used to perform mosaic processing on the response area in the heatmap image to obtain a mosaic image; or to perform overexposure processing on the response area in the heatmap image to obtain an overexposure image; or to perform image overlay processing on the response area in the heatmap image to obtain an image after image overlay.

[0128] In a preferred embodiment, the electronic device further includes:

[0129] The conversion unit 410 is used to convert the heat map corresponding to the plurality of second feature maps into a mask map;

[0130] The processing unit 405 is specifically used to perform mean blurring on the pixels of the mask image to obtain the mosaic image.

[0131] In a preferred embodiment, the electronic device further includes: a marking unit 411 and a storage unit 412;

[0132] The marking unit 411 is used to mark the heat map image or heat map image frame with a serial number;

[0133] The storage unit 412 is used to store the heat map image or heat map image frame marked with the serial number.

[0134] In a preferred embodiment, the electronic device further includes: a training unit 413 for constructing a sample dataset, wherein the sample dataset consists of several images of known target categories; and a unit for constructing a convolutional neural network, wherein the convolutional neural network is trained using the sample dataset to obtain the content moderation neural network model.

[0135] It should be noted that the electronic device provided in the above embodiments, when performing content review and blocking on the object to be reviewed, is only illustrated by the division of the above program modules. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the electronic device provided in the above embodiments and the image processing method embodiments provided above belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0136] Figure 5 This is a comparative illustration of Grad-CAM in this application, as shown below. Figure 5 As shown: including the original figure 501, Grad-CAM figure 502 and multi-scale Grad-CAM figure 503, wherein Grad-CAM figure 502 is the result of Grad-CAM performed by directly inputting the ResNeSt-50 basic network structure into the fully connected layer in the prior art, and multi-scale Grad-CAM figure 503 is the result of Grad-CAM performed using the multi-scale fusion network structure provided in this application.

[0137] Depend on Figure 5 It can be seen that the multi-scale fusion network structure proposed in this application can obtain a better local response region after feature fusion at multiple scales and gradient feedback, making the target localization more accurate and solving the problem that Grad-CAM performs poorly when there are multiple identical objects in an image.

[0138] Figure 6 This is a schematic diagram of the structural composition of the electronic device in this application. Figure 2 Electronic device 600 can be a computer, digital broadcasting terminal, information transceiver, medical device, fitness equipment, personal digital assistant, or other terminal or server. Here, server includes, but is not limited to, physical server and cloud server. Figure 6The illustrated electronic device 600 includes at least one processor 601, a memory 602, at least one network interface 604, and a user interface 603. The various components in the electronic device 600 are coupled together via a bus system 605. It is understood that the bus system 605 is used to implement communication between these components. In addition to a data bus, the bus system 605 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 6 The general designated all buses as Bus System 605.

[0139] The user interface 603 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, or touch screen.

[0140] It is understood that memory 602 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 602 described in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0141] In this embodiment, the memory 602 is used to store various types of data to support the operation of the electronic device 600. Examples of such data include: any computer program used to operate on the electronic device 600, such as the operating system 6021 and application program 6022; contact data; phonebook data; messages; pictures; audio, etc. The operating system 6021 includes various system programs, such as the framework layer, core library layer, driver layer, etc., used to implement various basic services and handle hardware-based tasks. The application program 6022 may include various applications, such as a media player, browser, etc., used to implement various application services. Programs implementing the methods of this embodiment may be included in the application program 6022.

[0142] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 601. Processor 601 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 601 or by instructions in software form. The processor 601 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 601 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 602. Processor 601 reads the information in memory 602 and combines it with its hardware to complete the steps of the aforementioned method.

[0143] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0144] In an exemplary embodiment, this application also provides a computer-readable storage medium, such as a memory 602 including a computer program, which can be executed by a processor 901 of an electronic device 600 to complete the steps described in the aforementioned method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM; it may also be various devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0145] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs any of the method steps of any of the above-described device binding methods.

[0146] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0147] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0148] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0149] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0150] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0151] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method includes: The object to be reviewed is input into the content review neural network model, resulting in multiple first feature maps output by multiple convolutional layers; The multiple first feature maps are upsampled and stitched together to obtain multiple second feature maps; the multiple second feature maps are all of different scales; The class activation map corresponding to the multiple second feature maps is fused with the object to be reviewed to generate a heat map image or heat map image frame corresponding to the multiple second feature maps. The heatmap image or the heatmap image frame is used to block the illegal content in the object to be reviewed; The method further includes, prior to the image fusion of the class activation maps corresponding to the plurality of second feature maps with the object to be reviewed: The multiple second feature maps are input into a global mean pooling layer for image stitching to obtain a third feature map; The third feature map is input into a fully connected layer for category classification to obtain the category score of the third feature map; If the category score meets the activation condition, gradient-weighted class activation mapping is performed on the category score to generate class activation mapping maps corresponding to the multiple second feature maps.

2. The method according to claim 1, characterized in that, The step of using the heatmap image or the heatmap image frame to block illegal content in the object to be reviewed includes: The response region in the heatmap image or the heatmap image frame is occluded to obtain an occluded image; wherein the response region represents the illegal content; Replace the corresponding target image or target image frame in the object to be reviewed with the occluded image.

3. The method according to claim 1, characterized in that, The method further includes: Compare the category score with a score threshold; If the comparison result indicates that the category score is greater than or equal to the score threshold, it is determined that the category score satisfies the activation condition.

4. The method according to claim 2, characterized in that, The process of occluding the response region in the heatmap image to obtain an occluded image includes at least one of the following methods: The response regions in the heatmap image are pixelated to obtain a mosaic image; The response region in the heatmap image is overexposed to obtain an overexposed image; The response areas in the heatmap image are overlaid to obtain the overlaid image.

5. The method according to claim 4, characterized in that, The step of performing mosaic processing on the response region in the heatmap image to obtain a mosaic image includes: Convert the heatmaps corresponding to the multiple second feature maps into mask images; The pixel points of the mask image are subjected to mean blurring to obtain the mosaic image.

6. The method according to claim 1, characterized in that, The method further includes: The heatmap images or heatmap image frames are numbered. Save the heatmap image or heatmap image frame marked with the serial number.

7. The method according to claim 1, characterized in that, The method further includes: Construct a sample dataset, which consists of several images with known target categories; A convolutional neural network is constructed, and the convolutional neural network is trained using the sample dataset to obtain the content moderation neural network model.

8. An electronic device, characterized in that, The electronic device includes: The input unit is used to input the object to be reviewed into the content review neural network model to obtain multiple first feature maps output by multiple convolutional layers; The stitching unit is used to perform image upsampling and image stitching on the multiple first feature maps respectively to obtain multiple second feature maps; the multiple second feature maps are all of different scales; The fusion unit is used to perform image fusion between the class activation map corresponding to the multiple second feature maps and the object to be reviewed, and generate a heat map image or heat map image frame corresponding to the multiple second feature maps. A shielding unit is used to shield the illegal content in the object to be reviewed using the heat map image or the heat map image frame; The stitching unit is further configured to input the multiple second feature maps into a global mean pooling layer for image stitching to obtain a third feature map; the input unit is further configured to input the third feature map into a fully connected layer for category classification to obtain a category score for the third feature map; The electronic device further includes a computing unit, configured to perform gradient-weighted class activation mapping on the class scores if the class scores meet the activation conditions, and generate class activation mapping maps corresponding to the plurality of second feature maps.

9. An electronic device, characterized in that, The electronic device includes: a processor and a memory for storing computer programs capable of running on the processor. When the processor runs the computer program, it executes any one of the image processing methods described in claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform any one of the image processing methods described in claims 1 to 7.

Images