Screenshot content detection method, device and computer-readable storage medium

Abstract

The present invention relates to a deep learning algorithm, and provides a method, device and computer-readable storage medium for detecting the content of a screenshot. The present invention first performs regional segmentation on the screenshot to be identified, so that the text part and the image part in the screenshot can be distinguished, thereby avoiding the low recognition accuracy caused by uniformly identifying the text and the image in the screenshot; by detecting and identifying the segmented text area and the image area respectively, the two different types of areas can be accurately identified in a targeted manner; the regional recognition results are automatically determined by preset audit standards, so that efficient and accurate auditing of the screenshot can be achieved. In addition, the present invention also relates to blockchain technology, and the regional recognition results after the screenshot is identified can be stored in the blockchain.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology in artificial intelligence, and in particular to a screenshot content detection method, device and computer-readable storage medium. Background Art

[0002] In today's society where social networks are highly developed, sharing on WeChat Moments has become an important way of commercial promotion. Merchants achieve their promotional purposes by asking customers to share their designated content on Moments and reach a certain number of likes. After completing the sharing on the Moments designated by the merchant, customers send corresponding screenshots through client feedback to obtain gifts from the merchant, which generates a large number of user feedback screenshots. The existing review methods for feedback screenshots are manual review and deep learning detection and recognition. If the review is conducted by manual review, the number of feedback screenshots is often very large in actual situations, and there are many screenshots that do not meet the merchant's preset standards. The workload of manually reviewing them is large, so the accuracy is unstable, which leads to the technical problem of low accuracy of the existing screenshot review method. Summary of the invention

[0003] The main purpose of the present invention is to provide a screenshot content detection method, device and computer-readable storage medium, aiming to solve the technical problem of low accuracy of existing screenshot review methods.

[0004] To achieve the above object, the present invention provides a screenshot content detection method, which comprises the following steps:

[0005] Receiving a screenshot to be identified sent by a client, performing text-image segmentation on the screenshot to be identified, and generating a text area picture set and an image area picture set;

[0006] Based on a preset deep learning algorithm and a preset target graphic estimation algorithm, the text area picture set and the image area picture set are respectively detected and identified to obtain a region recognition result after the text area picture set and the image area picture set are summarized;

[0007] Based on the preset review standard and the area recognition result, the review result corresponding to the screenshot to be recognized is determined.

[0008] Optionally, the text area picture set includes a time text area and a content text area.

[0009] The steps of receiving the screenshot to be identified sent by the client, performing text-image segmentation on the screenshot to be identified, and generating a text area picture set and an image area picture set include:

[0010] Receiving a screenshot to be identified sent by a client, and determining a time text area in the screenshot to be identified based on a preset multi-scale template matching algorithm;

[0011] Detect pixel value features of the screenshot to be identified, and determine the content text area and image area in the screenshot to be identified according to the pixel value features to obtain the text area picture set and the image area picture set.

[0012] Optionally, the step of receiving a screenshot to be identified sent by a client, and determining the time text area in the screenshot to be identified based on a preset multi-scale template matching algorithm includes:

[0013] Obtaining the image resolution of the screenshot to be identified, and scaling the size of a preset matching template according to the image resolution to adapt the matching template to the screenshot to be identified;

[0014] The matching template is used to perform local mask matching on the screenshot to be identified, and the time column coordinates in the screenshot to be identified are located, so that the area corresponding to the time column coordinates is used as the time text area.

[0015] Optionally, the image area picture set includes a content picture area and a like picture area.

[0016] The step of detecting the pixel value features of the screenshot to be identified, and determining the content text area and image area in the screenshot to be identified according to the pixel value features to obtain the text area picture set and the image area picture set comprises:

[0017] Detecting and obtaining the pixel value distribution of the screenshot to be identified, the difference between the extreme values ​​of the row and column pixels and the average values ​​of the pixels in the adjacent rows, as the pixel value features;

[0018] Determining a content image area in the to-be-identified screenshot according to the pixel value distribution;

[0019] Determining a text area of ​​the content in the screenshot to be identified according to the row and column pixel extreme values;

[0020] According to the difference in average values ​​of pixels in adjacent rows, the like picture area in the screenshot to be identified is determined to obtain a text area picture set including a time text area and a content text area and an image area picture set including a content picture area and a like picture area.

[0021] Optionally, the step of detecting and identifying the text area picture set and the image area picture set based on a preset deep learning algorithm and a preset target graphic estimation algorithm, and obtaining a region recognition result after the text area picture set and the image area picture set are aggregated includes:

[0022] According to a preset text recognition model based on a deep learning algorithm, the text area picture set is recognized to obtain a corresponding text recognition result;

[0023] According to a preset target graphic estimation algorithm, the image area picture set is identified to obtain an image recognition result corresponding to the image area picture set;

[0024] The text recognition result and the image recognition result are aggregated to generate the region recognition result, wherein the region recognition result is stored in the blockchain.

[0025] Optionally, the target image estimation algorithm includes an avatar frame size estimation algorithm, and the image recognition result includes a statistical number of likes.

[0026] The step of identifying the image area picture set according to a preset target graphic estimation algorithm to obtain an image recognition result corresponding to the image area picture set includes:

[0027] Locating the avatar frame area in the image area picture set, and dividing the avatar frame area by row according to the avatar frame size estimation algorithm to obtain an avatar frame set;

[0028] The preset avatar frame interval is used to count the number of avatar frames in the avatar frame set to serve as the number of likes.

[0029] Optionally, the audit result includes a first audit result and a second audit result.

[0030] The step of determining the audit result corresponding to the screenshot to be identified based on the preset audit standard and the area recognition result includes:

[0031] Determine whether the regional identification result meets the preset review standard;

[0032] If the area recognition result meets the preset audit standard, then generate audit pass information as the first audit result;

[0033] If the area recognition result does not meet the preset audit standard, error information in the audit process is collected, and audit failure information is generated based on the error information as the second audit result.

[0034] Optionally, after the step of determining the audit result corresponding to the screenshot to be identified based on the preset audit standard and the area recognition result, the step further includes:

[0035] Extracting label information from the region recognition result and obtaining user information of the client;

[0036] A user portrait model is established based on the tag information and the user information, so as to utilize the user portrait model to select targeted push content for the client.

[0037] In addition, to achieve the above-mentioned purpose, the present invention further provides a screenshot content detection device, the screenshot content detection device comprising:

[0038] The text image segmentation module is used to receive the screenshot to be identified sent by the client, perform text image segmentation on the screenshot to be identified, and generate a text area picture set and an image area picture set;

[0039] A region detection and recognition module is used to detect and recognize the text region picture set and the image region picture set respectively based on a preset deep learning algorithm and a preset target graphic estimation algorithm, and obtain a region recognition result after the text region picture set and the image region picture set are summarized;

[0040] The recognition result review module is used to determine the review result corresponding to the screenshot to be recognized based on a preset review standard and the area recognition result.

[0041] Optionally, the text area picture set includes a time text area and a content text area.

[0042] The text image segmentation module comprises:

[0043] The template matching unit is used to receive the screenshot to be identified sent by the client, and determine the time text area in the screenshot to be identified based on a preset multi-scale template matching algorithm;

[0044] The feature recognition unit is used to detect the pixel value features of the screenshot to be recognized, and determine the content text area and image area in the screenshot to be recognized according to the pixel value features to obtain the text area picture set and the image area picture set.

[0045] Optionally, the template matching unit is further used for:

[0046] Obtaining the image resolution of the screenshot to be identified, and scaling the size of a preset matching template according to the image resolution to adapt the matching template to the screenshot to be identified;

[0047] The matching template is used to perform local mask matching on the screenshot to be identified, and the time column coordinates in the screenshot to be identified are located, so that the area corresponding to the time column coordinates is used as the time text area.

[0048] Optionally, the image area picture set includes a content picture area and a like picture area.

[0049] The feature recognition unit is also used for:

[0050] Detecting and obtaining the pixel value distribution of the screenshot to be identified, the difference between the extreme values ​​of the row and column pixels and the average values ​​of the pixels in the adjacent rows, as the pixel value features;

[0051] Determining a content image area in the to-be-identified screenshot according to the pixel value distribution;

[0052] Determining a text area of ​​the content in the screenshot to be identified according to the row and column pixel extreme values;

[0053] According to the difference in average values ​​of pixels in adjacent rows, the like picture area in the screenshot to be identified is determined to obtain a text area picture set including a time text area and a content text area and an image area picture set including a content picture area and a like picture area.

[0054] Optionally, the area detection and identification module includes:

[0055] A text recognition unit, used to recognize the text area picture set according to a preset text recognition model based on a deep learning algorithm to obtain a corresponding text recognition result;

[0056] An image recognition unit, used to recognize the image area picture set according to a preset target graphic estimation algorithm to obtain an image recognition result corresponding to the image area picture set;

[0057] A result aggregation unit is used to aggregate the text recognition result and the image recognition result to generate the area recognition result, wherein the area recognition result is stored in the blockchain.

[0058] Optionally, the target image estimation algorithm includes an avatar frame size estimation algorithm, and the image recognition result includes a statistical number of likes.

[0059] The image recognition unit is also used for:

[0060] Locating the avatar frame area in the image area picture set, and dividing the avatar frame area by row according to the avatar frame size estimation algorithm to obtain an avatar frame set;

[0061] The preset avatar frame interval is used to count the number of avatar frames in the avatar frame set to serve as the number of likes.

[0062] Optionally, the audit result includes a first audit result and a second audit result.

[0063] The recognition result review module includes:

[0064] A standard judgment unit, used to judge whether the area recognition result meets the preset review standard;

[0065] A first review unit, configured to generate review pass information as a first review result if the area recognition result meets a preset review standard;

[0066] The second audit unit is used to collect error information in the audit process if the area recognition result does not meet the preset audit standard, and generate audit failure information based on the error information as the second audit result.

[0067] Optionally, the screenshot content detection device further includes:

[0068] A tag extraction unit, used to extract tag information from the region recognition result and obtain user information of the client;

[0069] The directional push unit is used to establish a user portrait model based on the tag information and the user information, so as to use the user portrait model to select directional push content for the client.

[0070] In addition, to achieve the above-mentioned purpose, the present invention also provides an electronic device, which includes a processor, a memory, and a screenshot content detection program stored in the memory and executable by the processor, wherein when the screenshot content detection program is executed by the processor, the steps of the screenshot content detection method as described above are implemented.

[0071] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a screenshot content detection program is stored, wherein when the screenshot content detection program is executed by a processor, the steps of the screenshot content detection method as described above are implemented.

[0072] The present invention provides a screenshot content detection method, device and computer-readable storage medium. The screenshot content detection method first performs regional segmentation on a screenshot to be identified, so that the text part and the image part in the screenshot can be distinguished, thereby avoiding the low recognition accuracy caused by uniformly identifying the text and the image in the screenshot; by detecting and identifying the segmented text area and image area respectively, the two different types of areas can be accurately identified in a targeted manner; the regional recognition result is automatically determined according to a preset review standard, so that efficient and accurate review of the screenshot can be achieved, thereby solving the technical problem of low accuracy of the existing screenshot review method. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 A schematic diagram of the hardware structure of an electronic device involved in an embodiment of the present invention;

[0074] Figure 2 It is a flowchart of the first embodiment of the screenshot content detection method of the present invention;

[0075] Figure 3 It is a schematic diagram of the functional modules of the screenshot content detection device of the present invention.

[0076] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0077] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0078] The screenshot content detection method involved in the embodiment of the present invention is mainly applied to electronic devices, which may be devices with display and processing functions such as PCs, portable computers, and mobile terminals.

[0079] Reference Figure 1 , Figure 1 Schematic diagram of the hardware structure of the electronic device involved in the embodiment of the present invention. In the embodiment of the present invention, the electronic device may include a processor 1001 (such as a CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components; the user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard); the network interface 1004 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface); the memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory, and the memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.

[0080] Those skilled in the art will understand that Figure 1 The hardware structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0081] Continue to refer to Figure 1 , Figure 1 The memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, and a screenshot content detection program.

[0082] exist Figure 1 In the embodiment, the network communication module is mainly used to connect to the server and perform data communication with the server; and the processor 1001 can call the screenshot content detection program stored in the memory 1005 and execute the screenshot content detection method provided in the embodiment of the present invention.

[0083] Based on the above hardware structure, various embodiments of the screenshot content detection method of the present invention are proposed.

[0084] In today's society where social networks are highly developed, sharing on WeChat Moments has become an important way of commercial promotion. Merchants achieve their promotional purposes by asking customers to share their designated content on Moments and reach a certain number of likes. After completing the sharing on the Moments designated by the merchant, customers send corresponding screenshots through client feedback to obtain gifts from the merchant, which generates a large number of user feedback screenshots. The existing review methods for feedback screenshots are manual review and deep learning detection and recognition. If the review is conducted by manual review, the number of feedback screenshots is often very large in actual situations, and there are many screenshots that do not meet the merchant's preset standards. The workload of manually reviewing them is large, so the accuracy is unstable, which leads to the technical problem of low accuracy of the existing screenshot review method.

[0085] To solve the above problems, the present invention provides a screenshot content detection method, namely, by first performing regional segmentation on the screenshot to be identified, so that the text part and the image part in the screenshot can be distinguished, thereby avoiding the low recognition accuracy caused by uniformly identifying the text and the image in the screenshot; by detecting and identifying the segmented text area and image area respectively, it is possible to accurately identify these two different types of areas in a targeted manner; the regional recognition results are automatically judged according to preset review standards, so that efficient and accurate review of the screenshot can be achieved, thereby solving the technical problem of low accuracy of the existing screenshot review method.

[0086] Reference Figure 2 , Figure 2 Schematic diagram of the process of the first embodiment of the screenshot content detection method of the present invention.

[0087] A first embodiment of the present invention provides a screenshot content detection method, the screenshot content detection method comprising the following steps:

[0088] Step S10, receiving a screenshot to be identified sent by a client, performing text-image segmentation on the screenshot to be identified, and generating a text area picture set and an image area picture set;

[0089] In this embodiment, the screenshot to be identified is a screenshot sent by the user to the platform through the client, which may be a screenshot of WeChat Moments or other webpage screenshots containing text and image content. The client may be a terminal device such as a mobile phone, a tablet, or a computer. The text area picture set may contain one or more text area images containing only text content, and similarly, the image area picture set may contain one or more image area pictures containing only image content. For example, if the current user takes a screenshot of WeChat Moments on a mobile phone, and sends the screenshot from the mobile phone to the screenshot content detection system, the system will use template matching and other methods to perform text and image segmentation operations on the screenshot after receiving the WeChat Moments screenshot to be identified sent by the current user, so as to separate the text area from the image area in the screenshot, and generate a text area picture set containing only text content, and an image area picture set containing only image content.

[0090] Step S20, based on a preset deep learning algorithm and a preset target graphic estimation algorithm, respectively detecting and identifying the text area picture set and the image area picture set, to obtain a region recognition result after the text area picture set and the image area picture set are aggregated;

[0091] In this embodiment, the preset deep learning algorithm may specifically be a convolutional neural network, a recurrent neural network, etc. The preset graphic estimation algorithm is an algorithm for locating certain specific graphics in the screenshot, such as the avatar frame, the time bar, etc. in the WeChat Moments screenshot. The area recognition result is a final recognition result that summarizes a type of recognition result of the system for the text area picture set and a type of recognition result of the image area picture set, that is, the recognition result corresponding to the entire screenshot to be recognized. Specifically, the system recognizes the text area picture set of the WeChat Moments screenshot through a deep learning-based text recognition model, and recognizes the image area picture set of the WeChat Moments screenshot through a target graphic estimation algorithm, and obtains the text recognition results corresponding to the text area picture set, and summarizes the image recognition results corresponding to the image area picture set to obtain the final recognition result corresponding to the entire WeChat Moments screenshot, that is, the above-mentioned area recognition result.

[0092] Step S30, determining the review result corresponding to the screenshot to be identified based on the preset review standard and the area recognition result.

[0093] In this embodiment, the preset audit standard is used to determine whether the recognition result corresponding to the screenshot meets the requirements of the screenshot activity initiator. For example, for WeChat Moments screenshots, the corresponding audit standards may be that the number of likes exceeds the preset threshold, the image content in the screenshot is activity-related content, and the time information in the screenshot is within the effective time limit of the activity. The audit result is the result after the regional recognition result is judged by the audit standard, which is usually divided into two categories, one is that the audit is passed and the screenshot meets the requirements of the activity initiator; the other is that the audit is not passed and the screenshot does not meet the requirements of the activity initiator. Specifically, the system uses the audit standard sent in advance by the activity initiator to judge the currently generated recognition result corresponding to the WeChat screenshot, judge whether it meets the standard, and send the audit result that meets the standard or does not meet the standard. In addition, after the system obtains the audit results, it can also collect these audit results, analyze the audit results of different clients separately, and obtain the user characteristics of each client user, so as to formulate corresponding push strategies for different characteristics of MoYi users.

[0094] In this embodiment, the present invention receives a screenshot to be identified sent by a client, performs text-image segmentation on the screenshot to be identified, generates a text area picture set and an image area picture set; based on a preset deep learning algorithm and a preset target graphic estimation algorithm, detects and identifies the text area picture set and the image area picture set respectively, and obtains the area identification result after the text area picture set and the image area picture set are summarized; based on the preset audit standard and the area identification result, determines the audit result corresponding to the screenshot to be identified. In the above manner, the present invention first performs area segmentation on the screenshot to be identified, so that the text part and the image part in the screenshot can be distinguished, thereby avoiding the low recognition accuracy caused by uniformly identifying the text and the image in the screenshot; by detecting and identifying the segmented text area and the image area respectively, the two different types of areas can be accurately identified in a targeted manner; the area identification result is automatically determined by the preset audit standard, so that the screenshot can be efficiently and accurately audited, thereby solving the technical problem of low accuracy of the existing screenshot audit method.

[0095] Furthermore, based on the above Figure 2 The first embodiment shown in the figure proposes a second embodiment of the screenshot content detection method of the present invention. In this embodiment, the text area picture set includes a time text area and a content text area, and step S10 includes:

[0096] Receiving a screenshot to be identified sent by a client, and determining a time text area in the screenshot to be identified based on a preset multi-scale template matching algorithm;

[0097] In this embodiment, the preset multi-scale matching algorithm is preferably a multi-scale matching algorithm to be masked. The time text area refers to the area showing time information in the screenshot to be identified.

[0098] Further, the receiving of the screenshot to be identified sent by the client, and determining the time text area in the screenshot to be identified based on a preset multi-scale template matching algorithm includes:

[0099] Obtaining the image resolution of the screenshot to be identified, and scaling the size of a preset matching template according to the image resolution to adapt the matching template to the screenshot to be identified;

[0100] The matching template is used to perform local mask matching on the screenshot to be identified, and the time column coordinates in the screenshot to be identified are located, so that the area corresponding to the time column coordinates is used as the time text area.

[0101] In this embodiment, the screenshot to be identified is a screenshot of WeChat Moments, and the time text area is the time column in the WeChat Moments screenshot. The WeChat Moments screenshot can be a normal mode screenshot or an album mode screenshot. The system locates the time column coordinates of the Moments screenshot through template matching based on opencv (a cross-platform computer vision library). The matching method used is square difference matching, that is, finding: R(x, y) = ∑ x′,y′ (T(x′,y′)-I(x+x′,y+y′)) 2 . Where T represents the template image, I represents the original image, (x, y) represents the coordinates of the template image corresponding to the pixel matrix, and (x′, y′) represents the coordinates of the original image corresponding to the pixel matrix. Since the screenshot sizes of different mobile phones are different, the resulting screenshot resolutions are also different. On the one hand, the image height is unified to make the image resolution as consistent as possible, and on the other hand, the template is scaled by multiple factors [0.8, 0.9, 1.1, 1.2] to match images with different resolutions. When matching, a local Mask matching method is used. For example, the key position of the time bar is on the right side of the image, so the horizontal right 1 / 4 area of ​​the entire image is masked for template matching. In this way, the matching calculation amount is only 1 / 4 of the entire image matching. If the system locates only one time bar in the Moments screenshot, the time bar position is that position; if multiple time bars are located, the business logic can determine that the time bar near 2 / 3 of the height of the entire image is the target time bar. Once the time bar is determined, the upper and lower parts of the Moments can be divided.

[0102] Detect pixel value features of the screenshot to be identified, and determine the content text area and image area in the screenshot to be identified according to the pixel value features to obtain the text area picture set and the image area picture set.

[0103] Furthermore, the step of detecting the pixel value features of the screenshot to be identified, and determining the content text area and image area in the screenshot to be identified according to the pixel value features to obtain the text area picture set and the image area picture set includes:

[0104] Detecting and obtaining the pixel value distribution of the screenshot to be identified, the difference between the extreme values ​​of the row and column pixels and the average values ​​of the pixels in the adjacent rows, as the pixel value features;

[0105] Determining a content image area in the to-be-identified screenshot according to the pixel value distribution;

[0106] Determining a text area of ​​the content in the screenshot to be identified according to the row and column pixel extreme values;

[0107] According to the difference in average values ​​of pixels in adjacent rows, the like picture area in the screenshot to be identified is determined to obtain a text area picture set including a time text area and a content text area and an image area picture set including a content picture area and a like picture area.

[0108] In this embodiment, the pixel value features include: the distribution of pixel values ​​in the image, the difference between the maximum and minimum values ​​of the pixels in a row or column and the average value of the pixels in the adjacent rows. Taking the screenshot to be identified as a screenshot of WeChat Moments as an example, since the content image area in the screenshot has the characteristic of wide pixel distribution, the system can detect and obtain the pixel value features of the screenshot, and locate the boundary coordinates of the content image area according to the widely distributed pixel values; since the content text area in the screenshot has the characteristic of large pixel difference between black text and white background, the system can obtain the pixel values ​​of each pixel in the screenshot and calculate the difference between the pixel values, so as to locate the boundary coordinates of the content text area where there is a large difference; since the like area of ​​the WeChat Moments screenshot is distributed in the gray background board, the system can detect and obtain the average difference of the pixels in the adjacent rows of the screenshot to locate the boundary coordinates of the like area. By detecting and acquiring these pixel value features, the system can locate the boundary coordinates of the content text area, content image area and like area, and then determine the content text area, content image area and like area. Combined with the above-mentioned time text area, a text area picture set containing the time text area and content text area and an image area picture set containing the content picture area and like picture area can be obtained.

[0109] Furthermore, by adjusting the size of the matching template according to the actual image resolution, the matching template can be adapted to the screenshot to be identified; by performing local mask matching on the screenshot, the time text area in the screenshot can be more efficiently located; by detecting various pixel value features in the screenshot, the content picture area, content text area and like picture area in the screenshot can be located according to the features.

[0110] Furthermore, based on the above Figure 2 The first embodiment shown here is a third embodiment of the screenshot content detection method of the present invention. In this embodiment, step S20 includes:

[0111] According to a preset text recognition model based on a deep learning algorithm, the text area picture set is recognized to obtain a corresponding text recognition result;

[0112] In this embodiment, the time text area in the text area picture set and whether there is a partially visible icon are taken as an example. The system detects whether there is a partially visible icon in the time area in the area picture set through template matching, and inputs the time bar area into the preset text recognition model that has been trained by deep learning to obtain time information, that is, the time when the circle of friends is sent.

[0113] According to a preset target graphic estimation algorithm, the image area picture set is identified to obtain an image recognition result corresponding to the image area picture set;

[0114] Furthermore, the step of identifying the image area picture set according to a preset target graphic estimation algorithm to obtain an image recognition result corresponding to the image area picture set includes:

[0115] Locating the avatar frame area in the image area picture set, and dividing the avatar frame area by row according to the avatar frame size estimation algorithm to obtain an avatar frame set;

[0116] The preset avatar frame interval is used to count the number of avatar frames in the avatar frame set to serve as the number of likes.

[0117] In this embodiment, the target graphic estimation algorithm is the avatar frame estimation algorithm, and the likes counting method is the avatar counting method. This likes counting method is applicable to the case where the proportion of text color relative to the gray background is low according to the name text color of the like area, that is, the screenshot to be identified is a WeChat Moments screenshot in the ordinary screenshot mode. Because the avatar has a random personalized feature, it is impossible to use a target detection method. If a rectangular frame is found, there is a situation where the color of a part of the avatar is gray and merges with the background. Therefore, the system can adopt an avatar frame size estimation method. The estimation method is to divide the avatar area into rows. The segmentation condition is the gray and white lines between the avatar rows. According to the square characteristics of the avatar, the height of each row of avatars is the height of the avatar. The obvious advantage of this estimation method is that multiple avatars in a row will not have color abnormalities at the same time, and the abnormality of a single avatar will not affect the height estimation of the row. With the avatar size, the avatar interval size can also be calculated by the empirical value (1 / 10 of the avatar), and then the number of avatars N that are liked is calculated based on the length of each row of avatars. The formula is: Where n represents the total number of rows, Li It represents the length of the avatar in the i-th row, W represents the avatar width, S represents the avatar interval size, and cell represents rounding up.

[0118] In addition, when the percentage of text color relative to the gray background is high according to the name text color statistics in the like area, that is, the screenshot to be recognized is a WeChat Moments screenshot in album mode. The system can input the name part of the like area into the text detection model, and input the obtained text segmentation result into the text recognition model to obtain the actual text information, and count the commas in the actual text information as the number of likes.

[0119] The text recognition result and the image recognition result are aggregated to generate the region recognition result, wherein the region recognition result is stored in the blockchain.

[0120] In this embodiment, after obtaining the text recognition results of the text area picture set and the image recognition results of the image area picture set, the system summarizes them to obtain the final area recognition result. For example, for the WeChat Moments screenshot, the system obtains the text recognition result that the release time is July 15, 2020, the Moments is visible to some people, and the image recognition result is that the number of like avatar frames is 10. The system summarizes this information as the area recognition result of the Moments screenshot. It should be emphasized that in order to further ensure the privacy and security of the above-mentioned area recognition results, the above-mentioned area recognition results are stored in a node of a blockchain.

[0121] Further, in this embodiment, the audit result includes a first audit result and a second audit result, and step S30 includes:

[0122] Determine whether the regional identification result meets the preset review standard;

[0123] If the area recognition result meets the preset audit standard, then generate audit pass information as the first audit result;

[0124] If the area recognition result does not meet the preset audit standard, error information in the audit process is collected, and audit failure information is generated based on the error information as the second audit result.

[0125] In this embodiment, the system's audit results for the regional identification results are generally divided into two categories, one is a first audit result representing a passed audit, and the other is a second audit result representing a failed audit. It should be noted that for the second audit result that failed the audit, it is necessary to collect error information about which specific conditions failed in the standard determination process, and summarize this information as the second audit result, so as to analyze the specific reasons for the failed audit.

[0126] Furthermore, in this embodiment, after the step of determining the audit result corresponding to the screenshot to be identified based on the preset audit standard and the region recognition result, the method further includes:

[0127] Extracting label information from the region recognition result and obtaining user information of the client;

[0128] A user portrait model is established based on the tag information and the user information, so as to utilize the user portrait model to select targeted push content for the client.

[0129] In this embodiment, the tag information may be location information, activity type information, etc., and the user information may be gender information, age information, etc. The system may use the screenshots of the circle of friends fed back by the user together with the corresponding user information as a training data set, and adopt a supervised learning method, such as the classification and regression algorithm of machine learning (Bayes, decision tree, logistic regression, support vector machine, etc.), to learn a user portrait model for prediction from the labeled training data. The construction of this user portrait model can be completed through three steps: target analysis, system construction and portrait establishment. The first step is target analysis: all user-related data is divided into static information data and dynamic information data. Static information data is relatively stable information of users, such as demographic attributes, business attributes and other aspects. Dynamic information data is constantly changing behavioral information, which needs to be acquired and analyzed in detail; the second step is system construction. The current mainstream labeling systems are hierarchical. First, the labels are divided into several major categories, and each major category is subdivided layer by layer. When constructing labels, you only need to build the lowest level label to map to the upper two levels of labels. The bottom-level labels are generally used for advertising and precision marketing. In addition, the granularity of the labels should also be noted. If the granularity of the labels is too coarse, there will be no differentiation. If the granularity is too fine, the label system will be too complex and not universal. The first thing to build based on the original data is the fact label, which can be directly obtained from the database (such as registration information) or obtained through simple statistics. This type of label is easy to build and has a clear practical meaning. Some labels can be used as basic features for subsequent label mining (such as the number of product purchases can be used as input feature data for user shopping preferences). The construction process of fact labels is also a process of deepening the understanding of data. While statistics are being taken on the data, not only the data processing and processing are completed, but also a certain understanding of the distribution of the data is obtained, which is ready for the construction of advanced labels. The model label is the core of the label system and the part with the largest workload in user portraits. The core of most user labels is the model label. The construction of model labels requires the use of machine learning and natural language processing technologies. The last thing to construct is the advanced label, which is obtained by statistical modeling based on fact labels and model labels. Its construction is often closely related to actual business indicators. Only after the construction of basic labels is completed can advanced labels be constructed. The model used to build advanced tags can be simple data statistics or complex machine learning models. According to the constructed tag system, the tags corresponding to the user dynamic information data are extracted; the third step is to establish a portrait, and establish a user portrait based on the tags corresponding to the user data. The user portrait model can be a user value model, a customer activity model, a user loyalty model, and a user shopping type model. The commonly used modeling algorithm is usually a clustering algorithm, which can be a K-means clustering algorithm and a system clustering algorithm.

[0130] Merchants can use the user portrait model to predict the types of promotional information that similar client users prefer to share, and then push promotional information of interest to these client users in subsequent push plans to increase the probability that users share this promotional information. Specifically, if the system obtains through the trained user portrait model that the promotional information sharing tendency of the female client user group aged 18-30 is beauty and food, then these two types of promotional information can be pushed to new client users of the same type in the WeChat public account to increase the probability that the promotional information can be shared by users.

[0131] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0132] Furthermore, by adopting a text recognition model based on a deep learning algorithm and a preset target graphic estimation algorithm respectively, the text area picture set and the image area picture set are recognized respectively, so that the screenshot recognition operation is more targeted and the recognition result is more accurate; by adopting an avatar frame size estimation algorithm to count the avatar frames in the like area, the number of likes reflected in the screenshot can be accurately obtained; the area recognition results are reviewed by preset review standards, so that the review results can be obtained automatically and efficiently; by extracting the label information in the area recognition results to establish the corresponding user portrait model, the collected user-related data can be fully utilized to specify push strategies suitable for different users.

[0133] In addition, if Figure 3 As shown, in order to achieve the above-mentioned purpose, the present invention also provides a screenshot content detection device, the screenshot content detection device comprising:

[0134] The text image segmentation module 10 is used to receive the screenshot to be identified sent by the client, perform text image segmentation on the screenshot to be identified, and generate a text area picture set and an image area picture set;

[0135] A region detection and recognition module 20 is used to detect and recognize the text region picture set and the image region picture set respectively based on a preset deep learning algorithm and a preset target graphic estimation algorithm, and obtain a region recognition result after the text region picture set and the image region picture set are summarized;

[0136] The recognition result review module 30 is used to determine the review result corresponding to the screenshot to be recognized based on a preset review standard and the area recognition result.

[0137] Optionally, the text area picture set includes a time text area and a content text area.

[0138] The text image segmentation module 10 comprises:

[0139] The template matching unit is used to receive the screenshot to be identified sent by the client, and determine the time text area in the screenshot to be identified based on a preset multi-scale template matching algorithm;

[0140] The feature recognition unit is used to detect the pixel value features of the screenshot to be recognized, and determine the content text area and image area in the screenshot to be recognized according to the pixel value features to obtain the text area picture set and the image area picture set.

[0141] Optionally, the template matching unit is further used for:

[0142] Obtaining the image resolution of the screenshot to be identified, and scaling the size of a preset matching template according to the image resolution to adapt the matching template to the screenshot to be identified;

[0143] The matching template is used to perform local mask matching on the screenshot to be identified, and the time column coordinates in the screenshot to be identified are located, so that the area corresponding to the time column coordinates is used as the time text area.

[0144] Optionally, the image area picture set includes a content picture area and a like picture area.

[0145] The feature recognition unit is also used for:

[0146] Detecting and obtaining the pixel value distribution of the screenshot to be identified, the difference between the extreme values ​​of the row and column pixels and the average values ​​of the pixels in the adjacent rows, as the pixel value features;

[0147] Determining a content image area in the to-be-identified screenshot according to the pixel value distribution;

[0148] Determining a text area of ​​the content in the screenshot to be identified according to the row and column pixel extreme values;

[0149] According to the difference in average values ​​of pixels in adjacent rows, the like picture area in the screenshot to be identified is determined to obtain a text area picture set including a time text area and a content text area and an image area picture set including a content picture area and a like picture area.

[0150] Optionally, the region detection and identification module 20 includes:

[0151] A text recognition unit, used to recognize the text area picture set according to a preset text recognition model based on a deep learning algorithm to obtain a corresponding text recognition result;

[0152] An image recognition unit, used to recognize the image area picture set according to a preset target graphic estimation algorithm to obtain an image recognition result corresponding to the image area picture set;

[0153] A result aggregation unit is used to aggregate the text recognition result and the image recognition result to generate the area recognition result, wherein the area recognition result is stored in the blockchain.

[0154] Optionally, the target image estimation algorithm includes an avatar frame size estimation algorithm, and the image recognition result includes a statistical number of likes.

[0155] The image recognition unit is also used for:

[0156] Locating the avatar frame area in the image area picture set, and dividing the avatar frame area by row according to the avatar frame size estimation algorithm to obtain an avatar frame set;

[0157] The preset avatar frame interval is used to count the number of avatar frames in the avatar frame set to serve as the number of likes.

[0158] Optionally, the audit result includes a first audit result and a second audit result.

[0159] The recognition result review module 30 includes:

[0160] A standard judgment unit, used to judge whether the area recognition result meets the preset review standard;

[0161] A first review unit, configured to generate review pass information as a first review result if the area recognition result meets a preset review standard;

[0162] The second audit unit is used to collect error information in the audit process if the area recognition result does not meet the preset audit standard, and generate audit failure information based on the error information as the second audit result.

[0163] Optionally, the screenshot content detection device further includes:

[0164] A tag extraction unit, used to extract tag information from the region recognition result and obtain user information of the client;

[0165] The directional push unit is used to establish a user portrait model based on the tag information and the user information, so as to use the user portrait model to select directional push content for the client.

[0166] The invention also provides an electronic device.

[0167] The electronic device includes a processor, a memory, and a screenshot content detection program stored in the memory and executable on the processor, wherein when the screenshot content detection program is executed by the processor, the steps of the screenshot content detection method described above are implemented.

[0168] Among them, the method implemented when the screenshot content detection program is executed can refer to the various embodiments of the screenshot content detection method of the present invention, and will not be repeated here.

[0169] In addition, an embodiment of the present invention further provides a computer-readable storage medium.

[0170] The computer-readable storage medium of the present invention stores a screenshot content detection program, wherein when the screenshot content detection program is executed by a processor, the steps of the screenshot content detection method as described above are implemented.

[0171] Among them, the method implemented when the screenshot content detection program is executed can refer to the various embodiments of the screenshot content detection method of the present invention, and will not be repeated here.

[0172] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0173] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0174] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0175] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A screenshot content detection method, characterized in that: The screenshot content detection method comprises the following steps: Receiving a screenshot to be identified sent by a client, performing text-image segmentation on the screenshot to be identified, and generating a text area picture set and an image area picture set; Based on a preset deep learning algorithm and a preset target graphic estimation algorithm, the text area picture set and the image area picture set are respectively detected and identified to obtain a region recognition result after the text area picture set and the image area picture set are summarized; Determine the audit result corresponding to the screenshot to be identified based on the preset audit standard and the area recognition result; The text area picture set includes a time text area and a content text area. The steps of receiving the screenshot to be identified sent by the client, performing text-image segmentation on the screenshot to be identified, and generating a text area picture set and an image area picture set include: Receiving a screenshot to be identified sent by a client, and determining a time text area in the screenshot to be identified based on a preset multi-scale template matching algorithm; Detecting pixel value features of the screenshot to be identified, and determining a content text area and an image area in the screenshot to be identified according to the pixel value features, so as to obtain the text area picture set and the image area picture set; The image area picture set includes a content picture area and a like picture area, The step of detecting the pixel value features of the screenshot to be identified, and determining the content text area and image area in the screenshot to be identified according to the pixel value features to obtain the text area picture set and the image area picture set comprises: Detecting and obtaining the pixel value distribution of the screenshot to be identified, the difference between the extreme values ​​of the row and column pixels and the average values ​​of the pixels in the adjacent rows, as the pixel value features; Determining a content image area in the to-be-identified screenshot according to the pixel value distribution; Determining a text area of ​​the content in the screenshot to be identified according to the row and column pixel extreme values; Determine the like picture area in the to-be-identified screenshot according to the average value difference of the pixels in the adjacent rows, so as to obtain a text area picture set including a time text area and a content text area and an image area picture set including a content picture area and a like picture area; By detecting and obtaining the pixel value features of the screenshot, and locating the boundary coordinates of the content image area based on the widely distributed pixel values; since the content text area in the screenshot has the characteristic of large pixel difference between black text and white background, the system can obtain the pixel values ​​of each pixel in the screenshot and calculate the difference between the pixel values, so as to locate the boundary coordinates of the content text area in the area where there is a large difference.

2. The screenshot content detection method according to claim 1, characterized in that: The step of receiving the screenshot to be identified sent by the client and determining the time text area in the screenshot to be identified based on a preset multi-scale template matching algorithm comprises: Obtaining the image resolution of the screenshot to be identified, and scaling the size of a preset matching template according to the image resolution to adapt the matching template to the screenshot to be identified; The matching template is used to perform local mask matching on the screenshot to be identified, and the time column coordinates in the screenshot to be identified are located, so that the area corresponding to the time column coordinates is used as the time text area.

3. The screenshot content detection method according to claim 1, characterized in that: The step of detecting and identifying the text area picture set and the image area picture set based on the preset deep learning algorithm and the preset target graphic estimation algorithm, and obtaining the area recognition result after the text area picture set and the image area picture set are summarized comprises: According to a preset text recognition model based on a deep learning algorithm, the text area picture set is recognized to obtain a corresponding text recognition result; According to a preset target graphic estimation algorithm, the image area picture set is identified to obtain an image recognition result corresponding to the image area picture set; The text recognition result and the image recognition result are aggregated to generate the region recognition result, wherein the region recognition result is stored in the blockchain.

4. The screenshot content detection method according to claim 3, characterized in that: The target image estimation algorithm includes an avatar frame size estimation algorithm, and the image recognition result includes the number of likes statistics. The step of identifying the image area picture set according to a preset target graphic estimation algorithm to obtain an image recognition result corresponding to the image area picture set includes: Locating the avatar frame area in the image area picture set, and dividing the avatar frame area by row according to the avatar frame size estimation algorithm to obtain an avatar frame set; The preset avatar frame interval is used to count the number of avatar frames in the avatar frame set to serve as the number of likes.

5. The screenshot content detection method according to claim 1, characterized in that: The audit results include a first audit result and a second audit result. The step of determining the audit result corresponding to the screenshot to be identified based on the preset audit standard and the area recognition result includes: Determine whether the regional identification result meets the preset review standard; If the area recognition result meets the preset audit standard, then generate audit pass information as the first audit result; If the area recognition result does not meet the preset audit standard, error information in the audit process is collected, and audit failure information is generated based on the error information as the second audit result.

6. The screenshot content detection method according to any one of claims 1 to 5, characterized in that: After the step of determining the audit result corresponding to the screenshot to be identified based on the preset audit standard and the area recognition result, the method further includes: Extracting label information from the region recognition result and obtaining user information of the client; A user portrait model is established based on the tag information and the user information, so as to utilize the user portrait model to select targeted push content for the client.

7. An electronic device, characterized in that: The electronic device includes a processor, a memory, and a screenshot content detection program stored in the memory and executable by the processor, wherein when the screenshot content detection program is executed by the processor, the steps of the screenshot content detection method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a screenshot content detection program, wherein when the screenshot content detection program is executed by the processor, the steps of the screenshot content detection method according to any one of claims 1 to 6 are implemented.

Images