Image recognition method, device, storage medium and computer equipment

Abstract

The embodiments of the present disclosure relate to the field of computer technology, and more specifically, to image recognition methods, devices, storage media, and computer equipment. The image recognition method includes: obtaining a picture area containing text from the image to be recognized; performing feature extraction on the picture area to obtain picture features of the corresponding picture area; determining the similarity between the picture features and the features of illegal pictures, wherein the illegal picture features are taken from a library of illegal picture features; in response to the similarity being greater than a first threshold, determining the image to be recognized as an illegal image. The illegal image recognition scheme disclosed in the present disclosure can efficiently identify illegal images with abnormal text content, reduce dependence on image semantic information, and can cope with a variety of text types to achieve accurate and comprehensive recognition of illegal images.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and more specifically, to an image recognition method, apparatus, storage medium, and computer equipment. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims, and no statement herein is admitted to be prior art by inclusion in this section.

[0003] Currently, illegal image identification primarily relies on image classification or text recognition algorithms. Image classification algorithms determine whether an image contains illegal content based on its semantic information and are unable to process images with unusual text content. Text recognition algorithms convert text in images into text to determine whether it contains illegal content and are unable to handle text types that attackers can deliberately synthesize to attack text recognition algorithms. Summary of the Invention

[0004] The embodiments of the present disclosure provide an image recognition method, apparatus, storage medium, and computer device that can efficiently identify illegal images with abnormal text content, reduce dependence on image semantic information, and cope with a variety of text types to achieve accurate and comprehensive recognition of illegal images.

[0005] According to one aspect of the present disclosure, there is provided an image recognition method, comprising: obtaining a picture region containing text from an image to be recognized; performing feature extraction on the picture region to obtain picture features of the corresponding picture region; determining a similarity between the picture features and features of a violation picture, the violation picture features being taken from a violation picture feature library; and determining the image to be recognized as a violation image in response to the similarity being greater than a first threshold.

[0006] In an exemplary embodiment of the present disclosure, obtaining a picture area containing text from an image to be recognized includes: performing text detection on the image to be recognized to determine a text box corresponding to a corresponding text sequence based on the detection result; in response to the number of text boxes being multiple and the corresponding multiple text boxes meeting a merging condition, merging the multiple text boxes to obtain a target text box, and intercepting the picture area from the image to be recognized based on the target text box; wherein the merging condition is determined based on the distance between the text boxes and / or the degree of overlap between the text boxes.

[0007] In an exemplary embodiment of the present disclosure, the step of obtaining a picture region containing text from the image to be recognized further includes: in response to there being only one text box, intercepting the picture region from the image to be recognized according to the text box.

[0008] In an exemplary embodiment of the present disclosure, merging the multiple text boxes to obtain a target text box includes: enlarging the multiple text boxes according to a preset ratio to obtain corresponding extended text boxes; and merging the extended text boxes to obtain the target text box.

[0009] In an exemplary embodiment of the present disclosure, the merging of the extended text boxes to obtain the target text box includes: determining the extended text box with the largest area as the base text box; determining the degree of overlap between the base text box and other extended text boxes, and determining a candidate text box based on the overlap result; wherein the candidate text box is an extended text box among the other extended text boxes, the degree of overlap between the base text box and the candidate text box exceeding a second threshold; merging the base text box with the candidate text box to obtain the target text box; and repeating the above steps for the remaining extended text boxes until no new target text box can be obtained in this way.

[0010] In an exemplary embodiment of the present disclosure, the merging of the extended text boxes to obtain the target text box includes: determining an extended text box whose area is greater than a third threshold as a reference text box; determining the degree of overlap between each reference text box and each associated extended text box, and determining a candidate text box for each reference text box based on the overlap result; wherein, an extended text box associated with a reference text box is an extended text box whose distance from the reference text box is less than a fourth threshold; a candidate text box for the reference text box is an extended text box among the extended text boxes associated with the reference text box, whose degree of overlap with the reference text box exceeds a second threshold; merging each reference text box with its candidate text boxes to obtain the target text box; and repeating the above steps for the remaining extended text boxes until no new target text box can be obtained in this way.

[0011] In an exemplary embodiment of the present disclosure, the feature extraction of the image area includes extracting features of the image area through a feature extraction model; wherein the network structure of the feature extraction model in the application stage includes: a backbone network, which is used to extract features of the image area and obtain basic features of the image area; and a measurement network, which is connected to the backbone network and is used to perform feature processing on the basic features to obtain image features of the image area.

[0012] In an exemplary embodiment of the present disclosure, the network structure of the feature extraction model in the training phase also includes a classification network connected to the backbone network; the training phase of the feature extraction model includes: obtaining labeled samples, each labeled sample carries a metric label and / or a classification label; inputting the labeled samples into the backbone network, obtaining metric features corresponding to the metric labels output by the metric network and classification results corresponding to the classification labels output by the classification network; and optimizing the parameters of the backbone network, the metric network, and the classification network based on the total loss including the metric loss and the classification loss.

[0013] In an exemplary embodiment of the present disclosure, the metric label is a triplet label including a sample label, a positive sample label and a negative sample label. In a group of sample images carrying associated triplet labels: a current sample image carrying the sample label is randomly selected from a sample image set; a positive sample image carrying the positive sample label is obtained by performing data enhancement on the current sample image; and a negative sample image carrying the negative sample label is selected from a remaining sample image group in the sample image set except at least the current sample image.

[0014] In an exemplary embodiment of the present disclosure, during the training phase of the feature extraction model, the parameters of the backbone network and the metric network are optimized so that the distance between the metric features corresponding to the current sample image and the metric features of its positive sample image output by the metric network is reduced, and the distance between the metric features corresponding to the current sample image and the metric features of its negative sample image output by the metric network is increased.

[0015] In an exemplary embodiment of the present disclosure, the metric loss of the metric features of each group of sample images is calculated using a triplet loss function.

[0016] In an exemplary embodiment of the present disclosure, when the current sample image does not carry a similar image label, the remaining sample image group is composed of sample images in the sample image set except the current sample image; when the current sample image carries the similar image label, the positive sample image can also be randomly selected from the similar sample image group in the sample image set, and the sample images in the similar sample image group all carry the similar image label, and the remaining sample image group is composed of sample images in the sample image set except the similar sample image group.

[0017] In an exemplary embodiment of the present disclosure, the negative sample image carrying the negative sample label is selected from the remaining sample image group of the sample image set, including: extracting features of each sample image in the sample image set; calculating the distance between the features of the current sample image and the features of each remaining sample image in the remaining sample image group; and randomly selecting a remaining sample image from the remaining sample images whose distance is less than a distance threshold as the negative sample image.

[0018] In an exemplary embodiment of the present disclosure, the classification labels include: a first category label, used to characterize that the corresponding sample image comes from real business; a second category label, used to characterize that the corresponding sample image is synthesized by a machine; a third category label, used to characterize that the corresponding sample image comes from an external resource platform; a fourth category label, used to characterize that the corresponding sample image does not contain text content; wherein, the sample images corresponding to the first category label, the second category label and the third category label all contain text content.

[0019] In an exemplary embodiment of the present disclosure, the classification loss of the sample images corresponding to the first category label, the second category label and the third category label is calculated using a classification loss function including a cross entropy loss function and a center loss function; the classification loss of the sample image corresponding to the fourth category label is calculated using the cross entropy loss function.

[0020] In an exemplary embodiment of the present disclosure, determining the similarity between the image feature and the illegal image feature includes calculating the similarity between the image feature and the illegal image feature using Euclidean distance.

[0021] In an exemplary embodiment of the present disclosure, determining the similarity between the image features and the illegal image features includes: each time determining the similarity between the image features of a picture region and a group of illegal image features, comparing the similarity with the first threshold; wherein a group of illegal image features comes from a illegal picture; or determining the similarity between the image features of each picture region and each group of illegal image features, taking the minimum similarity and comparing it with the first threshold.

[0022] According to one aspect of the present disclosure, an image recognition device is provided, comprising: a region acquisition module for acquiring a picture region containing text from an image to be recognized; a feature extraction module for performing feature extraction on the picture region to obtain picture features of the corresponding picture region; a feature comparison module for determining a similarity between the picture features and features of a violation picture, the violation picture features being taken from a violation picture feature library; and a violation recognition module for determining the image to be recognized as a violation image in response to the similarity being greater than a first threshold.

[0023] According to one aspect of the present disclosure, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image recognition method as described in any of the above embodiments is implemented.

[0024] According to one aspect of the present disclosure, a computer device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the image recognition method as described in any of the above embodiments by executing the executable instructions.

[0025] The image recognition method, apparatus, storage medium, and computer device of the disclosed embodiments, after obtaining an image region containing text in an image to be recognized, directly extract the image features of the image region, and determine whether the image to be recognized is illegal based on the similarity between the image features and the features of the illegal image. This method can efficiently identify illegal images with abnormal text content, reduce dependence on image semantic information, and does not need to rely on the text content of the text region. It can cope with various text types through the similarity of image features, breaking through the limitations of text recognition. In addition, through the dynamic update and maintenance of the illegal image feature library, the illegal image recognition scheme disclosed in the present invention can quickly respond to new attack types and achieve efficient, accurate, and comprehensive recognition of illegal images. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings, in which several embodiments of the present disclosure are shown by way of example and not limitation, wherein:

[0027] Figure 1 A schematic diagram schematically illustrates a flow chart of an image recognition method according to an embodiment of the present disclosure;

[0028] Figure 2 A schematic diagram schematically illustrates the effect of merging text boxes according to an embodiment of the present disclosure;

[0029] Figure 3 Schematically illustrating a flow chart of merging text boxes according to an embodiment of the present disclosure;

[0030] Figure 4 Schematically illustrating a flow chart of merging text boxes according to yet another embodiment of the present disclosure;

[0031] Figure 5 A schematic diagram of a network structure of a feature extraction model according to an embodiment of the present disclosure is schematically shown;

[0032] Figure 6Schematically illustrates a training process diagram of a feature extraction model according to an embodiment of the present disclosure;

[0033] Figure 7 Schematically illustrates a process diagram of image feature extraction according to an embodiment of the present disclosure;

[0034] Figure 8 Schematically illustrates a schematic diagram of an image recognition process according to an embodiment of the present disclosure;

[0035] Figure 9 Schematically illustrates a module architecture diagram of an image recognition device according to an embodiment of the present disclosure;

[0036] Figure 10 A schematic diagram schematically illustrates a storage medium according to an embodiment of the present disclosure;

[0037] Figure 11 The module architecture diagram of a computer device according to one embodiment of the present disclosure is schematically shown.

[0038] In the drawings, the same or corresponding reference numerals denote the same or corresponding parts. DETAILED DESCRIPTION

[0039] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided solely to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0040] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, apparatus, device, method, or computer program product. Therefore, the present disclosure may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software.

[0041] According to embodiments of the present disclosure, an image recognition method, apparatus, storage medium, and computer device are provided.

[0042] In this document, any number of elements in the drawings is for illustration and not for limitation, and any naming is for distinction only and does not have any limiting meaning.

[0043] The principles and spirit of the present disclosure are described in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION

[0045] Illegal images are those that contain prohibited content such as advertisements. Currently, the main methods for identifying illegal images include image classification algorithms and text recognition algorithms.

[0046] Image classification algorithms train image classification models to predict the probability that an image contains illegal content and then determine whether the image is illegal based on the predicted probability. Image classification models rely on semantic information in the image to predict violations, such as the presence of people, special objects, and text in the image. Because they cannot accurately identify text content, they are unable to process images with abnormal text content, which is often the main cause of image violations. Furthermore, because image classification models rely on semantic information to predict violations, even if text content is extracted as high-level semantic information, as long as an attacker creates new attack text (for example, adding or removing a few strokes from existing attack text), the semantic information is destroyed, and the image classification model needs to be retrained to recognize the new semantics, making it impossible to quickly respond to new attack types.

[0047] Text recognition algorithms primarily refer to OCR (Optical Character Recognition) algorithms. OCR algorithms identify text in images, convert it into text, and then use text understanding models or predefined keywords to determine whether it contains illegal content. As OCR technology matures, attackers will exploit text types that are difficult for OCR algorithms to recognize and synthesize new attack text (for example, by artificially adding occlusion to text to hide the attack content within the text) to bypass the OCR algorithm. The newly synthesized attack text changes the text content of the original text. Recognizing the new attack text requires retraining the OCR algorithm model to identify the same type of attack text. This is costly and requires a long update cycle, making it difficult to achieve a rapid response.

[0048] In view of the above, the basic idea of ​​this disclosure is:

[0049] For the image to be identified, after obtaining the image area containing text in the image, the image features of the image area are directly extracted. The similarity between the image features and the features of the illegal image is used to determine whether the image to be identified is illegal. It can efficiently identify illegal images with abnormal text content, reduce dependence on image semantic information, and do not need to rely on the text content of the text area. It can cope with various text types through the similarity of image features, breaking through the limitations of text recognition. In addition, it can quickly respond to new attack types through the dynamic update and maintenance of the illegal image feature library, and achieve efficient, accurate and comprehensive identification of illegal images.

[0050] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0051] Exemplary Methods

[0052] The following combination Figure 1 The image recognition method according to the exemplary embodiment of the present disclosure is described. Figure 1 As shown, the image recognition method may include the following steps:

[0053] S110 , obtaining an image region containing text from the image to be recognized.

[0054] The image to be identified is an image to be determined whether it contains illegal content. The image region obtained from the image to be identified may include one or more regions.

[0055] S120: Extract features from the image region to obtain image features of the corresponding image region.

[0056] When there are multiple image regions, the image features of each image region are obtained respectively.

[0057] S130, determining the similarity between the image features and the illegal image features, where the illegal image features are taken from the illegal image feature library.

[0058] The illegal image feature library stores known illegal image features. The higher the similarity between an image feature and an illegal image feature, the higher the likelihood that the corresponding image feature is illegal.

[0059] S140 : In response to the similarity being greater than a first threshold, determining the image to be identified as an illegal image.

[0060] The first threshold is a similarity threshold, and the specific value of the first threshold can be set as needed. For one or more image regions containing text obtained from the image to be identified, if the similarity of any image region is greater than the first threshold, the image to be identified can be determined to be an illegal image.

[0061] The image recognition method of the aforementioned embodiment, after acquiring the image region containing text within the image to be identified, directly extracts image features. Based on the similarity between these features and the features of known illegal images, it determines whether the image to be identified is illegal. This method can efficiently identify illegal images with abnormal text content, reducing reliance on image semantic information. Without relying on the text content of the text region, the method can address a wide variety of text types through the similarity of image features, thus overcoming the limitations of text recognition. Furthermore, the image recognition method of the aforementioned embodiment can also rapidly respond to new attack types through the dynamic updating and maintenance of the illegal image feature library, achieving efficient, accurate, and comprehensive recognition of illegal images.

[0062] In one exemplary embodiment, obtaining an image region containing text from an image to be recognized may include: performing text detection on the image to be recognized to determine a text box corresponding to a corresponding text sequence based on the detection results; responsive to the presence of multiple text boxes and the multiple text boxes meeting a merging condition, merging the multiple text boxes to obtain a target text box, and extracting an image region from the image to be recognized based on the target text box; wherein the merging condition is determined based on the distance between the text boxes and / or the degree of overlap between the text boxes. Furthermore, responsive to the presence of only one text box, extracting an image region from the image to be recognized based on the text box.

[0063] Text detection can be performed using an existing text detection algorithm. The algorithm first identifies individual text pixels, then aggregates these pixels into a single line of text. This generates a detection result containing one or more text boxes, each corresponding to a single line of text. Text boxes are typically rectangular; however, they can also take other shapes depending on the configuration of the text detection algorithm.

[0064] Since the feature differences of a single line of text are small and the discrimination is low, directly applying it to the subsequent violation identification based on similarity comparison will have the problem of low accuracy. Therefore, when there are multiple text boxes, the multiple text boxes are further merged. The target text boxes obtained after merging not only have greater differences, which can ensure the accuracy of subsequent violation identification, but also save the subsequent feature extraction calculation amount, and there is no need to extract features for the image areas corresponding to a large number of text boxes. In addition, it can also save feature storage space, and there is no need to store the features of the image areas corresponding to a large number of text boxes.

[0065] For example, a certain illegal content is suspected to be a vulgar advertisement, which contains three parts of information. The first part of the information is the advertising content (for example, a date), the second part of the information is the operation guide (for example, click on the website link), and the third part of the information is the specific website link. The attacker uses the text detection algorithm to identify the characteristics of a single line of text, and splits the above illegal content into three lines of text corresponding to the three parts of information. The three lines of text are separated by a certain distance, so that when the text detection algorithm performs text detection on the image to be identified containing the above illegal content, three text boxes corresponding to the above three parts of information are obtained. Subsequently, if the violation is identified based on each text box separately, since the violation level corresponding to a single text box is not high, the recognition result that the image to be identified is an illegal image will not be obtained; only by merging the three text boxes into a target text box and extracting the image features based on the image area corresponding to the target text box for similarity comparison can the illegal characteristics of the image to be identified be accurately determined.

[0066] Furthermore, merging the multiple text boxes to obtain the target text box includes: enlarging the multiple text boxes according to a preset ratio to obtain corresponding extended text boxes; and merging the extended text boxes to obtain the target text box.

[0067] Figure 2 The effect of merging text boxes in an exemplary embodiment is shown, referring to Figure 2 As shown, in the image 200 to be identified, after text detection, five text boxes 201, 202, 203, 204, and 205 are obtained. Among them, the three text boxes 201, 202, and 203 in the upper left corner correspond to a group of related content (actually illegal content, such as the illegal content split into three lines of text in the above example), and the two text boxes 204 and 205 in the middle and lower part correspond to a group of related content (normal non-illegal content). For text boxes 201, 202, and 203, because the attacker deliberately split the related illegal content into three lines of text at a certain distance, the three text boxes 201, 202, and 203 obtained by detection are spaced apart from each other, resulting in the three related text boxes being likely not to be merged together when the text boxes are merged, thereby affecting the subsequent violation identification.

[0068] Therefore, before merging the text boxes, first enlarge the text boxes according to a preset ratio. The preset ratio can be set as needed. For example, in this example, each text box is enlarged by 1.5 times the aspect ratio. Figure 2 In the figure, text boxes 201-205 illustrate text boxes obtained through text detection, while expanded text boxes 201', 202', 203', 204', and 205' illustrate expanded text boxes obtained by enlarging their length and width in proportion to each other. Expanding text boxes not only ensures that related text boxes are merged together when merging, but also incorporates background information surrounding the text, facilitating the acquisition of more comprehensive image features during subsequent image feature extraction.

[0069] After obtaining the extended text boxes, the extended text boxes can be merged based on the distance between the text boxes (for example, merging the extended text boxes with close distances), the overlap between the text boxes (for example, merging the extended text boxes that overlap), or a combination of distance and overlap. The final merge effect is shown in the figure below. Figure 2 As shown, the expanded text boxes 201 ′, 202 ′, and 203 ′ are merged into the target text box 200 a , and the expanded text boxes 204 ′ and 205 ′ are merged into the target text box 200 b .

[0070] Figure 3 The process of merging text boxes in an exemplary embodiment is shown, referring to Figure 3 As shown in the figure, in this example, the non-maximum suppression algorithm is used to merge the extended text boxes, including:

[0071] S310: Determine the extended text box with the largest area as the reference text box.

[0072] S320: Determine the overlap between the reference text frame and other extended text frames, and determine candidate text frames based on the overlap, wherein the candidate text frames are extended text frames whose overlap with the reference text frame exceeds a second threshold value.

[0073] The overlap can be calculated using the intersection-and-union ratio (IRU). This ratio indicates the degree of overlap between two polygons and ranges from [0, 1], where 0 represents no overlap and 1 represents complete overlap. A higher IRU indicates a higher degree of overlap. The second threshold can be set as needed; in this example, it is set to 0.1.

[0074] S330: Merge the reference text box and the candidate text box to obtain a target text box.

[0075] The target text box can surround the reference text box and its candidate text boxes, for example, it is the minimum bounding rectangle of the reference text box and its candidate text boxes.

[0076] S340: Repeat the above steps in the remaining extended text boxes until no new target text box can be obtained in this way.

[0077] When the algorithm is actually running, an empty result set and an empty candidate set can be initialized. Each time a merge is performed, the expanded text frame with the largest area from the initial set of expanded text frames is selected as the base frame. The intersection-of-union (IoU) of the remaining expanded text frames with this base frame is calculated. If any expanded text frame has an IoU greater than 0.1, the corresponding expanded text frame is moved to the candidate set. After the IoU calculation is completed, if the candidate set is empty, the base frame is moved to the result set. If the candidate set is not empty, the base frame is moved to the candidate set, and the minimum enclosing rectangle of all text frames in the candidate set is calculated. This is added to the result set as the target text frame, and the candidate set is cleared. This process is repeated until the initial set of expanded text frames is empty. The resulting result set is the final merge result.

[0078] Figure 4 Another exemplary embodiment shows a process for merging text boxes, referring to Figure 4 As shown, in this example, when merging, multiple extended text boxes with larger areas can be selected each time, and each selected extended text box and its neighboring extended text boxes can be merged to improve the merging efficiency. Specifically, merging the extended text boxes to obtain the target text box includes:

[0079] S410: Determine an extended text box with an area greater than a third threshold as a reference text box.

[0080] The third threshold value can be determined as needed. The third threshold value can be dynamically determined in combination with the areas of all extended text boxes, so that an appropriate number of reference text boxes can be selected each time the text boxes are merged.

[0081] S420: Determine the degree of overlap between each base text frame and each associated extended text frame, and determine candidate text frames for each base text frame based on the overlap results. An extended text frame associated with a base text frame is an extended text frame whose distance from the base text frame is less than a fourth threshold; a candidate text frame for the base text frame is an extended text frame associated with the base text frame whose degree of overlap with the base text frame exceeds a second threshold.

[0082] The distance between text boxes can be calculated based on their center points. The fourth threshold setting must ensure that related text boxes can be merged together without introducing too many unrelated text boxes and increasing the computational complexity. Regarding the degree of overlap, this example also uses the intersection-over-union ratio for calculation. For the specific principles and setting of the second threshold, refer to the description in the above example.

[0083] S430: Merge each reference text frame with its candidate text frame to obtain a target text frame.

[0084] When the reference text box has candidate text boxes, the minimum enclosing rectangle or other suitable enclosing rectangle surrounding the reference text box and its candidate text boxes is used as the target text box; if the reference text box does not have candidate text boxes, the reference text box is directly used as the target text box.

[0085] S440: Repeat the above steps in the remaining extended text boxes until no new target text box can be obtained in this way.

[0086] After obtaining the target text box, a picture area is cut out from the image to be recognized according to the target text box.

[0087] The algorithm for extracting text-containing regions from the image to be identified can be implemented using a text region detection model. This model is used to extract text-containing regions from the image. Subsequently, the model extracts features from these regions to determine whether the image to be identified is a violation.

[0088] In an exemplary embodiment of the present disclosure, the target text box obtained includes information such as the text box shape, picture area background, and the format, distribution and proportion of the text; thus, subsequent feature extraction of the picture area corresponding to the target text box can obtain picture features containing multiple feature information such as the text box shape, picture area background, and the format, distribution and proportion of the text. Based on the picture features containing multiple feature information, violations of the image to be identified can be accurately and efficiently identified.

[0089] In addition, in an exemplary embodiment, after the target text box is obtained by merging, it can also be determined whether the image to be identified is an illegal image based on the shape features of the target text box. Specifically, considering that normal text content is usually neatly arranged in lines, and the previous line is turned to the next line after it is full; while illegal text content such as advertisements often adopt special formatting such as line breaks for sensitive words to avoid detection, after the text boxes corresponding to the illegal text content are merged, target text boxes with special shapes such as trapezoidal, diamond-shaped, T-shaped, etc. will be obtained. Therefore, in one example, some specific shapes can be preset as the basis for identifying illegal text boxes; after the target text box is obtained by merging, the shape of the target text box can be compared to match the preset specific shape, then the target text box can be determined to be an illegal text box, and the image to be identified can be further determined to be an illegal image. By determining whether the image to be identified is an illegal image based on the shape features of the target text box, the processing of complex image features can be avoided, and the illegal identification of the image to be identified can be performed more quickly.

[0090] In an exemplary embodiment, feature extraction is performed on an image region, including extracting features from the image region through a feature extraction model; wherein the network structure of the feature extraction model in the application stage includes: a backbone network, which is used to extract features from the image region and obtain basic features of the image region; a measurement network, which is connected to the backbone network and is used to perform feature processing on the basic features and obtain image features of the image region.

[0091] Convolutional neural networks can be used for both the backbone network and the metric network. The backbone network is a deep convolutional neural network that extracts basic, common features from image regions. The metric network is a lightweight convolutional neural network that further processes the basic features extracted by the backbone network and outputs key, important image features. The feature extraction model is trained to accurately extract image features from input image regions.

[0092] In an exemplary embodiment, the network structure of the feature extraction model during the training phase further includes a classification network connected to the backbone network. The classification network may also be a convolutional neural network, which is a lightweight convolutional neural network.

[0093] Figure 5 The network structure of the feature extraction model in an exemplary embodiment is shown, referring to Figure 5 As shown, the network structure of the feature extraction model during the training phase includes a backbone network 510, a metric network 520, and a classification network 530. The classification network 530 is primarily used to accelerate model convergence during the training phase and will be described in detail below. After training is complete, the network structure 500 of the feature extraction model during the application phase only requires the backbone network 510 and the metric network 520.

[0094] The following describes the training process of the feature extraction model. It should be noted that the sample images referred to below are those processed by the aforementioned text region detection model. Specifically, they are the key regions containing text captured from the original image after performing operations such as text detection and text box merging.

[0095] In an exemplary embodiment, the training process of the feature extraction model includes: obtaining labeled samples, each labeled sample carries a metric label and / or a classification label; inputting the labeled samples into the backbone network, obtaining metric features corresponding to the metric labels output by the metric network and classification results corresponding to the classification labels output by the classification network; and optimizing the parameters of the backbone network, the metric network, and the classification network based on the total loss including the metric loss and the classification loss.

[0096] The training process of the above feature extraction model uses a metric learning approach. Through the combined constraints of metric loss and classification loss, the feature extraction model achieves rapid convergence and accurately learns the similarity between images. The metric loss is used to bring similar images closer together in feature space and dissimilar images further apart. Typically, in image metric learning, images of the same category are considered positive samples (i.e., similar), while images of different categories are considered negative samples (i.e., dissimilar). Therefore, the classification loss, as an auxiliary loss, can provide additional supervisory information, accelerating training and improving training results.

[0097] During model training, the classification labels use category labels corresponding to multiple categories, and the classification loss is used to constrain sample images belonging to the same category to be in the same category space; the metric labels use triple labels including sample labels, positive sample labels, and negative sample labels. The metric loss is used to reduce the distance between the metric features of the current sample image (carrying the sample label) output by the metric network and the metric features of its positive sample image (carrying the associated positive sample label), and increase the distance between the metric features of the current sample image output by the metric network and the metric features of its negative sample image (carrying the associated negative sample label).

[0098] Figure 6 The training process of the feature extraction model in an exemplary embodiment is shown, referring to Figure 6As shown, a triplet sample (i.e., a set of sample images with associated triplet labels) is used as an example, including a sample image 610 with a sample label "I" and a classification label "A," a positive sample image 620 with a positive sample label "+I" and a classification label "A," and a negative sample image 630 with a negative sample label "-I" and a classification label "B." The triplet sample is input into the network structure 500' of the feature extraction model during the training phase. The backbone network extracts basic features for each sample image; the classification network maps the basic features extracted by the backbone network to the category space, obtaining classification results 6101 for sample image 610, classification results 6201 for positive sample image 620, and classification results 6301 for negative sample image 630; and the metric network further processes the basic features extracted by the backbone network to obtain metric features 6102 for sample image 610, metric features 6202 for positive sample image 620, and metric features 6302 for negative sample image 630.

[0099] In this example, the metric network outputs 512-dimensional metric features for each sample image. In other examples, the metric features can have dimensions ranging from 256 to 2048. Considering the balance between accuracy and speed, a feature dimension of 512 achieves better performance.

[0100] After obtaining the model output results, the classification loss 6601 is calculated for the classification results 6101, 6201 and 6301; the metric loss 6602 is calculated for the metric features 6102, 6202 and 6302; and then, based on the total loss including the metric loss and the classification loss, the parameters of the network structure 500' are adjusted to constrain the sample image 610 and the positive sample image 620 to be in the category space "A" (the negative sample image 630 will also be constrained to the category space "B" with other images of the same category), and to make the distance between the sample image 610 and the positive sample image 620 in the feature space close, and to make the distance between the sample image 610 and the negative sample image 630 in the feature space far apart.

[0101] Therefore, through the classification task corresponding to the classification loss, sample images of different categories are distinguished, and global, coarse-grained category space constraints of sample images are achieved; through the measurement task corresponding to the measurement loss, the distance between positive samples is reduced and the distance between negative samples is increased, so that more fine-grained constraints on sample images in each category space are achieved; through the joint constraints of the classification task and the measurement task, the model convergence is accelerated, and a feature extraction model that can accurately extract image features is obtained.

[0102] The classification task and measurement task are explained below respectively.

[0103] For classification tasks, in the exemplary embodiments of the present disclosure, the training data for the feature extraction model consists of sample images containing multiple lines of text, merged into text boxes. This lacks the category information required for conventional image classification, making it difficult to construct positive and negative sample pairs and classification categories for the classification task using conventional methods. The category information depends on the classification task. For animal classification, the category information can be cat, dog, bird, etc.; for plant classification, the category information can be flower, number, grass, etc.; however, for sample images corresponding to multiple lines of text, it is difficult to define their categories. Specifically, the attributes of a single text character include font, color, and decorative lines. Font is the most distinguishing of these attributes, but there are many font categories (over 200), including artistic fonts and handwriting, making font classification extremely expensive. Furthermore, in the exemplary embodiments of the present disclosure, to improve computational efficiency and make individual images more distinguishable, a multi-line text aggregation approach was used. This means that when an image contains multiple lines of text, the attributes of different text characters are likely to conflict, making it difficult to distinguish categories based on text attributes. Without specific categories, it is difficult to mine positive samples (i.e., similar sample pairs).

[0104] Therefore, the present disclosure proposes a classification method for coarse-grained distinction based on data source. Specifically, in an exemplary embodiment, the classification labels include: a first category label, used to characterize that the corresponding sample image comes from real business; a second category label, used to characterize that the corresponding sample image is synthesized by a machine; a third category label, used to characterize that the corresponding sample image comes from an external resource platform; a fourth category label, used to characterize that the corresponding sample image does not contain text content; wherein, the sample images corresponding to the first category label, the second category label, and the third category label all contain text content and are positive samples of the classification task, and the sample images that do not contain text content are negative samples of the classification task.

[0105] Data from different sources will have certain differences in feature expression. By marking classification labels according to the data source, sample images from different data sources can be constrained to the corresponding category space.

[0106] In one exemplary embodiment, the classification loss for sample images corresponding to the first, second, and third category labels is calculated using a classification loss function comprising a cross-entropy loss function and a center-loss function, with the cross-entropy loss and center-loss function weighted, for example, at 1:1. The classification loss for sample images corresponding to the fourth category label is calculated using the cross-entropy loss function. Sample images without text content, i.e., background images, are broad and open spaces that cannot be aggregated, and therefore center-loss is not used to calculate their classification loss.

[0107] For metric tasks, since the category labels used in the aforementioned classification tasks are coarse-grained, unlike conventional metric learning with category labels, in the exemplary embodiments of this disclosure, images within the same category are highly diverse and cannot serve as positive examples for each other. Therefore, this disclosure proposes a "self-supervised" approach to metric learning.

[0108] Specifically, in an exemplary embodiment, the metric label is a triplet label including a sample label, a positive sample label, and a negative sample label. In a group of sample images carrying associated triplet labels: the current sample image carrying the sample label is randomly selected from the sample image set; the positive sample image carrying the positive sample label is obtained by performing data enhancement on the current sample image; the negative sample image carrying the negative sample label is selected from the remaining sample image group in the sample image set except at least the current sample image.

[0109] Data augmentation can include random cropping, random changes in brightness or contrast, random swapping of color channels, normalization, and other operations. For a sample image, the image after data augmentation can be used as its positive sample, and all sample images other than the sample image can be used as its negative samples.

[0110] In one exemplary embodiment, a small number of sample images in the sample image set may carry similar image labels, providing difficult examples to aid model training. In this case, in addition to obtaining positive sample images by performing data augmentation on the current sample image, when the current sample image carries a similar image label, positive sample images can also be randomly selected from a group of similar sample images in the sample image set, where all sample images in the group carry similar image labels.

[0111] In addition, when the current sample image does not carry a similar image label, the remaining sample image group is composed of sample images in the sample image set except the current sample image; when the current sample image carries a similar image label, the remaining sample image group is composed of sample images in the sample image set except the similar sample image group; the negative sample image is selected from the remaining sample image group.

[0112] In an exemplary embodiment, a negative sample image carrying a negative sample label is selected from the remaining sample image group of the sample image set, which may specifically include: extracting features of each sample image in the sample image set; calculating the distance between the features of the current sample image and the features of each remaining sample image in the remaining sample image group; and randomly selecting a remaining sample image as a negative sample image from the remaining sample images whose distance is less than a distance threshold (for example, from the top 20% of the remaining sample images sorted in ascending order of distance).

[0113] Furthermore, in one exemplary embodiment, a triplet-loss function is used to calculate the metric loss of the metric features of each set of sample images carrying triple labels. Through metric learning, a mapping from image to feature space is achieved, and the image features output by the metric network are able to make similar images close in feature space and dissimilar images far apart in feature space.

[0114] The weighting of the total loss, which includes both the metric loss and the classification loss, can be adjusted as needed. Typically, the metric loss is weighted more heavily than the classification loss, allowing the feature extraction model to be trained with the metric task as the primary task and the classification task as the secondary task. In one exemplary embodiment, the weighting of the metric loss and the classification loss is 1:0.5.

[0115] In summary, during the aforementioned model training process, the classification task constrains sample images to specific categories within the corresponding category space, which not only accelerates the model's convergence but also learns how to distinguish between text and non-text. The measurement task then performs a more fine-grained differentiation of sample images within the corresponding category space, enabling clustering of similar images. After model training, the measurement network is able to output accurate image features, such that if the image to be identified actually contains illegal content, the output image features are close in feature space to the illegal image features, allowing the image to be subsequently identified as an illegal image. If the image to be identified does not actually contain illegal content, the image features output by the measurement network are far away in feature space from the illegal image features, preventing the image from being subsequently identified as an illegal image.

[0116] Figure 7 The image feature extraction process in an exemplary embodiment is shown, referring to Figure 7 As shown, in combination with the description of the above embodiments, when extracting image features of the image to be identified 700: first, a text area detection model 710 is used to perform operations such as text detection and text box merging to obtain an image area containing text in the image to be identified 700; then a feature extraction model 720 including a backbone network and a metric network is used to extract image features 700' of the image area.

[0117] After obtaining the image features, the Euclidean distance can be used to calculate the similarity between the image features and the illegal image features. The closer the distance, the higher the similarity.

[0118] In terms of algorithm implementation, the similarity between image features and illegal image features can be achieved through a similarity comparison module. The similarity comparison module can compare it with the threshold value every time a similarity is obtained, or compare it with the threshold value after all similarities are calculated. Specifically, determining the similarity between image features and illegal image features includes: each time determining the similarity between the image features of an image area and a group of illegal image features, comparing the similarity with a first threshold value; wherein a group of illegal image features comes from an illegal image; or, determining the similarity between the image features of each image area and each group of illegal image features, taking the minimum similarity and comparing it with the first threshold value.

[0119] The first threshold is highly related to the feature extraction model. The training dataset is required to calculate the precision and recall values ​​under different thresholds, and the optimal threshold on the training dataset is used as the final first threshold.

[0120] Figure 8 An exemplary embodiment of the image recognition process is shown, referring to Figure 8 As shown, in combination with the description of the above embodiments, in a specific example, the image recognition process includes:

[0121] S810, construct a feature library of illegal pictures. Specifically, extract the features of known illegal pictures (see Figure 7 The image feature extraction process shown in the figure is used to obtain the features of the illegal images and store them in the illegal image feature library. The illegal image feature library stores the extracted low-dimensional features rather than the high-dimensional images themselves, which can reduce the amount of calculation for similarity comparison and speed up the calculation. In addition, when there are new violation identification needs or when new attack types are discovered in real business, the illegal image features in the illegal image feature library can be added at any time; and when it is necessary to restore the features of a certain type of illegal image to normal, the illegal image features in the illegal image feature library can be deleted at any time; the combination of the updated illegal image feature library and the image features to be identified for violation identification can adapt to rapid changes in demand and constantly changing attack types.

[0122] By dynamically maintaining the illegal image feature library, there is no need for image classification algorithms, which rely on image semantic information to predict violations. When an attacker destroys the original semantic information and constructs a new attack type, the image classification model needs to be retrained to identify the new attack type. There is also no need for text recognition algorithms, which rely on text content to identify violations. When an attacker destroys the original text content and constructs a new attack type, the text recognition model needs to be retrained to identify the new attack type.

[0123] After obtaining the text area detection model and training the feature extraction model, we can build a feature library of illegal images to prepare a comparison basis for illegal identification of the images to be identified.

[0124] S820: Extract image features from the image to be recognized to obtain image features. The image features are also low-dimensional feature vectors, which facilitate fast distance calculation.

[0125] S830: Calculate the similarity between the image features and the features of the illegal images using a similarity comparison module. If any image feature of the image to be identified is similar to any illegal image feature in the illegal image feature library (the similarity is less than a first threshold), the image to be identified is determined to be an illegal image. Conversely, if the image features are similar or dissimilar to the illegal image features, the image to be identified is determined to be a normal image.

[0126] When there are multiple images to be identified, a streaming processing method can be used to add all the images to be identified into a queue, and then extract image features and compare similarities in sequence.

[0127] In summary, the above-mentioned image recognition method, after obtaining the image area containing text in the image to be identified through text area detection, directly extracts the image features of the image area, and determines whether the image to be identified is illegal through the similarity between the image features and the illegal image features. It can efficiently identify illegal images with abnormal text content, reduce dependence on image semantic information, and does not need to rely on the text content of the text area. It can cope with various text types through the similarity of image features, breaking through the limitations of text recognition; in addition, it can also dynamically update and maintain the illegal image feature library. For example, when a new attack type is discovered in real business, the illegal image features corresponding to the new attack type are added to the illegal image feature library, so that subsequent images with similar attack types can be identified. There is no need to retrain the model, and it can quickly respond to new attack types, thereby achieving efficient, accurate and comprehensive recognition of illegal images.

[0128] Exemplary devices

[0129] After introducing the image recognition method of the exemplary embodiment of the present disclosure, the following will be combined with Figure 9 An image recognition device according to an exemplary embodiment of the present disclosure will be described.

[0130] The image recognition device provided in the embodiments of the present disclosure can be used to implement the image recognition method described in the above corresponding embodiments. The features and principles of the image recognition method described in any of the above embodiments can be applied to the corresponding image recognition device embodiments below.

[0131] Reference Figure 9As shown, an image recognition device 900 according to an exemplary embodiment of the present disclosure includes a region acquisition module 910, a feature extraction module 920, a feature comparison module 930, and a violation identification module 940. Region acquisition module 910 may be used to obtain a region of an image containing text from an image to be identified; feature extraction module 920 may be used to extract features from the image region to obtain image features of the corresponding image region; feature comparison module 930 may be used to determine the similarity between the image features and features of violation images, where the violation image features are obtained from a violation image feature library; and violation identification module 940 may be used to determine that the image to be identified is a violation image in response to the similarity being greater than a first threshold.

[0132] The above-mentioned image recognition device 900, after obtaining the image area containing text in the image to be recognized, directly extracts the image features of the image area, and determines whether the image to be recognized is illegal through the similarity between the image features and the illegal image features. It can efficiently identify illegal images with abnormal text content, reduce dependence on image semantic information, and does not need to rely on the text content of the text area. It can cope with various text types through the similarity of image features, breaking through the limitations of text recognition; in addition, it can also quickly respond to new attack types through the dynamic update and maintenance of the illegal image feature library, and realize efficient, accurate and comprehensive recognition of illegal images.

[0133] According to an exemplary embodiment of the present disclosure, the area acquisition module 910 may include: a text detection module, used to perform text detection on the image to be recognized, so as to determine a text box corresponding to the corresponding text sequence based on the detection result; a first interception module, used to merge the multiple text boxes in response to the fact that there are multiple text boxes and the corresponding multiple text boxes meet the merging condition, to obtain a target text box, and intercept a picture area from the image to be recognized according to the target text box; wherein the merging condition is determined based on the distance between the text boxes, and / or the overlap between the text boxes.

[0134] According to an exemplary embodiment of the present disclosure, the region acquisition module 910 may further include: a second interception module, configured to intercept a picture region from the image to be recognized according to the text box in response to there being only one text box.

[0135] According to an exemplary embodiment of the present disclosure, in the first interception module, the module for merging multiple text boxes to obtain a target text box may include: a text box magnification module for magnifying multiple text boxes according to a preset ratio to obtain corresponding extended text boxes; and a text box merging module for merging the extended text boxes to obtain the target text box.

[0136] According to an exemplary embodiment of the present disclosure, a text box merging module may include: a first base frame determination module, used to determine the extended text box with the largest area as the base text box; a first candidate frame determination module, used to determine the degree of overlap between the base text box and other extended text boxes, and determine the candidate text box based on the overlap result; wherein the candidate text box is an extended text box among the other extended text boxes, the degree of overlap between the base text box and the candidate text box exceeding a second threshold; a first merging module, used to merge the base text box with the candidate text box to obtain a target text box; and a first loop module, used to repeat the above steps in the remaining extended text boxes until a new target text box can no longer be obtained in this way.

[0137] According to an exemplary embodiment of the present disclosure, a text box merging module may include: a second base frame determination module, used to determine an extended text box whose area is greater than a third threshold as a base text box; a second candidate frame determination module, used to determine the degree of overlap between each base text box and each associated extended text box, and determine the candidate text box of each base text box based on the overlap result; wherein, the extended text box associated with a base text box is an extended text box whose distance from the base text box is less than a fourth threshold; the candidate text box of the base text box is an extended text box among the extended text boxes associated with the base text box, whose degree of overlap with the base text box exceeds the second threshold; a second merging module, used to merge each base text box with its candidate text box to obtain a target text box; and a second loop module, used to repeat the above steps in the remaining extended text boxes until no new target text box can be obtained in this way.

[0138] According to an exemplary embodiment of the present disclosure, the feature extraction module 920 can be used to extract features from the image area through a feature extraction model; wherein, the network structure of the feature extraction model in the application stage includes: a backbone network, which is used to extract features from the image area and obtain basic features of the image area; a measurement network, which is connected to the backbone network and is used to perform feature processing on the basic features and obtain image features of the image area.

[0139] According to an exemplary embodiment of the present disclosure, the network structure of the feature extraction model in the training stage also includes a classification network connected to the backbone network; the modules for training the feature extraction model include: a sample acquisition module for obtaining labeled samples, each labeled sample carries a metric label and / or a classification label; a model operation module for inputting the labeled samples into the backbone network to obtain metric features corresponding to the metric labels output by the metric network and classification results corresponding to the classification labels output by the classification network; a model adjustment module for optimizing the parameters of the backbone network, the metric network and the classification network based on the total loss including the metric loss and the classification loss.

[0140] According to an exemplary embodiment of the present disclosure, the metric label is a triplet label including a sample label, a positive sample label and a negative sample label. In a group of sample images carrying associated triplet labels: the current sample image carrying the sample label is randomly selected from the sample image set; the positive sample image carrying the positive sample label is obtained by performing data enhancement on the current sample image; the negative sample image carrying the negative sample label is selected from the remaining sample image group in the sample image set except at least the current sample image.

[0141] According to an exemplary embodiment of the present disclosure, the model adjustment module reduces the distance between the metric features of the current sample image output by the metric network and the metric features of its positive sample image, and increases the distance between the metric features of the current sample image output by the metric network and the metric features of its negative sample image by optimizing the parameters of the backbone network and the metric network.

[0142] According to an exemplary embodiment of the present disclosure, the model adjustment module is configured to use a triplet loss function to calculate the metric loss of the metric features of each group of sample images.

[0143] According to an exemplary embodiment of the present disclosure, when the current sample image does not carry a similar image label, the remaining sample image group is composed of sample images in the sample image set except the current sample image; when the current sample image carries a similar image label, the positive sample image can also be randomly selected from the similar sample image group in the sample image set, and the sample images in the similar sample image group all carry similar image labels, and the remaining sample image group is composed of sample images in the sample image set except the similar sample image group.

[0144] According to an exemplary embodiment of the present disclosure, a module for obtaining a negative sample image carrying a negative sample label includes: a feature acquisition module for extracting features of each sample image in a sample image set; a distance calculation module for calculating the distance between the features of the current sample image and the features of each remaining sample image in the remaining sample image group; and an image selection module for randomly selecting a remaining sample image as a negative sample image from the remaining sample images whose distance is less than a distance threshold.

[0145] According to an exemplary embodiment of the present disclosure, the classification labels include: a first category label, used to characterize that the corresponding sample image comes from real business; a second category label, used to characterize that the corresponding sample image is synthesized by a machine; a third category label, used to characterize that the corresponding sample image comes from an external resource platform; a fourth category label, used to characterize that the corresponding sample image does not contain text content; wherein, the sample images corresponding to the first category label, the second category label and the third category label all contain text content.

[0146] According to an exemplary embodiment of the present disclosure, the model adjustment module uses a classification loss function including a cross-entropy loss function and a center loss function to calculate the classification loss of the sample images corresponding to the first category label, the second category label and the third category label; the model adjustment module uses the cross-entropy loss function to calculate the classification loss of the sample image corresponding to the fourth category label.

[0147] According to an exemplary embodiment of the present disclosure, the feature comparison module 930 uses the Euclidean distance to calculate the similarity between the image features and the illegal image features.

[0148] According to an exemplary embodiment of the present disclosure, the feature comparison module 930 may include: a first comparison module, configured to determine the similarity between the image features of each image region and a group of illegal image features, and compare the similarity with a first threshold; wherein a group of illegal image features comes from one illegal image; or a second comparison module, configured to determine the similarity between the image features of each image region and each group of illegal image features, and take the minimum similarity for comparison with the first threshold.

[0149] The various functional modules of the image recognition device in the above-mentioned embodiment of the present disclosure have the same principles and features as the image recognition method described in detail in the above-mentioned embodiment of the present disclosure. The specific features and principles of each functional module can be referred to the description of the above-mentioned image recognition method embodiment, and will not be repeated here.

[0150] In summary, the image recognition device provided by the exemplary embodiment of the present disclosure obtains the picture area containing text in the image to be identified through text area detection, directly extracts the picture features of the picture area, and determines whether the image to be identified is illegal through the similarity between the picture features and the illegal picture features. It can efficiently identify illegal images with abnormal text content, reduce dependence on image semantic information, and does not need to rely on the text content of the text area. It can cope with various text types through the similarity of picture features, breaking through the limitations of text recognition; in addition, it can also achieve rapid response to new attack types without retraining the model through dynamic update and maintenance of the illegal picture feature library, thereby achieving efficient, accurate and comprehensive recognition of illegal images.

[0151] Exemplary Storage Media

[0152] After introducing the image recognition method and apparatus according to the exemplary embodiment of the present disclosure, Figure 10 A storage medium according to an exemplary embodiment of the present disclosure is described.

[0153] Reference Figure 10As shown, a storage medium 1000 for implementing the above-mentioned image recognition method according to an embodiment of the present disclosure is described. The storage medium 1000 may be a portable compact disk read-only memory (CD-ROM) and includes program code, and can be run on a device such as a personal computer. However, the storage medium of the present disclosure is not limited thereto. In this document, a storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0154] The storage medium can be any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0155] A readable signal medium may include a data signal transmitted in baseband or as part of a carrier wave, which carries readable program code. Such a transmitted data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0156] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0157] The program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0158] Exemplary Computer Devices

[0159] After introducing the storage medium of the exemplary embodiment of the present disclosure, next, referring to Figure 11 A computer device according to an exemplary embodiment of the present disclosure will be described.

[0160] Figure 11 The computer device 1100 shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0161] like Figure 11 As shown, computer device 1100 is implemented as a general-purpose computing device. Components of computer device 1100 may include, but are not limited to, the aforementioned at least one processing unit 1110, the aforementioned at least one storage unit 1120, a bus 1130 connecting various system components (including storage unit 1120 and processing unit 1110), and a display unit 1140.

[0162] The storage unit 1120 stores program codes, which can be executed by the processing unit 1110 to enable the processing unit 1110 to perform the steps of the image recognition method according to various exemplary embodiments of the present disclosure described in the exemplary method section above.

[0163] The storage unit 1120 may include a volatile storage unit, such as a random access memory unit (RAM) 11201 and / or a cache memory unit 11202 , and may further include a read-only memory unit (ROM) 11203 .

[0164] The storage unit 1120 may also include a program / utility 11204 having a set (at least one) of program modules 11205, such program modules 11205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0165] The bus 1130 may include a data bus, an address bus, and a control bus.

[0166] The computer device 1100 can also communicate with one or more external devices (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), and such communication can be performed via an input / output (I / O) interface 1150. The computer device 1100 also includes a display unit 1140, which is connected to the input / output (I / O) interface 1150 for display. Furthermore, the computer device 1100 can also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1160. As shown, the network adapter 1160 communicates with other modules of the computer device 1100 via a bus 1130. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the computer device 1100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0167] It should be noted that although several modules or submodules of the image recognition device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in a single unit / module. Conversely, the features and functions of a single unit / module described above can be further divided and embodied by multiple units / modules.

[0168] Furthermore, although the operations of the disclosed method are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0169] Although the spirit and principles of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the specific embodiments disclosed, and the division into various aspects does not mean that the features in these aspects cannot be combined to benefit. Such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the appended claims.

Claims

1. An image recognition method, characterized in that: include: Performing text detection on the image to be recognized to determine a text box corresponding to a corresponding text sequence based on the detection result; In response to the plurality of text boxes, and the plurality of text boxes meeting a merging condition, merging the plurality of text boxes to obtain a target text box, and intercepting a picture region containing text from the image to be recognized based on the target text box; wherein the merging condition is determined based on a distance between the text boxes and / or a degree of overlap between the text boxes; In response to the text box being one, intercepting a picture area containing text from the image to be recognized according to the text box; Performing feature extraction on the image region to obtain image features of the corresponding image region; Determining a similarity between the image feature and a violation image feature, wherein the violation image feature is obtained from a violation image feature library; In response to the similarity being greater than a first threshold, the image to be identified is determined as an illegal image.

2. The image recognition method according to claim 1, wherein: Merging the multiple text boxes to obtain a target text box includes: Enlarging the plurality of text boxes according to a preset ratio to obtain corresponding extended text boxes; The extended text boxes are merged to obtain the target text box.

3. The image recognition method according to claim 2, wherein: The merging of the extended text boxes to obtain the target text box includes: Determine the extended text frame with the largest area as the base text frame; Determining the degree of overlap between the reference text frame and each of the other extended text frames, and determining candidate text frames based on the overlap results; The candidate text box is an extended text box among other extended text boxes, the degree of overlap between which and the reference text box exceeds a second threshold; Merging the reference text frame with the candidate text frame to obtain the target text frame; Repeat the above steps in the remaining extended text boxes until you can no longer obtain a new target text box.

4. The image recognition method according to claim 2, wherein: The merging of the extended text boxes to obtain the target text box includes: determining an extended text frame whose area is greater than a third threshold as a reference text frame; Determining the degree of overlap between each reference text frame and each associated extended text frame, and determining a candidate text frame for each reference text frame based on the overlap result; The extended text box associated with the reference text box is an extended text box whose distance from the reference text box is less than a fourth threshold; the candidate text box of the reference text box is an extended text box associated with the reference text box whose overlap with the reference text box exceeds a second threshold. Merging each of the reference text frames with its candidate text frames to obtain the target text frame; Repeat the above steps in the remaining extended text boxes until you can no longer obtain a new target text box.

5. The image recognition method according to claim 1, wherein: The extracting features from the image region includes extracting features from the image region using a feature extraction model; The network structure of the feature extraction model in the application stage includes: A backbone network is used to extract features from the image region to obtain basic features of the image region; The metric network is connected to the backbone network and is used to perform feature processing on the basic features to obtain image features of the image area.

6. The image recognition method according to claim 5, wherein: The network structure of the feature extraction model in the training phase also includes a classification network connected to the backbone network; The training phase of the feature extraction model includes: Obtain labeled samples, each labeled sample carries a metric label and / or a classification label; Inputting the labeled sample into the backbone network to obtain the metric feature corresponding to the metric label output by the metric network and the classification result corresponding to the classification label output by the classification network; Parameters of the backbone network, the metric network, and the classification network are optimized according to a total loss including a metric loss and a classification loss.

7. The image recognition method according to claim 6, wherein: The metric label is a triplet label including a sample label, a positive sample label, and a negative sample label. In a set of sample images carrying the associated triplet label: A current sample image carrying the sample label is randomly selected from a sample image set; A positive sample image carrying the positive sample label is obtained by performing data enhancement on the current sample image; The negative sample image carrying the negative sample label is selected from the remaining sample image group of the sample image set except at least the current sample image.

8. The image recognition method according to claim 7, wherein: During the training phase of the feature extraction model, the parameters of the backbone network and the metric network are optimized so that the distance between the metric features of the current sample image output by the metric network and the metric features of its positive sample image is reduced, and the distance between the metric features of the current sample image output by the metric network and the metric features of its negative sample image is increased.

9. The image recognition method according to claim 7, wherein: The metric loss of the metric features of each group of sample images is calculated using the triplet loss function.

10. The image recognition method according to claim 7, wherein: When the current sample image does not carry a similar image label, the remaining sample image group is composed of sample images in the sample image set except the current sample image; When the current sample image carries the similar image label, the positive sample image can also be randomly selected from the similar sample image group in the sample image set, and the sample images in the similar sample image group all carry the similar image label, and the remaining sample image group is composed of sample images in the sample image set except the similar sample image group.

11. The image recognition method according to claim 7, wherein: The negative sample image carrying the negative sample label is selected from the remaining sample image group of the sample image set, comprising: Extracting features of each sample image in the sample image set; Calculating the distance between the feature of the current sample image and the feature of each remaining sample image in the remaining sample image group; A remaining sample image is randomly selected from the remaining sample images whose distance is less than the distance threshold as the negative sample image.

12. The image recognition method according to claim 6, wherein: The classification labels include: The first category label is used to indicate that the corresponding sample image comes from real business; The second category label is used to characterize that the corresponding sample image is synthesized by the machine; The third category label is used to indicate that the corresponding sample image comes from an external resource platform; The fourth category label is used to indicate that the corresponding sample image does not contain text content; The sample images corresponding to the first category label, the second category label, and the third category label all contain text content.

13. The image recognition method according to claim 12, wherein: The classification loss of the sample images corresponding to the first category label, the second category label, and the third category label is calculated using a classification loss function including a cross entropy loss function and a center loss function; The classification loss of the sample image corresponding to the fourth category label is calculated using the cross entropy loss function.

14. The image recognition method according to claim 1, wherein: Determining the similarity between the image feature and the illegal image feature includes calculating the similarity between the image feature and the illegal image feature using Euclidean distance.

15. The image recognition method according to claim 1, wherein: Determining the similarity between the image feature and the illegal image feature includes: Each time determining the similarity between the image feature of an image region and a set of illegal image features, comparing the similarity with the first threshold; Wherein, a set of illegal image features is derived from an illegal image; or Determine the similarity between the image features of each image region and the features of each group of illegal images, and take the minimum similarity to compare with the first threshold.

16. An image recognition device, characterized in that: include: A text detection module, configured to perform text detection on the image to be recognized, and determine a text box corresponding to a corresponding text sequence based on the detection result; a first interception module configured to, in response to the plurality of text boxes meeting a merging condition, merge the plurality of text boxes to obtain a target text box, and intercept a picture region containing text from the image to be recognized based on the target text box; wherein the merging condition is determined based on a distance between the text boxes and / or a degree of overlap between the text boxes; a second interception module, configured to intercept a picture region containing text from the image to be recognized according to the text frame in response to the text frame being one; A feature extraction module is used to extract features from the image area to obtain image features of the corresponding image area; a feature comparison module, configured to determine similarity between features of the image and features of illegal images, wherein the features of the illegal images are obtained from a library of illegal image features; The violation identification module is configured to determine the image to be identified as a violation image in response to the similarity being greater than a first threshold.

17. The image recognition device according to claim 16, wherein: In the first interception module, the module for merging the multiple text boxes to obtain a target text box includes: A text box magnification module, configured to magnify the plurality of text boxes according to a preset ratio to obtain corresponding extended text boxes; The text frame merging module is configured to merge the extended text frames to obtain the target text frame.

18. The image recognition device according to claim 17, wherein: The text box merging module includes: A first reference frame determining module, configured to determine the extended text frame with the largest area as the reference text frame; A first candidate frame determination module is configured to determine the overlap between the reference text frame and each of the other extended text frames, and determine a candidate text frame based on the overlap result; The candidate text box is an extended text box among other extended text boxes, the degree of overlap between which and the reference text box exceeds a second threshold; A first merging module, configured to merge the reference text frame and the candidate text frame to obtain the target text frame; The first loop module is used to repeat the above steps in the remaining extended text boxes until no new target text box can be obtained in this way.

19. The image recognition device according to claim 17, wherein: The text box merging module includes: a second reference frame determining module, configured to determine an extended text frame having an area greater than a third threshold as a reference text frame; a second candidate frame determination module, configured to determine a degree of overlap between each reference text frame and each associated extended text frame, and determine a candidate text frame for each reference text frame based on the overlap result; The extended text box associated with the reference text box is an extended text box whose distance from the reference text box is less than a fourth threshold; the candidate text box of the reference text box is an extended text box associated with the reference text box whose overlap with the reference text box exceeds a second threshold. a second merging module, configured to merge each of the reference text frames with its candidate text frames to obtain the target text frame; The second loop module is used to repeat the above steps in the remaining extended text boxes until no new target text box can be obtained in this way.

20. The image recognition device according to claim 16, wherein The feature extraction module is used to extract features from the image area using a feature extraction model; The network structure of the feature extraction model in the application stage includes: A backbone network is used to extract features from the image region to obtain basic features of the image region; The metric network is connected to the backbone network and is used to perform feature processing on the basic features to obtain image features of the image area.

21. The image recognition device according to claim 20, wherein: The network structure of the feature extraction model in the training phase also includes a classification network connected to the backbone network; The modules for training the feature extraction model include: A sample acquisition module is used to obtain labeled samples, each labeled sample carries a metric label and / or a classification label; A model operation module, configured to input the label sample into the backbone network, obtain the metric feature corresponding to the metric label output by the metric network and the classification result corresponding to the classification label output by the classification network; A model adjustment module is used to optimize the parameters of the backbone network, the metric network and the classification network according to the total loss including the metric loss and the classification loss.

22. The image recognition device according to claim 21, wherein: The metric label is a triplet label including a sample label, a positive sample label, and a negative sample label. In a set of sample images carrying the associated triplet label: A current sample image carrying the sample label is randomly selected from a sample image set; A positive sample image carrying the positive sample label is obtained by performing data enhancement on the current sample image; The negative sample image carrying the negative sample label is selected from the remaining sample image group of the sample image set except at least the current sample image.

23. The image recognition device according to claim 22, wherein: The model adjustment module optimizes the parameters of the backbone network and the metric network so as to reduce the distance between the metric features of the current sample image output by the metric network and the metric features of its positive sample image, and increase the distance between the metric features of the current sample image output by the metric network and the metric features of its negative sample image.

24. The image recognition device according to claim 22, wherein: The model adjustment module is used to calculate the metric loss of the metric features of each group of sample images using a triplet loss function.

25. The image recognition device according to claim 22, wherein: When the current sample image does not carry a similar image label, the remaining sample image group is composed of sample images in the sample image set except the current sample image; When the current sample image carries the similar image label, the positive sample image can also be randomly selected from the similar sample image group in the sample image set, and the sample images in the similar sample image group all carry the similar image label, and the remaining sample image group is composed of sample images in the sample image set except the similar sample image group.

26. The image recognition device according to claim 22, wherein: The module for obtaining a negative sample image carrying the negative sample label includes: A feature acquisition module, configured to extract features of each sample image in the sample image set; a distance calculation module, configured to calculate the distance between the feature of the current sample image and the feature of each remaining sample image in the remaining sample image group; The image selection module is used to randomly select a remaining sample image from the remaining sample images whose distance is less than a distance threshold as the negative sample image.

27. The image recognition device according to claim 21, wherein: The classification labels include: The first category label is used to indicate that the corresponding sample image comes from real business; The second category label is used to characterize that the corresponding sample image is synthesized by the machine; The third category label is used to indicate that the corresponding sample image comes from an external resource platform; The fourth category label is used to indicate that the corresponding sample image does not contain text content; The sample images corresponding to the first category label, the second category label, and the third category label all contain text content.

28. The image recognition device according to claim 27, wherein: The model adjustment module calculates the classification loss of the sample images corresponding to the first category label, the second category label, and the third category label using a classification loss function including a cross entropy loss function and a center loss function; The model adjustment module uses the cross entropy loss function to calculate the classification loss of the sample image corresponding to the fourth category label.

29. The image recognition device according to claim 16, wherein: The feature comparison module uses Euclidean distance to calculate the similarity between the image feature and the illegal image feature.

30. The image recognition device according to claim 16, wherein: The feature comparison module includes: a first comparison module, configured to determine a similarity between an image feature of an image region and a set of illegal image features, and compare the similarity with the first threshold; Wherein, a set of illegal image features is derived from an illegal image; or The second comparison module is used to determine the similarity between the image features of each image area and the features of each group of illegal images, and compare the minimum similarity with the first threshold.

31. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image recognition method according to any one of claims 1 to 15 is implemented.

32. A computer device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to execute the image recognition method according to any one of claims 1 to 15 by executing the executable instructions.

Images