Display content anomaly detection method and system

Abstract

The invention relates to the technical field of image detection, in particular to a display content anomaly detection method and system, and the method can effectively isolate the influence of background noise by segmenting a to-be-displayed image and carrying out anomaly detection by taking a target image block as a unit, thereby improving the accuracy of image anomaly detection. And forming a second image block set containing the first reference image blocks of the normal category and the second reference image blocks of the abnormal category, classifying the second image block set by adopting a twin network model, and cooperatively determining an anomaly detection result of the target image block through multiple groups of second image block sets. And the generalization capability and reliability of image anomaly detection are improved, so that the accuracy of image anomaly detection is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image detection, and in particular to a method and system for detecting abnormalities in displayed content. Background Art

[0002] At present, display technology is developing rapidly, and display screens have been widely used in various public places. However, if the display screen is subjected to human error or malicious attack during use, it may cause the display screen to broadcast illegal abnormal images or videos, thereby causing adverse effects.

[0003] In the existing technology, a neural network model for classification is usually used to detect anomalies in the image to be displayed, so that anomalies can be detected in the image to be displayed before it is put on the display screen for broadcast. When the anomaly detection result is that the image is abnormal, the image is automatically blocked to avoid playback accidents.

[0004] However, there are many possible factors for image anomalies, but the number of abnormal images that can be collected is relatively small, which will lead to an imbalance in the training samples, resulting in poor generalization ability of the neural network model. Moreover, there are more possible factors for image normality, and it is difficult to improve the neural network model's ability to recognize abnormal images through effective learning of normal images. In addition, there is usually a large amount of background noise information in the image, which will include a large number of redundant features or invalid features, making it difficult to train and converge the neural network model. Even if the training of the neural network model can converge, its accuracy and generalization ability are still poor, resulting in poor accuracy in image anomaly detection.

[0005] Therefore, how to improve the accuracy of image anomaly detection has become an urgent problem to be solved. Summary of the Invention

[0006] In order to solve the above technical problems, the present invention adopts a technical solution of a display content anomaly detection method, which includes the following steps: S101 , dividing an image to be displayed into blocks according to a preset block size to obtain M target image blocks, where M is a positive integer.

[0007] S102 : For any target image block, form N first image block sets with the target image block and N preset first reference image blocks, where the first reference image blocks correspond to normal categories, and N is a positive integer.

[0008] S103 , for any second reference image block among the K preset second reference image blocks, add the second reference image block to N first image block sets to obtain N second image block sets, where K is a positive integer.

[0009] S104 , traverse all second reference image blocks in K preset second reference image blocks to obtain a set of K×N second image blocks, where the second reference image blocks correspond to abnormal categories.

[0010] S105 , for any second image block set, input the second image block set into the trained twin network model to obtain a reference category and a reference category probability corresponding to the second image block set.

[0011] S106 : If the reference categories and reference category probabilities respectively corresponding to all second image block sets do not satisfy a first preset condition, then the target image block is determined to be an abnormal image block; otherwise, the target image block is determined to be a normal image block.

[0012] S107: If any target image block is an abnormal image block, determine that the image to be displayed is an abnormal image, and generate abnormal information.

[0013] The present invention also provides a display content anomaly detection system, the display content anomaly detection system comprising: An image segmentation module is used to segment the image to be displayed according to a preset block size to obtain M target image blocks, where M is a positive integer; A first set construction module is configured to form N first image block sets by combining any target image block with N preset first reference image blocks, where the first reference image blocks correspond to a normal category and N is a positive integer; a second set construction module, configured to add any second reference image block among the K preset second reference image blocks to the N first image block sets, to obtain N second image block sets, where K is a positive integer; a set traversal module, configured to traverse all second reference image blocks in K preset second reference image blocks to obtain a set of K×N second image blocks, where the second reference image blocks correspond to abnormal categories; A category prediction module is used to input any second image block set into the trained twin network model to obtain a reference category and a reference category probability corresponding to the second image block set; an abnormal image block determining module, configured to determine that the target image block is an abnormal image block if the reference categories and the reference category probabilities corresponding to all the second image block sets respectively do not satisfy a first preset condition; otherwise, determine that the target image block is a normal image block; The abnormality information generating module is configured to determine that the image to be displayed is an abnormal image and generate abnormality information if any target image block is an abnormal image block.

[0014] The present invention has at least the following beneficial effects: by segmenting the image to be displayed and performing anomaly detection on the target image block as a unit, the influence of background noise can be effectively isolated, thereby improving the accuracy of image anomaly detection, forming a second image block set including a first reference image block of a normal category and a second reference image block of an abnormal category, and then using a twin network model to classify the second image block set. The anomaly detection result of the target image block is collaboratively determined by multiple groups of second image block sets, thereby improving the generalization ability and reliability of image anomaly detection, thereby improving the accuracy of image anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 A flowchart of a method for detecting abnormalities in display content provided in the first embodiment of the present invention; Figure 2 This is a structural diagram of a display content anomaly detection system provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0018] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It is understood that, where appropriate, the above-mentioned terms used to distinguish similar objects can be interchanged so that the present invention can also implement other embodiments other than the above-mentioned illustrated embodiments or described embodiments. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0019] Example 1 This embodiment provides a method for detecting abnormalities in displayed content. Figure 1 FIG. 1 is a flow chart of a method for detecting abnormalities in displayed content provided by an embodiment of the present invention. The method for detecting abnormalities in displayed content includes: S101, dividing the image to be displayed into blocks according to a preset block size to obtain M target image blocks, where M is a positive integer; S102, for any target image block, respectively, the target image block and N preset first reference image blocks to form N first image block sets, where the first reference image blocks correspond to a normal category, and N is a positive integer; S103, for any second reference image block among the K preset second reference image blocks, adding the second reference image block to the N first image block sets to obtain N second image block sets, where K is a positive integer; S104, traversing all second reference image blocks among K preset second reference image blocks to obtain a set of K×N second image blocks, where the second reference image blocks correspond to an abnormality category; S105: For any second image block set, input the second image block set into the trained Siamese network model to obtain a reference category and a reference category probability corresponding to the second image block set; S106, if the reference categories and reference category probabilities corresponding to all the second image block sets respectively do not satisfy the first preset condition, then determining that the target image block is an abnormal image block; otherwise, determining that the target image block is a normal image block; S107: If any target image block is an abnormal image block, determine that the image to be displayed is an abnormal image, and generate abnormal information.

[0020] The image to be displayed may refer to an image to be played on a display screen, or may refer to an image frame in video data.

[0021] The preset block size can be set by the implementer, but the number of blocks and the block size need to be considered when setting it, so that the block size can cover most image elements and ensure that the number of blocks is not too large. In this embodiment, taking the image size of 512×512 as an example, the block size can be set to 16×16 or 32×32. It should be noted that, under normal circumstances, regardless of whether the width and height of the image size are the same, the width and height of the block size are the same.

[0022] The memory may further store a first reference image block set and a second reference image block set, N first reference image blocks are randomly selected from the first reference image block set, and K second reference image blocks are randomly selected from the second reference image block set.

[0023] The reference category may be a normal category or an abnormal category, and the reference category probability may refer to the probability that the target image block predicted according to the second image block set is of the reference category.

[0024] The first image block set includes a target image block and a first reference image block of a normal category, and the second image block set includes a target image block, a first reference image block of a normal category, and a second reference image block of an abnormal category.

[0025] In a specific embodiment, the anomaly detection method further includes: Acquire a first base image; Inputting the first basic image into a trained classification model to obtain a first basic category and a first basic category probability; Dividing the first basic image according to the preset block size to obtain M first basic image blocks; For any first basic image block, perform mask processing on the first basic image block in the first basic image to obtain a first intermediate image; Inputting the first intermediate image into the trained classification model to obtain a first intermediate probability; If the difference between the first basic category probability and the first intermediate probability is greater than a preset probability threshold, the first basic image block is determined to be the first reference image block.

[0026] Among them, the first basic category is the normal category. The first basic image is input into the trained classification model to obtain a first prediction value that the first basic image belongs to the normal category and a second prediction value that the first basic image belongs to the abnormal category. The first prediction value and the second prediction value are mapped through the softmax function to obtain a first prediction probability corresponding to the first prediction value as the first basic category probability.

[0027] The first intermediate probability may refer to the predicted probability that the first intermediate image belongs to the normal category.

[0028] Specifically, when the difference between the first basic category probability and the first intermediate probability is greater than the preset probability threshold, it means that the current first basic image block makes an important contribution to the classification as a normal category. Therefore, the current first basic image block can be used as the first reference image block, thereby isolating other background noise. The preset probability threshold can be set to 0.1, and the implementer can adjust the preset probability threshold according to actual conditions.

[0029] In a specific embodiment, the anomaly detection method further includes: acquiring a second base image; Inputting the second basic image into the trained classification model to obtain a second basic category and a second basic category probability; Dividing the second basic image according to the preset block size to obtain M second basic image blocks; For any second basic image block, perform mask processing on the second basic image block in the second basic image to obtain a second intermediate image; Inputting the second intermediate image into the trained classification model to obtain a second intermediate probability; If the difference between the second basic category probability and the second intermediate probability is greater than the preset probability threshold, the second basic image block is determined to be the second reference image block.

[0030] Among them, the second basic category is the abnormal category. The second basic image is input into the trained classification model to obtain a first prediction value that the second basic image belongs to the normal category and a second prediction value that the first basic image belongs to the abnormal category. The first prediction value and the second prediction value are mapped through the softmax function to obtain the second prediction probability corresponding to the second prediction value as the second basic category probability.

[0031] The second intermediate probability may refer to the predicted probability that the second intermediate image belongs to an abnormal category.

[0032] Specifically, when the difference between the second basic category probability and the second intermediate probability is greater than the preset probability threshold, it indicates that the current second basic image block makes an important contribution to classification as an abnormal category, and therefore the current second basic image block can be used as the second reference image block.

[0033] In a specific embodiment, it is characterized in that the training process of the classification model includes: A sample set is formed by I first sample images labeled as normal categories and J second sample images labeled as abnormal categories, where I and J are both positive integers, I>>J; The classification model is trained according to the sample set and the first loss function to obtain the trained classification model.

[0034] The sample set may be used to train a classification model in batches, and the first loss function may be a binary cross entropy loss function.

[0035] In a specific embodiment, for any second image block set, inputting the second image block set into the trained Siamese network model to obtain the reference category and reference category probability corresponding to the second image block set includes: For any second image block set, inputting a target image block and a first reference image block in the second image block set into the trained Siamese network model to obtain a first reference distance between the target image block and the first reference image block; Inputting the target image block and the second reference image block into the trained Siamese network model to obtain a second reference distance between the target image block and the second reference image block; A reference category and a reference category probability corresponding to the second image block set are determined according to the first reference distance and the second reference distance.

[0036] Among them, the twin network model can include a first encoder, a second encoder and a distance measurement unit. The parameters of the first encoder and the second encoder are the same. The twin network model can be used to measure the degree of feature difference between the two input data.

[0037] The first reference distance may represent a degree of feature difference between the target image block and the first reference image block, and the second reference distance may represent a degree of feature difference between the target image block and the second reference image block.

[0038] In a specific embodiment, determining the reference category and the reference category probability corresponding to the second image block set according to the first reference distance and the second reference distance includes: calculating the sum of the first reference distance and the second reference distance as a reference comprehensive distance; If the first reference distance is greater than the second reference distance, the reference category is determined to be an abnormal category, and the ratio of the first reference distance to the reference comprehensive distance is used as the reference category probability; If the first reference distance is less than or equal to the second reference distance, the reference category is determined to be the normal category, and the ratio of the second reference distance to the reference comprehensive distance is used as the reference category probability.

[0039] When the first reference distance is greater than the second reference distance, it is considered that the feature difference between the target image block and the second reference image block is smaller, and thus the reference category is determined to be an abnormal category.

[0040] When the first reference distance is less than or equal to the second reference distance, it is considered that the feature difference between the target image block and the first reference image block is smaller, and thus the reference category is determined to be the normal category.

[0041] In a specific embodiment, the first preset condition is: The reference categories corresponding to all the second image block sets are normal categories, and the reference category probabilities corresponding to the second image block sets are all greater than a preset classification probability threshold.

[0042] In this embodiment, the classification probability threshold can be set to 0.8. The implementer can adjust the classification probability threshold according to actual conditions, but it is recommended to set it within the range of [0.8, 1] to avoid misjudgment.

[0043] It should be noted that, under normal circumstances, the twin network model does not determine the reference category probability when it is applied. This embodiment determines the reference category probability by the ratio of the smaller value of the first reference distance and the second reference distance to the reference comprehensive distance. The reason is that, based on a priori knowledge, there are far more image samples of the normal category than image samples of the abnormal category. Combined with the subsequent special training method of the twin network model in this embodiment, it can be considered that the twin network model can basically learn the feature space range formed by the features of all normal category image samples. Therefore, when the target image block is a normal category, it should be satisfied that the first reference distance is small and the second reference distance is large at the same time, so that it can be determined whether the target image block is a normal category based on the reference category probability combined with the classification probability threshold.

[0044] In a specific embodiment, the training process of the twin network model includes: Determining a plurality of sample pairs based on a plurality of first reference image blocks labeled as normal categories and a plurality of second reference image blocks labeled as abnormal categories; For any sample pair, the sample pair is input into the twin network model to obtain the sample distance; Determine a selection coefficient according to labels corresponding to two reference image blocks included in the sample pair; Calculating a training loss based on the sample distance, the selection coefficient, a preset distance reference value, and a second loss function; The twin network model is trained according to the training loss until the training loss converges to obtain the trained twin network model.

[0045] In a specific embodiment, the training loss is calculated as follows: y=(x(x+1) / 2)×(max(0,Ld)) 2 +d 2 ×((1-x)(x+1)) / 2, where x is the selection coefficient, d is the sample distance, L is the distance reference value, and y is the training loss.

[0046] The selection coefficient is determined according to the labels corresponding to the two reference image blocks included in the sample pair. This means that when the labels corresponding to the two reference image blocks included in the sample pair are different, the selection coefficient is determined to be 1, and in this case y=(max(0,Ld)) 2 , which means that when d is close to L, the training loss is small, that is, the feature distance between reference image blocks of different categories is far away, reaching near L.

[0047] When the labels corresponding to the two reference image blocks contained in the sample pair are both normal categories, the selection coefficient is determined to be -1. At this time, y=0, indicating that the feature distance between the reference image blocks of the same normal category is not constrained, so that the features of the reference image blocks of the normal category are not restricted in the range of the feature space.

[0048] When the labels corresponding to the two reference image blocks contained in the sample pair are both abnormal categories, the selection coefficient is determined to be 0. At this time, y=d 2 , indicating that the feature distance between reference image blocks of the same abnormal category should be small, so that the training loss is small, so that the features of the reference image blocks of the abnormal category are as close as possible in the feature space.

[0049] In the first embodiment of the present invention, by dividing the image to be displayed and performing anomaly detection on the target image block as a unit, the influence of background noise can be effectively isolated, thereby improving the accuracy of image anomaly detection, forming a second image block set including a first reference image block of a normal category and a second reference image block of an abnormal category, and then using a twin network model to classify the second image block set. The anomaly detection result of the target image block is collaboratively determined by multiple groups of second image block sets, thereby improving the generalization ability and reliability of image anomaly detection, thereby improving the accuracy of image anomaly detection.

[0050] Example 2 This embodiment 2 provides a display content abnormality detection system, such as Figure 2 FIG. 1 is a schematic diagram of a structure of a display content anomaly detection system provided in a second embodiment of the present invention. The display content anomaly detection system includes: The image segmentation module 201 is used to segment the image to be displayed according to a preset block size to obtain M target image blocks, where M is a positive integer; A first set building module 202 is configured to form N first image block sets with respect to any target image block and N preset first reference image blocks, where the first reference image blocks correspond to a normal class and N is a positive integer; A second set construction module 203 is configured to add any second reference image block among the K preset second reference image blocks to the N first image block sets to obtain N second image block sets, where K is a positive integer; A set traversal module 204 is configured to traverse all second reference image blocks in the K preset second reference image blocks to obtain a set of K×N second image blocks, where the second reference image blocks correspond to abnormal categories; The category prediction module 205 is used to input any second image block set into the trained twin network model to obtain a reference category and a reference category probability corresponding to the second image block set; an abnormal image block determining module 206 configured to determine that the target image block is an abnormal image block if the reference categories and reference category probabilities corresponding to all sets of second image blocks do not satisfy a first preset condition; otherwise, determine that the target image block is a normal image block; The abnormality information generating module 207 is configured to determine that the image to be displayed is an abnormal image and generate abnormality information if any target image block is an abnormal image block.

[0051] It should be noted that the specific limitations of the display content anomaly detection system can be found in the limitations of the display content anomaly detection method described above and will not be repeated here. The information interaction and execution process between the above modules, etc., are based on the same concept as the method embodiments of the present invention. Their specific functions and technical effects can be found in the method embodiments and will not be repeated here.

[0052] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any form. Although the present invention has been disclosed as above in terms of preferred embodiments, they are not intended to limit the present invention. Any technician familiar with this profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A method for detecting abnormality in display content, characterized in that: The method for detecting abnormality of displayed content includes: S101, dividing the image to be displayed into blocks according to a preset block size to obtain M target image blocks, where M is a positive integer; S102, for any target image block, respectively, the target image block and N preset first reference image blocks to form N first image block sets, where the first reference image blocks correspond to a normal category, and N is a positive integer; S103, for any second reference image block among the K preset second reference image blocks, adding the second reference image block to the N first image block sets to obtain N second image block sets, where K is a positive integer; S104, traversing all second reference image blocks among K preset second reference image blocks to obtain a set of K×N second image blocks, where the second reference image blocks correspond to an abnormality category; S105: For any second image block set, input the second image block set into the trained Siamese network model to obtain a reference category and a reference category probability corresponding to the second image block set; S106, if the reference categories and reference category probabilities corresponding to all the second image block sets respectively do not satisfy the first preset condition, then determining that the target image block is an abnormal image block; otherwise, determining that the target image block is a normal image block; S107: If any target image block is an abnormal image block, determine that the image to be displayed is an abnormal image, and generate abnormal information.

2. The method for detecting abnormality in display content according to claim 1, wherein: The anomaly detection method further includes: Acquire a first base image; Inputting the first basic image into a trained classification model to obtain a first basic category and a first basic category probability; Dividing the first basic image according to the preset block size to obtain M first basic image blocks; For any first basic image block, perform mask processing on the first basic image block in the first basic image to obtain a first intermediate image; Inputting the first intermediate image into the trained classification model to obtain a first intermediate probability; If the difference between the first basic category probability and the first intermediate probability is greater than a preset probability threshold, the first basic image block is determined to be the first reference image block.

3. The method for detecting abnormality in display content according to claim 2, wherein: The anomaly detection method further includes: acquiring a second base image; Inputting the second basic image into the trained classification model to obtain a second basic category and a second basic category probability; Dividing the second basic image according to the preset block size to obtain M second basic image blocks; For any second basic image block, perform mask processing on the second basic image block in the second basic image to obtain a second intermediate image; Inputting the second intermediate image into the trained classification model to obtain a second intermediate probability; If the difference between the second basic category probability and the second intermediate probability is greater than the preset probability threshold, the second basic image block is determined to be the second reference image block.

4. The method for detecting abnormality of display content according to claim 3, wherein: The training process of the classification model includes: A sample set is formed by I first sample images labeled as normal categories and J second sample images labeled as abnormal categories, where I and J are both positive integers, I>>J; The classification model is trained according to the sample set and the first loss function to obtain the trained classification model.

5. The method for detecting abnormality of display content according to claim 1, wherein: For any second image block set, inputting the second image block set into the trained twin network model to obtain a reference category and a reference category probability corresponding to the second image block set includes: For any second image block set, inputting a target image block and a first reference image block in the second image block set into the trained Siamese network model to obtain a first reference distance between the target image block and the first reference image block; Inputting the target image block and the second reference image block into the trained Siamese network model to obtain a second reference distance between the target image block and the second reference image block; A reference category and a reference category probability corresponding to the second image block set are determined according to the first reference distance and the second reference distance.

6. The method for detecting abnormality in display content according to claim 5, wherein: The determining, based on the first reference distance and the second reference distance, a reference category and a reference category probability corresponding to the second image block set includes: calculating the sum of the first reference distance and the second reference distance as a reference comprehensive distance; If the first reference distance is greater than the second reference distance, the reference category is determined to be an abnormal category, and the ratio of the first reference distance to the reference comprehensive distance is used as the reference category probability; If the first reference distance is less than or equal to the second reference distance, the reference category is determined to be the normal category, and the ratio of the second reference distance to the reference comprehensive distance is used as the reference category probability.

7. The method for detecting abnormality in display content according to claim 1, wherein: The first preset condition is: The reference categories corresponding to all the second image block sets are normal categories, and the reference category probabilities corresponding to the second image block sets are all greater than a preset classification probability threshold.

8. The method for detecting abnormality in display content according to claim 1, wherein: The training process of the twin network model includes: Determining a plurality of sample pairs based on a plurality of first reference image blocks labeled as normal categories and a plurality of second reference image blocks labeled as abnormal categories; For any sample pair, the sample pair is input into the twin network model to obtain the sample distance; Determine a selection coefficient according to labels corresponding to two reference image blocks included in the sample pair; Calculating a training loss based on the sample distance, the selection coefficient, a preset distance reference value, and a second loss function; The twin network model is trained according to the training loss until the training loss converges to obtain the trained twin network model.

9. The method for detecting abnormality in display content according to claim 8, wherein: The training loss is calculated as: y=(x(x+1) / 2)×(max(0,Ld)) 2 +d 2 ×((1-x)(x+1)) / 2, where x is the selection coefficient, d is the sample distance, L is the distance reference value, and y is the training loss.

10. A display content anomaly detection system, characterized in that: The display content anomaly detection system includes: An image segmentation module is used to segment the image to be displayed according to a preset block size to obtain M target image blocks, where M is a positive integer; A first set construction module is configured to form N first image block sets by combining any target image block with N preset first reference image blocks, where the first reference image blocks correspond to a normal category and N is a positive integer; a second set construction module, configured to add any second reference image block among the K preset second reference image blocks to the N first image block sets, to obtain N second image block sets, where K is a positive integer; a set traversal module, configured to traverse all second reference image blocks in K preset second reference image blocks to obtain a set of K×N second image blocks, where the second reference image blocks correspond to abnormal categories; A category prediction module is used to input any second image block set into the trained twin network model to obtain a reference category and a reference category probability corresponding to the second image block set; an abnormal image block determining module, configured to determine that the target image block is an abnormal image block if the reference categories and the reference category probabilities corresponding to all the second image block sets respectively do not satisfy a first preset condition; otherwise, determine that the target image block is a normal image block; The abnormality information generating module is configured to determine that the image to be displayed is an abnormal image and generate abnormality information if any target image block is an abnormal image block.

Images