Method for automatically monitoring static image of display

Abstract

The invention provides an automatic monitoring method for a static image of a display, and belongs to the technical field of testing, and the method comprises the steps: monitoring an image of a monitored display through employing a network camera, and obtaining video data collected in real time; converting the obtained video data into a plurality of single-frame images, and performing de-noising gray processing on the plurality of single-frame images and the calibration image; comparing each single-frame image after denoising gray processing with the calibration image to obtain a difference area; for each single-frame image, carrying out image processing on the difference region; determining the contour position of the monitored display according to the shape characteristics of the contour of the original single-frame image, and intercepting each image after image processing; and comparing each intercepted image with the calibration image, and judging whether an abnormal image exists or not based on a preset measurement standard. According to the scheme of the invention, when the abnormal image occurs, the platform can automatically remind the record, does not occupy redundant time of a tester, and improves the working efficiency.

Description

Technical Field

[0001] The present application relates to the field of testing technology, and in particular to a method for automatically monitoring static images of a display. Background Art

[0002] With the continuous development of the aviation field, the mission system is becoming more and more complex, and the requirements for basic image display are getting higher and higher. In order to ensure the reliability of the displayed image, the displayed image must be tested. Only by testing the displayed image for a long time can we judge whether the image display of this product can meet the needs and whether the quality can be guaranteed. Therefore, automatic image monitoring, which is produced with the emergence of artificial intelligence, is an indispensable part of automatic image monitoring.

[0003] The use of monitors in actual work may inevitably encounter some quality problems. Previously, manual visual inspection and quality analysis were mainly carried out. Since the display content is large, human fatigue may occur when analyzing by the human eye. At the same time, the test lasts for a long time, and there are personnel changes, which makes it inevitable that some test conditions will be lost during the test, leading to quality problems. In addition, during the test, relevant personnel need to pay attention to the operating status all the time. In addition to personnel fatigue, it will also take up a lot of personnel execution and observation time, resulting in a shortage of available human resources. Summary of the invention

[0004] In view of this, an embodiment of the present application provides a method for automatically monitoring static images of displays, which can enable display static image display testing, detection and other tasks to be carried out unmanned around the clock, and when abnormal images occur, the platform can automatically remind and record, without taking up extra time of testers, thereby improving work efficiency.

[0005] The present application provides a method for automatically monitoring a static image on a display, the method comprising:

[0006] Use network cameras to monitor the images of the monitored display and obtain real-time collected video data;

[0007] Convert the acquired video data into multiple single-frame images, and perform denoising grayscale processing on the multiple single-frame images and the calibration image;

[0008] Each single frame image after denoising grayscale processing is compared with the calibration image to obtain the difference area;

[0009] For each single frame image, image processing is performed on the difference area;

[0010] According to the shape characteristics of the outline of the original single-frame image, the outline position of the monitored display is determined, and each image after image processing is intercepted;

[0011] Each captured image is compared with the calibration image to determine whether there are abnormal images based on preset criteria.

[0012] According to a specific implementation method of an embodiment of the present application, the image processing is performed on the difference area for each single frame image, including: for each single frame image, edge detection in the x direction, image thresholding operation, closing operation and median filtering are performed on the difference area in sequence.

[0013] According to a specific implementation of an embodiment of the present application, the closing operation includes corrosion and expansion.

[0014] According to a specific implementation of the embodiment of the present application, the preset measurement standard includes mean square error, structural similarity index and normalized cross-correlation coefficient.

[0015] According to a specific implementation of the embodiment of the present application, the calculation formula of the mean square error is:

[0016]

[0017] Among them, I1 and I2 are the grayscale values ​​of the corresponding pixels in the two images, n is the number of pixels, and MSE is the mean square error.

[0018] According to a specific implementation of the embodiment of the present application, the calculation formula of the structural similarity index is:

[0019] SSIM(x,y)=(2*ux*uy+C1)*(2*wxy+C2) / (ux 2 +uy 2 +C1)*(wx 2 +wy 2 +C2),

[0020] Among them, SSIM is the structural similarity index, x and y are the two images to be compared, ux and uy represent the means of the two images respectively, and wx 2 and wy 2 They represent the variance of the two images, wxy represents the covariance of the two images, and C1 and C2 are constants less than 1.

[0021] According to a specific implementation of the embodiment of the present application, the calculation formula of the normalized cross-correlation coefficient is:

[0022]

[0023] Where NCC is the normalized correlation coefficient, f(x, y), t(x, y) are two vectors or samples, n is the vector dimension or the size of the window, σ is the standard deviation of the corresponding sample, and μ is the mean of the corresponding sample.

[0024] According to a specific implementation of the embodiment of the present application, comparing each captured image with the calibration image and determining whether there is an abnormal image based on a preset measurement standard includes:

[0025] Setting the value range of the mean square error to a first range, setting the value range of the structural similarity index to a second range, and setting the value range of the normalized cross-correlation coefficient to a third range;

[0026] For each captured image, the mean square error, the structural similarity index and the normalized mutual correlation coefficient between the image and the calibration image are calculated. When the calculated mean square error, the structural similarity index and the normalized mutual correlation coefficient are all within the corresponding ranges, it is judged that the image is not an abnormal image; otherwise, the image is judged to be an abnormal image.

[0027] According to a specific implementation method of an embodiment of the present application, the abnormal image includes a noise image, a bright line image, a black screen image, a blue screen image, a white screen image and a flower screen image.

[0028] According to a specific implementation of the embodiment of the present application, the model of the network camera is iDS-2DF5220S-D4 / JY.

[0029] Beneficial effects:

[0030] The method for automatically monitoring static images of displays in the embodiments of the present application analyzes the current test requirements of various types of display images, forms a solution for automatically capturing static images of aircraft-mounted display images, and develops and verifies the development through the development of a network camera in Python. It can enable testing, detection and other work to be carried out around the clock, and when problems occur, the platform can automatically remind and record, without taking up extra time for testers, and improving work efficiency. It has the following beneficial effects:

[0031] 1. High efficiency of monitoring quality: In the process of debugging, troubleshooting and aging, it reduces the disadvantages of human visual observation, can accurately identify and handle the problems such as noise, bright lines, black screen, blue screen, white screen, and flower screen in the picture, and remind users of the problem;

[0032] 2. Save human resource costs: During the debugging, troubleshooting and aging process, the monitoring platform can be used for 24-hour unattended monitoring. The user only needs to open the platform at the beginning and confirm the final result at the end, and the whole process only takes 10 minutes;

[0033] 3. Improve verification methods: It can clearly remind the number and time of problems during use and record the screen when the fault occurred, which can be used directly for troubleshooting. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 is a flow chart of a method for automatically monitoring a static image on a display according to an embodiment of the present invention;

[0036] Figure 2 This is a real-time preview image of a network camera according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0038] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0039] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present application, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0040] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The illustrations only show the components related to the present application rather than being drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0041] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0042] By analyzing the current testing requirements of various types of display images, a solution for automatic static image capture of aircraft-mounted display images is formed, and development and verification are carried out through python development + opencv + network camera. It can enable testing and detection work to be carried out around the clock, and when problems occur, the platform can automatically remind and record them, which does not take up extra time for testers and improves work efficiency.

[0043] Based on the above content, the embodiment of the present application provides a method for automatically monitoring static images on a display, which is described in detail below with reference to the drawings. The method comprises the following steps:

[0044] Step 1: Use a network camera to monitor the screen of the monitored display and obtain real-time collected video data;

[0045] Step 2: convert the obtained video data into multiple single-frame images, and perform denoising grayscale processing on the multiple single-frame images and the calibration image;

[0046] Step 3, compare each single frame image after denoising grayscale processing with the calibration image to obtain the difference area;

[0047] Step 4: for each single frame image, perform image processing on the difference area;

[0048] Step 5: Determine the outline position of the monitored display according to the shape characteristics of the outline of the original single-frame image, and intercept each image after image processing;

[0049] Step 6: Compare each captured image with the calibration image and determine whether there is an abnormal image based on a preset measurement standard.

[0050] In specific implementation, image recognition technology is a technology that uses computers to analyze and understand images. It uses computer vision, pattern recognition, artificial intelligence and other related technologies to extract and match features of images, find target objects or patterns in images, and perform tasks such as classification, detection, and tracking. As image recognition technology becomes more and more mature and popular, it is widely used in various fields, especially in the fields of intelligent security. Cameras and image recognition analysis can completely replace manual observation. The method of this embodiment is as follows:

[0051] 1. According to specific usage requirements and scenarios, summarize and refine fault models, including common display image problems (noise, bright lines, black screen, blue screen, white screen, flower screen, etc.), complete the display image monitoring platform demand report. And according to the actual usage scenario, choose to develop based on Hikvision IPC network camera (model: iDS-2DF5220S-D4 / JY).

[0052] 2. Independently develop a monitoring platform, perform image processing and positioning, image recognition and analysis based on OpenCV, realize identification based on fault type, and implement real-time monitoring, storage and abnormal status reminders in the platform.

[0053] 3. Continuously improve the recognition algorithm, adjust the main parameters such as mean square error (MSE), structural similarity index (SSIM), normalized cross correlation (NCC), and improve the fault type to improve the algorithm recognition rate.

[0054] This example is based on Hikvision IPC network camera (model: iDS-2DF5220S-D4 / JY), refer to Figure 2 , using Hikvision's built-in real-time preview module, first initialize and register the IPC network camera, define the code stream callback function, decode the returned video YUV data, convert the YUV data into a jpg image and save it locally. Because image processing is time-consuming, the decoded data needs to be copied to other threads outside the callback function for processing to avoid blocking the callback and causing decoding frame loss.

[0055] In one embodiment, the performing image processing on the difference region for each single frame image includes: for each single frame image, sequentially performing edge detection in the x direction, image thresholding operation, closing operation and median filtering on the difference region.

[0056] Furthermore, the closing operation includes corrosion and expansion.

[0057] In the specific implementation, the image (single-frame image) converted from the single-frame data collected by the camera in real time and the image collected during calibration (calibration image) are first subjected to image denoising grayscale processing. Then, the difference between the real-time collected image and the calibration image is obtained, and edge detection (enhancing edge information), image thresholding operation (obtaining a binary image) and closing operation are performed on it in the x direction. The closing operation involves erosion and dilation. Erosion is like soil erosion. This operation will erode the boundary of the foreground object. The convolution kernel slides along the image. If all the pixel values ​​of the original image corresponding to the convolution kernel are 1, then the central element maintains the original pixel value, otherwise it becomes zero. Dilation is the opposite of erosion. As long as one of the pixel values ​​of the original image corresponding to the convolution kernel is 1, the pixel value of the central element is 1. After the closing operation is completed, median filtering (denoising) is performed again to obtain the contour of the original image collected by the camera in real time. Then, according to the shape characteristics of the contour, the contour position of the display is determined and the image to be processed subsequently is intercepted to prevent the image outside the display screen from affecting image recognition.

[0058] The image recognition and analysis module is used to identify abnormal images in the captured images. The reference image is the image collected after power-on during calibration. Abnormal images are found by image comparison, and three measurement criteria are mainly considered comprehensively. The first is the mean square error MSE: calculate the difference of each pixel and then find the average value. The smaller the MSE, the more similar the images are. The second is the structural similarity index (SSIM): considering the sensitivity of the human eye to image details, it can better reflect the similarity of images than MSE. The SSIM calculation formula includes comprehensive consideration of factors such as brightness, contrast, and structure. The last one is the normalized mutual correlation coefficient (NCC): mainly calculates the mutual correlation coefficient of two images. The closer it is to 1, the more similar the images are. For example, when NCC is -1, the two vectors are negatively correlated, when NCC is 0, the two vectors are uncorrelated, and when NCC is 1, the two vectors are positively correlated. Therefore, the larger the NCC value, the more similar the two vectors are, and the smaller the NCC value, the more dissimilar they are.

[0059] In one embodiment, the preset metrics include mean square error, structural similarity index and normalized cross-correlation coefficient.

[0060] In one embodiment, the calculation formula of the mean square error is:

[0061]

[0062] Among them, I1 and I2 are the grayscale values ​​of the corresponding pixels in the two images, n is the number of pixels, and MSE is the mean square error.

[0063] In one embodiment, the calculation formula of the structural similarity index is:

[0064] SSIM(x,y)=(2*ux*uy+C1)*(2*wxy+C2) / (ux 2 +uy 2 +C1)*(wx 2 +wy 2 +C2),

[0065] Among them, SSIM is the structural similarity index, x and y are the two images to be compared, ux and uy represent the means of the two images respectively, and wx 2 and wy 2 They represent the variance of the two images respectively, wxy represents the covariance of the two images, C1 and C2 are constants less than 1, which are used to avoid the situation where the denominator is 0. The value of the constant varies according to the color range and is less than 1.

[0066] In one embodiment, the calculation formula of the normalized cross-correlation coefficient is:

[0067]

[0068] Where NCC is the normalized correlation coefficient, f(x, y), t(x, y) are two vectors or samples, n is the vector dimension or the size of the window, σ is the standard deviation of the corresponding sample, and μ is the mean of the corresponding sample.

[0069] In one embodiment, comparing each captured image with the calibration image and determining whether there is an abnormal image based on a preset criterion includes:

[0070] Setting the value range of the mean square error to a first range, setting the value range of the structural similarity index to a second range, and setting the value range of the normalized cross-correlation coefficient to a third range;

[0071] For each captured image, the mean square error, the structural similarity index and the normalized mutual correlation coefficient between the image and the calibration image are calculated. When the calculated mean square error, the structural similarity index and the normalized mutual correlation coefficient are all within the corresponding ranges, it is judged that the image is not an abnormal image; otherwise, the image is judged to be an abnormal image.

[0072] In specific implementation, the specified ranges of the three parameters in the preset measurement criteria are continuously iterated through actual usage scenarios and fault models to achieve a balance. The image is judged to be abnormal based on the values ​​of MSE, SSIM, and NCC. For example, if the MSE value is within 12, the SSIM value is within the range of [0.9, 1], and the NCC value is within [0.85, 1], the image is judged to be normal. On the contrary, if any parameter value exceeds the range, the image is judged to be abnormal and an abnormality is prompted.

[0073] In one embodiment, the abnormal images include noise images, bright line images, black screen images, blue screen images, white screen images and flower screen images.

[0074] In one embodiment, the model of the network camera is iDS-2DF5220S-D4 / JY.

[0075] The operation of the operating platform for implementing the method of the present application includes the following specific steps:

[0076] 1. Run the platform. First, place the product next to the network camera. Use the platform console and real-time preview to adjust the front, back, left, and right positions of the camera so that the display screen faces the camera and occupies half or more of the entire image.

[0077] 2. Power off the monitor by default, click the "Calibrate" button, and follow the prompts to calibrate the monitor screen;

[0078] 3. After the calibration is completed, the display screen is continuously monitored. When the monitored screen is abnormal (including color change, screen change, black screen, wire pulling, etc.), the platform will prompt the abnormal screen and record it in the list box;

[0079] 4. When an abnormal picture appears in the list box, you can click the "Locate Picture" button to locate the faulty picture and observe and save it.

[0080] The automatic display image test method provided by the present invention can enable the display static image display test and detection to be carried out unmanned around the clock, and when an abnormal image occurs, the platform can automatically remind and record it, which does not take up the tester's extra time and improves work efficiency. It has the following characteristics:

[0081] 1. High efficiency of monitoring quality: In the process of debugging, troubleshooting and aging, it reduces the disadvantages of human visual observation, can accurately identify and handle problems such as noise, bright lines, black screen, blue screen, white screen, and flower screen in the picture, and remind users of the problem;

[0082] 2. Save human resource costs: During the debugging, troubleshooting and aging process, the monitoring platform can be used for 24-hour unattended monitoring. The user only needs to open the platform at the beginning and confirm the final result at the end, and the whole process only takes 10 minutes;

[0083] 3. Improve verification methods: It can clearly remind the number and time of problems during use and record the screen when the fault occurred, which can be used directly for troubleshooting.

[0084] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for automatically monitoring static images on a display, characterized in that: The method comprises: Use network cameras to monitor the images of the monitored display and obtain real-time collected video data; The obtained video data is converted into a plurality of single-frame images, and a denoising grayscale process is performed on the plurality of single-frame images and the calibration image; Each single frame image after denoising grayscale processing is compared with the calibration image to obtain the difference area; For each single frame image, image processing is performed on the difference area; According to the shape characteristics of the outline of the original single-frame image, the outline position of the monitored display is determined, and each image after image processing is intercepted; Each captured image is compared with the calibration image to determine whether there are abnormal images based on preset criteria.

2. The method for automatically monitoring static images of a display according to claim 1, characterized in that: The image processing is performed on the difference area for each single frame image, including: for each single frame image, edge detection in the x direction, image thresholding operation, closing operation and median filtering are performed on the difference area in sequence.

3. The method for automatically monitoring static images of a display according to claim 2, characterized in that: The closing operations include erosion and dilation.

4. The method for automatically monitoring static images of a display according to claim 1, characterized in that: The preset measurement criteria include mean square error, structural similarity index and normalized cross-correlation coefficient.

5. The method for automatically monitoring static images of a display according to claim 4, characterized in that: The calculation formula of the mean square error is: Among them, I1 and I2 are the grayscale values ​​of the corresponding pixels in the two images, n is the number of pixels, and MSE is the mean square error.

6. The method for automatically monitoring static images of a display according to claim 4, characterized in that: The calculation formula of the structural similarity index is: SSIM(x,y)=(2*ux*uy+C1)*(2*wxy+C2) / (ux 2 +uy 2 +C1)*(wx 2 +wy 2 +C2), Among them, SSIM is the structural similarity index, x and y are the two images to be compared, ux and uy represent the means of the two images respectively, and wx 2 and wy 2 They represent the variance of the two images, wxy represents the covariance of the two images, and C1 and C2 are constants less than 1.

7. The method for automatically monitoring static images of a display according to claim 4, characterized in that: The calculation formula of the normalized cross-correlation coefficient is: Where NCC is the normalized correlation coefficient, f(x, y), t(x, y) are two vectors or samples, n is the vector dimension or the size of the window, σ is the standard deviation of the corresponding sample, and μ is the mean of the corresponding sample.

8. The method for automatically monitoring static images of a display according to claim 4, characterized in that: The step of comparing each captured image with the calibration image and determining whether there is an abnormal image based on a preset measurement standard includes: Setting the value range of the mean square error to a first range, setting the value range of the structural similarity index to a second range, and setting the value range of the normalized cross-correlation coefficient to a third range; For each captured image, the mean square error, the structural similarity index and the normalized mutual correlation coefficient between the image and the calibration image are calculated. When the calculated mean square error, the structural similarity index and the normalized mutual correlation coefficient are all within the corresponding ranges, it is judged that the image is not an abnormal image; otherwise, the image is judged to be an abnormal image.

9. The method for automatically monitoring static images of a display according to claim 1, characterized in that: The abnormal images include noise images, bright line images, black screen images, blue screen images, white screen images and flower screen images.

10. The method for automatically monitoring static images of a display according to any one of claims 1 to 9, characterized in that: The model of the network camera is iDS-2DF5220S-D4 / JY.

Images