A fault detection method and device for data acquisition terminal based on artificial intelligence

Abstract

The present invention discloses a fault detection method and device for a data acquisition terminal based on artificial intelligence. The present invention is based on an image captured by a camera, and utilizes information such as grayscale, color, gradient, and inter-frame difference between adjacent frame images in the image to identify camera video quality faults and lens damage faults; at the same time, combined with machine vision technology, target detection is performed on the image captured after the camera is rotated, so as to utilize the target detection result to identify the pan / tilt damage fault of the camera; finally, the fault detection result of the camera can be obtained using the aforementioned different types of fault identification results; thus, the present invention combines image processing and machine vision technology to realize camera fault detection, which not only reduces the workload, but also improves the detection efficiency and real-time detection compared to traditional technologies, and is therefore very suitable for large-scale application and promotion.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a fault detection method and device for a data acquisition terminal based on artificial intelligence. Background Art

[0002] With the rapid development of Internet of Things technology, more and more Internet of Things devices have been applied to sanitation work. Therefore, smart sanitation has come into being. Among them, smart sanitation is a technology that relies on Internet of Things technology and mobile Internet technology to conduct real-time management of the entire process of people, vehicles, objects and things involved in sanitation management. It can reasonably plan the sanitation management model, improve the quality of sanitation operations, and reduce sanitation operating costs. In actual applications, all smart sanitation services are deployed on the smart city management cloud and connected to the smart city network. It provides managers and operators with the required services at any time in the form of cloud services. Therefore, it greatly improves the convenience and efficiency of sanitation operations.

[0003] Among the services provided by smart sanitation, a video surveillance center can usually be provided for the sanitation management department. The video surveillance center can conduct multi-dimensional scene monitoring of the streets and realize visual management of street sanitation, thereby assisting the sanitation management department in scheduling sanitation work. In actual application, the video surveillance center is built based on camera terminals arranged at various monitoring points. As the demand for video surveillance centers continues to grow, the monitoring range also continues to expand, and the number of cameras also increases. Therefore, in order to ensure the reliable operation of the video surveillance center, the staff needs to promptly grasp the working status of the front-end camera equipment so as to deal with it in time when a failure occurs.

[0004] At present, manual methods are mostly used to detect the working status of cameras, that is, the video images of each camera are called out, the imaging quality of the camera is manually judged, and the inspection and analysis results are recorded to form an operation and maintenance report; therefore, the aforementioned existing technology has the following shortcomings: large workload, low efficiency and poor real-time performance, and equipment failures cannot be discovered in time; based on this, how to provide a fault detection method with high efficiency, low workload and high real-time performance has become a problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the present invention is to provide a fault detection method and device for a data acquisition terminal based on artificial intelligence, so as to solve the problems of large workload, low efficiency and poor real-time performance that cannot be solved by the prior art.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, a fault detection method for a data acquisition terminal based on artificial intelligence is provided, comprising:

[0008] Acquire a first monitoring image, a second monitoring image, and a third monitoring image captured by a target camera, wherein the first monitoring image and the second monitoring image are two adjacent frames of images in the monitoring video of the target camera, and the third monitoring image is an image captured after the target camera rotates according to the rotation instruction;

[0009] Performing color conversion processing on the first monitoring image or the second monitoring image to obtain a color converted image;

[0010] grayscale the first monitoring image and the second monitoring image to obtain a first grayscale image and a second grayscale image;

[0011] constructing an inter-frame difference image between the first grayscale image and the second grayscale image based on the first grayscale image and the second grayscale image;

[0012] According to the color conversion image, the first grayscale image and the inter-frame difference image, a first fault type identification process is performed on the target camera to obtain a first fault identification result of the target camera, wherein the first fault type includes a lens damage fault and a video quality fault;

[0013] Performing target detection processing on the third monitoring image to obtain a target detection result, and based on the target detection result, performing second fault type identification processing on the target camera to obtain a second fault identification result of the target camera, wherein the second fault type includes a pan / tilt damage fault;

[0014] The first fault identification result and the second fault identification result are used to obtain a fault detection result of the target camera.

[0015] Based on the above disclosed content, the present invention first obtains monitoring images of two adjacent frames in the monitoring video taken by the target camera, and the image taken after the position is rotated according to the rotation instruction; then, any image in the aforementioned two adjacent frames of monitoring images is color converted to obtain a color converted image; then, the monitoring images of the two adjacent frames are gray-scale transformed to obtain a corresponding gray-scale image; then, based on the two gray-scale images, an inter-frame difference image is obtained; in this way, the automatic fault identification of the target camera can be realized based on the aforementioned images, and the process is: first, based on the color conversion image, the first gray-scale image and the inter-frame difference image, the video quality fault identification and lens damage fault identification of the target camera are performed to obtain a first fault identification result; then, using machine vision technology, the image taken after the position is rotated is subjected to target detection to obtain a target detection result; then, based on the target detection result, the camera's pan / tilt damage fault is identified to obtain a second fault identification result; finally, the fault detection result of the target camera can be obtained based on the aforementioned first and second fault identification results.

[0016] Through the above design, the present invention identifies camera video quality faults and lens damage faults based on the images captured by the camera, and uses information such as grayscale, color, and frame differences between adjacent frame images in the image; at the same time, combined with machine vision technology, it performs target detection on the images captured after the camera is rotated, and then uses the target detection results to identify the camera's pan / tilt damage fault; finally, using the aforementioned different types of fault identification results, the camera's fault detection results can be obtained; thus, the present invention combines image processing and machine vision technology to automatically realize camera fault detection. Compared with traditional technologies, it not only reduces the workload, but also improves detection efficiency and real-time detection. Therefore, it is very suitable for large-scale application and promotion.

[0017] In a possible design, performing a first fault type identification process on the target camera according to the color conversion image, the first grayscale image, and the inter-frame difference image includes:

[0018] Based on the color conversion image, calculating the color cast characteristic value of the target camera;

[0019] According to the first grayscale image, a noise interference characteristic value, a brightness characteristic value, a clarity characteristic value, and a signal clutter characteristic value of the target camera are calculated, and jitter characteristic information of the target camera is determined;

[0020] Using the inter-frame difference image, determining the picture freezing characteristic value and the water ripple characteristic value of the target camera;

[0021] Based on the color cast characteristic value, the noise interference characteristic value, the brightness characteristic value, the clarity characteristic value, the signal clutter characteristic value, the jitter characteristic information, the picture freezing characteristic value, and the water ripple characteristic value, a video quality fault detection process is performed on the target camera to obtain a video quality fault detection result;

[0022] Using a machine vision-based damage detection algorithm and utilizing the first grayscale image, a lens damage detection process is performed on the target camera to obtain a lens damage fault detection result;

[0023] Based on the video quality fault detection result and the lens damage fault detection result, a first fault identification result of the target camera is obtained.

[0024] In a possible design, the color conversion image is a Lab image, wherein the color cast characteristic value of the target camera is calculated based on the color conversion image, including:

[0025] Calculating the mean and variance of the a color component and the mean and variance of the b color component in the color conversion image;

[0026] Based on the mean and variance of the a color component and the mean and variance of the b color component, and according to the following formula (1), the color cast characteristic value of the target camera is calculated;

[0027]

[0028] In the above formula (1), color p represents the color cast feature value, represents the mean of the a color component and the mean of the b color component, var a ,var b They represent the variance of the a color component and the variance of the b color component respectively, and γ represents a constant greater than 0.

[0029] In a possible design, calculating the noise interference characteristic value and the brightness characteristic value of the target camera according to the first grayscale image includes:

[0030] Calculating the grayscale mean of all pixels in the first grayscale image, and determining the brightness characteristic value of the target camera based on the grayscale mean;

[0031] Performing filtering on the first grayscale image to obtain a filtered image;

[0032] Subtracting the filtered image from the first grayscale image to obtain a difference image, and calculating the grayscale variance of all pixels in the difference image;

[0033] According to the grayscale variance of all pixels in the difference image, the noise interference characteristic value is calculated using the following formula (2);

[0034]

[0035] In the above formula (2), Z1 represents the noise interference characteristic value, σ1 represents the grayscale variance of all pixels in the difference image, Represents the characteristic factor.

[0036] In a possible design, calculating the clarity feature value and the signal clutter feature value of the target camera according to the first grayscale image includes:

[0037] Obtaining a first gradient histogram in the horizontal direction and a second gradient histogram in the vertical direction of the first grayscale image, and screening out a histogram with the largest gradient value from the first gradient histogram and the second gradient histogram as a target histogram;

[0038] Counting the number of pixel points whose gradient values ​​are less than or equal to a first gradient threshold from the target histogram, and calculating the gradient mean of each pixel point whose gradient value is greater than the first gradient threshold;

[0039] Filtering out the number of pixel points whose gradient values ​​are less than a second gradient threshold from the second gradient histogram;

[0040] The clarity feature value is calculated based on the gradient mean and the number of pixels whose gradient values ​​are less than or equal to the first gradient threshold value, and the signal clutter feature value is calculated based on the number of pixels whose gradient values ​​are less than the second gradient threshold value, and the following formula (4) is used;

[0041]

[0042] In the above formula (3), Q represents the clarity characteristic value, represents the gradient mean, r1 represents the number of pixels whose gradient values ​​are less than or equal to the first gradient threshold, and M and N represent the width and height of the first grayscale image respectively;

[0043] In the above formula (4), Z2 represents the signal clutter characteristic value, and r2 represents the number of pixel points whose gradient values ​​are less than the second gradient threshold.

[0044] In a possible design, determining the jitter feature information of the target camera according to the first grayscale image includes:

[0045] Performing Fourier transform processing on the first grayscale image to obtain a Fourier transform signal;

[0046] Based on the Fourier transform signal, deriving amplitude information and phase information of the signal;

[0047] Using the amplitude information and phase information of the signal to form the jitter characteristic information;

[0048] Correspondingly, using the inter-frame difference image to determine the picture freezing feature value and the water ripple feature value of the target camera includes:

[0049] Based on the inter-frame difference image, constructing a grayscale histogram of the inter-frame difference image;

[0050] According to the grayscale histogram, counting the number of pixels whose grayscale is less than or equal to the grayscale threshold and the number of pixels whose grayscale is within the calibration range;

[0051] The image freezing characteristic value is calculated according to the number of pixels whose grayscale is less than or equal to the grayscale threshold, and the water ripple characteristic value is calculated according to the number of pixels whose grayscale is within the calibration range.

[0052] In a possible design, a damage detection algorithm based on machine vision is adopted, and the first grayscale image is used to perform lens damage detection processing on the target camera to obtain a lens damage fault detection result, including:

[0053] Performing Gaussian smoothing on the first grayscale image to obtain a Gaussian smoothed image;

[0054] Based on the Gaussian smoothed image, determining a grayscale distribution characteristic value of the Gaussian smoothed image;

[0055] Calculating a spatially adjacent Gaussian factor according to the grayscale distribution characteristic value;

[0056] Obtaining a convolution kernel, and performing a convolution operation on the Gaussian smoothed image using the convolution kernel to obtain a convolution image;

[0057] Based on the convolution image, a pixel value distribution Gaussian factor is calculated;

[0058] Using the spatial proximity Gaussian factor and the pixel value distribution Gaussian factor, filtering the Gaussian smoothed image to obtain a filtered Gaussian smoothed image;

[0059] Divide the filtered Gaussian smoothed image into a number of detection sub-images, and determine the grayscale threshold of each detection sub-image;

[0060] The lens damage fault detection result is determined according to each detection sub-image and the grayscale threshold of each detection sub-image.

[0061] In a possible design, the target detection result is the presence or absence of a calibration reference object in the third monitoring image, wherein the calibration reference object is obtained by performing target detection on a calibration image, and the calibration image is obtained by photographing the target camera when it rotates to a calibration position;

[0062] Wherein, based on the target detection result, performing a second fault type identification process on the target camera to obtain a second fault identification result of the target camera includes:

[0063] Determining whether the target detection result indicates that a calibration reference object exists;

[0064] If yes, obtaining the position of the calibration reference object in the third monitoring image, and obtaining the position of the calibration reference object in the calibration image;

[0065] Based on the position of the calibration reference object in the third monitoring image and the position of the calibration reference object in the calibration image, a second fault recognition result of the target camera is determined.

[0066] In a second aspect, a fault detection device for a data acquisition terminal based on artificial intelligence is provided, comprising:

[0067] an acquisition unit, configured to acquire a first surveillance image, a second surveillance image, and a third surveillance image captured by a target camera, wherein the first surveillance image and the second surveillance image are two adjacent frames of images in a surveillance video of the target camera, and the third surveillance image is an image captured after the target camera rotates in accordance with a rotation instruction;

[0068] An image processing unit, configured to perform color conversion processing on the first monitoring image or the second monitoring image to obtain a color converted image;

[0069] An image processing unit, used for performing grayscale processing on the first monitoring image and the second monitoring image to obtain a first grayscale image and a second grayscale image;

[0070] The image processing unit is further used to construct an inter-frame difference image between the first grayscale image and the second grayscale image based on the first grayscale image and the second grayscale image;

[0071] a fault identification unit, configured to perform a first fault type identification process on the target camera according to the color conversion image, the first grayscale image and the inter-frame difference image, to obtain a first fault identification result of the target camera, wherein the first fault type includes a lens damage fault and a video quality fault;

[0072] a fault identification unit, configured to perform target detection processing on the third monitoring image to obtain a target detection result, and based on the target detection result, perform second fault type identification processing on the target camera to obtain a second fault identification result of the target camera, wherein the second fault type includes a pan / tilt damage fault;

[0073] The fault identification unit is further used to obtain a fault detection result of the target camera using the first fault identification result and the second fault identification result.

[0074] According to a third aspect, another fault detection device for a data acquisition terminal based on artificial intelligence is provided. Taking the device as an electronic device as an example, the device includes a memory, a processor and a transceiver which are communicatively connected in sequence, wherein the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer program to execute the fault detection method for the data acquisition terminal based on artificial intelligence as in the first aspect or any possible design in the first aspect.

[0075] In a fourth aspect, a storage medium is provided, on which instructions are stored. When the instructions are executed on a computer, the fault detection method for the artificial intelligence-based data acquisition terminal as described in the first aspect or any possible design of the first aspect is executed.

[0076] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, causes the computer to execute the fault detection method for a data acquisition terminal based on artificial intelligence as described in the first aspect or any possible design of the first aspect.

[0077] Beneficial effects:

[0078] (1) The present invention is based on the image captured by the camera, and uses information such as grayscale, color, and inter-frame differences between adjacent frame images to identify camera video quality faults and lens damage faults; at the same time, combined with machine vision technology, it performs target detection on the image captured after the camera is rotated, and then uses the target detection results to identify the camera's pan / tilt damage fault; finally, using the aforementioned different types of fault identification results, the camera's fault detection results can be obtained; thus, the present invention combines image processing and machine vision technology to automatically realize camera fault detection. Compared with traditional technologies, it not only reduces the workload, but also improves detection efficiency and real-time detection. Therefore, it is very suitable for large-scale application and promotion. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 A schematic diagram of the steps of a fault detection method for a data acquisition terminal based on artificial intelligence provided by an embodiment of the present invention;

[0080] Figure 2 A schematic diagram of the structure of a fault detection device for a data acquisition terminal based on artificial intelligence provided by an embodiment of the present invention;

[0081] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0082] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0083] It should be understood that although the terms first, second, etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another unit. For example, a first unit can be referred to as a second unit, and similarly, a second unit can be referred to as a first unit without departing from the scope of the exemplary embodiments of the present invention.

[0084] It should be understood that the term "and / or" that may appear in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, B exists alone, and A and B exist at the same time. The term " / and" that may appear in this article describes another type of association object relationship, indicating that two relationships may exist. For example, A / and B can represent two situations: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this article generally indicates that the previous and next associated objects are in an "or" relationship.

[0085] Example:

[0086] See also Figure 1As shown, the fault detection method for the data acquisition terminal based on artificial intelligence provided in this embodiment is based on the image captured by the camera, and uses the grayscale, color, gradient and inter-frame difference between adjacent frame images in the image to identify the camera video quality fault and lens damage fault; at the same time, combined with machine vision technology, the image captured after the camera is rotated is used for target detection, so as to use the target detection result to identify the pan / tilt damage fault of the camera; finally, using the above-mentioned different types of fault identification results, the fault detection result of the camera can be obtained; thus, the method combines image processing and machine vision technology to realize camera fault detection, which not only reduces the workload, but also improves the detection efficiency and real-time detection compared with traditional technologies. Therefore, it is very suitable for large-scale application and promotion; among them, for example, the method can be but not limited to running on the server side of smart sanitation. It can be understood that the above-mentioned execution subject does not constitute a limitation on the embodiments of the present application. Accordingly, the operation steps of the method can be but not limited to the following steps S1 to S7.

[0087] S1. Obtain a first monitoring image, a second monitoring image and a third monitoring image captured by a target camera, wherein the first monitoring image and the second monitoring image are two adjacent frames of images in the monitoring video of the target camera, and the third monitoring image is an image captured by the target camera after the position is rotated according to the rotation instruction; in this embodiment, each target camera used for sanitation monitoring can communicate with a server to upload monitoring videos, and then the server can obtain the video stream captured by the target camera; then, the video stream is processed frame by frame, and images of two adjacent frames are taken as the aforementioned first monitoring image and the second monitoring image; at the same time, after completing the aforementioned first and second monitoring images After the image is collected, the server can send a rotation instruction to the target camera, so that the target camera rotates its position after receiving the rotation instruction, and then captures the image after the rotation to obtain the third monitoring image; of course, the video stream can also be captured after the position rotation, and then the first frame image is taken as the third monitoring image; in this way, the aforementioned first monitoring image and second monitoring image are used to identify the video quality failure and lens damage failure of the target camera, while the third monitoring image is used to identify the pan / tilt damage failure of the target camera, that is, if the pan / tilt of the target camera is damaged, there must be an error in its position rotation. Based on this, the third monitoring image can be compared with the calibration image to obtain the corresponding fault identification result.

[0088] Optionally, the fault detection process of the target camera may be but is not limited to the following steps S2 to S7.

[0089] S2. Perform color conversion on the first monitoring image or the second monitoring image to obtain a color conversion image; in this embodiment, the first monitoring image or the second monitoring image is converted into Lab space, and then the converted Lab image (i.e., the color conversion image) is used as one of the data bases for video quality fault identification; at the same time, this embodiment also needs to grayscale the first monitoring image and the second monitoring image, and calculate the frame-to-frame difference between the two; finally, the frame-to-frame difference, the first grayscale image (which can also be the second grayscale image) and the aforementioned color conversion image are used to identify the video quality fault and lens damage fault of the target camera; wherein, the aforementioned grayscale process can be but is not limited to as shown in the following step S3.

[0090] S3. grayscale the first monitoring image and the second monitoring image to obtain a first grayscale image and a second grayscale image; in specific applications, for example but not limited to, the following formula can be used to obtain the grayscale value of each pixel in the first monitoring image and the second monitoring image, thereby completing the grayscale processing of the two.

[0091] Taking any pixel in the first monitoring image as an example, its grayscale conversion formula is:

[0092] Y=R×0.3+G×0.59+B×0.11 (5)

[0093] In formula (5), R, G, and B are the red component value, green component value, and blue component value of any pixel point, respectively, and Y represents the grayscale value of any pixel point.

[0094] In this way, through the above formula (5), the grayscale of the first monitoring image and the second monitoring image can be completed, thereby obtaining corresponding grayscale images; and after obtaining the corresponding grayscale images of the two, the inter-frame difference images of the two can be constructed based on this, and the process can be but not limited to as shown in the following step S4.

[0095] S4. Based on the first grayscale image and the second grayscale image, construct an inter-frame difference image between the first grayscale image and the second grayscale image; in this embodiment, the inter-frame difference image is expressed as: f′(x,y)=|f1(x,y)-f2(x,y)|, wherein f′(x,y), f1(x,y), f2(x,y) respectively represent the inter-frame difference image, the first grayscale image and the second grayscale image, || represents the absolute value, and the subtraction of the two images in the formula is actually the subtraction of the grayscale values ​​of the pixels at the corresponding positions; in this way, based on the above formula, the inter-frame difference image can be obtained; then, the above color conversion image and the first grayscale image can be combined to perform video quality fault identification and lens damage fault identification of the target camera; wherein the identification process can be but is not limited to the following step S5.

[0096] S5. Perform a first fault type identification process on the target camera according to the color conversion image, the first grayscale image and the inter-frame difference image to obtain a first fault identification result of the target camera, wherein the first fault type includes a lens damage fault and a video quality fault; in specific applications, examples of video quality faults may include but are not limited to: color cast fault, noise interference fault, shooting brightness fault, shooting clarity fault, no signal fault, black screen fault, jitter fault and transmission fault, etc., and a lens damage fault refers to the coating damage caused by external factors (such as high temperature, timeout, scratches) when the lens of the target camera is used for a long time; further, examples may include but are not limited to the following steps S51 to S56 to obtain the first fault identification result of the target camera.

[0097] S51. Based on the color conversion image, the color cast characteristic value of the target camera is calculated; in this embodiment, if the target camera has poor line contact, external interference and other reasons, it may cause a single color cast of the image or a banded color cast fault of multiple colors. Therefore, this embodiment calculates the color cast characteristic value of the target camera based on the color conversion image, so as to identify the color cast fault in the video quality fault according to the color cast characteristic value. Of course, the identification process is described in detail below; at the same time, the specific calculation process of the aforementioned color cast characteristic value can be but is not limited to: first calculate the mean and variance of the a color component and the mean and variance of the b color component in the color conversion image; then, based on the mean and variance of the a color component and the mean and variance of the b color component, calculate the color cast characteristic value of the target camera.

[0098] Optionally, the color cast characteristic value may be calculated by, for example but not limited to, using the following formula (1).

[0099]

[0100] In the above formula (1), color p represents the color cast characteristic value, represents the mean of the a color component and the mean of the b color component, var a ,var b They represent the variance of the a color component and the variance of the b color component respectively, and γ represents a constant greater than 0. In the present embodiment, γ is less than 0.1, and its specific value can be set according to actual use and is not specifically limited here.

[0101] In this way, based on the aforementioned formula (1), the color cast characteristic value of the target camera can be obtained; then, the remaining fault features can be extracted according to the first grayscale image, so as to subsequently identify the video quality fault based on the extracted fault features; wherein, the extraction process of the remaining fault features can be but is not limited to as shown in the following step S52.

[0102] S52. According to the first grayscale image, the noise interference characteristic value, brightness characteristic value, clarity characteristic value and signal clutter characteristic value of the target camera are calculated, and the jitter characteristic information of the target camera is determined; in the present embodiment, for the noise interference characteristic value, for example, the grayscale mean of all pixels in the first grayscale image can be calculated first; then, based on the grayscale mean, the brightness characteristic value of the target camera is determined; specifically, the brightness characteristic value can be obtained by multiplying the brightness characteristic value by 100 and then dividing it by 255, but is not limited to; in this way, the brightness fault of the shooting can be identified according to the brightness characteristic value.

[0103] At the same time, in this embodiment, the noise interference characteristic value is mainly used to identify whether the target camera has a noise interference fault; wherein, when there are factors such as electromagnetic interference, messy flying points, thorns, lines and other interferences will appear in the image taken by the target camera, and these phenomena are all regarded as noise phenomena; in this way, the noise interference characteristic value in the first grayscale image can be extracted to identify the aforementioned fault.

[0104] In this embodiment, the noise interference characteristic value may be obtained by, for example but not limited to, using the following steps S52a to S52c.

[0105] S52a. Filter the first grayscale image to obtain a filtered image. In this embodiment, the first grayscale image may be filtered by, but not limited to, a median filtering method to obtain a filtered image. Then, a difference image may be obtained based on the filtered image and the first grayscale image. The process may be, but not limited to, as shown in the following step S52b.

[0106] S52b. Use the first grayscale image to subtract the filtered image to obtain a difference image, and calculate the grayscale variance of all pixels in the difference image; in this embodiment, the first grayscale image is subtracted from the filtered image, which is actually the subtraction of the grayscale values ​​of the pixels at corresponding positions; in this way, after obtaining the difference image, the grayscale variance of the difference image can be calculated, so that the noise interference characteristic value can be calculated based on the grayscale variance later, and the process can be but not limited to the following step S52c.

[0107] S52c. Calculate the noise interference characteristic value based on the grayscale variance of all pixels in the difference image and using the following formula (2).

[0108]

[0109] In the above formula (2), Z1 represents the noise interference characteristic value, σ1 represents the grayscale variance of all pixels in the difference image, Indicates the characteristic factor; in a specific application, the characteristic factor can be, but is not limited to, set to 100.

[0110] In this way, the principle of the aforementioned steps S52a~S52c is: based on the difference operation results of the images before and after denoising, to analyze the image noise anomaly, so as to obtain the corresponding noise interference characteristic value; based on this, the noise interference characteristic value can be used to identify the noise interference fault of the target camera; of course, the specific identification process is explained below.

[0111] Furthermore, the aforementioned clarity feature value is used to identify the shooting clarity failure of the target camera; and the signal noise feature value is used to identify the no-signal failure. Among them, when the target camera has a focusing problem, it may cause image blur, which is reflected in the image as a problem of weakened edge clarity and loss of details in the high-frequency area. Therefore, the clarity feature value can be extracted based on the gradient information of the aforementioned first grayscale image. Similarly, when the target camera has no signal, white noise and black and white stripes will appear in the image it captures. In this way, the entire image is equivalent to horizontal noise with no signal. Therefore, based on the aforementioned reasons, the signal noise feature value and clarity feature value in the first grayscale image can be extracted, so that the no-signal failure and shooting clarity failure can be identified based on the aforementioned feature values.

[0112] In a specific implementation, for example but not limited to, the following steps S52d to S52g may be used to determine the clarity characteristic value and the signal clutter characteristic value of the target camera.

[0113] S52d. Obtain a first gradient histogram in the horizontal direction and a second gradient histogram in the vertical direction of the first grayscale image, and select a histogram with the largest gradient value from the first gradient histogram and the second gradient histogram as a target histogram; in specific applications, the gradient histogram is a commonly used technology for image processing, and its acquisition process is no longer repeated; at the same time, this embodiment is equivalent to determining the maximum gradient value from the gradient values ​​of the first grayscale image in the horizontal and vertical directions, and then, using the histogram corresponding to the direction of the maximum gradient value as the target histogram; for example, if the maximum gradient value appears in the gradient in the horizontal direction, then the first gradient histogram is used as the target histogram.

[0114] After the histogram in the direction of the maximum gradient value is screened, the gradient may be screened to filter out the pixels that cannot represent edge information. The process may be, but is not limited to, as shown in the following step S52e.

[0115] S52e. Count the number of pixel points whose gradient values ​​are less than or equal to the first gradient threshold from the target histogram, and calculate the gradient mean of each pixel point whose gradient value is greater than the first gradient threshold; in this embodiment, the abscissa of the target histogram is equivalent to the gradient value, and the ordinate is the distribution number of the gradient value. For example, if the abscissa is 1 and the ordinate is 5, it means that there are 5 gradients with a gradient value of 1, that is, there are 5 pixels with a gradient of 1; of course, the above examples are only to help understanding and are not regarded as limitations on this embodiment; at the same time, the first gradient threshold can be, but is not limited to, set to 20; in this way, the number of pixel points with a gradient value less than or equal to 20 can be counted from the target histogram; then, the mean of all remaining gradients is calculated to obtain the gradient mean.

[0116] After the gradient mean is obtained, the gradient can be screened again from the second gradient histogram so as to subsequently extract the signal clutter feature value; wherein the aforementioned screening process can be but is not limited to as shown in the following step S52f.

[0117] S52f. Filter out the number of pixel points whose gradient values ​​are less than the second gradient threshold from the second gradient histogram; in the present embodiment, the second gradient threshold may be, but is not limited to, 50; thus, after completing the aforementioned gradient screening in the target histogram and the second gradient histogram, the aforementioned clarity eigenvalue and signal clutter eigenvalue may be determined based on this; wherein, the calculation process of the aforementioned eigenvalue may be, but is not limited to, as shown in the following step S52g.

[0118] S52g. Based on the gradient mean and the number of pixels whose gradient values ​​are less than or equal to the first gradient threshold, the clarity characteristic value is calculated using the following formula (3), and based on the number of pixels whose gradient values ​​are less than the second gradient threshold, the signal clutter characteristic value is calculated using the following formula (4).

[0119]

[0120] In the above formula (3), Q represents the clarity characteristic value, represents the gradient mean, r1 represents the number of pixels whose gradient values ​​are less than or equal to the first gradient threshold, and M and N represent the width and height of the first grayscale image respectively.

[0121] In the above formula (4), Z2 represents the signal clutter characteristic value, and r2 represents the number of pixel points whose gradient values ​​are less than the second gradient threshold.

[0122] Thus, through the aforementioned steps S52d to S52g, the clarity characteristic value and the signal clutter characteristic value of the target camera can be calculated; then, the jitter characteristic information of the target camera can be extracted, and the process is as follows.

[0123] In specific applications, when the lens shakes, it is manifested in the image as periodic small shakes of the entire image. According to the displacement theorem, as long as the characteristic information of the target (i.e., the photographed object) itself does not change, then after Fourier transform, the amplitude of the signal does not change, but the phase will change. Therefore, based on the above principle, this embodiment performs Fourier transform processing on the first grayscale image to obtain a Fourier transform signal; then, based on the Fourier transform signal, the amplitude information and phase information of the signal are obtained; finally, the amplitude information and phase information of the signal can be used to form the jitter characteristic information.

[0124] Furthermore, in this embodiment, when performing video quality fault identification, it is also necessary to extract the image freeze feature value and the water ripple feature value of the target camera, and the process can be but is not limited to the following step S53.

[0125] S53. Using the inter-frame difference image, determine the screen freeze feature value and water ripple feature value of the target camera; in this embodiment, the screen freeze feature value is used to detect the black screen or blue screen fault of the target camera, while the water ripple feature value is used to detect the transmission fault of the target camera; wherein, when the target camera has a black screen or a blue screen, the two adjacent frames of the image will not change, therefore, the corresponding features can be extracted based on the inter-frame difference image; similarly, when a transmission fault occurs (such as caused by aging of the transmission line), there will be water ripple interference in the image captured by the target camera, therefore, the traces of the water ripples can be extracted by monitoring the difference image of two adjacent frames of the video (i.e., the inter-frame difference), thereby obtaining the corresponding feature value; wherein, for example, but not limited to, the following steps S53a to S53c can be used to calculate the aforementioned screen freeze feature value and water ripple feature value.

[0126] S53a. Based on the inter-frame difference image, construct a grayscale histogram of the inter-frame difference image; in the present embodiment, it is equivalent to performing statistics on the grayscale in the inter-frame difference image to form a grayscale histogram with the horizontal axis being the grayscale level and the vertical axis being the number of pixels corresponding to the grayscale level; then, based on the grayscale histogram, pixel points can be screened, thereby calculating the picture freezing feature value and the water ripple feature value based on the number of screened pixels.

[0127] The pixel selection process may be, but is not limited to, as shown in the following step S53b.

[0128] S53b. According to the grayscale histogram, the number of pixels whose grayscale is less than or equal to the grayscale threshold and the number of pixels whose grayscale is within the calibration range are counted; in the present embodiment, the grayscale threshold may be, but not limited to, 5, and the calibration range may be, but not limited to, [5, 10]; thus, after obtaining the number of pixels whose grayscale is less than or equal to the grayscale threshold and the number of pixels whose grayscale is within the calibration range based on the aforementioned screening criteria, the aforementioned features may be calculated based on this, and the process may be, but not limited to, as shown in the following step S53c.

[0129] S53c. Calculate the image freeze characteristic value according to the number of pixels whose grayscale is less than or equal to the grayscale threshold, and calculate the water ripple characteristic value according to the number of pixels whose grayscale is within the calibration range.

[0130] In this embodiment, the picture freezing feature value may be calculated by, for example but not limited to, using the following formula (6).

[0131]

[0132] In the above formula (6), D represents the image freezing characteristic value, and r3 is the number of pixels whose grayscale is less than or equal to the grayscale threshold.

[0133] Similarly, the water ripple characteristic value can be calculated by, for example but not limited to, using the following formula (7).

[0134]

[0135] In the above formula (7), S represents the water ripple characteristic value, and r4 represents the number of pixels whose grayscale is within the calibration range.

[0136] In this way, based on the aforementioned S53a~S53c, the picture freezing characteristic value and water ripple characteristic value of the target camera can be calculated; then, the characteristic values ​​calculated in the aforementioned steps S52 and S51 can be combined to identify the video quality failure of the target camera, and the process can be but not limited to as shown in the following step S54.

[0137] S54. Based on the color cast characteristic value, the noise interference characteristic value, the brightness characteristic value, the clarity characteristic value, the signal clutter characteristic value, the jitter characteristic information, the picture freezing characteristic value and the water ripple characteristic value, the target camera is subjected to video quality fault detection processing to obtain a video quality fault detection result; in this embodiment, corresponding characteristic thresholds are set for the aforementioned color cast characteristic value, noise interference characteristic value, brightness characteristic value, clarity characteristic value, signal clutter characteristic value, picture freezing characteristic value and water ripple characteristic value; in this way, video quality fault identification can be performed based on the aforementioned characteristic thresholds, and the process is as follows.

[0138] When the brightness feature value is greater than the brightness feature threshold, it is determined that the target camera has a shooting brightness fault; when the color cast feature value is greater than the color cast feature threshold, it is determined that the target camera has a color cast fault; when the noise interference feature value is less than the noise feature threshold, it is determined that the target camera has a noise interference fault; when the clarity feature value is less than the clarity feature threshold, it is determined that the target camera has a shooting clarity fault; when the signal clutter feature value is less than the signal clutter feature threshold, it is determined that the target camera has a no signal fault; when the picture freeze feature value is less than the picture freeze feature threshold, it is determined that the target camera has a black screen or blue screen fault (i.e., picture freeze fault); and when the water ripple feature value is greater than the water ripple feature threshold, it is determined that the target camera has a transmission fault.

[0139] At the same time, it has been explained above that the jitter characteristic information includes the phase and amplitude information of the Fourier transform signal corresponding to the first grayscale image. Therefore, the standard image taken by the target camera without jitter failure can be obtained first; then, the phase and amplitude information of the Fourier transform signal corresponding to the standard image is obtained as the standard phase and standard amplitude; finally, the phase and amplitude information of the Fourier transform signal corresponding to the first grayscale image is compared with the standard phase and standard amplitude. If the amplitude is the same but the phase is different, it means that the target camera has a jitter failure.

[0140] In this way, the various types of fault identification results mentioned above can be used to form the video quality fault detection results of the target camera; based on this, through the aforementioned steps S51 to S54, the video quality fault detection of the target camera can be achieved; then, the first grayscale image can be used, combined with machine vision technology, to complete the lens loss detection of the target camera, and the process can be but not limited to as shown in the following step S55.

[0141] S55. Using a damage detection algorithm based on machine vision and utilizing the first grayscale image, perform lens damage detection processing on the target camera to obtain a lens damage fault detection result; in this embodiment, when the lens is damaged (such as the coating damage mentioned above), the image captured by it will have defects. Therefore, this embodiment performs lens damage detection by detecting defect points in the first grayscale image, and the process can be but is not limited to the following steps S55a to S55h.

[0142] S55a. Perform Gaussian smoothing on the first grayscale image to obtain a Gaussian smoothed image. In the present embodiment, Gaussian smoothing is performed on the first grayscale image mainly to reduce noise points in the image and reduce the level of details. Gaussian smoothing is a commonly used technique in image processing, and its principle will not be described in detail.

[0143] After obtaining the Gaussian smoothed image, the grayscale distribution characteristic value in the image can be determined so as to calculate the subsequent spatial adjacent Gaussian factor based on the grayscale distribution characteristic value. The process can be but is not limited to the following step S55b.

[0144] S55b. Based on the Gaussian smoothed image, determine the grayscale distribution characteristic value of the Gaussian smoothed image; in specific applications, for example, but not limited to, the following formula (8) can be used to calculate the aforementioned initial grayscale distribution characteristic value; then, based on the initial grayscale characteristic value, obtain the grayscale distribution characteristic value of the Gaussian smoothed image.

[0145]

[0146] In the above formula (8), L represents the initial grayscale distribution characteristic value, α, β represent the grayscale, ρ represents the maximum grayscale in the Gaussian smoothed image, and P(α, β, d, θ) represents the probability of occurrence of a pixel pair with grayscale values ​​of α and β, distance d, and angle θ. The total number of all pixel pairs in the Gaussian smoothed image can be obtained first. Then, the number of pixel pairs with grayscale values ​​of α and β, distance d, and angle θ is divided by the total number to obtain the above probability. For example, assuming α is 1, β is 1, d is 1, and θ is 0, then the probability of the occurrence of a pixel pair with a gray value of 1, a distance of 1, and an angle of 0 is calculated. In this way, the number of pixel pairs with a gray value of 1, a distance of 1, and an angle of 0 can be counted first, and finally, the counted number is divided by the total number of all pixel pairs to obtain the probability of the occurrence of a pixel pair with a gray value of 1, a distance of 1, and an angle of 0. Furthermore, for example, the values ​​of d are 1, 2, and 3, and the values ​​of θ are The angle θ between two pixels in a pixel pair is the angle between the line connecting the two pixels and the x-axis.

[0147] Therefore, based on the above formula (8), it can be known that there are 12 initial grayscale distribution eigenvalues ​​in total, and then, the average of the 12 initial grayscale distribution eigenvalues ​​can be taken as the grayscale distribution eigenvalue of the Gaussian smoothed image.

[0148] After the grayscale distribution characteristic value of the Gaussian smoothed image is obtained, the spatial adjacent Gaussian factor can be calculated based on it. The process can be but not limited to the following step S55c.

[0149] S55c. Calculate the spatial proximity Gaussian factor according to the grayscale distribution characteristic value; in specific implementation, the spatial proximity Gaussian factor can be calculated by, for example but not limited to, using the following formula (9).

[0150]

[0151] In the above formula (9), σ k represents the space near the Gaussian factor, represents the grayscale distribution characteristic value.

[0152] After calculating the spatial proximity Gaussian factor based on the above formula (9), it is also necessary to calculate the pixel value distribution Gaussian factor, and the process can be but is not limited to the following steps S55d and S55e.

[0153] S55d. Obtain a convolution kernel, and use the convolution kernel to perform a convolution operation on the Gaussian smoothed image to obtain a convolution image; in this embodiment, the convolution kernel may be, but is not limited to:

[0154] Where J represents the convolution kernel.

[0155] In this way, after performing a convolution operation on the Gaussian smoothed image based on the aforementioned convolution kernel, a convolution image can be obtained; then, the Gaussian factor of the pixel value distribution can be calculated based on the convolution image, and the process can be but not limited to as shown in the following step S55e.

[0156] S55e. Based on the convolution image, a Gaussian factor of pixel value distribution is calculated; in a specific application, the following formula (10) may be used, for example but not limited to, to calculate the Gaussian factor of pixel value distribution.

[0157]

[0158] In the above formula (10), σ srepresents the Gaussian factor of pixel value distribution, V represents the convolution image, ρV| represents the determinant operation on the convolution image, wherein V is actually a matrix (each element of the matrix is ​​the pixel value of a pixel point), and the determinant operation is to convert the matrix into a numerical value, such as a 2×2 matrix, whose first row elements are a1 and a2, and the second row elements are a3 and a4, then the determinant operation result is: a1×a4-a2×a4.

[0159] Thus, based on the aforementioned formula (10), the Gaussian factor of the pixel value distribution can be calculated; then, the Gaussian smoothed image can be filtered in combination with the aforementioned spatially adjacent Gaussian factor, and the process can be but is not limited to the following step S55f.

[0160] S55f. The Gaussian smoothed image is filtered using the spatial proximity Gaussian factor and the pixel value distribution Gaussian factor to obtain a filtered Gaussian smoothed image. In this embodiment, the spatial proximity Gaussian factor is first used to construct a spatial proximity Gaussian function, and then the pixel value distribution Gaussian factor is used to construct a pixel value similarity Gaussian function. Then, based on the aforementioned function, the Gaussian smoothed image is filtered to obtain a filtered Gaussian smoothed image.

[0161] In this embodiment, the following formula (11) may be used, but is not limited to, to perform filtering processing on any pixel point in the Gaussian smoothed image.

[0162]

[0163] In the above formula (11), (u,c) represents the coordinates of any pixel point in the Gaussian smoothed image, u represents the horizontal coordinate of any pixel point, c represents the vertical coordinate of any pixel point, O(u,c) represents the neighborhood area centered on the pixel point with coordinates (u,c) (the size can be 3×3), f(l,m) represents the grayscale value of the pixel point with coordinates (l,m) in the neighborhood area, ω(u,c,l,m) represents the filtered Gaussian function, and g(u,c) represents any pixel point after filtering (the grayscale value after filtering). Among them, ω(u,c,l,m)=ω1×ω2, and ω1,ω2 are the spatial proximity Gaussian function and the pixel value similarity Gaussian function respectively.

[0164] In this embodiment, the spatial proximity Gaussian function is:

[0165]

[0166] In the above formula (12), l, m are the horizontal coordinate and the vertical coordinate of the target pixel point respectively, and the target pixel point is the pixel point with coordinates (l, m) in the neighborhood area.

[0167] Similarly,

[0168] In the above formula (13), f(u,c) represents the grayscale value of any pixel point, and |||| represents the norm.

[0169] Therefore, through the above formulas (11)-(13), the spatial proximity Gaussian factor and the pixel value distribution Gaussian factor can be used to complete the filtering of the Gaussian smoothed image; then, the defect points can be identified based on the filtered Gaussian smoothed image, and the process can be but not limited to the following steps S55g and S55h.

[0170] S55g. Divide the filtered Gaussian smoothed image into several detection sub-images, and determine the grayscale threshold of each detection sub-image; in this embodiment, for example, but not limited to, first calculate the grayscale mean and grayscale standard deviation of each detection sub-image; then, use the control coefficient to multiply the grayscale standard deviation of each detection sub-image to obtain the adaptive threshold adjustment parameter of each detection sub-image; finally, use the grayscale mean of each detection sub-image to subtract the corresponding adaptive threshold adjustment parameter to obtain the grayscale threshold of each detection sub-image; optionally, the aforementioned control system is taken as a constant, which can be specifically set according to actual use.

[0171] After obtaining the grayscale threshold of each detection sub-image, it can be used to determine whether there are defective points in each detection sub-image, so as to obtain the lens damage fault detection result based on the judgment result. The process can be but is not limited to the following step S55h.

[0172] S55h. Determine the lens damage fault detection result based on each detection sub-image and the grayscale threshold of each detection sub-image; in the present embodiment, for any detection sub-image, if there is any pixel whose grayscale value is less than the grayscale threshold corresponding to the any detection sub-image, it can be determined that there is a defective point in the any detection sub-image; in this way, the any detection sub-image can be used as a defective image; based on this, when there are defective images in all detection sub-images, it can be determined that the target camera has a lens damage fault, and the lens damage fault detection result is that there is a lens damage fault; otherwise, there is no lens damage fault.

[0173] In this way, after obtaining the lens damage fault detection result, combined with the aforementioned video quality fault detection result, the first fault identification result of the target camera can be obtained, and the process is shown in the following step S56.

[0174] S56. Based on the video quality fault detection result and the lens damage fault detection result, a first fault identification result of the target camera is obtained; in this embodiment, the first fault identification result includes the aforementioned video quality fault detection result and the lens damage fault detection result.

[0175] Therefore, through the aforementioned steps S51 to S56 and their sub-steps, the video quality fault identification and lens damage fault identification of the target camera can be completed; then, it is necessary to perform pan / tilt damage fault identification of the target camera based on the third monitoring image, and the process can be but is not limited to as shown in the following step S6.

[0176] S6. Perform target detection processing on the third monitoring image to obtain a target detection result, and based on the target detection result, perform second fault type identification processing on the target camera to obtain a second fault identification result of the target camera, wherein the second fault type includes a gimbal damage fault; in this embodiment, for example, but not limited to, a CNN neural network or a YOLOv4 neural network is used to perform target detection processing on the third monitoring image to obtain a target detection result, wherein, performing target detection on the third monitoring image is essentially to detect whether there is a calibration reference object in its image, and the calibration reference object is obtained by performing target detection on the calibration image, and the calibration image is obtained by taking the target camera when it is rotated to the calibration position; in this way, it is equivalent to first obtaining the image taken when the target camera is rotated to the calibration position in this embodiment, and then performing target detection on the target, and obtaining an object that only exists in the image corresponding to the calibration position, thereby serving as a calibration reference object; of course, the calibration reference object can also be manually marked and stored in the server.

[0177] In this way, the target detection result is whether the calibrated reference object exists in the third monitoring image or not; based on this, the gimbal damage fault can be identified according to the aforementioned target detection result, and the process can be but not limited to as shown in the following steps S61 to S63.

[0178] S61. Determine whether the target detection result shows that there is a calibration reference object; in the present embodiment, when there is no calibration reference object in the third monitoring image, it means that the calibration reference object has not been captured, and therefore, there is an error in its rotation, and there must be a gimbal damage fault; on the contrary, if there is a calibration reference object, it is necessary to further judge based on its position, and the process may be but is not limited to the following steps S62 to S63.

[0179] S62. If yes, then obtain the position of the calibration reference object in the third monitoring image, and obtain the position of the calibration reference object in the calibration image; in this embodiment, the position of the calibration reference object in the third monitoring image may be the position of the corresponding detection frame when performing target detection; similarly, the position in the calibration image is also the same; based on this, the gimbal damage fault may be identified according to the position of the calibration reference object in the third monitoring image and the position in the calibration image, and the process may be but is not limited to as shown in the following step S63.

[0180] S63. Based on the position of the calibration reference object in the third monitoring image and its position in the calibration image, determine the second fault identification result of the target camera; in this embodiment, if the position of the calibration reference object in the third monitoring image is the same as its position in the calibration image, then it is judged that the target camera does not have a gimbal damage fault, that is, the second fault identification result is no gimbal fault; conversely, if the two are different, then it is judged that the target camera has a gimbal damage fault (that is, the position is not turned to the right position), that is, the second fault identification result is a gimbal fault.

[0181] In this way, through the aforementioned steps S61 to S63, the pan / tilt damage fault identification of the target camera can be completed; then, combined with the first fault identification result, the actual fault detection result of the target camera can be obtained, and the process is shown in the following step S7.

[0182] S7. Use the first fault identification result and the second fault identification result to obtain the fault detection result of the target camera; in this embodiment, the first fault identification result and the second fault identification result can be used but not limited to form the fault detection result; at the same time, if the obtained fault detection result contains one or more of the aforementioned types of faults (such as pan / tilt damage fault, color cast fault, lens loss fault and / or no signal fault, etc.), an alarm prompt is sent to the video monitoring center of smart sanitation to prompt the staff to perform equipment maintenance in a timely manner, thereby ensuring the reliability of equipment operation.

[0183] Therefore, through the fault detection method of the data acquisition terminal based on artificial intelligence described in detail in the aforementioned steps S1 to S7, the present invention identifies camera video quality faults and lens damage faults based on the image captured by the camera, and uses information such as grayscale, color, gradient in the image, and frame differences between adjacent frame images; at the same time, combined with machine vision technology, it performs target detection on the image captured after the camera is rotated, and then uses the target detection result to identify the pan / tilt damage fault of the camera; finally, using the aforementioned different types of fault identification results, the fault detection result of the camera can be obtained; therefore, the present invention combines image processing and machine vision technology to realize camera fault detection, which not only reduces the workload, but also improves the detection efficiency and real-time detection compared to traditional technologies. Therefore, it is very suitable for large-scale application and promotion.

[0184] like Figure 2 As shown, the second aspect of this embodiment provides a hardware device for implementing the fault detection method of the data acquisition terminal based on artificial intelligence described in the first aspect of the embodiment, including:

[0185] An acquisition unit is used to acquire a first surveillance image, a second surveillance image and a third surveillance image captured by a target camera, wherein the first surveillance image and the second surveillance image are two adjacent frames of images in the surveillance video of the target camera, and the third surveillance image is an image captured after the target camera rotates in position according to a rotation instruction.

[0186] The image processing unit is used to perform color conversion processing on the first monitoring image or the second monitoring image to obtain a color converted image.

[0187] The image processing unit is used to perform grayscale processing on the first monitoring image and the second monitoring image to obtain a first grayscale image and a second grayscale image.

[0188] The image processing unit is further used to construct an inter-frame difference image between the first grayscale image and the second grayscale image based on the first grayscale image and the second grayscale image.

[0189] A fault identification unit is used to perform a first fault type identification process on the target camera according to the color conversion image, the first grayscale image and the inter-frame difference image, so as to obtain a first fault identification result of the target camera, wherein the first fault type includes a lens damage fault and a video quality fault.

[0190] A fault identification unit is used to perform target detection processing on the third monitoring image to obtain a target detection result, and based on the target detection result, perform a second fault type identification processing on the target camera to obtain a second fault identification result of the target camera, wherein the second fault type includes a pan / tilt damage fault.

[0191] The fault identification unit is further used to obtain a fault detection result of the target camera using the first fault identification result and the second fault identification result.

[0192] The working process, working details and technical effects of the device provided in this embodiment can be found in the first aspect of the embodiment and will not be described in detail here.

[0193] like Figure 3 As shown, the third aspect of this embodiment provides another fault detection device for a data acquisition terminal based on artificial intelligence. Taking the device as an electronic device as an example, it includes: a memory, a processor and a transceiver that are communicatively connected in sequence, wherein the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer program to execute the fault detection method for the data acquisition terminal based on artificial intelligence as described in the first aspect of the embodiment.

[0194] For example, the memory may include, but is not limited to, random access memory (RAM), read only memory (ROM), flash memory, first input first output (FIFO) and / or first in last out (FILO), etc. Specifically, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). At the same time, the processor may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state.

[0195] In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. For example, the processor may not be limited to a microprocessor of the STM32F105 series, a reduced instruction set computer (RISC) microprocessor, an X86 architecture processor, or a processor with an integrated embedded neural network processing unit (NPU); the transceiver may be, but is not limited to, a wireless fidelity (WIFI) wireless transceiver, a Bluetooth wireless transceiver, a general packet radio service technology (General Packet Radio Service, GPRS) wireless transceiver, a ZigBee protocol (a low-power local area network protocol based on the IEEE802.15.4 standard, ZigBee) wireless transceiver, a 3G transceiver, a 4G transceiver and / or a 5G transceiver, etc. In addition, the device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0196] The working process, working details and technical effects of the electronic device provided in this embodiment can be found in the first aspect of the embodiment and will not be described in detail here.

[0197] The fourth aspect of this embodiment provides a storage medium storing instructions for the fault detection method for a data acquisition terminal based on artificial intelligence as described in the first aspect of the embodiment, that is, the storage medium stores instructions, and when the instructions are run on a computer, the fault detection method for a data acquisition terminal based on artificial intelligence as described in the first aspect of the embodiment is executed.

[0198] The storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, a CD, a hard disk, a flash memory, a USB flash drive and / or a memory stick, etc. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0199] The working process, working details and technical effects of the storage medium provided in this embodiment can be found in the first aspect of the embodiment and will not be described in detail here.

[0200] The fifth aspect of this embodiment provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the fault detection method for a data acquisition terminal based on artificial intelligence as described in the first aspect of the embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0201] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A fault detection method for a data acquisition terminal based on artificial intelligence, characterized in that: include: Acquire a first monitoring image, a second monitoring image, and a third monitoring image captured by a target camera, wherein the first monitoring image and the second monitoring image are two adjacent frames of images in the monitoring video of the target camera, and the third monitoring image is an image captured after the target camera rotates according to the rotation instruction; Performing color conversion processing on the first monitoring image or the second monitoring image to obtain a color converted image; grayscale the first monitoring image and the second monitoring image to obtain a first grayscale image and a second grayscale image; constructing an inter-frame difference image between the first grayscale image and the second grayscale image based on the first grayscale image and the second grayscale image; According to the color conversion image, the first grayscale image and the inter-frame difference image, a first fault type identification process is performed on the target camera to obtain a first fault identification result of the target camera, wherein the first fault type includes a lens damage fault and a video quality fault; Performing target detection processing on the third monitoring image to obtain a target detection result, and based on the target detection result, performing second fault type identification processing on the target camera to obtain a second fault identification result of the target camera, wherein the second fault type includes a pan / tilt damage fault; Obtaining a fault detection result of a target camera using the first fault identification result and the second fault identification result; According to the color conversion image, the first grayscale image and the inter-frame difference image, a first fault type identification process is performed on the target camera, including: Based on the color conversion image, calculating the color cast characteristic value of the target camera; According to the first grayscale image, a noise interference characteristic value, a brightness characteristic value, a clarity characteristic value, and a signal clutter characteristic value of the target camera are calculated, and jitter characteristic information of the target camera is determined; Using the inter-frame difference image, determining the picture freezing characteristic value and the water ripple characteristic value of the target camera; Based on the color cast characteristic value, the noise interference characteristic value, the brightness characteristic value, the clarity characteristic value, the signal clutter characteristic value, the jitter characteristic information, the picture freezing characteristic value, and the water ripple characteristic value, a video quality fault detection process is performed on the target camera to obtain a video quality fault detection result; Using a damage detection algorithm based on machine vision and utilizing the first grayscale image, a lens damage detection process is performed on the target camera to obtain a lens damage fault detection result; Based on the video quality fault detection result and the lens damage fault detection result, obtaining a first fault identification result of the target camera; The color conversion image is a Lab image, wherein the color cast characteristic value of the target camera is calculated based on the color conversion image, including: Calculating the mean and variance of the a color component and the mean and variance of the b color component in the color conversion image; Based on the mean and variance of the a color component and the mean and variance of the b color component, and in accordance with the following formula (1), the color cast characteristic value of the target camera is calculated; （1） In the above formula (1), represents the color cast characteristic value, represents the mean of the a color component and the mean of the b color component, respectively. Represents the variance of the a color component and the variance of the b color component respectively, Represents a constant greater than 0; A damage detection algorithm based on machine vision is adopted, and the first grayscale image is used to perform lens damage detection processing on the target camera to obtain a lens damage fault detection result, including: Performing Gaussian smoothing on the first grayscale image to obtain a Gaussian smoothed image; Based on the Gaussian smoothed image, determining a grayscale distribution characteristic value of the Gaussian smoothed image; Calculating a spatially adjacent Gaussian factor according to the grayscale distribution characteristic value; Obtaining a convolution kernel, and performing a convolution operation on the Gaussian smoothed image using the convolution kernel to obtain a convolution image; Based on the convolution image, a pixel value distribution Gaussian factor is calculated; Using the spatial proximity Gaussian factor and the pixel value distribution Gaussian factor, filtering the Gaussian smoothed image to obtain a filtered Gaussian smoothed image; Divide the filtered Gaussian smoothed image into a number of detection sub-images, and determine the grayscale threshold of each detection sub-image; The lens damage fault detection result is determined according to each detection sub-image and the grayscale threshold of each detection sub-image.

2. The method according to claim 1, characterized in that Calculating the noise interference characteristic value and the brightness characteristic value of the target camera according to the first grayscale image includes: Calculating the grayscale mean of all pixels in the first grayscale image, and determining the brightness characteristic value of the target camera based on the grayscale mean; Performing filtering on the first grayscale image to obtain a filtered image; Subtracting the filtered image from the first grayscale image to obtain a difference image, and calculating the grayscale variance of all pixels in the difference image; According to the grayscale variance of all pixels in the difference image, the noise interference characteristic value is calculated using the following formula (2); （2） In the above formula (2), represents the noise interference characteristic value, represents the grayscale variance of all pixels in the difference image, Represents the characteristic factor.

3. The method according to claim 1, characterized in that Calculating the clarity feature value and the signal clutter feature value of the target camera according to the first grayscale image includes: Obtaining a first gradient histogram in the horizontal direction and a second gradient histogram in the vertical direction of the first grayscale image, and screening out a histogram with the largest gradient value from the first gradient histogram and the second gradient histogram as a target histogram; Counting the number of pixel points whose gradient values ​​are less than or equal to a first gradient threshold from the target histogram, and calculating the gradient mean of each pixel point whose gradient value is greater than the first gradient threshold; Filtering out the number of pixel points whose gradient values ​​are less than a second gradient threshold from the second gradient histogram; The clarity feature value is calculated based on the gradient mean and the number of pixels whose gradient values ​​are less than or equal to the first gradient threshold value, and the signal clutter feature value is calculated based on the number of pixels whose gradient values ​​are less than the second gradient threshold value, and the following formula (4) is used to calculate the signal clutter feature value; （3） （4） In the above formula (3), represents the clarity feature value, represents the gradient mean, Indicates the number of pixels whose gradient values ​​are less than or equal to the first gradient threshold. represent the width and height of the first grayscale image in sequence; In the above formula (4), represents the signal clutter eigenvalue, Indicates the number of pixels whose gradient values ​​are less than the second gradient threshold.

4. The method according to claim 1, characterized in that: Determining jitter feature information of the target camera according to the first grayscale image includes: Performing Fourier transform processing on the first grayscale image to obtain a Fourier transform signal; Based on the Fourier transform signal, deriving amplitude information and phase information of the signal; Using the amplitude information and phase information of the signal to form the jitter characteristic information; Correspondingly, using the inter-frame difference image to determine the picture freezing feature value and the water ripple feature value of the target camera includes: Based on the inter-frame difference image, constructing a grayscale histogram of the inter-frame difference image; According to the grayscale histogram, counting the number of pixels whose grayscale is less than or equal to the grayscale threshold and the number of pixels whose grayscale is within the calibration range; The image freezing characteristic value is calculated according to the number of pixels whose grayscale is less than or equal to the grayscale threshold, and the water ripple characteristic value is calculated according to the number of pixels whose grayscale is within the calibration range.

5. The method according to claim 1, characterized in that The target detection result is the presence or absence of a calibration reference object in the third monitoring image, wherein the calibration reference object is obtained by performing target detection on a calibration image, and the calibration image is obtained by taking a picture when the target camera rotates to a calibration position; Wherein, based on the target detection result, performing a second fault type identification process on the target camera to obtain a second fault identification result of the target camera includes: Determining whether the target detection result indicates that a calibration reference object exists; If yes, obtaining the position of the calibration reference object in the third monitoring image, and obtaining the position of the calibration reference object in the calibration image; Based on the position of the calibration reference object in the third monitoring image and the position of the calibration reference object in the calibration image, a second fault recognition result of the target camera is determined.

6. A fault detection device for a data acquisition terminal based on artificial intelligence, characterized in that: include: an acquisition unit, configured to acquire a first surveillance image, a second surveillance image, and a third surveillance image captured by a target camera, wherein the first surveillance image and the second surveillance image are two adjacent frames of images in a surveillance video of the target camera, and the third surveillance image is an image captured after the target camera rotates in accordance with a rotation instruction; An image processing unit, configured to perform color conversion processing on the first monitoring image or the second monitoring image to obtain a color converted image; An image processing unit, used for performing grayscale processing on the first monitoring image and the second monitoring image to obtain a first grayscale image and a second grayscale image; The image processing unit is further used to construct an inter-frame difference image between the first grayscale image and the second grayscale image based on the first grayscale image and the second grayscale image; a fault identification unit, configured to perform a first fault type identification process on the target camera according to the color conversion image, the first grayscale image and the inter-frame difference image, to obtain a first fault identification result of the target camera, wherein the first fault type includes a lens damage fault and a video quality fault; a fault identification unit, configured to perform target detection processing on the third monitoring image to obtain a target detection result, and based on the target detection result, perform second fault type identification processing on the target camera to obtain a second fault identification result of the target camera, wherein the second fault type includes a pan / tilt damage fault; The fault identification unit is further used to obtain a fault detection result of the target camera by using the first fault identification result and the second fault identification result; According to the color conversion image, the first grayscale image and the inter-frame difference image, a first fault type identification process is performed on the target camera, including: Based on the color conversion image, calculating the color cast characteristic value of the target camera; According to the first grayscale image, a noise interference characteristic value, a brightness characteristic value, a clarity characteristic value, and a signal clutter characteristic value of the target camera are calculated, and jitter characteristic information of the target camera is determined; Using the inter-frame difference image, determining the picture freezing characteristic value and the water ripple characteristic value of the target camera; Based on the color cast characteristic value, the noise interference characteristic value, the brightness characteristic value, the clarity characteristic value, the signal clutter characteristic value, the jitter characteristic information, the picture freezing characteristic value, and the water ripple characteristic value, a video quality fault detection process is performed on the target camera to obtain a video quality fault detection result; Using a damage detection algorithm based on machine vision and utilizing the first grayscale image, a lens damage detection process is performed on the target camera to obtain a lens damage fault detection result; Based on the video quality fault detection result and the lens damage fault detection result, obtaining a first fault identification result of the target camera; The color conversion image is a Lab image, wherein the color cast characteristic value of the target camera is calculated based on the color conversion image, including: Calculating the mean and variance of the a color component and the mean and variance of the b color component in the color conversion image; Based on the mean and variance of the a color component and the mean and variance of the b color component, and in accordance with the following formula (1), the color cast characteristic value of the target camera is calculated; （1） In the above formula (1), represents the color cast characteristic value, represents the mean of the a color component and the mean of the b color component, respectively. Represents the variance of the a color component and the variance of the b color component respectively, Represents a constant greater than 0; A damage detection algorithm based on machine vision is adopted, and the first grayscale image is used to perform lens damage detection processing on the target camera to obtain a lens damage fault detection result, including: Performing Gaussian smoothing on the first grayscale image to obtain a Gaussian smoothed image; Based on the Gaussian smoothed image, determining a grayscale distribution characteristic value of the Gaussian smoothed image; Calculating a spatially adjacent Gaussian factor according to the grayscale distribution characteristic value; Obtaining a convolution kernel, and performing a convolution operation on the Gaussian smoothed image using the convolution kernel to obtain a convolution image; Based on the convolution image, a pixel value distribution Gaussian factor is calculated; Using the spatial proximity Gaussian factor and the pixel value distribution Gaussian factor, filtering the Gaussian smoothed image to obtain a filtered Gaussian smoothed image; Divide the filtered Gaussian smoothed image into a number of detection sub-images, and determine the grayscale threshold of each detection sub-image; The lens damage fault detection result is determined according to each detection sub-image and the grayscale threshold of each detection sub-image.

7. A computer program product comprising instructions, characterized in that When the instruction is executed on a computer, the computer is caused to execute the fault detection method for a data acquisition terminal based on artificial intelligence as described in any one of claims 1 to 5.

Images