Internet-based intelligent VR inspection system and method

Abstract

The application relates to the technical field of image enhancement, and discloses an intelligent VR (Virtual Reality) inspection system and method based on the Internet, which comprises the following steps: collecting an inspection video and reading image frames frame by frame to obtain different superpixel regions in a gray image frame corresponding to the image frames; according to the gradient difference and texture difference between edge pixel points on the edges of the superpixel regions and the number of pixel points contained in the superpixel regions, the window adjustment degree of the superpixel regions is obtained, the optimal side length of the superpixel regions is calculated, and the superpixel regions are divided into different local regions to be enhanced; the image is enhanced according to the difference between the pixel values of all pixel points in the local regions to be enhanced, local enhanced regions are obtained, and image frames after image enhancement are obtained according to all the local enhanced regions corresponding to the same image frame; and VR inspection is realized according to the image frames after image enhancement. The application can improve the rationality and stability of the VR inspection result.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image enhancement technology, in particular to an intelligent VR inspection system and method based on the Internet. BACKGROUND

[0002] VR inspection can allow staff to remotely perform inspection in a safe place, avoid exposure to dangerous environments, improve safety, reduce risks, and improve inspection efficiency and quality. During VR inspection, in order to improve the quality of video images, strong light needs to be used during video acquisition, which often leads to overexposure of the acquired video, resulting in loss of detail information in each frame of the video and affecting the modeling of the inspection environment.

[0003] CLAHE histogram equalization algorithm can be used for image enhancement of each frame of the video, highlighting the detail information of the image and improving the accuracy of the modeling of the inspection environment. However, the image enhancement effect of the histogram equalization algorithm is often affected by the preset gray value threshold. When the value of the gray value threshold is unreasonable, the area around the overexposed position is excessively enhanced, which is prone to edge artifacts, resulting in poor image enhancement effect, and further leading to a large difference between the modeling result of the inspection environment and the real environment, affecting the inspection effect. SUMMARY

[0004] The present application provides an intelligent VR inspection system and method based on the Internet to solve the problem of unreasonable preset gray value threshold of the image enhancement algorithm, poor image enhancement effect, large difference between the modeling result of the inspection environment and the real environment, and affecting the inspection effect. The technical solution adopted is as follows:

[0005] In a first aspect, an embodiment of the present application provides an intelligent VR inspection method based on the Internet, which comprises the following steps:

[0006] Acquire the inspection video and read the image frame by frame to obtain different superpixel regions in the gray image frame corresponding to the image frame;

[0007] According to the difference between the gradient and texture of the edge pixel points on the edge of the superpixel region, the edge gradient variance and edge LBP variance of the superpixel region are determined respectively, and the window adjustment degree of the superpixel region is obtained by combining the number of pixel points contained in the superpixel region. According to the window adjustment degree of the superpixel region, the optimal edge length of the superpixel region is calculated, and the superpixel region is divided into different local enhanced regions according to the optimal edge length;

[0008] According to the difference between the pixel values of all pixel points in the local enhanced region, the image is enhanced to obtain a local enhanced region, and the image frame after image enhancement is obtained according to all local enhanced regions corresponding to the same image frame;

[0009] According to the image-enhanced image frame, VR inspection is realized.

[0010] Further, the specific method for obtaining the edge gradient variance of the superpixel region is:

[0011] The variance of the gradient amplitudes of all edge pixel points on the edge of the superpixel region is denoted as the edge gradient variance of the superpixel region.

[0012] Further, the method for determining the edge LBP variance is:

[0013] The variance of the LBP values of all edge pixel points on the edge of the superpixel region is denoted as the edge LBP variance of the superpixel region.

[0014] Further, the method for obtaining the window adjustment degree of the superpixel region is:

[0015] The circularity of the superpixel region is calculated according to the number of pixel points included in the edge of the superpixel region as the perimeter of the superpixel region and the number of pixel points included in the superpixel region as the area of the superpixel region.

[0016] According to the edge gradient variance and the edge LBP variance of the superpixel region, the edge feature value of the superpixel region is obtained.

[0017] According to the preset third weight coefficient and the preset fourth weight coefficient, the circularity and the edge feature value of the superpixel region are weighted and summed to obtain the window adjustment degree of the superpixel region.

[0018] Further, the method for obtaining the edge feature value is:

[0019] According to the preset first weight coefficient and the preset second weight coefficient, the edge gradient variance and the edge LBP variance of the superpixel region are weighted and summed to obtain the edge feature value of the superpixel region.

[0020] Further, the calculation formula of the optimal side length of the superpixel region is:

[0021]

[0022] wherein, represents the optimal side length of the superpixel region . represents the preset window base value of the superpixel region; represents the preset window adjustment coefficient; represents the window adjustment degree of the superpixel region . represents the upward rounding function.

[0023] Further, the method of dividing the superpixel region into different local regions to be enhanced according to the optimal edge length comprises the following specific steps:

[0024] The superpixel region is divided into different local regions to be enhanced according to the optimal edge length of the superpixel region.

[0025] Further, the method of obtaining the image frame after image enhancement comprises the following specific steps:

[0026] The product of the standard deviation of the pixel values of all the pixel points in the local region to be enhanced and the number 2 is taken as an integer, and the result is recorded as the clipping threshold of the local region to be enhanced; the clipping threshold of the local region to be enhanced is used as the clipping threshold of the histogram specification algorithm, the histogram specification algorithm is used to perform image enhancement on the local region to be enhanced, and a local enhanced region is obtained.

[0027] The image formed by all the local enhanced regions corresponding to the same image frame is smoothed, and an image frame after image enhancement is obtained.

[0028] Further, the method of realizing VR inspection according to the image frame after image enhancement comprises the following specific steps:

[0029] The image frame after image enhancement is restored to an RGB image, and is merged into a VR inspection video frame by frame; the VR inspection video is transmitted to an intelligent VR device, and the inspection environment is reconstructed.

[0030] In the second aspect, the embodiments of the present application further provide an intelligent VR inspection system based on the Internet, which comprises a memory, a processor, and a computer program stored in the memory and running on the processor; when the processor executes the computer program, the steps of the method described in any one of the above aspects are realized.

[0031] The present application has the following beneficial effects:

[0032] The application first reads the inspection video frame by frame, obtains image frames, and divides the gray image frames corresponding to the image frames into different superpixel regions according to the overexposure features. Further, considering that the exposure area formed by the mirror reflection is a strip of light with clear boundaries, and the overexposure area formed by the direct light is an approximately circular white area with a relatively blurred edge, the window adjustment degree of the superpixel region is determined according to the clear degree of the edge of the superpixel region, the richness of the edge texture, and the shape of the superpixel region. According to the window adjustment degree of the superpixel region, the superpixel region is divided into different local enhanced regions. The more details contained in the superpixel region, the smaller the edge length of the local enhanced region divided in the image enhancement process. According to the contrast of the local enhanced region, the gray value threshold is adaptively set and the local enhanced region is image enhanced to obtain a local enhanced region. According to all the local enhanced regions corresponding to the same image frame, an image frame after image enhancement is obtained, which solves the problem of unreasonable preset gray value threshold value of the image enhancement algorithm, poor image enhancement effect, too large difference between the modeling result of the inspection environment and the real environment, and affects the inspection effect. Finally, according to the image frame after image enhancement, VR inspection is realized, and the rationality and stability of the VR inspection result are improved. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0034] Figure 1 The flowchart of the intelligent VR inspection method based on the Internet provided by an embodiment of the present application;

[0035] Figure 2 The window adjustment degree acquisition flowchart provided by an embodiment of the present application. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0037] Please refer to Figure 1Fig. 1 shows a flow chart of an Internet-based intelligent VR inspection method according to an embodiment of the present application, which comprises the following steps:

[0038] In step S001, the inspection video is collected and the image frames are read frame by frame to obtain different superpixel regions in the gray image frames corresponding to the image frames.

[0039] The inspection video is collected by using a dual high-definition camera of a UAV, and the inspection video is transmitted by using a wireless transmission technology, and each frame of image frame in the inspection video is read frame by frame.

[0040] In this embodiment, the video frame rate of the inspection video is set to 30 FPS, and each frame of image frame in the inspection video is read frame by frame by using the OpenCV library function. In actual application, as other implementation manners, the implementer can determine the value of the video frame rate of the inspection video according to the actual situation, and the present application does not make special limitation.

[0041] When the image frame contains an exposed region, the pixel value of the pixel point in the exposed region is generally greater than or equal to 200, the image details are missing, and the exposure conditions in different exposed regions are different. Specifically, the exposure condition difference is, for example, the smooth surface such as metal and glass is easy to cause strong light reflection and appear exposure phenomenon under strong light environment, resulting in that there is often a strip of light in the captured image, but the boundary of the exposed region is clear; the image captured by the direct light angle of view will appear an overexposed region, which is a white and hazy circle, and the edge is relatively blurred. Therefore, it is necessary to divide different regions in the image frame according to the characteristics of overexposure and perform image enhancement.

[0042] The image frame is subjected to gray processing to obtain a gray image frame, and the SLIC superpixel segmentation algorithm is used for image segmentation on the gray image frame to obtain a superpixel region.

[0043] In this embodiment, the number of superpixel regions is 4, and the specific calculation process of the number of superpixel regions is , The down arrow symbol represents the down arrow symbol.

[0044] At this point, the superpixel region in each frame of gray image frame is obtained.

[0045] In step S002, the edge gradient variance and the edge LBP variance of the superpixel region are respectively determined according to the gradient and the texture difference between the edge pixel points on the edge of the superpixel region, the window adjustment degree of the superpixel region is obtained in combination with the number of pixel points contained in the superpixel region, the optimal edge length of the superpixel region is calculated according to the window adjustment degree of the superpixel region, and the superpixel region is divided into different local regions to be enhanced according to the optimal edge length.

[0046] The Sobel operator is used to process the edge pixels on the edge of the superpixel region, to obtain the gradient amplitude of each edge pixel, and the variance of the gradient amplitudes of all the edge pixels on the edge of the superpixel region is denoted as the edge gradient variance of the superpixel region. The use of the Sobel operator to calculate the gradient amplitude of a pixel is a known technique and will not be described again.

[0047] The edge gradient variance of the superpixel region reflects the degree of gradient change of the edge of the superpixel region. When the gradient change of the edge of the superpixel region is greater, the edge of the superpixel region is clearer, and the possibility of the superpixel region corresponding to an exposure region formed by specular reflection is greater, and the possibility of the superpixel region corresponding to an overexposure region formed by direct light is smaller.

[0048] The LBP value of each edge pixel on the edge of the superpixel region is calculated, and the variance of the LBP values of all the edge pixels on the edge of the superpixel region is denoted as the edge LBP variance of the superpixel region. The calculation of the LBP value of a pixel is a known technique and will not be described again.

[0049] The edge LBP variance of the superpixel region reflects the richness of the edge texture of the superpixel region. When the richness of the edge texture of the superpixel region is greater, the possibility of the superpixel region corresponding to an exposure region formed by specular reflection is greater, and the possibility of the superpixel region corresponding to an overexposure region formed by direct light is smaller.

[0050] The circularity of the superpixel region is calculated. The calculation of the circularity of the superpixel region is a known technique and will not be described again. In this embodiment, the circularity is calculated based on the perimeter and area of the superpixel region. The perimeter of the superpixel region is the number of pixels included in the edge of the superpixel region, and the area of the superpixel region is the number of pixels included in the superpixel region.

[0051] It can be understood that when the circularity of the superpixel region is greater, if the superpixel region is an exposure region, the shape of the superpixel region is closer to a circle, the possibility of the superpixel region corresponding to an overexposure region formed by direct light is greater, and the possibility of the superpixel region corresponding to an exposure region formed by specular reflection is smaller.

[0052] The edge gradient variance and the edge LBP variance of the superpixel region are weighted and summed to obtain an edge feature value of the superpixel region, and the circularity of the superpixel region is combined to obtain a window adjustment degree of the superpixel region.

[0053] The preset first weight coefficient and the preset second weight coefficient are respectively taken as weights of the edge gradient variance and the edge LBP variance of the superpixel region, the edge gradient variance and the edge LBP variance of the superpixel region are weighted and summed, and an edge feature value of the superpixel region is obtained. The preset third weight coefficient and the preset fourth weight coefficient are respectively taken as weights of the circularity and the edge feature value of the superpixel region, the circularity and the edge feature value of the superpixel region are weighted and summed, and a window adjustment degree of the superpixel region is obtained.

[0054] The sum of the first weight coefficient and the second weight coefficient should be 1, and the sum of the third weight coefficient and the fourth weight coefficient should be 1. In the embodiment, the values of the first weight coefficient, the second weight coefficient, the third weight coefficient and the fourth weight coefficient are 0.6, 0.4, 0.5 and 0.5 respectively.

[0055] When the edge feature value of the superpixel region is larger, the richness of the edge texture of the superpixel region is greater, the gradient change of the edge is greater, the possibility of the superpixel region corresponding to the exposure region formed by the specular reflection is greater, and the possibility of the superpixel region corresponding to the overexposure region formed by the direct light is smaller. At this time, the superpixel region contains more detail information, and the value of the side length of the corresponding region in the image enhancement process should be smaller. The window adjustment degree acquisition flow chart is as shown in Figure 2

[0056] According to the window adjustment degree of the superpixel region, the optimal side length of the superpixel region is calculated, and the calculation formula is:

[0057]

[0058] wherein, represents the optimal side length of the superpixel region; represents the preset window base value of the superpixel region, and the value of the window base value of the superpixel region in the embodiment is 5; represents the preset window adjustment coefficient, and the value of the window adjustment coefficient in the embodiment is 0.5; represents the window adjustment degree of the superpixel region; represents the upward rounding function, and the function of the upward rounding function is to ensure that the optimal side length of the superpixel region is an integer.

[0059] At this point, the optimal side length of each superpixel region is obtained.

[0060] The optimal side length of the superpixel region is taken as the side length of the local region to be enhanced, and the superpixel region is divided into different local regions to be enhanced.

[0061] ​​​​At this point, each superpixel region is divided into different local regions to be enhanced.

[0062] In step S003, the image is enhanced according to the difference between the pixel values of all pixel points in the local region to be enhanced, and a local enhanced region is obtained. According to all local enhanced regions corresponding to the same image frame, an image frame after image enhancement is obtained.

[0063] When overexposure occurs in the local region to be enhanced, the numerical distribution in the gray histogram of the local region to be enhanced is not uniform, and a large number of gray values are concentrated in the region corresponding to the highlight. Therefore, the part with a high frequency of gray values can be transferred to the position of the frequency of other gray values by cropping to limit the problem of excessive contrast of the local region. When cropping, a threshold needs to be set to crop the frequency of the gray value higher than the threshold, and the cropped frequency is evenly distributed to the frequency of other gray values. It should be noted that when the threshold value is too large, the region with high contrast cannot be effectively processed, and the details of the exposure region cannot be effectively restored. When the threshold value is too small, the pixel values of the overexposure region will be excessively cropped, which may exacerbate the halo effect or artifacts of the overexposure region. Therefore, a suitable threshold for cropping needs to be set according to the contrast of the local region to be enhanced.

[0064] The down-rounded value of the product of the standard deviation of the pixel values of all pixel points in the local region to be enhanced and the number 2 is recorded as the cropping threshold of the local region to be enhanced.

[0065] The cropping threshold of the local region to be enhanced is used as the cropping threshold of the histogram specification algorithm, and the histogram specification algorithm is used to enhance the image of the local region to be enhanced to obtain a local enhanced region.

[0066] At this point, the local enhanced region corresponding to each local region to be enhanced divided by each superpixel region is obtained.

[0067] The image composed of all local enhanced regions corresponding to the same image frame is processed using the histogram equalization algorithm and the bilinear interpolation smoothing method to obtain an image frame after image enhancement, thereby realizing image enhancement of the image frame.

[0068] The processing of the image using the histogram equalization algorithm and the bilinear interpolation smoothing method is a known technology and will not be described in detail.

[0069] At this point, the image frame after image enhancement is obtained.

[0070] In step S004, VR inspection is realized according to the image frame after image enhancement.

[0071] The image frame after image enhancement is restored to an RGB image using the OpenCV library function cv2.cvtColor, and the RGB image restored is merged into a VR inspection video frame by frame using the library function cv2.VideoWriter, the VR inspection video is transmitted to the intelligent VR device through WiFi wireless transmission technology, and the SGM multi-view three-dimensional modeling technology is used to reconstruct the inspection environment, so that the VR device can display the inspection area corresponding to the VR inspection video in real time, clearly and completely.

[0072] Wherein, using OpenCV library function cv2.cvtColor to restore the image frame to RGB image, using library function cv2.VideoWriter to merge RGB image into video frame by frame, transmitting video through WiFi wireless transmission technology, SGM multi-view three-dimensional modeling technology to reconstruct the inspection environment, are all known technologies, and will not be repeated. As other embodiments, on the basis of achieving the purpose of reconstructing the inspection environment according to the image frame after image enhancement, the implementer can realize it by other methods of prior art, and the present application does not make special limitation.

[0073] Thus, the VR inspection is realized.

[0074] Based on the same inventive concept as the above method, the embodiments of the present application also provide an intelligent VR inspection system based on the Internet, which comprises a memory, a processor and a computer program stored in the memory and running on the processor, and the processor implements the steps of any one of the methods in the above intelligent VR inspection method based on the Internet.

[0075] The above is only the preferred embodiment of the present application, and is not used to limit the present application, any modification, equivalent replacement, improvement, etc. within the principles of the present application, should be included in the protection scope of the present application.

Claims

1. An internet-based intelligent VR inspection method, characterized in that, The method comprises the following steps: Collecting a patrol video and reading image frames frame by frame to obtain different superpixel regions in a gray image frame corresponding to the image frame; According to the differences between the gradients and textures of the edge pixel points on the edge of the superpixel region, the edge gradient variance and the edge LBP variance of the superpixel region are determined respectively, the window adjustment degree of the superpixel region is obtained by combining the number of pixel points contained in the superpixel region, the optimal edge length of the superpixel region is calculated according to the window adjustment degree of the superpixel region, and the superpixel region is divided into different local regions to be enhanced according to the optimal edge length; According to the differences between the pixel values of all pixel points in the local region to be enhanced, image enhancement is performed to obtain a local enhanced region, and an image frame after image enhancement is obtained according to all local enhanced regions corresponding to the same image frame; VR patrol is realized according to the image frame after image enhancement. The method for obtaining the window adjustment degree of the superpixel region comprises the following steps: The circularity of the superpixel region is calculated according to the number of pixel points contained in the edge of the superpixel region as the perimeter of the superpixel region and the number of pixel points contained in the superpixel region as the area of the superpixel region; The edge feature value of the superpixel region is obtained according to the edge gradient variance and the edge LBP variance of the superpixel region; The circularity and the edge feature value of the superpixel region are weighted and summed according to the third preset weight coefficient and the fourth preset weight coefficient to obtain the window adjustment degree of the superpixel region; The calculation formula of the optimal edge length of the superpixel region is: wherein, represents an optimal side length of the superpixel region ; represents a preset window base value of the superpixel region; represents a preset window adjustment coefficient; represents a window adjustment degree of the superpixel region ; represents a ceiling function; The specific method for dividing the superpixel region into different local regions to be enhanced according to the optimal edge length comprises the following steps: The optimal edge length of the superpixel region is taken as the edge length of the local region to be enhanced, and the superpixel region is divided into different local regions to be enhanced; The method for obtaining the image frame after image enhancement comprises the following steps: The standard deviation of the pixel values of all pixel points in the local region to be enhanced is multiplied by 2, and the down-rounded value is taken as the clipping threshold value of the local region to be enhanced; the clipping threshold value of the local region to be enhanced is used as the clipping threshold value of the histogram specification algorithm, the local region to be enhanced is subjected to image enhancement by using the histogram specification algorithm, and a local enhanced region is obtained; The image formed by all local enhanced regions corresponding to the same image frame is subjected to smoothing processing to obtain an image frame after image enhancement. 2.The Internet-based intelligent VR inspection method according to claim 1, wherein, The specific method for obtaining the edge gradient variance of the superpixel region comprises the following steps: The variance of the gradient amplitudes of all edge pixel points on the edge of the superpixel region is taken as the edge gradient variance of the superpixel region. 3.The Internet-based intelligent VR inspection method of claim 1, wherein, The specific method for determining the edge LBP variance comprises the following steps: The variance of the LBP values of all edge pixel points on the edge of the superpixel region is taken as the edge LBP variance of the superpixel region. 4.The Internet-based intelligent VR inspection method of claim 1, wherein, The specific method for obtaining the edge feature value of the superpixel region comprises the following steps: The edge gradient variance and the edge LBP variance of the superpixel region are weighted and summed according to the first preset weight coefficient and the second preset weight coefficient to obtain the edge feature value of the superpixel region. 5.The Internet-based intelligent VR inspection method of claim 1, wherein, The specific method for realizing VR patrol according to the image frame after image enhancement comprises the following steps: The image frame after image enhancement is restored as an RGB image, and is merged frame by frame into a VR inspection video, and the VR inspection video is transmitted to a smart VR device to reconstruct the inspection environment. 6.An Internet-based intelligent VR inspection system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, The processor implements the steps of the method of any one of claims 1-5 when executing the computer program.

Images