Internet-based intelligent VR inspection system and method

Abstract

The invention relates to the technical field of image enhancement, and provides an intelligent VR inspection system and method based on the Internet, and the method comprises the steps: collecting an inspection video, reading image frames frame by frame, and obtaining different superpixel regions in gray level image frames corresponding to the image frames; according to the gradient and texture difference between edge pixel points on the edge of the super-pixel area and the number of pixel points contained in the super-pixel area, the window adjustment degree of the super-pixel area is obtained, the optimal side length of the super-pixel area is calculated, and the super-pixel area is divided into different local areas to be enhanced; performing image enhancement according to the difference between the pixel values of all the pixel points in the local to-be-enhanced area, obtaining a local enhanced area, and obtaining an image frame after image enhancement according to all the local enhanced areas corresponding to the same image frame; and realizing VR inspection according to the image frame after image enhancement. According to the invention, the reasonability and stability of VR inspection results can be improved.

Description

Technical Field

[0001] The present invention relates to the field of image enhancement technology, and in particular to an Internet-based intelligent VR inspection system and method. Background Art

[0002] VR inspections allow workers to conduct inspections remotely from a safe location, avoiding exposure to hazardous environments. This improves safety, reduces risks, and enhances inspection efficiency and quality. During VR inspections, to enhance video image quality, strong lighting is required during video capture. This often results in overexposure of the captured video, causing loss of detail in each frame and compromising the modeling of the inspection environment.

[0003] The CLAHE histogram equalization algorithm can be used to enhance each frame in the video, highlighting image details and improving the accuracy of the inspection environment modeling. However, the image enhancement effect of the histogram equalization algorithm is often affected by the preset grayscale threshold. When the grayscale threshold is not set properly, the area around the overexposed position is over-enhanced, which can easily produce edge artifacts, resulting in poor image enhancement results. This can further cause the inspection environment modeling results to differ significantly from the actual environment, affecting the inspection results. Summary of the Invention

[0004] The present invention provides an Internet-based intelligent VR inspection system and method to solve the problem that the preset grayscale value threshold of the image enhancement algorithm is unreasonable, resulting in poor image enhancement effect, causing the modeling result of the inspection environment to differ greatly from the actual environment, and affecting the inspection effect. The technical solutions adopted are as follows: In a first aspect, an embodiment of the present invention provides an Internet-based intelligent VR inspection method, the method comprising the following steps: Collect inspection videos and read image frames frame by frame to obtain different super-pixel areas in the grayscale image frame corresponding to the image frame; Based on the differences in gradient and texture between edge pixels on the edge of the superpixel region, the edge gradient variance and edge LBP variance of the superpixel region are determined respectively. Combined with the number of pixels contained in the superpixel region, the window adjustment degree of the superpixel region is obtained. Based on the window adjustment degree of the superpixel region, the optimal side length of the superpixel region is calculated. Based on the optimal side length, the superpixel region is divided into different local areas to be enhanced; Perform image enhancement based on the difference between the pixel values ​​of all pixels in the local area to be enhanced, obtain the local enhanced area, and obtain the image frame after image enhancement based on all the local enhanced areas corresponding to the same image frame; VR inspection is achieved based on the image frames after image enhancement.

[0005] Furthermore, the specific method for obtaining the edge gradient variance of the superpixel region is: The variance of the gradient amplitudes of all edge pixels on the edge of the super-pixel region is recorded as the edge gradient variance of the super-pixel region.

[0006] Furthermore, the method for determining the edge LBP variance is: The variance of the LBP values ​​of all edge pixels on the edge of the superpixel region is recorded as the edge LBP variance of the superpixel region.

[0007] Furthermore, the method for obtaining the window adjustment degree of the superpixel area is: The circularity of the superpixel region is calculated by taking the perimeter of the superpixel region as the number of pixels contained in the edge of the superpixel region and the area of ​​the superpixel region as the number of pixels contained in the superpixel region; Obtain the edge feature value of the superpixel region based on the edge gradient variance and edge LBP variance of the superpixel region; According to the preset third weight coefficient and the preset fourth weight coefficient, the circularity and edge feature value of the super pixel area are weighted and summed to obtain the window adjustment degree of the super pixel area.

[0008] Furthermore, the method for obtaining the edge feature value is: According to the preset first weight coefficient and the preset second weight coefficient, the edge gradient variance and the edge LBP variance of the super pixel area are weighted and summed to obtain the edge feature value of the super pixel area.

[0009] Furthermore, the optimal side length of the superpixel region is calculated as: in, Represents superpixel area The optimal side length of Represents the window base value of the preset superpixel area; Indicates the preset window adjustment coefficient; Represents superpixel area Window adjustment degree; Represents the ceiling function.

[0010] Furthermore, the method of dividing the superpixel region into different local regions to be enhanced according to the optimal side length includes the following specific methods: The optimal side length of the superpixel region is used as the side length of the local region to be enhanced, and the superpixel region is divided into different local regions to be enhanced.

[0011] Furthermore, the method for acquiring the image frame after image enhancement is: The product of the standard deviation of the pixel values ​​of all pixels in the local area to be enhanced and the number 2 is rounded down to an integer, and recorded as the clipping threshold of the local area to be enhanced; the clipping threshold of the local area to be enhanced is used as the clipping threshold of the histogram regularization algorithm, and the local area to be enhanced is image enhanced using the histogram regularization algorithm to obtain a local enhanced area; The image composed of all local enhanced regions corresponding to the same image frame is smoothed to obtain an image frame after image enhancement.

[0012] Furthermore, the VR inspection is implemented based on the image frames after image enhancement, including the following specific methods: The enhanced image frames are restored to RGB images and merged frame by frame into VR inspection videos. The VR inspection videos are transmitted to the intelligent VR device to reconstruct the inspection environment.

[0013] In a second aspect, an embodiment of the present invention also provides an Internet-based intelligent VR inspection system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of any one of the above methods when executing the computer program.

[0014] The beneficial effects of the present invention are: The present application first reads the inspection video frame by frame to obtain the image frame, and divides the grayscale image frame corresponding to the image frame into different super-pixel areas according to the over-exposure characteristics. Furthermore, considering that the exposure area formed by the mirror reflection is a strip of bright light with a clear boundary, while the over-exposure area formed by direct light is a nearly circular white area with a relatively blurred edge, the window adjustment degree of the super-pixel area is determined according to the clarity of the edge of the super-pixel area, the richness of the edge texture, and the shape of the super-pixel area, and the super-pixel area is divided into different local areas to be enhanced according to the window adjustment degree of the super-pixel area. The more detail information contained in the super-pixel area, the better the image quality. During the image enhancement process, the side length of the local area to be enhanced into which the superpixel area is divided should be as small as possible; according to the contrast of the local area to be enhanced, the grayscale value threshold is adaptively set and the image enhancement is performed on the local area to be enhanced to obtain the local enhanced area. According to all the local enhanced areas corresponding to the same image frame, the image frame after image enhancement is obtained, so as to solve the problem that the preset grayscale value threshold of the image enhancement algorithm is unreasonable, resulting in poor image enhancement effect, causing the modeling result of the inspection environment to be too different from the real environment, affecting the inspection effect; finally, based on the image frame after image enhancement, VR inspection is realized to improve the rationality and stability of VR inspection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 A flowchart of an Internet-based intelligent VR inspection method provided by one embodiment of the present invention; Figure 2 A flowchart of obtaining window adjustment degree provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0018] See also Figure 1 , which shows a flow chart of an Internet-based intelligent VR inspection method provided by one embodiment of the present invention, the method includes the following steps: Step S001: collect inspection videos and read image frames frame by frame to obtain different super-pixel areas in the grayscale image frame corresponding to the image frame.

[0019] Use a drone's binocular high-definition camera to collect inspection videos, use wireless transmission technology to transmit the inspection videos, and read each image frame in the inspection video frame by frame.

[0020] In this embodiment, the video frame rate of the inspection video is set to 30FPS, and each image frame in the inspection video is read frame by frame through the OpenCV library function. In actual application, as other implementation methods, the implementer can determine the value of the video frame rate of the inspection video according to actual conditions, and this application does not impose any special restrictions.

[0021] When an image frame contains an exposed area, the pixel values ​​of the pixels in the exposed area are generally greater than or equal to 200, the image details are lost, and there are differences in exposure conditions in different exposed areas. Specifically, the difference in exposure conditions is, for example, that smooth surfaces such as metal and glass are prone to strong reflections in strong light environments, resulting in an overexposure phenomenon, which often causes strips of bright light in the captured image, but the boundaries of the exposed area are clear; images captured from a direct light perspective will have overexposed areas, which are approximately circular white areas with blurred edges. Therefore, it is necessary to divide the different areas in the image frame according to the characteristics of overexposure and perform image enhancement.

[0022] The image frame is grayscaled to obtain a grayscale image frame, and the grayscale image frame is segmented using the SLIC superpixel segmentation algorithm to obtain a superpixel area.

[0023] In this embodiment, the number of super pixel regions is set to 4, and the specific calculation process of the number of super pixel regions is as follows: , Indicates the floor symbol.

[0024] At this point, the superpixel region in each grayscale image frame is obtained.

[0025] Step S002: Determine the edge gradient variance and edge LBP variance of the superpixel region based on the differences in gradient and texture between edge pixels on the edge of the superpixel region, obtain the window adjustment degree of the superpixel region in combination with the number of pixels contained in the superpixel region, calculate the optimal side length of the superpixel region based on the window adjustment degree of the superpixel region, and divide the superpixel region into different local areas to be enhanced based on the optimal side length.

[0026] The edge pixels on the edge of the superpixel region are processed using the Sobel operator to obtain the gradient magnitude of each edge pixel. The variance of the gradient magnitudes of all edge pixels on the edge of the superpixel region is recorded as the edge gradient variance of the superpixel region. The use of the Sobel operator to calculate the gradient magnitude of a pixel is a well-known technique and will not be described in detail here.

[0027] The edge gradient variance of the superpixel region reflects the degree of gradient change of the edge of the superpixel region. When the gradient change of the edge of the superpixel region is greater, the edge of the superpixel region is clearer, the superpixel region is more likely to correspond to the exposure area formed by mirror reflection, and the possibility of corresponding to the overexposure area formed by direct light is smaller.

[0028] Calculate the LBP value of each edge pixel on the edge of the superpixel region, and record the variance of the LBP values ​​of all edge pixels on the edge of the superpixel region as the edge LBP variance of the superpixel region. Calculating the LBP value of a pixel is a well-known technique and will not be described in detail here.

[0029] The edge LBP variance of the superpixel region reflects the richness of the edge texture of the superpixel region. When the richness of the edge texture of the superpixel region is greater, the possibility that the superpixel region corresponds to the exposed area formed by mirror reflection is greater, and the possibility that the superpixel region corresponds to the overexposed area formed by direct light is smaller.

[0030] Calculate the circularity of the superpixel region. Calculating the circularity of a superpixel region is a well-known technique and will not be described in detail. This embodiment calculates the circularity based on the perimeter and area of ​​the superpixel region. The perimeter of the superpixel region is the number of pixels contained in the edge of the superpixel region, and the area of ​​the superpixel region is the number of pixels contained in the superpixel region.

[0031] It can be understood that when the circularity of the superpixel area is greater, if the superpixel area is an exposed area, the shape of the superpixel area is closer to a circle, the superpixel area is more likely to correspond to an overexposed area directly illuminated by light, and the superpixel area is less likely to correspond to an area formed by mirror reflection exposure.

[0032] The edge gradient variance and edge LBP variance of the super-pixel region are weightedly summed to obtain the edge feature value of the super-pixel region. Combined with the circularity of the super-pixel region, the window adjustment degree of the super-pixel region is obtained.

[0033] The preset first weight coefficient and the preset second weight coefficient are used as the weights of the edge gradient variance and the edge LBP variance of the superpixel region, respectively, and the edge gradient variance and the edge LBP variance of the superpixel region are weighted summed to obtain the edge eigenvalue of the superpixel region. The preset third weight coefficient and the preset fourth weight coefficient are used as the weights of the circularity and the edge eigenvalue of the superpixel region, respectively, and the circularity and the edge eigenvalue of the superpixel region are weighted summed to obtain the window adjustment degree of the superpixel region.

[0034] Among them, the sum of the first weight coefficient and the second weight coefficient should be 1, and the sum of the third weight coefficient and the fourth weight coefficient should be 1. In this embodiment, the values ​​of the first weight coefficient, the second weight coefficient, the third weight coefficient and the fourth weight coefficient are 0.6, 0.4, 0.5 and 0.5 respectively.

[0035] The larger the edge feature value of the super-pixel region, the richer the edge texture of the super-pixel region, the greater the gradient change of the edge, the more likely the super-pixel region is to correspond to the exposed area formed by mirror reflection, and the smaller the possibility of the over-exposed area formed by direct light. At this time, the more detail information contained in the super-pixel region, the smaller the side length of the corresponding region should be during the image enhancement process. The flow chart for obtaining the window adjustment degree is as follows: Figure 2 shown.

[0036] According to the window adjustment degree of the super-pixel area, the optimal side length of the super-pixel area is calculated as follows: in, Represents superpixel area The optimal side length of The window base value of the preset super pixel area is 5 in this embodiment. represents the preset window adjustment coefficient. In this embodiment, the value of the window adjustment coefficient is 0.5; Represents superpixel area Window adjustment degree; Represents the upward rounding function, which is used to ensure the super pixel area The optimal side length of is an integer.

[0037] At this point, the optimal side length of each superpixel area is obtained.

[0038] The optimal side length of the superpixel region is used as the side length of the local region to be enhanced, and the superpixel region is divided into different local regions to be enhanced.

[0039] At this point, each superpixel area is divided into different local areas to be enhanced.

[0040] Step S003 , performing image enhancement based on the difference between the pixel values ​​of all pixels in the local area to be enhanced, obtaining a local enhanced area, and obtaining an image frame after image enhancement based on all the local enhanced areas corresponding to the same image frame.

[0041] When overexposure occurs in the local area to be enhanced, the value distribution in the grayscale histogram of the local area to be enhanced is not uniform, and a large number of grayscale values ​​are concentrated in the corresponding highlighted area. Therefore, the part with higher grayscale value frequency can be transferred to the frequency position of other grayscale values ​​through clipping, so as to limit the problem of excessive contrast in the local area. When clipping, a threshold needs to be set to clip the frequencies of grayscale values ​​higher than the threshold, and evenly distribute the clipped frequencies to the frequencies of other grayscale values. It should be noted that when the threshold value is set too large, it is impossible to effectively process the area with too high contrast, and it is impossible to effectively restore the details of the exposed area; when the threshold value is set too small, the pixel values ​​of the overexposed area will be over-clipped, which may aggravate the halo effect or artifacts in the overexposed area. Therefore, it is necessary to set an appropriate threshold for clipping according to the contrast of the local area to be enhanced.

[0042] The product of the standard deviation of the pixel values ​​of all pixels in the local area to be enhanced and the number 2 is rounded down to an integer, which is recorded as the clipping threshold of the local area to be enhanced.

[0043] The clipping threshold of the local area to be enhanced is used as the clipping threshold of the histogram specification algorithm, and the histogram specification algorithm is used to perform image enhancement on the local area to be enhanced to obtain the local enhanced area.

[0044] At this point, the local enhancement region corresponding to each local area to be enhanced divided by all superpixel regions is obtained.

[0045] The image composed of all local enhanced regions corresponding to the same image frame is processed using a histogram equalization algorithm and a bilinear interpolation smoothing method to obtain an image frame after image enhancement, thereby achieving image enhancement of the image frame.

[0046] Among them, using the histogram equalization algorithm and the bilinear interpolation smoothing method to process the image is a well-known technology and will not be described in detail.

[0047] At this point, the enhanced image frame is obtained.

[0048] Step S004: Implement VR inspection based on the image frames after image enhancement.

[0049] The OpenCV library function cv2.cvtColor is used to restore the enhanced image frames to RGB images, and the library function cv2.VideoWriter is used to merge the restored RGB images frame by frame into VR inspection videos. The VR inspection videos are transmitted to smart VR devices via WiFi wireless transmission technology, and the SGM multi-view 3D modeling technology is used to reconstruct the inspection environment, so that the VR device can display the inspection area corresponding to the VR inspection video in real time, clearly and completely.

[0050] The use of the OpenCV library function cv2.cvtColor to restore image frames to RGB images, the library function cv2.VideoWriter to merge RGB images frame by frame into a video, the use of WiFi wireless transmission technology to transmit the video, and the SGM multi-view 3D modeling technology to reconstruct the inspection environment are all well-known technologies and will not be described in detail. As other implementation methods, based on achieving the goal of reconstructing the inspection environment based on the image frames after image enhancement, implementers may adopt other existing methods to achieve this goal, and this application does not impose any special restrictions.

[0051] At this point, VR inspection is realized.

[0052] Based on the same inventive concept as the above method, an embodiment of the present invention also provides an Internet-based intelligent VR inspection system, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the steps of any one of the above-mentioned Internet-based intelligent VR inspection methods are implemented.

[0053] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. The Internet-based intelligent VR inspection method is characterized by: The method comprises the following steps: Collect inspection videos and read image frames frame by frame to obtain different super-pixel areas in the grayscale image frame corresponding to the image frame; Based on the differences in gradient and texture between edge pixels on the edge of the superpixel region, the edge gradient variance and edge LBP variance of the superpixel region are determined respectively. Combined with the number of pixels contained in the superpixel region, the window adjustment degree of the superpixel region is obtained. Based on the window adjustment degree of the superpixel region, the optimal side length of the superpixel region is calculated. Based on the optimal side length, the superpixel region is divided into different local areas to be enhanced; Perform image enhancement based on the difference between the pixel values ​​of all pixels in the local area to be enhanced, obtain the local enhanced area, and obtain the image frame after image enhancement based on all the local enhanced areas corresponding to the same image frame; VR inspection is achieved based on the image frames after image enhancement.

2. The Internet-based intelligent VR inspection method according to claim 1, characterized in that: The specific method for obtaining the edge gradient variance of the superpixel region is: The variance of the gradient amplitudes of all edge pixels on the edge of the super-pixel region is recorded as the edge gradient variance of the super-pixel region.

3. The Internet-based intelligent VR inspection method according to claim 1, characterized in that: The method for determining the edge LBP variance is: The variance of the LBP values ​​of all edge pixels on the edge of the superpixel region is recorded as the edge LBP variance of the superpixel region.

4. The Internet-based intelligent VR inspection method according to claim 1, characterized in that: The method for obtaining the window adjustment degree of the super pixel area is: The circularity of the superpixel region is calculated by taking the perimeter of the superpixel region as the number of pixels contained in the edge of the superpixel region and the area of ​​the superpixel region as the number of pixels contained in the superpixel region; Obtain the edge feature value of the superpixel region based on the edge gradient variance and edge LBP variance of the superpixel region; According to the preset third weight coefficient and the preset fourth weight coefficient, the circularity and edge feature value of the super pixel area are weighted and summed to obtain the window adjustment degree of the super pixel area.

5. The Internet-based intelligent VR inspection method according to claim 4 is characterized in that: The method for obtaining the edge feature value is: According to the preset first weight coefficient and the preset second weight coefficient, the edge gradient variance and the edge LBP variance of the super pixel area are weighted and summed to obtain the edge feature value of the super pixel area.

6. The Internet-based intelligent VR inspection method according to claim 1, characterized in that: The calculation formula for the optimal side length of the superpixel area is: in, Represents superpixel area The optimal side length of Represents the window base value of the preset superpixel area; Indicates the preset window adjustment coefficient; Represents superpixel area Window adjustment degree; Represents the ceiling function.

7. The Internet-based intelligent VR inspection method according to claim 1, characterized in that: The method of dividing the super pixel area into different local areas to be enhanced according to the optimal side length includes: The optimal side length of the superpixel region is used as the side length of the local region to be enhanced, and the superpixel region is divided into different local regions to be enhanced.

8. The Internet-based intelligent VR inspection method according to claim 1, characterized in that: The method for acquiring the image frame after image enhancement is: The product of the standard deviation of the pixel values ​​of all pixels in the local area to be enhanced and the number 2 is rounded down to the integer value, which is recorded as the clipping threshold of the local area to be enhanced; The clipping threshold of the local area to be enhanced is used as the clipping threshold of the histogram specification algorithm, and the histogram specification algorithm is used to perform image enhancement on the local area to be enhanced to obtain a local enhanced area; The image composed of all local enhanced regions corresponding to the same image frame is smoothed to obtain an image frame after image enhancement.

9. The Internet-based intelligent VR inspection method according to claim 1, characterized in that: The VR inspection is implemented based on the image frames after image enhancement, including the following specific methods: The enhanced image frames are restored to RGB images and merged frame by frame into VR inspection videos. The VR inspection videos are transmitted to the intelligent VR device to reconstruct the inspection environment.

10. An Internet-based intelligent VR inspection system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

Images