Image detail enhancement method and system for binocular near-to-eye display equipment

Abstract

The invention relates to the technical field of image processing, in particular to an image detail enhancement method and system for binocular near-to-eye display equipment. According to the technical scheme of the invention, the method comprises the steps: obtaining a first image and a second image collected by a binocular near-to-eye display device under a current display task, dividing the first image and the second image into a plurality of regions, and determining an overlapping region and a non-overlapping region according to the feature similarity between the regions divided by the first image and the second image; respectively configuring a first enhancement coefficient and a second enhancement coefficient for each divided region in the overlapping region and the non-overlapping region; and executing a current display task of the binocular near-to-eye display equipment according to the first enhancement coefficient and the second enhancement coefficient. According to the technical scheme, the corresponding enhancement coefficients are respectively configured based on the overlapping region and the non-overlapping region of the binocular near-eye image, so that more accurate enhancement processing of the binocular near-eye image can be realized, and the display quality of the binocular near-eye image is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image detail enhancement method and system for a binocular near-eye display device. Background Art

[0002] Binocular near-eye display devices are devices designed to provide users with near-eye display experience, such as AR (Augmented Reality) glasses and VR (Virtual Reality) helmets. In recent years, they have been widely used in entertainment, education, medical care, industry and other fields. These devices provide users with an immersive visual experience by simulating or enhancing the real environment. However, due to the limitations of display technology, binocular near-eye display devices have problems such as insufficient resolution when presenting image details, which affects the user's visual experience. Summary of the invention

[0003] In order to solve the technical problem of how to improve the display quality of binocular near-eye images, the object of the present invention is to provide an image detail enhancement method and system for a binocular near-eye display device. The technical solution adopted is as follows: In a first aspect, an embodiment of the present invention provides an image detail enhancement method for a binocular near-eye display device, the method comprising: Acquire a first image and a second image captured by a binocular near-eye display device in a current display task, wherein the first image and the second image are two images captured at different angles at the same time in the current display task; Dividing the first image and the second image into a plurality of regions, and determining overlapping regions and non-overlapping regions according to feature similarities between the regions divided by the first image and the second image; For each divided area in the overlapping area and the non-overlapping area, respectively configure a first enhancement coefficient and a second enhancement coefficient; The current display task of the binocular near-eye display device is performed according to the first enhancement coefficient and the second enhancement coefficient.

[0004] In an optional embodiment, determining the overlapping area and the non-overlapping area according to the feature similarity between the areas divided by the first image and the second image includes: Determine the grayscale value of the region and the shape similarity of each region between the first image and the second image according to the grayscale distribution characteristics of the pixels of the region divided by the first image and the second image; The overlapping area and the non-overlapping area between the first image and the second image are determined according to the regional grayscale values ​​and the shape similarity.

[0005] In an optional embodiment, determining the grayscale value of the region according to the grayscale distribution characteristics of the pixels of the region divided by the first image and the second image includes: Obtain the pixel grayscale, the number of valid pixels and the number of grayscale values ​​of each divided area on the first image and the second image; According to the formula , obtain the Image The gray value of the area ,in, For the Image The grayscale of the pixels in the area, For the Image The number of effective pixels in an area, For the Image The number of gray values ​​in a region, is 1 or 2, is a natural number greater than 0, is the normalization function.

[0006] In an optional embodiment, determining the shape similarity of each region between the first image and the second image according to the grayscale distribution characteristics of the pixels of the regions divided by the first image and the second image includes: Constructing an edge chain code of the corresponding area according to edge pixel points of each divided area on the first image and the second image; According to the edge chain code of each divided area, determine the angle difference between adjacent pixel lines in each area, wherein the pixel line is the line between the geometric center of the edge chain code and the edge pixel point; Pixel sampling and screening are performed according to the angle difference between adjacent pixel lines in each area to obtain the target edge point of each area; Perform polar coordinate histogram transformation on all target edge points in each region to obtain the shape feature vector of the corresponding region; The shape similarity is obtained according to the similarity between the shape feature vector of the divided area on the first image and the shape feature vector of the divided area on the second image.

[0007] In an optional embodiment, pixel sampling and screening are performed according to the angle difference between adjacent pixel lines in each region to obtain the target edge point of each region, including: When the angle difference between the current adjacent pixel lines is greater than a preset first threshold, determining that each edge pixel point corresponding to the current adjacent pixel line is a target edge point; When the angle difference between the current adjacent pixel lines is less than or equal to the first threshold, and the continuous cumulative number is greater than a preset second threshold, a preset number of edge pixel points are screened out from the current adjacent pixel lines as target edge points.

[0008] In an optional embodiment, obtaining shape similarity according to the similarity between the shape feature vector of the divided area on the first image and the shape feature vector of the divided area on the second image includes: According to the formula , get the first image and the second image In the region The distribution similarity of target edge points ,in, The first image is divided into In the region The shape feature vector of the target edge points, The second image is divided into In the region The shape feature vector of the target edge points, cos represents the cosine similarity function, The first image is divided into in the area The target edge point is The number of pixels in a histogram region, The second image is divided into in the area The target edge point is The number of pixels in a histogram region, The number of histogram regions after the polar coordinate histogram is divided into regions; According to the formula , get the first image and the second image The shape similarity of the regions ,in, For the The number of polar histograms in the bin.

[0009] In an optional embodiment, determining the overlapping area and the non-overlapping area between the first image and the second image according to the regional grayscale value and the shape similarity includes: Obtaining a regional grayscale difference value according to a difference in regional grayscale values ​​between any two regions between the first image and the second image; Obtaining a characteristic matching result of the position of the region according to the regional grayscale difference and shape similarity of the region divided by the first image and the second image; Determine the area corresponding to all the characteristic matching results greater than the third threshold as the overlapping area between the first image and the second image; An image region where the first image and the second image are located outside the overlapping region is determined as a non-overlapping region.

[0010] In an optional embodiment, the area divided in the overlapping area is the first area, and the area divided in the non-overlapping area is the second area; for each divided area in the overlapping area and the non-overlapping area, a first enhancement coefficient and a second enhancement coefficient are respectively configured, including: The grayscale values ​​of all pixels in each divided area are averaged to obtain the average grayscale value of each divided area; Obtaining a first enhancement coefficient of the current first region according to the number of regions of the first region, an average grayscale value of the current first region, and an average grayscale value of first regions adjacent to the current first region; A second enhancement coefficient of the current second region is obtained according to the average gray value of the current second region, the average gray value of the target first region, and the first enhancement coefficient, wherein the target first region is the first region closest to the current second region.

[0011] In an optional embodiment, executing a current display task of a binocular near-eye display device according to the first enhancement coefficient and the second enhancement coefficient includes: Determine a first enhanced grayscale value of each pixel in the overlapping area according to the first enhancement coefficient and the grayscale value of each pixel in the overlapping area; Determine a second enhanced grayscale value of each pixel in the non-overlapping area according to the second enhancement coefficient and the grayscale value of each pixel in the non-overlapping area; According to the first enhanced grayscale value and the second enhanced grayscale value, the binocular near-eye display device is controlled to perform stereo matching display.

[0012] In a second aspect, an embodiment of the present invention further provides an image detail enhancement system for a binocular near-eye display device, the system comprising: A shooting terminal is used to obtain a first image and a second image captured by a binocular near-eye display device in a current display task, wherein the first image and the second image are two images captured at different angles at the same time in the current display task; A processing terminal connected to the shooting terminal, the processing terminal is used to divide the first image and the second image into a plurality of regions, and determine the overlapping region and the non-overlapping region according to the feature similarity between the regions divided by the first image and the second image; and respectively configure the first enhancement coefficient and the second enhancement coefficient for each divided region in the overlapping region and the non-overlapping region; The display terminal is connected to the processing terminal, and the display terminal is used to execute the current display task of the binocular near-eye display device according to the first enhancement coefficient and the second enhancement coefficient.

[0013] Compared with the prior art, the image detail enhancement method and system of a binocular near-eye display device provided by the present invention has the following advantages: The technical solution of the present invention obtains the first image and the second image captured by the binocular near-eye display device under the current display task. Since the first image and the second image are two images captured at different angles at the same time in the current display task, the first image and the second image can be divided into multiple areas, and the overlapping area and the non-overlapping area are determined according to the feature similarity between the areas divided by the first image and the second image; for each divided area in the overlapping area and the non-overlapping area, the first enhancement coefficient and the second enhancement coefficient are respectively configured; according to the first enhancement coefficient and the second enhancement coefficient, the current display task of the binocular near-eye display device is executed. The technical solution configures corresponding enhancement coefficients based on the overlapping area and the non-overlapping area of ​​the binocular near-eye image, respectively, and can realize more accurate enhancement processing of the binocular near-eye image, so that the stereoscopic sense of the binocular near-eye image display is stronger, thereby improving the display quality of the binocular near-eye image. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 A flowchart of an image detail enhancement method for a binocular near-eye display device provided by one embodiment of the present invention; Figure 2 A schematic diagram of the principle of image acquisition by a binocular camera provided by one embodiment of the present invention; Figure 3 A schematic diagram of the principle of capturing a first image and a second image provided by an embodiment of the present invention; Figure 4 A schematic diagram of a 4-bit chain code provided by an embodiment of the present invention; Figure 5 A schematic diagram of an 8-bit chain code provided by an embodiment of the present invention; Figure 6 A schematic diagram of a polar coordinate histogram provided by an embodiment of the present invention; Figure 7 A schematic diagram of dividing a first image into regions provided by an embodiment of the present invention; Figure 8 A schematic diagram of dividing a second image into regions provided by an embodiment of the present invention; Fig. 9A schematic diagram of overlapping areas and non-overlapping areas on a binocular near-eye image provided by an embodiment of the present invention; Fig.10 A schematic diagram of configuring enhancement coefficients for a first image provided by an embodiment of the present invention; Fig.11 A schematic diagram of configuring enhancement coefficients for a second image provided by an embodiment of the present invention; Fig.12 A diagram showing the implementation steps of an image detail enhancement method provided by an embodiment of the present invention; Fig.13 A schematic structural diagram of an image detail enhancement system for a binocular near-eye display device provided by an embodiment of the present invention.

[0016] Explanation of reference numerals: 21 - left camera, 22 - right camera, 23 - first image, 24 - second image, 25 - photographed object, 26 - overlapping area, 27 - non-overlapping area, 28 - binocular near-eye image. DETAILED DESCRIPTION

[0017] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the image detail enhancement method and system of a binocular near-eye display device proposed by the present invention, its specific implementation method, structure, features and effects, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0018] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0019] At present, when enhancing the images collected by binocular near-eye display devices, stereo matching is usually performed first, and then image enhancement is performed. When the original image collected has poor effect, the result of its stereo matching may be biased, so that the effect after image enhancement deviates from the effect that should be produced. The present invention analyzes and processes the two images collected under the current display task to identify the overlapping area and non-overlapping area of ​​the two images. Then, the image enhancement coefficients are calculated respectively to complete the image enhancement processing, and the stereo matching is further used to present the results in the binocular near-eye display device, so that the features of the binocular near-eye image are more distinct when displayed, which helps to more accurately identify and match the corresponding feature points in the subsequent stereo matching process, thereby improving the accuracy of stereo image display matching, so that the display effect presented by the binocular near-eye display device is better. The following is a specific scheme of an image detail enhancement method and system for a binocular near-eye display device provided by the present invention in conjunction with the accompanying drawings.

[0020] See also Figure 1 , Figure 1 A flowchart of an image detail enhancement method for a binocular near-eye display device provided by an embodiment of the present invention. The method can be applied to the operation of a binocular near-eye display device. The binocular near-eye display device can be a terminal device such as AR glasses and VR helmets. It only needs to be able to run the enhancement method. The device type is not specifically limited here. The method specifically includes: S11. Acquire a first image and a second image captured by a binocular near-eye display device in a current display task, wherein the first image and the second image are two images captured at different angles at the same time in the current display task.

[0021] Specifically, the binocular near-eye display device is provided with a left camera and a right camera (or binocular camera), see Figure 2 , 21 represents the left camera, 22 represents the right camera, the left camera is used to capture the user's left eye's observation field, and the right camera 22 is used to capture the user's right eye's observation field. 23 represents the first image, 24 represents the second image, the image captured by the left camera is the first image, and the image captured by the right camera is the second image. Since the quality of the lens during shooting directly affects the clarity and accuracy of the image, the use of the lens here needs to be selected according to the specific application scenario. Usually, two cameras are used to collect images in different position relationships (such as on a straight line, on a plane, and in three-dimensional distribution) to achieve a three-dimensional visual effect.

[0022] After acquiring the first image and the second image, they can also be denoised. Preprocessing can improve the image quality and provide a better basis for subsequent processing. When processing the two images collected by the binocular near-eye display device, because when wearing the binocular near-eye display device, people pay more attention to the middle part of the current viewing angle, and pay less attention to the two side areas. Therefore, the enhancement requirements for pixels at different positions on the image are different. It is necessary to analyze the two images collected by the binocular camera to improve the accuracy of image enhancement.

[0023] S12: Divide the first image and the second image into a plurality of regions, and determine overlapping regions and non-overlapping regions according to feature similarities between the regions divided into the first image and the second image.

[0024] Specifically, when the two images collected by the binocular near-eye display device are processed, because the left camera and the right camera shoot the same part, they are limited by the different shooting angles, and the two images collected will have a certain position deviation, so it is impossible to directly use the feature matching algorithm to match the two images collected by the binocular near-eye display device. However, from the perspective of the local whole on the two images, the feature performance of different positions remains basically unchanged. Based on this feature, the two images can be divided into regions, and the regional features of each region can be analyzed to help complete the matching between the images. The first image and the second image can be divided into multiple regions based on the region growing method, or the image partition model can be used to implement the regional division of the first image and the second image. The two images can be divided into regions based on the same division method, and no specific restrictions are made here. It can be understood that since the two images are divided into regions based on the same method, the divided regions have similar features, which is convenient for subsequent accurate processing.

[0025] After completing the region division of the two images, whether it is an overlapping area can be determined based on the feature similarity between the divided areas. When the feature similarity between the divided areas of the first image and the divided areas of the second image is relatively close, it means that the two areas are overlapping areas; otherwise, it means that the two areas are non-overlapping areas.

[0026] Exemplarily, step S12 includes sub-steps S12-1 to S12-2, which are specifically described as follows: S12-1. Determine the regional grayscale value and the shape similarity of each region between the first image and the second image based on the grayscale distribution characteristics of the pixel points of the regions divided by the first image and the second image. For regions with the same characteristic expressions on the two images, there is a certain similarity in their grayscale expressions. The regional grayscale value represents the grayscale expression of the divided region. After grayscale processing is performed on both the first image and the second image, the regional grayscale value can be calculated based on the grayscale value of the pixel points in each region. The shape similarity represents the degree of shape similarity between each region between the first image and the second image. The shape similarity can be determined based on the shape presentation characteristics of the divided regions.

[0027] In a specific implementation, determining the grayscale value of the region includes S12-1-1 and S12-1-2, which are specifically described as follows: S12-1-1. Obtain the grayscale of the pixels, the number of valid pixels, and the number of grayscale values ​​for each divided area on the first image and the second image. The grayscale of the pixel characterizes the grayscale value of the pixel in each divided area, which can be obtained based on the grayscale processing results of the first image and the second image; the number of valid pixels is the number of pixels with grayscale values ​​in the divided area, which can be obtained based on the grayscale statistics of the pixels in each area; the number of grayscale values ​​is the number of grayscale values ​​included in a single area, which can also be obtained based on the grayscale statistics of the pixels. For example, a certain area contains 20 pixels, of which 10 pixels have a grayscale value of 210, 5 pixels have a grayscale value of 150, 3 pixels have a grayscale value of 100, and 2 pixels have a grayscale value of 0, then the number of valid pixels is 18 and the number of grayscale values ​​is 3.

[0028] S12-1-2, according to the formula , obtain the Image The gray value of the area ,in, For the Image The grayscale of the pixels in the area, For the Image The number of effective pixels in an area, For the Image The number of gray values ​​in a region, is 1 or 2, is a natural number greater than 0. 1 hour That is, the first image The grayscale value of the region, 2 o'clock That is, the The grayscale value of the region, is the normalization function. It can be understood that the normalization function can make the number of gray values In the preset range, prevent the gray value of the area Exceeds the grayscale value range 0-255.

[0029] See also Figure 3 , 23 represents the first image, 24 represents the second image, and 25 represents the photographed object. Since the first image and the second image are partially similar, the above analysis is mainly calculated using the grayscale representation of the image collected by the binocular near-eye display device. There are overlapping parts in the images collected by the binocular near-eye display device. The grayscale distribution of these image features has spatial errors, but they still have great similarity in grayscale representation.

[0030] Analyzing similar parts of the image captured by the binocular near-eye display device, it is not accurate to analyze only the grayscale expression on the image, and it may not be the same area due to other different expressions. For each area on the image captured by the binocular near-eye display device, the shape expression of the same object under a slight change in viewing angle may be relatively similar. Based on this, in a specific embodiment, determining the shape similarity of each area between the first image and the second image includes S12-1-3 to S12-1-5, which are specifically described as follows: S12-1-3. Based on the edge pixels of each divided area on the first image and the second image, construct an edge chain code for the corresponding area. The edge pixel points are the pixels located at the edge of the divided area. The edge chain code can be constructed based on the edge pixel points. Figure 4 and Figure 5 , can be based on Image The edge pixels in the area are selected, and the first pixel on the leftmost vertical line is selected as the starting pixel, and the edge chain code is constructed in the clockwise direction. Figure 4 and Figure 5 The 4-bit chain code and the 8-bit chain code are shown respectively.

[0031] S12-1-4. Determine the angle difference between adjacent pixel lines in each region according to the edge chain code of each divided region, wherein the pixel line is the line between the geometric center of the edge chain code (i.e., the region center) and the edge pixel point. The angle change between a pixel line and its adjacent pixel line is recorded as , based on the angle change between adjacent pixel lines , to the first Image The edge pixels of the region are sampled to calculate the angle difference between the lines connecting adjacent pixels: , , Respectively Image Region pixel lines and The angle between the pixel line and the reference line, which can be a horizontal line or a vertical line. is a natural number greater than 1.

[0032] S12-1-5, pixel sampling and screening are performed according to the angle difference between adjacent pixel lines in each region to obtain the target edge point of each region. Through pixel sampling and screening, effective shape features of the region can be obtained, reducing data calculation redundancy.

[0033] The method of pixel sampling and screening can be selected based on actual needs. For example, when the angle difference of the current adjacent pixel lines is greater than a preset first threshold, it is determined that all edge pixel points corresponding to the current adjacent pixel lines are target edge points; when the angle difference of the current adjacent pixel lines is less than or equal to the first threshold, and the continuous cumulative number is greater than a preset second threshold, a preset number of edge pixel points in the current adjacent pixel lines are screened out as target edge points.

[0034] Specifically, set the first threshold ,when When , it means that the angle change between adjacent pixel lines is large, and the corresponding edge pixels need to be retained; otherwise, it means that the angle change between adjacent pixel lines is relatively gentle, and they can be deleted during sampling. Calculated by adjacent pixel lines , then in this continuous Select from pixels Pixels are deleted and the remaining edge pixels are sampled. In the above method, characteristic edge pixels can be screened out from all edge pixels and used as target edge points.

[0035] S12-1-6. Perform polar coordinate histogram transformation on all target edge points in each region to obtain the shape feature vector of the corresponding region.

[0036] Specifically, a polar coordinate system with each sampled target edge point as the center is constructed. In the polar coordinate system, the other target edge points of the current target edge point on the edge are divided according to angle and distance, and the number of target edge points in each divided area is counted to form a polar coordinate histogram. For example, a region contains 8 target edge points. A polar coordinate system is constructed with each target edge point as the center. Multiple line segments are obtained based on the connection between the current target edge point and other target edge points. Multiple angles are obtained through the angles between the multiple line segments and the horizontal axis or vertical axis of the polar coordinate system. The multiple angles are counted to obtain the histogram of the current target edge point. Please refer to Figure 6 , angle division according to each Divide once; the distance division is divided according to the division distance of the angle in the polar coordinate system, so that each small area divided can be a square or rectangle. Figure 6 The horizontal axis is the angle , the vertical axis is the distance After dividing the polar coordinate histogram into regions, we can get regions, each small region in the polar coordinate histogram is recorded as , is a natural number greater than 1.

[0037] The polar coordinate histograms of all target edge points in each region are combined to form a high-dimensional feature vector, namely the shape feature vector (or shape context descriptor). It can be understood that since the shape feature vector is obtained based on the polar coordinate histogram transformation, each target edge point has a corresponding histogram, so the shape feature vector can capture the shape edge structure and relative position information of the region. The shape feature vector is recorded as .

[0038] , where Indicates Image Region No. The histogram vector of target edge points, Indicates the first The number of pixels in a small area.

[0039] S12-1-7, obtaining shape similarity based on the similarity between the shape feature vector of the divided area on the first image and the shape feature vector of the divided area on the second image. The shape similarity characterizes the degree of similarity between the divided area on the first image and the divided area on the second image, and the shape similarity can be obtained based on the cosine similarity between the two shape feature vectors.

[0040] The shape similarity of two regions can be calculated based on the following methods: The first step is to use the formula , get the first image and the second image In the region The distribution similarity of target edge points ,in, The first image is divided into In the region The shape feature vector of the target edge points, The second image is divided into In the region The shape feature vector of the target edge points, cos represents the cosine similarity function, The first image is divided into in the area The target edge point is The number of pixels in a histogram region, The second image is divided into in the area The target edge point is The number of pixels in a histogram region, The number of histogram regions after the polar coordinate histogram is divided into regions. is a natural number greater than 1.

[0041] It can be understood that the above formula is calculated based on the similarity between the histogram vectors of the target edge points reflected on the collected images and the Euclidean distance between the two shape feature vectors. For the shape similarity analysis of the same area on two images, the higher the similarity of the vector features corresponding to a single dimension and the smaller the Euclidean distance calculated by the corresponding histogram, the smaller the difference in shape features of a certain area on the two images in this dimension and the higher the shape similarity of the two areas.

[0042] The second step is to use the formula , get the first image and the second image The shape similarity of the regions ,in, For the The number of polar histograms in the bin.

[0043] This formula is based on the characteristic performance in a single dimension (the histogram corresponding to a target edge point) analyzed above, and calculates the Image The shape feature vector corresponding to the region is Image The shape similarity between the shape feature vectors corresponding to the regions. When analyzing the shape similarity between the entire region and another region, only analyzing with the features of a single dimension may lead to ignoring some information. Therefore, it is necessary to combine the feature performance of multiple dimensions (histograms corresponding to multiple target edge points) to analyze the shape feature performance of the region. This calculation method can improve the accuracy of the shape similarity calculation of two regions on two images.

[0044] Repeat the above steps to calculate, and then the grayscale value and shape similarity of each area on the first image and the second image can be obtained.

[0045] S12-2, determine the overlapping area and non-overlapping area between the first image and the second image according to the regional gray value and shape similarity. The regional gray value reflects the brightness of the corresponding area, and the shape similarity can measure the similarity of the regional shapes. By comparing the gray values ​​and shapes of these areas, find out the similar areas in the first image and the second image. These similar areas are overlapping areas, which means that they correspond to the same or similar parts in the image; and dissimilar areas are non-overlapping areas, that is, different parts in the image.

[0046] Sub-step S12-2 includes S12-2-1 and S12-2-4, which are described in detail as follows: S12-2-1. Obtain a regional grayscale difference value based on the difference in regional grayscale values ​​between any two regions of the first image and the second image. The regional grayscale value of the first image is recorded as , the regional gray value of the second image is recorded as , the difference between the two can be used to obtain the regional grayscale difference.

[0047] S12-2-2, according to the regional grayscale difference and shape similarity of the region divided by the first image and the second image, obtain the characteristic matching result of the region position. Figure 7 and Figure 8 , there are similar areas on the first image and the second image, Figure 7 FIG. 4 shows the region 1 and region 2 divided by the first image. Figure 8 , region 1 and region 2 divided by the second image are shown. Whether the two regions are similar can be determined based on the grayscale difference and shape similarity of the two regions on the two images. The degree of similarity between the two regions is characterized based on the feature matching result.

[0048] For example, based on the formula , calculate the characteristic matching results of the two regions , the shape similarity of two regions compared in the first image and the second image, For the first image The grayscale value of the region, For the second image The grayscale value of the region, is the normalization function.

[0049] It can be understood that the above formula is based on the analysis and calculation of the shape similarity and grayscale performance difference of two areas on the image captured by the binocular near-eye display device. For the overlapping parts of the image captured by the device, the shape similarity is high and the grayscale performance similarity is high, which means that a certain area of ​​the two images captured at two angles is relatively overlapped and the regional characteristics match well.

[0050] S12-2-3, determine the area corresponding to all feature matching results greater than the third threshold as the overlapping area between the first image and the second image. The third threshold can be set based on actual needs, and it only needs to be able to accurately distinguish the overlapping area of ​​the two images. Based on the calculation results analyzed above, the third threshold can be set , when the calculated When , it means that the regional features of the two regions match well. Then choose to calculate Construct a data sequence in the region of: . Based on the data sequence constructed each time ,Will The two areas on the corresponding two pictures are taken as corresponding areas. If a region on an image captured by a binocular near-eye display device matches the region characteristics calculated from another image, If the data sequence cannot be constructed, it means that there is no corresponding area on the image, and the area is not an overlapping area. Repeat the above steps to complete the matching and recognition of the overlapping areas on the two images.

[0051] S12-2-4, determine the image area of ​​the first image and the second image outside the overlapping area as a non-overlapping area. Fig. 9 , 26 represents the overlapping area, 27 represents the non-overlapping area, and 28 represents the binocular near-eye image. In the figure, area a, area b and area c are all in the overlapping area. The area enclosed by the dotted line is the overlapping area, and the area outside the dotted line is the non-overlapping area.

[0052] So far, the matching and identification of the overlapping parts of the two images captured by the binocular near-eye display device and the identification of the non-overlapping areas have been completed.

[0053] S13. For each divided area in the overlapping area and the non-overlapping area, respectively configure a first enhancement coefficient and a second enhancement coefficient.

[0054] Specifically, it is necessary to perform graphic enhancement on the image captured by the binocular near-eye display device based on the recognized image results. However, since people pay different attention to different positions on the image when wearing a binocular near-eye display device, the degree of enhancement of images at different positions is also different. Generally speaking, the image located in the middle of the line of sight will greatly attract the user's visual attention (that is, the overlapping part of the two images), and the degree of image enhancement in the overlapping area will be greater than that in the non-overlapping areas on both sides. The corresponding enhancement coefficient can be configured based on the regional grayscale values ​​of the overlapping area and the non-overlapping area. For example, the regional grayscale value is input into a preset calculation model, and the first enhancement coefficient of the divided area in the overlapping area and the second enhancement coefficient of the divided area in the non-overlapping area are obtained based on the output result of the calculation model.

[0055] In practical applications, since the first enhancement coefficient and the second enhancement coefficient are related to the enhanced display effect of the image, if there is a deviation in the configuration of each enhancement coefficient, it may cause problems such as abruptness when the overall image is displayed. Therefore, the image enhancement degree of the non-overlapping parts on both sides should be analyzed based on the middle overlapping part.

[0056] Exemplarily, the area divided in the overlapping area is the first area, and the area divided in the non-overlapping area is the second area; step S13 includes sub-steps S13-1 to S13-3, which are specifically described as follows: S13-1, perform mean processing on the grayscale values ​​of all pixels in each divided area to obtain the average grayscale value of each divided area. The average grayscale value of the pixel points in each divided area can be calculated based on the mean processing model, and the average grayscale value of each divided area can be obtained through the output result of the mean processing model.

[0057] S13-2, obtaining a first enhancement coefficient of the current first region according to the number of first regions, the average grayscale value of the current first region, and the average grayscale value of first regions adjacent to the current first region.

[0058] Specifically, we can use the formula , calculate the average gray value of the pixels in the overlapping area of ​​the two images , where is the number of overlapping areas between the two images, For the first image The grayscale value of the area.

[0059] Further, from The specific information of the overlapping areas on the image is analyzed and the first enhancement coefficient is calculated by the formula , calculate the overlapping area The first enhancement factor of the region , where For the Image The area next to The average grayscale value of the first region adjacent to the current first region, M is the total number of first regions adjacent to the current first region.

[0060] It can be understood that the image enhancement coefficients of all overlapping areas can be obtained by repeating the above steps. The above calculation method is to analyze the degree of enhancement of a certain area based on its basic conditions, mainly using the difference between the average grayscale value of a certain area and the average grayscale value of the overall overlapping part, as well as the difference with the average grayscale value of the surrounding area. When the grayscale value of a certain area on the image collected by the binocular near-eye display device is significantly different from the overall grayscale value, and is significantly different from the grayscale value of the surrounding area, it means that the area is more prominent in the image and is subject to greater image enhancement. Please refer to Fig.10 and Fig.11 , the divided areas of the overlapping areas in the figure can be configured with a larger first enhancement coefficient based on the above method.

[0061] S13-3. Obtain a second enhancement coefficient of the current second region according to the average grayscale value of the current second region, the average grayscale value of the target first region, and the first enhancement coefficient, wherein the target first region is the first region closest to the current second region.

[0062] Specifically, in order to prevent the image from being displayed abruptly, the image enhancement coefficient of the overlapping area can be used to calculate the first Image The image enhancement factor for non-overlapping regions.

[0063] For example, according to the formula , calculate the second enhancement coefficient of the current second area , that is, Image The second enhancement coefficient of non-overlapping regions is: For distance Image The enhancement coefficient calculated from the first area closest to the non-overlapping areas, that is, the first enhancement coefficient of the target first area; The enhancement factor is The average gray value of the corresponding area, that is, the average gray value of the first area of ​​the target; For the Image The average gray value of non-overlapping areas.

[0064] Based on the above formula, the enhancement coefficient of a non-overlapping area can be analyzed based on the enhancement coefficient of the nearby area that has been calculated, and then analyzed in combination with its grayscale performance. When the grayscale value of a certain area is greater than the grayscale value of the nearby area with a known enhancement coefficient, in order to ensure that the original grayscale performance law remains unchanged, it is subjected to a stronger degree of image enhancement.

[0065] Repeat the above steps to obtain the image enhancement coefficients of all regions on the two images. So far, the image enhancement coefficients of all regions on the two images have been obtained.

[0066] S14. Execute a current display task of the binocular near-eye display device according to the first enhancement coefficient and the second enhancement coefficient.

[0067] Specifically, after obtaining the image enhancement coefficients of all areas on the two images, the images can be enhanced based on the enhancement coefficients of each area, and the two enhanced images can be used to complete stereo matching, so that the stereo image can be presented to the user through a binocular near-eye display device.

[0068] Exemplarily, step S14 includes S14-1 to S14-3, which are specifically described as follows: S14-1. Determine the first enhanced grayscale value of each pixel in the overlapping area according to the first enhancement coefficient and the grayscale value of each pixel in the overlapping area. This can be based on the formula , calculate the first The first enhanced gray value of the pixel , In the overlapping area The gray value of a pixel, For the The first enhancement coefficient of pixels, is a natural number greater than 1.

[0069] S14-2, determining the second enhanced grayscale value of each pixel in the non-overlapping area according to the second enhancement coefficient and the grayscale value of each pixel in the non-overlapping area. Similarly, based on the formula , calculate the first The second enhanced gray value of the pixel , In the non-overlapping area The gray value of a pixel, For the The second enhancement coefficient of pixels, is a natural number greater than 1.

[0070] It should be noted that, when implementing the first enhanced gray value and the second enhanced gray value When calculating, normalization processing can be performed based on actual conditions so that the calculated enhanced grayscale value is within the grayscale value range of 0-255.

[0071] S14-3. According to the first enhanced grayscale value and the second enhanced grayscale value, control the binocular near-eye display device to perform stereo matching display. Specifically, an adaptive method and a specific similarity measurement factor can be used to complete the stereo matching calculation of the two images and obtain a disparity space image (DSI). The initial disparity image is obtained through the two steps of cost aggregation and disparity calculation. Further, the stereo matching of the two images is completed by proposing steps such as error disparity, smoothing processing, and sub-pixel precision optimization. Therefore, the depth image can be determined based on the disparity image, and the 3D information of the captured scene can be restored.

[0072] The following embodiments of the present invention will generally describe the image detail enhancement method of a binocular near-eye display device. Fig.12 , specifically including: S901, image acquisition: a first image is captured by the left camera, and a second image is captured by the right camera.

[0073] S902: Region division: Divide the first image and the second image into a plurality of regions.

[0074] S903, identifying overlapping areas and non-overlapping areas: Based on the grayscale features and shape similarities of the divided areas, the overlapping areas and non-overlapping areas of the first image and the second image are determined.

[0075] S904: Calculate image enhancement coefficients. Configure enhancement coefficients for the overlapping area and the non-overlapping area respectively.

[0076] S905: Image enhancement: Perform image enhancement on the first image and the second image based on the calculated image enhancement coefficient.

[0077] S906: Stereo matching: Perform stereo matching processing on the first image and the second image.

[0078] S907, presenting by a binocular near-eye display device.

[0079] Based on the same technical concept as the enhancement method, the embodiment of the present invention also provides an image detail enhancement system for a binocular near-eye display device, see Fig.13 , Fig.13 1 is a schematic diagram of the structure of the enhancement system. The enhancement system includes a shooting terminal 101, a processing terminal 102 and a display terminal 103.

[0080] The shooting terminal 101 is used to obtain the first image and the second image captured by the binocular near-eye display device under the current display task, wherein the first image and the second image are two images captured at different angles at the same time in the current display task; the processing terminal 102 is connected to the shooting terminal 101, and the processing terminal 102 is used to divide the first image and the second image into multiple areas, and determine the overlapping area and the non-overlapping area according to the feature similarity between the areas divided by the first image and the second image; and configure the first enhancement coefficient and the second enhancement coefficient for each divided area in the overlapping area and the non-overlapping area respectively; the display terminal 103 is connected to the processing terminal 102, and the display terminal 103 is used to execute the current display task of the binocular near-eye display device according to the first enhancement coefficient and the second enhancement coefficient.

[0081] It should be noted that the shooting terminal 101 may be a shooting device composed of a left camera and a right camera, the processing terminal 102 may be a device composed of a data processing chip, and the display terminal 103 may be a display screen capable of implementing stereoscopic display.

[0082] The technical solution provided in the embodiments of the present invention has at least the following technical effects or advantages: The technical solution of the embodiment of the present invention obtains the first image and the second image captured by the binocular near-eye display device under the current display task. Since the first image and the second image are two images captured at different angles at the same time in the current display task, the first image and the second image can be divided into multiple regions, and the overlapping region and the non-overlapping region are determined according to the feature similarity between the regions divided by the first image and the second image; the first enhancement coefficient and the second enhancement coefficient are respectively configured for each divided region in the overlapping region and the non-overlapping region; and the current display task of the binocular near-eye display device is executed according to the first enhancement coefficient and the second enhancement coefficient. The technical solution configures corresponding enhancement coefficients based on the overlapping region and the non-overlapping region of the binocular near-eye image, respectively, and can realize more accurate enhancement processing of the binocular near-eye image, so that the stereoscopic sense of the binocular near-eye image display is stronger, thereby improving the display quality of the binocular near-eye image.

[0083] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0084] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A method for enhancing image details of a binocular near-eye display device, characterized in that: The method comprises: Acquire a first image and a second image captured by a binocular near-eye display device in a current display task, wherein the first image and the second image are two images captured at different angles at the same time in the current display task; Dividing the first image and the second image into a plurality of regions, and determining overlapping regions and non-overlapping regions according to feature similarities between the regions divided into the first image and the second image; For each divided area in the overlapping area and the non-overlapping area, respectively configure a first enhancement coefficient and a second enhancement coefficient; A current display task of the binocular near-eye display device is performed according to the first enhancement coefficient and the second enhancement coefficient.

2. The image detail enhancement method of a binocular near-eye display device according to claim 1, characterized in that: The determining of overlapping areas and non-overlapping areas according to the feature similarity between the areas divided by the first image and the second image includes: Determine the grayscale value of the region and the shape similarity of each region between the first image and the second image according to the grayscale distribution characteristics of the pixel points of the regions divided by the first image and the second image; An overlapping area and a non-overlapping area between the first image and the second image are determined according to the area grayscale value and the shape similarity.

3. The image detail enhancement method of a binocular near-eye display device according to claim 2, characterized in that: Determining the grayscale value of the region according to the grayscale distribution characteristics of the pixels of the region divided by the first image and the second image includes: Obtaining the pixel grayscale, the number of valid pixels and the number of grayscale values ​​of each divided area on the first image and the second image; According to the formula , obtain the Image The gray value of the area ,in, For the Image The gray level of the pixels in the area, For the Image The number of effective pixels in an area, For the Image The number of gray values ​​in a region, is 1 or 2, is a natural number greater than 0, is the normalization function.

4. The image detail enhancement method of a binocular near-eye display device according to claim 2, characterized in that: Determining shape similarities of regions between the first image and the second image according to grayscale distribution characteristics of pixels of regions divided by the first image and the second image includes: Constructing an edge chain code of a corresponding area according to edge pixels of each divided area on the first image and the second image; Determine the angle difference between adjacent pixel lines in each area according to the edge chain code of each divided area, wherein the pixel line is a line connecting the geometric center of the edge chain code and the edge pixel point; Pixel sampling and screening are performed according to the angle difference between adjacent pixel lines in each area to obtain the target edge point of each area; Perform polar coordinate histogram transformation on all target edge points in each region to obtain the shape feature vector of the corresponding region; The shape similarity is obtained according to the similarity between the shape feature vector of the divided area on the first image and the shape feature vector of the divided area on the second image.

5. The image detail enhancement method of a binocular near-eye display device according to claim 4, characterized in that: The pixel sampling and screening is performed according to the angle difference between adjacent pixel lines in each area to obtain the target edge point of each area, including: When the angle difference between the current adjacent pixel lines is greater than a preset first threshold, determining that each edge pixel point corresponding to the current adjacent pixel line is a target edge point; When the angle difference between the current adjacent pixel lines is less than or equal to the first threshold, and the continuous cumulative number is greater than a preset second threshold, a preset number of edge pixel points are screened out from the current adjacent pixel lines as target edge points.

6. The method for enhancing image details of a binocular near-eye display device according to claim 4, characterized in that: The obtaining the shape similarity according to the similarity between the shape feature vector of the area divided on the first image and the shape feature vector of the area divided on the second image comprises: According to the formula , obtain the first image and the second image In the region The distribution similarity of target edge points ,in, The first image is divided into In the region The shape feature vector of the target edge points, The second image is divided into In the region The shape feature vector of the target edge points, cos represents the cosine similarity function, The first image is divided into in the area The target edge point is The number of pixels in a histogram region, The second image is divided into in the area The target edge point is The number of pixels in a histogram region, The number of histogram regions after the polar coordinate histogram is divided into regions; According to the formula , obtain the first image and the second image The shape similarity of the regions ,in, For the The number of polar histograms in the bin.

7. The image detail enhancement method of a binocular near-eye display device according to claim 2, characterized in that: The determining, according to the regional grayscale values ​​and the shape similarity, the overlapping region and the non-overlapping region between the first image and the second image comprises: Obtaining a regional grayscale difference value according to a difference in regional grayscale values ​​between any two regions of the first image and the second image; Obtaining a characteristic matching result of the position of the region according to the regional grayscale difference and shape similarity of the region divided by the first image and the second image; Determine an area corresponding to all characteristic matching results greater than a third threshold as an overlapping area between the first image and the second image; An image area of ​​the first image and the second image outside the overlapping area is determined as the non-overlapping area.

8. The method for enhancing image details of a binocular near-eye display device according to claim 1, characterized in that: The area divided in the overlapping area is the first area, and the area divided in the non-overlapping area is the second area; For each divided area in the overlapping area and the non-overlapping area, respectively configure a first enhancement coefficient and a second enhancement coefficient, including: Performing mean processing on the grayscale values ​​of all pixels in each divided area to obtain an average grayscale value of each divided area; Obtaining a first enhancement coefficient of the current first region according to the number of regions of the first region, an average grayscale value of the current first region, and an average grayscale value of first regions adjacent to the current first region; A second enhancement coefficient of the current second region is obtained according to the average gray value of the current second region, the average gray value of the target first region, and the first enhancement coefficient, wherein the target first region is the first region closest to the current second region.

9. The method for enhancing image details of a binocular near-eye display device according to claim 1, characterized in that: The performing a current display task of the binocular near-eye display device according to the first enhancement coefficient and the second enhancement coefficient includes: Determine a first enhanced grayscale value of each pixel in the overlapping area according to the first enhancement coefficient and the grayscale value of each pixel in the overlapping area; Determine a second enhanced grayscale value of each pixel in the non-overlapping area according to the second enhancement coefficient and the grayscale value of each pixel in the non-overlapping area; According to the first enhanced grayscale value and the second enhanced grayscale value, the binocular near-eye display device is controlled to perform stereo matching display.

10. An image detail enhancement system for a binocular near-eye display device, characterized in that: The system comprises: A shooting terminal, used to obtain a first image and a second image captured by a binocular near-eye display device in a current display task, wherein the first image and the second image are two images captured at different angles at the same time in the current display task; a processing terminal connected to the shooting terminal, the processing terminal being used to divide the first image and the second image into a plurality of regions, and determine overlapping regions and non-overlapping regions according to feature similarities between the regions divided by the first image and the second image; and respectively configure a first enhancement coefficient and a second enhancement coefficient for each divided region within the overlapping region and the non-overlapping region; A display terminal is connected to the processing terminal, and is used to execute a current display task of the binocular near-eye display device according to the first enhancement coefficient and the second enhancement coefficient.

Images