An image processing method, apparatus, device and storage medium

Abstract

The application provides an image processing method, device and equipment and a storage medium. The method comprises: acquiring binocular images captured by a head-mounted display device; determining edge curvature in a first image in the binocular images; determining edge points in the first image according to the edge curvature; determining a matching point corresponding to each edge point in a second image of the binocular images; constructing a three-dimensional perspective model corresponding to the binocular images according to the matching points; and determining a binocular display image corresponding to the head-mounted display device according to the three-dimensional perspective model. The application adaptively collects edge points under different curvatures according to edge curvature distribution, so as to greatly improve the edge distortion problem that may exist in the binocular display image. According to each matching point, a corresponding three-dimensional perspective model is constructed, the overall unity of the perspective scene is ensured, and the problem of distortion of the overall scene after perspective is avoided.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, device, and storage medium. Background Technology

[0002] With the widespread use of virtual reality (VR), augmented reality (AR), and mixed reality (MR) devices in daily life, corresponding see-through functions are usually configured to ensure user convenience, so that users can see the real environment when wearing such devices.

[0003] However, current video perspective techniques often suffer from overall scene distortion and edge aberration, thus necessitating the design of a more efficient and realistic perspective method. Summary of the Invention

[0004] This application provides an image processing method, apparatus, device, and storage medium to achieve accurate and effective processing of binocular display images and improve the realism of binocular images after perspective.

[0005] In a first aspect, embodiments of this application provide an image processing method applied to a head-mounted display device, the method comprising:

[0006] Acquire binocular images captured by a head-mounted display device;

[0007] Determine the edge curvature within the first image of the binocular images;

[0008] Based on the edge curvature, determine the edge points within the first image;

[0009] In the second image of the binocular image, determine the matching point corresponding to each edge point;

[0010] Based on the matching points, construct a three-dimensional perspective model corresponding to the binocular image;

[0011] Based on the three-dimensional model to be viewed, the binocular display image corresponding to the head-mounted display device is determined.

[0012] Secondly, embodiments of this application provide an image processing apparatus configured in a head-mounted display device, the apparatus comprising:

[0013] The image acquisition module is used to acquire binocular images captured by the head-mounted display device;

[0014] An edge curvature determination module is used to determine the edge curvature within the first image of the binocular images;

[0015] An edge point determination module is used to determine edge points within the first image based on the edge curvature.

[0016] A matching point determination module is used to determine the matching point corresponding to each edge point in the second image of the stereo image;

[0017] A 3D model construction module is used to construct a 3D perspective model corresponding to the binocular image based on the matching points.

[0018] The display image determination module is used to determine the binocular display image corresponding to the head-mounted display device based on the three-dimensional model to be viewed.

[0019] Thirdly, embodiments of this application provide an electronic device, which includes:

[0020] A processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to perform the image processing method provided in the first aspect of this application.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium for storing a computer program that causes a computer to perform the image processing method provided in the first aspect of this application.

[0022] Fifthly, embodiments of this application provide a computer program product, including a computer program / instructions, characterized in that, when the computer program / instructions are executed by a processor, they implement the image processing method provided in the first aspect of this application.

[0023] This application provides an image processing method, apparatus, device, and storage medium that acquires binocular images captured by a head-mounted display device. Then, based on the edge curvature within the first image of the binocular images, corresponding edge points are determined to ensure adaptive acquisition of edge points under different curvatures, thereby significantly improving the edge aliasing problem that may exist in binocular perspective images. Next, matching points corresponding to each edge point are determined in the second image of the binocular images, and a corresponding three-dimensional model to be perspectived is constructed based on each matching point, ensuring the overall uniformity of the scene to be perspectived. This, in turn, determines the binocular display image corresponding to the three-dimensional model to be perspectived, avoiding distortion of the overall scene after perspective and improving the realism of the binocular image after perspective. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating an image processing method according to an embodiment of this application;

[0026] Figure 2 This is a flowchart illustrating the method for constructing a three-dimensional model to be viewed from perspective, as shown in an embodiment of this application.

[0027] Figure 3 This is a schematic diagram illustrating the principle of binocular image perspective process in an embodiment of this application;

[0028] Figure 4 This is a flowchart illustrating the process of realizing perspective in a head-mounted display device for multiple binocular image scenes as shown in the embodiments of this application;

[0029] Figure 5 This is a schematic block diagram of an image processing apparatus according to an embodiment of this application;

[0030] Figure 6 This is a schematic block diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0033] Considering that existing perspective functions often suffer from overall scene distortion or edge aliasing, this application designs a novel perspective processing scheme for binocular images. By adaptively determining the corresponding edge points based on the edge curvature within the first image of the binocular image, the edge aliasing problem that may exist in binocular perspective images is significantly mitigated. Then, in the second image of the binocular image, matching points corresponding to each edge point are determined, and based on each edge point and matching point, a corresponding 3D model to be perspectived is constructed, ensuring the overall uniformity of the scene to be perspectived. This, in turn, determines the binocular display image corresponding to the 3D model to be perspectived, avoiding overall scene distortion after perspective and improving the realism of the binocular image after perspective.

[0034] Figure 1 This is a flowchart illustrating an image processing method according to an embodiment of this application. The method can be executed by the image processing apparatus provided in this disclosure, which can be implemented in any software and / or hardware manner. Exemplarily, the image processing apparatus can be applied to any electronic device, including but not limited to tablet computers, mobile phones (such as foldable phones, large-screen phones, etc.), wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), smart TVs, smart screens, high-definition TVs, 4K TVs, smart speakers, smart projectors, and other devices equipped with binocular cameras. This application does not limit the specific type of electronic device.

[0035] Specifically, such as Figure 1 As shown, the method may include the following steps:

[0036] S110: Acquire binocular images captured by a head-mounted display device.

[0037] After a user puts on a head-mounted display device, it is usually equipped with a see-through function to make it convenient for the user to use, so that the user can see the real environment while wearing the head-mounted display device.

[0038] Therefore, to ensure the accuracy of the perspective function in a head-mounted display device, a camera is typically installed at different locations on the device to represent the user's field of vision. In this case, real-time external images of the user's surrounding environment are captured by the cameras at different locations, serving as the binocular images in this application. For example, the image captured by the left camera is used as the left-eye image, and the image captured by the right camera is used as the right-eye image.

[0039] S120, determine the edge curvature within the first image of the binocular images.

[0040] For binocular images captured by a head-mounted display device, the first image can be any image in the binocular images, and the second image can be another image in the binocular images.

[0041] In this application, to address the potential edge distortion or warping issues during perspective viewing, any one of the binocular images is used as a reference image of the real-world external scene, i.e., the first image in this application. By analyzing the curvature distribution of each edge curve in the first image, the degree of curvature of the edge curve at different positions is determined, allowing for the adaptive determination of the appropriate number of edge points. Furthermore, since binocular images are simultaneously captured by cameras at different positions within the head-mounted display device of the same external environment, the same target point will exhibit parallax in the binocular images. Therefore, to ensure efficient and accurate binocular image processing, after acquiring the binocular images, this application first performs binocular correction on the first and second images to eliminate distortions and ensure that the epipolar lines of the binocular images are on the same horizontal line. Thus, for any one of the binocularly corrected first and second images, any feature point within that image will have its matching point on the other image on the same horizontal line.

[0042] S130, determine the edge points within the first image based on the edge curvature.

[0043] After determining the edge curvature within the first image, this application can adaptively determine the corresponding number of edge points according to the curvature of the edge curve at different positions in the first image.

[0044] In other words, a larger number of edge points can be identified where the edge curvature is high, while a smaller number of edge points are identified where the edge curvature is low. This ensures adaptive acquisition of edge points under different curvatures. Furthermore, a high edge curvature indicates a greater degree of curvature, so a larger number of edge points can more comprehensively refine the edge features in this area, thus minimizing edge distortion or warping during perspective. Conversely, a low edge curvature indicates a lower degree of curvature (i.e., a straighter surface), so even with a smaller number of edge points, the edge features can be accurately represented, thus avoiding edge distortion or warping during perspective.

[0045] As an optional implementation of this application, determining the edge points within the first image based on edge curvature can specifically include the following steps:

[0046] The first step is to extract edges from the first image to obtain the corresponding edge feature map.

[0047] This application employs an edge detection algorithm to analyze the grayscale changes of each pixel in a first image, and extracts all feature points reflecting grayscale changes from the first image using a preset edge operator. Then, some boundary points are removed or boundary discontinuities are filled from all extracted feature points. Finally, the remaining feature points are connected to form corresponding edge curves, resulting in the corresponding edge feature map.

[0048] The second step is to determine the corresponding edge points based on the edge curvature in the edge feature map.

[0049] By analyzing the curvature of each edge curve in the edge feature map at any feature point, the degree of curvature of the edge curve at that feature point can be represented. Then, based on the positive correlation between edge curvature and the number of edge points, an appropriate number of edge points are adaptively collected at various positions of the edge curve, so that more edge points can be identified in areas with large edge curvature, and fewer edge points can be identified in areas with small edge curvature.

[0050] S140, determine the matching point corresponding to each edge point in the second image of the stereo image.

[0051] After identifying the edge points in the first image, and considering that the first and second images were captured in real-time by two cameras of the same external scene, meaning that the same target object exists in both images, this application performs feature analysis on each pixel in the second image to accurately analyze the real external scene of the user's environment. Then, based on the feature information of each edge point in the first image, matching points for each edge point can be found in the second image. Here, edge points and matching points can be different representations of the same edge position points on the same target object in the real external scene, respectively, in the first and second images.

[0052] In this application, an optical flow algorithm is used to calculate the matching point of each corner point in the first image in the second image. Since the first and second images are obtained by capturing the same real-world scene using different cameras, the target positions of the two target objects in the real-world scene have the same difference in the first and second images. Therefore, for each edge point in the first image, this application can analyze the feature position difference between the edge point and its nearest corner point. Then, based on this feature position difference and the matching point of the nearest corner point in the second image, the matching point of the edge point in the second image can be determined. Thus, for each edge point in the first image, the same steps can be performed to determine the matching point of each edge point in the second image.

[0053] As an optional implementation of this application, binocular correction is performed using the first and second images to eliminate distortions in the binocular images and ensure that the epipolar lines of the binocular images are on the same horizontal line. Therefore, after binocular correction of the first and second images, it can be determined that each edge point in the corrected first image and its matching point in the corrected second image are on the same horizontal coordinate. Thus, determining the matching point corresponding to each edge point in the second image of the binocular image can be specifically achieved by performing horizontal optical flow calculations on each edge point in the second image to obtain its matching point.

[0054] In other words, by analyzing the row coordinates of each edge point in the second image, the horizontal optical flow of that edge point can be calculated directly at the same row coordinates in the second image to obtain the matching point. This transforms the two-dimensional optical flow algorithm into a one-dimensional optical flow calculation, significantly reducing the computational overhead of edge matching and improving its efficiency.

[0055] S150: Based on the matching points, construct a 3D perspective model corresponding to the binocular image.

[0056] Since the matching points in the second image are also edge points of the real target object, it means that the grayscale changes of other feature points in the second image are not obvious. Therefore, even if such feature points are stretched or deformed during perspective, it will not be obvious.

[0057] Therefore, this application can connect the matching points in the second image to extract the edge feature map within the second image. Then, by analyzing the depth information of each matching point in the second image, the second image can be transformed into a three-dimensional space according to the edge feature map, thus obtaining the corresponding three-dimensional spatial model. At this point, since the three-dimensional spatial model can prominently represent the three-dimensional line relationships existing in the real external scene, it can be used as the three-dimensional perspective model in this application.

[0058] S160, Based on the three-dimensional model to be viewed, determine the binocular display image corresponding to the head-mounted display device.

[0059] Since the two cameras representing the user's eyes are mounted in different positions, the rendering angles for the same 3D model to be viewed from different perspectives are also different. Therefore, the 3D model to be viewed from different perspectives can be projected onto the rendering angles represented by the different cameras, thereby obtaining corresponding binocular display images. Then, by displaying the binocular display images on the corresponding eye-level display interfaces, efficient and realistic binocular image perspective functionality can be achieved within the head-mounted display device.

[0060] As an optional implementation scheme in this application, in order to ensure the accuracy of the binocular display image, the coordinate system corresponding to each camera can be determined by analyzing the camera parameters of each installed camera as the corresponding binocular coordinate system.

[0061] At this point, the determination of the binocular display image corresponding to the head-mounted display device based on the three-dimensional model to be viewed in this application may specifically include the following steps: rasterizing the three-dimensional model to be viewed into the binocular coordinate system to obtain the binocular display image corresponding to the head-mounted display device.

[0062] By using a rasterization algorithm, the three-dimensional model to be viewed is transformed into corresponding pixels in the binocular coordinate system through coordinate transformation and geometric discretization in the binocular coordinate system, thereby obtaining the corresponding binocular display image.

[0063] The technical solution provided in this application acquires binocular images captured by a head-mounted display device. Then, based on the edge curvature within the first image of the binocular images, corresponding edge points are determined. This ensures adaptive acquisition of edge points under different curvatures, significantly improving the edge aliasing problem that may exist in binocular perspective images. Next, matching points are determined for each edge point in the second image of the binocular images. Based on each matching point, a corresponding 3D model to be viewed is constructed, ensuring the overall uniformity of the scene to be viewed. This, in turn, determines the binocular display image corresponding to the 3D model to be viewed, avoiding distortion of the overall scene after perspective and improving the realism of the binocular image after perspective.

[0064] As described above, this application constructs a corresponding 3D perspective model by matching points within the second image. This allows for the subsequent use of the rasterization of the 3D perspective model in a binocular coordinate system to achieve efficient and realistic perspective functionality of the binocular image within a head-mounted display device. The following section combines... Figure 2 The following is an exemplary description of the specific construction process of the three-dimensional model to be viewed in this application.

[0065] like Figure 2 As shown, the process of constructing this 3D model to be viewed from perspective may include the following steps:

[0066] S210, based on the matching points, perform triangulation on the second image under edge constraints to obtain the corresponding triangulated mesh image.

[0067] To accurately represent the 3D scene structure of the second image, this application can use each matching point within the second image as the vertices of a triangulation. Then, as... Figure 3 As shown, based on the edge constraint principle between each matching point, the second image is triangulated under this edge constraint (i.e., Delaunay triangulation), resulting in the corresponding triangulated mesh image. This triangulated mesh image can then optimally represent the three-dimensional structural relationships within the second image.

[0068] It should be noted that each vertex of the triangulated mesh image is a matching point within the second image. Furthermore, the grayscale variations within each mesh of the triangulated mesh image are not significant. Even if stretching or distortion occurs during subsequent perspective transformations, it does not affect the realistic perception of perspective. Therefore, it can greatly improve the problem of distortion in the overall scene after perspective transformation.

[0069] S220 uses the depth information of each matching point to convert the triangulated mesh image into a three-dimensional model to be viewed.

[0070] By determining the depth information of each matching point in the second image, the depth information of each grid vertex in the triangulation mesh image can be determined. Then, according to the depth information of each grid vertex in the triangulation mesh image, each grid vertex is transformed into three-dimensional space. At the same time, during the transformation of each grid vertex, other feature points in the triangulation mesh image can be transformed into corresponding three-dimensional spaces, thereby converting the triangulation mesh image into a three-dimensional model to be viewed from perspective.

[0071] The technical solution in this embodiment performs edge-constrained triangulation on the second image based on each matching point, and then converts the triangulated mesh image into a three-dimensional perspective model, ensuring the overall uniformity of the perspective scene and improving the realism of binocular perspective.

[0072] As an optional implementation in this application, to ensure the comprehensiveness of the scene view within the head-mounted display device, multiple sets of binocular cameras are pre-installed at different locations within the head-mounted display device to comprehensively capture the external scene of the user's environment. Taking a head-mounted display in a VR device as the head-mounted display device in this application, typically with four cameras installed, they are usually installed in the upper left, upper right, lower left, and lower right corners of the head-mounted display. In this case, the cameras in the upper left and upper right corners form one set of binocular cameras, used to capture the corresponding binocular images. The cameras in the lower left and lower right corners form another set of binocular cameras, also used to capture the corresponding binocular images.

[0073] Therefore, it can be seen that after installing multiple sets of binocular cameras, this application can also acquire two or more sets of binocular images. Then, for each set of binocular images, the image processing method provided in the above embodiments of this application can be executed to obtain a binocular display image corresponding to each set of binocular images. The following is in conjunction with... Figure 4 This paper explains the process of realizing the perspective function of binocular images in a head-mounted display device under multiple binocular image scenarios.

[0074] like Figure 4 As shown, the process of realizing perspective in a head-mounted display device for binocular images in multiple binocular image scenes may include the following steps:

[0075] S410 acquires binocular images captured by a head-mounted display device.

[0076] S420, determine the edge curvature within the first image of the binocular images.

[0077] S430, determine the edge points within the first image based on the edge curvature.

[0078] S440, determine the matching point corresponding to each edge point in the second image of the stereo image.

[0079] S450: Based on the matching points, construct a 3D perspective model corresponding to the binocular image.

[0080] S460 determines the binocular display image corresponding to the head-mounted display device based on the three-dimensional model to be viewed.

[0081] S470, if there are two or more sets of binocular images, then based on the binocular display images corresponding to each set of binocular images, determine the corresponding left-eye display image set and right-eye display image set.

[0082] If two or more sets of binocular images are acquired, then for each set of binocular images, steps S410-S460 are executed to obtain the binocular display image corresponding to each set of binocular images.

[0083] At this point, in order to ensure the realism and integrity of the binocular images after perspective, this application will merge the left and right display images in the binocular display images corresponding to multiple sets of binocular images to obtain the corresponding left display image set and right display image set.

[0084] S480: For each display image set in the left-eye display image set and the right-eye display image set, perform image fusion on each display image in the display image set to obtain an optimized binocular display image.

[0085] For each display image set in both the left and right eye display image sets, image fusion processing is required to obtain the optimized display image for either eye. Therefore, the overlapping regions between the display images within each display image set are first analyzed. Then, the Poisson blending algorithm is used to fuse the display images within each display image set according to these overlapping regions. After fusing the perspective images in both the left and right eye display image sets, the optimized binocular display image is obtained.

[0086] The technical solution provided in this application embodiment achieves optimized binocular display images by fusing images of each binocular display image corresponding to multiple sets of binocular images, thus ensuring the comprehensiveness of the scene after binocular image perspective.

[0087] Figure 5 This is a schematic block diagram of an image processing apparatus according to an embodiment of this application. Figure 5 As shown, the device 500 may include:

[0088] Image acquisition module 510 is used to acquire binocular images captured by a head-mounted display device;

[0089] The edge curvature determination module 520 is used to determine the edge curvature within the first image of the binocular image;

[0090] Edge point determination module 530 is used to determine edge points within the first image based on the edge curvature;

[0091] The matching point determination module 540 is used to determine the matching point corresponding to each edge point in the second image of the stereo image;

[0092] The 3D model construction module 550 is used to construct a 3D perspective model corresponding to the binocular image based on the matching points.

[0093] The image determination module 560 is used to determine the binocular display image corresponding to the head-mounted display device based on the three-dimensional model to be viewed.

[0094] In some embodiments of this application, the edge point determination module 530 can be specifically used for:

[0095] Edge extraction is performed on the first image to obtain the corresponding edge feature map;

[0096] The corresponding edge points are determined based on the edge curvature in the edge feature map.

[0097] In some embodiments of this application, the three-dimensional model construction module 550 can be specifically used for:

[0098] Based on the matching points, the second image is triangulated under edge constraints to obtain the corresponding triangulated mesh image;

[0099] Using the depth information of each matching point, the triangulated mesh image is converted into a three-dimensional model to be viewed.

[0100] In some embodiments of this application, the matching point determination module 540 can be specifically used for:

[0101] For each of the aforementioned edge points, horizontal optical flow calculations are performed in the second image to obtain the matching point of that edge point.

[0102] In some embodiments of this application, the image determination module 560 can be specifically used for:

[0103] The three-dimensional model to be viewed is rasterized into a binocular coordinate system to obtain the binocular display image corresponding to the head-mounted display device.

[0104] In some embodiments of this application, the image processing apparatus 500 may further include:

[0105] A binocular correction module is used to perform binocular correction on the first image and the second image.

[0106] In some embodiments of this application, the image processing apparatus 500 may further include:

[0107] The image set determination module is used to determine the corresponding left-eye display image set and right-eye display image set based on the binocular display image corresponding to each group of binocular images if there are two or more groups of binocular images.

[0108] The display image optimization module is used to perform image fusion on each display image set in the left-eye display image set and the right-eye display image set to obtain an optimized binocular display image.

[0109] In this embodiment, binocular images captured by a head-mounted display device are acquired. Then, according to the edge curvature within the first image of the binocular images, corresponding edge points are determined, thereby ensuring adaptive acquisition of edge points under different curvatures. This greatly improves the edge aliasing problem that may exist in binocular perspective images. Next, matching points corresponding to each edge point are determined in the second image of the binocular images. Based on each matching point, a corresponding 3D model to be perspectived is constructed, ensuring the overall uniformity of the scene to be perspectived. This, in turn, determines the binocular display image corresponding to the 3D model to be perspectived, avoiding distortion of the overall scene after perspective and improving the realism of the binocular image after perspective.

[0110] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 5 The apparatus 500 shown can execute any of the method embodiments provided in this application, and the foregoing and other operations and / or functions of each module in the apparatus 500 are respectively for implementing the corresponding processes in the various methods of the embodiments of this application. For the sake of brevity, they will not be described in detail here.

[0111] The apparatus 500 of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0112] Figure 6 This is a schematic block diagram of the electronic device 600 provided in the embodiments of this application.

[0113] like Figure 6 As shown, the electronic device 600 may include:

[0114] The system includes a memory 610 and a processor 620. The memory 610 stores computer programs and transfers the program code to the processor 620. In other words, the processor 620 can retrieve and run the computer program from the memory 610 to implement the methods described in the embodiments of this application.

[0115] For example, the processor 620 can be used to execute the above-described method embodiments according to instructions in the computer program.

[0116] In some embodiments of this application, the processor 620 may include, but is not limited to:

[0117] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0118] In some embodiments of this application, the memory 610 includes, but is not limited to:

[0119] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0120] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 610 and executed by the processor 620 to perform the method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0121] like Figure 6 As shown, the electronic device may also include:

[0122] Transceiver 630, which can be connected to processor 620 or memory 610.

[0123] The processor 620 can control the transceiver 630 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 630 may include a transmitter and a receiver. The transceiver 630 may further include antennas, and the number of antennas may be one or more.

[0124] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0125] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0126] When implemented using software, it can be implemented entirely or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0127] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0128] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0129] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0130] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, Applications in head-mounted display devices, including: Acquire binocular images captured by a head-mounted display device; Determine the edge curvature within the first image of the binocular images; Based on the edge curvature, determine the edge points within the first image; In the second image of the binocular image, determine the matching point corresponding to each edge point; Based on the matching points, construct a three-dimensional perspective model corresponding to the binocular image; Based on the three-dimensional model to be viewed, determine the binocular display image corresponding to the head-mounted display device; The step of determining the edge points within the first image based on the edge curvature includes: Edge extraction is performed on the first image to obtain the corresponding edge feature map; Based on the edge curvature in the edge feature map, a corresponding number of edge points are collected from the edge curve of the edge feature map, and the edge curvature is positively correlated with the number of edge points.

2. The method according to claim 1, characterized in that, The step of constructing a 3D perspective model corresponding to the binocular image based on the matching points includes: Based on the matching points, the second image is triangulated under edge constraints to obtain the corresponding triangulated mesh image; Using the depth information of each matching point, the triangulated mesh image is converted into a three-dimensional model to be viewed.

3. The method according to claim 1, characterized in that, Determining the matching point corresponding to each edge point in the second image of the binocular image includes: For each of the aforementioned edge points, horizontal optical flow calculations are performed in the second image to obtain the matching point of that edge point.

4. The method according to claim 1, characterized in that, The step of determining the binocular display image corresponding to the head-mounted display device based on the three-dimensional model to be viewed includes: The three-dimensional model to be viewed is rasterized into a binocular coordinate system to obtain the binocular display image corresponding to the head-mounted display device.

5. The method according to any one of claims 1-4, characterized in that, Before determining the edge curvature within the first image of the binocular images, the method further includes: Binocular correction is performed on the first and second images.

6. The method according to any one of claims 1-4, characterized in that, The method further includes: If there are two or more sets of binocular images, then based on the binocular display images corresponding to each set of binocular images, the corresponding left-eye display image set and right-eye display image set are determined. For each display image set in the left-eye display image set and the right-eye display image set, image fusion is performed on each display image in the display image set to obtain an optimized binocular display image.

7. An image processing apparatus, characterized in that, Configured for head-mounted display devices, including: The image acquisition module is used to acquire binocular images captured by the head-mounted display device; An edge curvature determination module is used to determine the edge curvature within the first image of the binocular images; An edge point determination module is used to determine edge points within the first image based on the edge curvature. A matching point determination module is used to determine the matching point corresponding to each edge point in the second image of the stereo image; A 3D model construction module is used to construct a 3D perspective model corresponding to the binocular image based on the matching points. The image determination module is used to determine the binocular display image corresponding to the head-mounted display device based on the three-dimensional model to be viewed. The edge point determination module is specifically used for: Edge extraction is performed on the first image to obtain the corresponding edge feature map; Based on the edge curvature in the edge feature map, a corresponding number of edge points are collected from the edge curve of the edge feature map, and the edge curvature is positively correlated with the number of edge points.

8. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store a computer program, the processor being used to invoke and run the computer program stored in the memory to perform the image processing method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the image processing method according to any one of claims 1-6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the image processing method according to any one of claims 1-6.

Images