Image correction method and device, electronic equipment and readable storage medium

Abstract

The invention discloses an image correction method and device, electronic equipment and a readable storage medium, and belongs to the technical field of electronic equipment. The method comprises the following steps: determining an image correction parameter according to azimuth angle information of a sight line when a user watches M mark points in a target image; the target image is a test image displayed by an optical machine of the XR equipment or a virtual reality image displayed by the XR equipment, and M is a positive integer; and according to the image correction parameter, carrying out image correction on a first image displayed by the XR equipment, and outputting a second image.

Description

Technical Field

[0001] The present application belongs to the technical field of electronic equipment, and specifically relates to an image correction method, device, electronic equipment and readable storage medium. Background Art

[0002] With the development of extended reality technology, XR devices have become key devices for providing a fully immersive experience. XR devices present two images with stereoscopic parallax on binocular cameras, and then form stereoscopic vision through binocular fusion of the human eye. However, there are a lot of equipment and assembly tolerances in the production process of XR devices, which may cause the geometric position of the binocular cameras to deviate from the design value, easily leading to unnecessary parallax of the binocular images, thus affecting the normal use of XR devices.

[0003] In the related art, usually during the production stage of the XR device, the image presented by the optical machine is captured by a camera, and the calibration parameters for binocular image position compensation are generated based on the difference in the obtained images, thereby realizing the correction of binocular parallax. However, the traditional binocular parallax correction scheme can only realize the calibration of binocular cameras in a fixed state, and the position of the binocular cameras will change when the interpupillary distance (IPD) of the XR device is adjusted or during use due to wearing pressure, aging of structural parts and other factors. When the position of the binocular cameras changes, the traditional parallax correction method cannot guarantee the matching of the stereoscopic parallax of the binocular images, resulting in the inability to correctly fuse the images, resulting in poor display effects of the XR device. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image correction method, device, electronic device and readable storage medium, which can match the stereoscopic parallax of the image, thereby improving the display effect of the XR device.

[0005] In a first aspect, an embodiment of the present application provides an image correction method, the method comprising: determining image correction parameters based on the azimuth information of the user's line of sight when he or she is looking at M marker points in a target image; the target image is a test image displayed by an optical machine of an XR device, or is a virtual reality image displayed by the XR device, and M is a positive integer; according to the image correction parameters, performing image correction on a first image displayed by the XR device, and outputting a second image.

[0006] In a second aspect, an embodiment of the present application provides an image correction device, comprising: a processing module; a processing module for determining image correction parameters based on the azimuth information of the user's line of sight when the user looks at M marker points in the target image; the target image is a test image displayed by the optical machine of the XR device, or is a virtual reality image displayed by the XR device, and M is a positive integer; the processing module is also used to perform image correction on the first image displayed by the XR device according to the image correction parameters, and output a second image.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0011] In an embodiment of the present application, the XR device determines the image correction parameters according to the azimuth information of the user's line of sight when the user is gazing at M marker points in the target image, the target image being a test image displayed by the optical machine of the XR device, or being a virtual reality image displayed by the XR device, M being a positive integer, and the XR device performs image correction on the first image displayed by the XR device according to the image correction parameters, and outputs a second image. Through this method, the XR device can accurately determine the image correction parameters according to the azimuth information of the user's line of sight when the user is gazing at the binocular image, so that when the optical machine position changes during the use of the XR device due to factors such as pupil distance adjustment, wearing pressure or aging of structural parts, the image correction parameters can be determined in a timely and accurate manner to correct the image. In addition, since the azimuth information of the user's line of sight can more accurately reflect the user's perspective, correcting the image based on the azimuth information of the user's line of sight can ensure that the user always sees a clear and accurate image, thereby matching the stereoscopic parallax of the image, thereby improving the display effect of the XR device. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1A schematic diagram of a flow chart of an image correction method provided in some embodiments of the present application;

[0013] Figure 2 A schematic diagram of a test image provided for some embodiments of the present application;

[0014] Figure 3 A schematic diagram of a marker point in a VST scene provided for some embodiments of the present application;

[0015] Figure 4 A schematic diagram of sight angles provided for some embodiments of the present application;

[0016] Figure 5 A schematic diagram of left and right eye images provided for some embodiments of the present application;

[0017] Figure 6 A schematic diagram of a test image provided for some embodiments of the present application;

[0018] Figure 7 A schematic diagram of the structure of an image correction device provided in some embodiments of the present application;

[0019] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0020] Fig. 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0022] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0023] The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0024] The following is an explanation of the terms involved in the embodiments of the present application.

[0025] Virtual Reality (VR): Virtual reality is a computer-generated environment that simulates the real world or the imaginary world. By using head-mounted displays, handles, sensors and other devices, users can interact with the virtual environment and get an immersive feeling. By simulating sensory experiences such as vision, hearing, and touch, virtual reality technology makes users feel that they are in an environment that is completely different from the real world.

[0026] VR glasses: short for "virtual reality head-mounted display device", also known as VR headset. VR glasses are an important part of virtual reality technology, which can provide users with an immersive visual experience. By wearing VR glasses, users can see the virtual environment generated by the computer and interact with the virtual environment through devices such as handles.

[0027] Mixed Reality (MR): Mixed Reality is a technology that builds an interactive feedback loop between the real world, the virtual world and the user, and enhances the realism of the user experience by introducing virtual scene information. Mixed Reality technology combines the characteristics of Augmented Reality (AR) and Virtual Reality, and can switch freely between virtual and real, providing users with a more natural, realistic and practical interactive experience.

[0028] MR glasses: MR glasses are a new device that combines augmented reality technology and virtual reality technology. It can integrate virtual information into the real world, bringing users a richer and more three-dimensional visual experience. Unlike traditional VR glasses, MR glasses do not need to be completely isolated from the external environment. Users can see objects and scenes in the real world through the glasses, and these objects and scenes in the real world can also interact with virtual information. For example, users can see virtual menus or information prompts on the walls of the real world through MR glasses. This technology provides users with a more intuitive and natural way of interaction, greatly expanding the application scenarios of mixed reality technology.

[0029] Stereopsis: Stereopsis, also known as stereoscopic vision or stereoscopic perception, is the depth perception based on binocular parallax. Specifically, since the distance between a person's eyes is about 65mm and they are almost parallel, a separate image is formed on the retina of each eye when observing external objects. The two images mostly overlap, but there are some slight differences, which are called parallax. When the nerve impulses caused by the two images with parallax are transmitted to the visual cortex of the brain, the fusion of the cerebral cortex will form a three-dimensional visual image.

[0030] The image correction method provided in the embodiments of the present application can be applied to XR game scenarios.

[0031] In an XR gaming device, in order to provide an immersive gaming experience, the device needs to ensure that the displayed image matches the user's line of sight and position. However, due to possible errors in the optical display system of the device, or visual differences in the user (such as pupil distance, vision, etc.), the displayed image may be distorted or distorted. In an embodiment of the present application, the XR gaming device displays a test image through a binocular camera, in which M markers are set. The M markers can be fixed image features, such as black dots or specific objects or characters in the game scene. When the user wears the XR gaming device and looks at these markers, the XR gaming device captures the user's line of sight information, such as the azimuth of the line of sight, through a built-in sensor, and by analyzing the line of sight azimuth when the user looks at each marker, the image correction parameters required for image correction can be calculated, and then the currently displayed game image is corrected according to the calculated image correction parameters, and the corrected image is output to ensure that the image matches the user's line of sight and position. In this way, by outputting the image corrected by the image truth parameter to the user, a more accurate and realistic gaming experience is provided, thereby improving the display effect of the XR gaming device.

[0032] The execution subject of the image correction method provided in the embodiment of the present application may be an image correction device, and the image correction device may be an electronic device, or a functional module or functional entity in an electronic device. The image correction method provided in the embodiment of the present application is described below by taking an XR device executing the image correction method as an example.

[0033] The image correction method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0034] Figure 1 A schematic diagram of the process of the image correction method provided in the embodiment of the present application is shown in FIG. Figure 1 The image correction method shown may include the following steps 201 and 202:

[0035] Step 201: The XR device determines image correction parameters based on the azimuth information of the user's line of sight when the user is gazing at M marker points in the target image.

[0036] Among them, the above-mentioned target image is a test image displayed by the optical machine of the XR device, or a virtual reality image displayed by the XR device, and M is a positive integer.

[0037] In some embodiments of the present application, the XR device may be any of the following: a Virtual Reality (VR) device, an Augmented Reality (AR) device, or a Mixed Reality (MR) device.

[0038] In some embodiments of the present application, the above-mentioned XR device may be an XR head-mounted device or XR glasses, etc.

[0039] In some embodiments of the present application, the above-mentioned M marker points may include one or more marker points, and the M marker points may be obvious feature points that are preset or naturally present in the target image, and are used to assist in image correction or positioning.

[0040] In some embodiments of the present application, the target image may be a test image specifically used for image correction displayed by the optical machine of the XR device, which is used to calibrate the display performance of the XR device, or it may be a virtual reality image displayed by the XR device, for users to interact and experience it.

[0041] It should be noted that the optical machine is a complex system that integrates multiple optical elements and technologies, and is used to accurately present virtual images in the user's eyes. Specifically, the optical machine can convert digital images or video signals into optical images, and display the optical images on the lenses of MR glasses, or directly project them onto the user's retina through some form of optical system.

[0042] In some embodiments of the present application, the target image may be a left-eye image, that is, an image corresponding to the left eye; or the target image may be a right-eye image, that is, an image corresponding to the right eye.

[0043] For ease of understanding, the left-eye image may also be referred to as the left-eye image, and the right-eye image may also be referred to as the right-eye image.

[0044] Exemplarily, the XR device can display the test image to the left eye and the right eye respectively, and the presentation of the test image includes simultaneous presentation to both eyes and independent presentation to the left and right eyes, such as Figure 2 As shown in (A) in FIG, the test images for the left eye and the right eye are presented simultaneously, and each of them includes four marking points, such as Figure 2 As shown in (B) in the figure, the left eye test image is first presented, which includes four marking points, such as Figure 2 As shown in (C) in the figure, a test image for the right eye is first presented, which includes four marking points, wherein the four marking points are distributed symmetrically with respect to the center of the test image.

[0045] It should be noted that Figure 2 The circle in the figure is used to illustrate the test image, and the black dots in the circle are used to illustrate the landmark points in the test image.

[0046] In some embodiments of the present application, the test image may be any image used for verification or calibration, such as a standard image containing known geometric shapes or patterns. For example, when the target image is a test image, the M identification points may be clearly specified and easily identifiable landmarks in the test image, such as specific corner points, circles, or intersections.

[0047] In some embodiments of the present application, the virtual reality image may be a frame of a video see-through technology (VST) scene or the entire virtual environment view. When the target image is a virtual reality image, the M markers may be specific objects with significant features or representativeness selected from the scene, decorations in the corners of a room, specific accessories on a virtual character, or fixed landmark buildings in the scene. Furthermore, the user may select specific objects in the scene as binocular gaze markers through the handle.

[0048] For example, Figure 3 A schematic diagram is selected for VST scene markers, such as Figure 3 As shown, the VST scene includes a square object A and a diamond object B. The user uses a handle or gestures to select position 1 (i.e., the upper left corner of square object A) and position 2 (i.e., the lower right corner of diamond object B) of the image in the VST scene seen by the left eye, and position 3 (i.e., the upper left corner of square object A) and position 4 (i.e., the lower right corner of diamond object B) of the image seen by the right eye. Based on the user's operation, the XR device determines the two marker points 1 and 2 corresponding to the left eye and the two marker points 3 and 4 corresponding to the right eye.

[0049] It should be noted that Figure 3 The object vertices are circled to indicate the landmarks selected from the VST scene.

[0050] In an embodiment of the present application, by capturing the user's binocular gaze information in the VST scene, image correction parameters are generated, so that the fusion deviation of the VST binocular camera combined with the binocular camera can be calculated, and image compensation parameters for this fusion deviation can be generated to achieve a calibration effect.

[0051] In some embodiments of the present application, the above-mentioned line of sight may be the line of sight of the user's left eye, and the above-mentioned target image may be the left-eye image; or, the above-mentioned line of sight may be the line of sight of the user's right eye, and the above-mentioned target image may be the right-eye image.

[0052] In some embodiments of the present application, the azimuth information of the above-mentioned line of sight is used to characterize the angle of the user's line of sight relative to a reference direction (such as the front of the XR device).

[0053] In some embodiments of the present application, the azimuth angle information of the above-mentioned line of sight may include a yaw angle, a pitch angle, a roll angle, etc.

[0054] It should be noted that in three-dimensional space, Euler Angles are usually used to describe the rotation of an object. Among them, the yaw angle is the angle of rotation of the object around its vertical axis (Y axis), which describes the rotation of the object around the vertical axis. When the object rotates to the right around the vertical axis, the yaw angle is positive; when it rotates to the left, the yaw angle is negative; the pitch angle is the angle of rotation of the object around the horizontal axis (X axis). When the object tilts forward or backward, a pitch angle is generated. When the object rotates in the positive direction around the axis, the pitch angle is positive; when it rotates in the negative direction, the pitch angle is negative; the roll angle is the angle of rotation around the longitudinal axis (Z axis). When the object rolls along the horizontal plane, a roll angle is generated. When the object rotates in the positive direction around the axis, the roll angle is positive; when it rotates in the negative direction, the roll angle is negative.

[0055] It should be noted that in the XR display scenario, the yaw angle can represent the left and right rotation angle of the user's head, and the pitch angle can represent the up and down swing angle of the user's head.

[0056] In some embodiments of the present application, the XR device can identify the azimuth of the user's line of sight through a camera or sensor system when the user is gazing at M marker points of the target image to obtain azimuth information of the line of sight.

[0057] Exemplarily, the XR device captures the user's eye image through a camera, and uses an image processing algorithm to extract key information such as the eye movement trajectory and pupil position from the eye image, and then calculates the user's line of sight direction based on the extracted eye movement trajectory and pupil position information.

[0058] Specifically, taking the XR device as MR glasses as an example, the gaze point system of the MR glasses can obtain the user's line of sight axis data when gazing at each landmark point, including the left eye: (pitch_L, yaw_L)_i and the right eye: (pitch_R, yaw_R)_i, where i represents the line of sight data pitch angle and yaw angle of the i-th point. Figure 4 Schematic diagram of sight angle.

[0059] In some embodiments of the present application, the XR device can calculate the parameters required for image correction, namely, image correction parameters, based on the azimuth information of the user's line of sight and the position information of the marker points.

[0060] It should be noted that the XR device uses a binocular display system, that is, it provides independent images for the left eye and the right eye respectively to simulate the visual difference between the two eyes of the human body, thereby creating a three-dimensional stereoscopic view. When a user wears an XR device that can correctly perform parallax fusion, the azimuth of the user's line of sight when looking at a certain point displayed by the XR device is determined. For example, when the user looks at the center of the image displayed by the XR device, the yaw angle and pitch angle of the user's line of sight should be 0 degrees. When the optical machine of the XR device is offset due to factors such as pupil distance adjustment or structural aging, the line of sight angle of the user when looking at the center position will change to a certain extent, making the yaw angle and pitch angle of the user's line of sight not equal to 0 degrees. Therefore, by obtaining the azimuth information of the user's line of sight when looking at a specific marker point in the image displayed by the XR device, the displayed image can be adjusted accordingly by detecting the change in the azimuth information to maintain the correct display of the binocular image and parallax fusion.

[0061] In an embodiment of the present application, the XR device can track the azimuth information of the user's line of sight in real time, and perform image correction based on the azimuth information to ensure that the displayed image is consistent with the user's line of sight.

[0062] In some embodiments of the present application, the above-mentioned image correction parameters may include at least one of the following: a horizontal displacement compensation parameter, a vertical displacement compensation parameter, a rotation compensation parameter, a size compensation parameter, etc.

[0063] In some embodiments of the present application, the horizontal displacement compensation parameters can be determined based on the translation amount of the target image in the horizontal direction; the vertical displacement compensation parameters can be determined based on the translation amount of the target image in the vertical direction; the rotation compensation parameters can be determined based on the rotation angle of the target image; and the size compensation parameters can be determined based on the scaling ratio of the target image.

[0064] Step 202: The XR device performs image correction on the first image displayed by the XR device according to the image correction parameters, and outputs a second image.

[0065] In some embodiments of the present application, the first image may be a video frame captured in real time or a pre-generated image.

[0066] In some embodiments of the present application, the first image may be an image corresponding to the left eye displayed by the XR device, or an image corresponding to the right eye displayed by the XR device.

[0067] It should be noted that in XR scenarios, in order to achieve stereoscopic vision, XR devices usually present the left eye image and the right eye image on the left and right optical machines respectively. There is a slight stereoscopic parallax between the two images. When the user wears VR glasses, the left and right eyes receive the corresponding images respectively, and the three-dimensional sense is formed through the fusion of the brain.

[0068] In some embodiments of the present application, the first image may be an image in a VST scene.

[0069] In some embodiments of the present application, the XR device reads image correction parameters through a rendering system, performs correction processing on the image displayed by the XR device, and realizes parallax correction of the image.

[0070] The image correction process is described below with reference to the accompanying drawings. Figure 5 Schematic diagram of binocular deviation. Figure 5 The left picture in (A) is the image seen by the left eye when the XR device is displayed normally, where points 1 and 2 are two landmark points. Figure 5 The right picture in (A) shows the up-down shifted image seen by the right eye after the optical position of the XR device is shifted, where points 3 and 4 are two landmark points; Figure 5 The left image in (B) is the image seen by the left eye when the XR device is displayed normally, where points 1 and 2 are two landmark points. Figure 5 The right picture in (B) is the rotated image seen by the right eye after the optical and mechanical position of the XR device is offset, where points 3 and 4 are two landmark points; Figure 5 In (C), the left image is the image seen by the left eye during normal display, and the right image is the image seen by the right eye after the optical and mechanical position of the XR device is offset. Points 1 and 2 are two marker points in the left eye image, and points 3 and 4 are two marker points in the right eye image. Figure 5 In (D), the left picture is the image seen by the left eye during normal display, and the right picture is the image seen by the right eye with a size deviation, where points 1 and 2 are two marker points in the left eye image, and points 3 and 4 are two marker points in the right eye image.

[0071] Exemplarily, when the first image has an upward offset, that is, a deviation in the vertical direction, the XR device can move the first image in the vertical direction according to the vertical displacement compensation parameters to achieve parallax compensation in the vertical direction; when the first image has a left-right offset, that is, a deviation in the horizontal direction, the XR device can move the first image in the horizontal direction according to the horizontal displacement compensation parameters to achieve parallax compensation in the horizontal direction; when the first image is offset due to rotation, the XR device can rotate the first image according to the rotational displacement compensation parameters to achieve image direction correction; when the size of the first image deviates, the XR device can scale the first image according to the size compensation parameters to achieve image size correction.

[0072] The above image correction process is illustrated below by using examples.

[0073] Exemplarily, taking the XR device as MR glasses, the MR glasses display test images for the left eye and the right eye respectively through a binocular camera display, and each test image includes two marker points. When the user wears the MR glasses, the MR glasses capture the azimuth of the user's left eye's line of sight through a built-in sensor, and calculate the image correction parameters required to correct the image of the left eye by analyzing the azimuth of the left eye's line of sight when the user looks at each marker point, and then perform correction processing on the left eye image displayed by the MR glasses based on the calculated image correction parameters, and output the corrected left eye image, and then, the MR glasses capture the azimuth of the user's right eye's line of sight through a built-in sensor, and calculate the image correction parameters required to correct the right eye's image by analyzing the azimuth of the right eye's line of sight when the user looks at each marker point, and then perform correction processing on the right eye image displayed by the MR glasses based on the calculated image correction parameters, and output the corrected right eye image. In this way, the left and right eye images are corrected respectively by the azimuth angles of the left and right eye lines of sight to ensure that the image matches the user's line of sight and position. By outputting the image corrected by the image trueness parameters to the user, the user can obtain the correct stereoscopic visual experience at different angles and positions, thereby providing a more natural and realistic visual experience, thereby improving the display effect of the MR glasses.

[0074] The image correction method provided in the embodiment of the present application is that the XR device determines the image correction parameters according to the azimuth information of the user's line of sight when the user is gazing at M marker points in the target image, and the target image is a test image displayed by the optical machine of the XR device, or is a first image displayed by the XR device for image correction, and outputs a second image. Through this method, the XR device can accurately determine the image correction parameters according to the azimuth information of the user's line of sight when the user is gazing at the binocular image. In this way, when the optical machine position of the XR device changes during use due to factors such as pupil distance adjustment, wearing pressure or aging of structural parts, the image correction parameters can be determined in a timely and accurate manner to correct the image. In addition, since the azimuth information of the user's line of sight can accurately reflect the user's viewing angle, correcting the image based on the azimuth information of the user's line of sight can ensure that the user always sees a clear and accurate image, thereby ensuring the stereoscopic parallax matching of the image, thereby improving the display effect of the XR device.

[0075] In some embodiments of the present application, the above step 202 can be implemented by the following step 202a.

[0076] Step 202a: The XR device performs a geometric transformation on the first image according to the image correction parameters and outputs a second image.

[0077] In some embodiments of the present application, the above-mentioned geometric transformation may include at least one of the following: translation, rotation, scaling, affine transformation, etc.

[0078] It should be noted that translating the first image refers to moving the first image in a specific direction; rotating the first image refers to rotating the first image around a center point; scaling the first image refers to increasing or reducing the size of the first image; and affine transforming the first image refers to combining transformations such as translation, rotation, scaling, and tilting to keep the parallelism of straight lines and the length ratio of the straight lines in the first image unchanged.

[0079] In an embodiment of the present application, the XR device can perform the above-mentioned geometric transformation on the first image according to the image correction parameters, thereby adjusting the first image to adapt to the user's binocular perspective, thereby providing a more realistic immersive experience.

[0080] In some embodiments of the present application, the M marking points include one marking point, the image correction parameter includes a horizontal displacement compensation parameter, and the azimuth angle information includes a yaw angle.

[0081] Exemplarily, the above step 201 may include the following steps 201a1 and 201a2:

[0082] Step 201a1: The XR device moves the target image horizontally according to the yaw angle of the user's line of sight when the user is looking at the marker point in the target image, until the yaw angle of the line of sight is 0°

[0083] Step 201a2: The XR device determines a horizontal displacement compensation parameter according to the distance the target image moves in the horizontal direction.

[0084] In some embodiments of the present application, the one marker point may be located at the center of the target image.

[0085] In some embodiments of the present application, the XR device may obtain the distance that the target image moves in the horizontal direction, and determine a horizontal displacement compensation parameter according to the distance.

[0086] Exemplarily, the user looks at the mark point at the center of the test image of the left eye through the XR device, and the built-in sensor (such as a camera, etc.) of the XR device detects the yaw angle of the user's left eye sight, and according to the detected yaw angle, moves the test image of the left eye in the horizontal direction to try to reduce the yaw angle. Further, by moving the test image, the yaw angle of the left eye sight is 0°, that is, the user's left eye sight is completely aligned with the mark point in the image, and then according to the distance the test image moves (i.e., the horizontal offset), the horizontal displacement compensation parameter corresponding to the left eye is determined. Similarly, the user looks at the mark point at the center of the test image of the right eye through the XR device, and the built-in sensor (such as a gyroscope, camera, etc.) of the XR device detects the yaw angle of the user's right eye sight, and according to the detected yaw angle, moves the test image of the right eye in the horizontal direction, so that the yaw angle of the right eye sight is 0°, and then according to the distance the test image moves (i.e., the horizontal offset), the horizontal displacement compensation parameter corresponding to the right eye is determined, thereby obtaining the horizontal displacement compensation parameter of the binoculars.

[0087] It should be noted that the horizontal displacement compensation parameter indicates the distance that the image needs to be moved in the horizontal direction in order to align the user's line of sight with the marker point in the image. After determining the horizontal displacement compensation parameter for the left eye or the right eye, the XR device can apply this parameter when subsequently displaying the left eye image or the right eye image to ensure that the binocular image can be automatically adjusted to a position aligned with the user's line of sight.

[0088] The process of determining the horizontal displacement compensation parameters for the left eye or the right eye is exemplarily described below with reference to the accompanying drawings.

[0089] For example, Figure 6 (A) is a schematic diagram of a test image, which includes a marker point 1, such as Figure 6 As shown in (A), the XR device can prompt the user to stare at the marker point 1 on the test image, then move the horizontal position of the image, and monitor the yaw angle of the user's line of sight in real time during the movement until the yaw angle of the user's line of sight is equal to 0, and then determine the offset of the image in the horizontal direction as the horizontal displacement compensation parameter.

[0090] It should be noted that when the yaw of the user's line of sight is equal to 0 degrees, it indicates that the user's head has not turned left or right. At this time, the user's line of sight is facing straight ahead. When the user looks at the marker point in the test image, the yaw angle of the line of sight is not 0 degrees, indicating that there is a deviation in the display of the test image. At this time, by moving the position of the test image left or right so that the user's line of sight is 0 degrees when looking at the marker point in the test image, the marker point in the test image can be adjusted to a position directly opposite the user's line of sight, thereby correcting the horizontal deviation of the image display, thereby eliminating stereoscopic parallax and improving the display effect of the XR device.

[0091] In an embodiment of the present application, by monitoring the yaw angle of the user's line of sight in real time and adjusting the image position horizontally, the XR device can ensure that the image is precisely aligned with the user's line of sight, thereby improving the accuracy of the image.

[0092] In some embodiments of the present application, the M marking points include one marking point, the image correction parameter includes a vertical displacement compensation parameter, and the azimuth angle information includes a pitch angle.

[0093] Exemplarily, the above step 201 may include the following steps 201b1 and 201b2:

[0094] Step 201b1: The XR device moves the target image in the vertical direction according to the pitch angle of the user's line of sight when the user is gazing at the marker point in the target image, until the yaw angle and pitch angle of the line of sight are 0°.

[0095] Step 201b2: The XR device determines a vertical displacement compensation parameter according to the distance the target image moves in the vertical direction.

[0096] In some embodiments of the present application, the one marker point may be located at the center of the target image.

[0097] In some embodiments of the present application, the XR device may obtain a distance that the target image moves in the vertical direction, and determine a horizontal displacement compensation parameter according to the distance.

[0098] Exemplarily, the user looks at the mark point of the image of the left eye through the XR device, and the built-in sensor of the XR device (such as a gyroscope, camera, etc.) detects the pitch angle of the user's left eye line of sight in real time, and according to the detected pitch angle, moves the target image in the vertical direction to try to reduce the pitch angle. Further, by moving the image, the pitch angle of the left eye line of sight is 0°, that is, the user's left eye line of sight is completely aligned with the mark point in the image, and then the vertical displacement compensation parameter corresponding to the left eye is determined according to the distance the test image moves (i.e., the vertical offset). Similarly, the user looks at the mark point at the center position of the test image of the right eye through the XR device, and the built-in sensor of the XR device (such as a gyroscope, camera, etc.) detects the pitch angle of the user's right eye line of sight, and according to the detected pitch angle, moves the test image of the right eye in the vertical direction, so that the pitch angle of the right eye line of sight is 0°, and then according to the distance the test image moves (i.e., the handling offset), the vertical displacement compensation parameter corresponding to the right eye is determined, thereby obtaining the vertical displacement compensation parameter of the binoculars.

[0099] It should be noted that the vertical displacement compensation parameter indicates the distance that the image needs to be moved in the vertical direction in order to align the user's line of sight with the marker point in the image. After determining the vertical displacement compensation parameter of the binocular eye, the XR device can use this parameter when subsequently displaying the left and right eye images to ensure that the binocular image can be automatically adjusted to a position aligned with the user's line of sight.

[0100] The process of determining the vertical displacement compensation parameters of the left eye or the right eye is exemplarily described below with reference to the accompanying drawings.

[0101] For example, in combination with the above Figure 6 In (A), the XR device can prompt the user to stare at the marker point 1 on the test image, then move the vertical position of the image, and monitor the pitch angle of the user's line of sight in real time during the movement until the pitch angle of the user's line of sight is equal to 0, and then determine the vertical movement distance of the image as the vertical displacement compensation parameter.

[0102] It should be noted that when the pitch of the user's line of sight is equal to 0 degrees, it means that the user's head is neither tilted up nor lowered. At this time, the user's line of sight is facing straight ahead. When the pitch angle of the line of sight is not 0 degrees when the user is looking at the mark point in the test image, it indicates that there is a deviation in the display of the test image. Specifically, when the line of sight is tilted upward, the pitch is a positive value. For example, if the line of sight is tilted upward by 10 degrees, the pitch angle is 10 degrees. On the contrary, when the line of sight is tilted downward, the pitch is a negative value. For example, if the line of sight is tilted downward by 10 degrees, the pitch is -10 degrees. Assuming that the current line of sight pitch = 10 degrees when the user is looking at the mark point, in order to make the user's line of sight face the mark point, the test image can be moved downward to adjust the pitch to 0 degrees (that is, the line of sight is horizontal).

[0103] In this way, by moving the position of the test image up or down so that the user's line of sight is 0 degrees when looking at the mark point in the test image, the mark point in the test image can be adjusted to a position directly opposite the user's line of sight, thereby correcting the vertical deviation of the image display, thereby eliminating stereoscopic parallax and improving the display effect of the XR device.

[0104] In an embodiment of the present application, by monitoring the pitch angle of the user's line of sight in real time and vertically adjusting the image position, the XR device can ensure that the image is precisely aligned with the user's line of sight, thereby improving the accuracy of the image.

[0105] In some embodiments of the present application, the M marker points include a first marker point and a second marker point symmetrically distributed in the vertical direction relative to the center point of the target image, the image correction parameters include rotation compensation parameters, and the azimuth angle information includes a yaw angle.

[0106] Exemplarily, the above step 201 may include the following steps 201c1 to 201c5:

[0107] Step 201c1: the XR device horizontally moves the target image according to the first yaw angle of the user's line of sight when gazing at the first marker point in the target image, until the first yaw angle is 0°.

[0108] Step 201c2: The XR device determines a horizontal displacement compensation parameter according to the distance the target image moves in the horizontal direction.

[0109] Step 201c3: The XR device rotates the target image with the first marker point as the rotation center according to the second yaw angle of the user's line of sight when the user gazes at the second marker point in the target image until the second yaw angle is 0.

[0110] Step 201c4: The XR device determines an angle compensation parameter according to the rotation angle of the target image.

[0111] Step 201c5: The XR device determines the rotation compensation parameter according to the horizontal displacement compensation parameter and the angle compensation parameter.

[0112] In some embodiments of the present application, the first marker point may be located in a top area of ​​the target image, and the second marker point may be located in a bottom area of ​​the target image.

[0113] In some embodiments of the present application, the XR device may obtain the distance that the target image moves in the horizontal direction, and determine the horizontal displacement compensation parameter based on the distance, and obtain the rotation angle of the target image, and determine the angle compensation parameter based on the rotation angle.

[0114] Exemplarily, the user looks at the first mark point of the image of the left eye through the XR device, and the built-in sensor of the XR device detects the yaw angle of the user's left eye line of sight in real time, and according to the detected yaw angle, moves the test image in the horizontal direction to try to reduce the yaw angle. Further, by moving the test image left and right, the yaw angle of the left eye line of sight is made to be 0°, that is, the user's left eye line of sight is completely aligned with the mark point in the test image, and then the horizontal displacement compensation parameter corresponding to the left eye is determined according to the distance the test image moves. Next, the user looks at the second mark point of the image of the left eye through the XR device, and the built-in sensor of the XR device detects the yaw angle of the user's left eye line of sight in real time, and rotates the test image with the first mark point as the rotation center until the yaw angle of the left eye line of sight is 0°, and then determines the angle compensation parameter of the left eye according to the angle of rotation of the test image, and determines the rotation compensation parameter of the left eye according to the horizontal displacement compensation parameter and angle compensation parameter of the left eye.

[0115] Similarly, the XR device can determine the horizontal displacement compensation parameters and angle compensation parameters corresponding to the right eye according to the above method, and then determine the rotation compensation parameters of the right eye according to the horizontal displacement compensation parameters and angle compensation parameters corresponding to the right eye, thereby obtaining the rotation compensation parameters of the binocular eyes.

[0116] The process of determining the vertical displacement compensation parameters of the left eye or the right eye is exemplarily described below with reference to the accompanying drawings.

[0117] For example, Figure 6 (B) in FIG. 1 is a schematic diagram of a test image, which includes a landmark point 1 and a landmark point 2 located in the top area. Figure 6 As shown in (B), the XR device can prompt the user to stare at the marker point 1 on the test image, then move the position of the image horizontally, and monitor the yaw angle of the user's line of sight in real time during the movement until the yaw angle of the user's line of sight is equal to 0, and then determine the horizontal displacement compensation parameter according to the horizontal offset of the image. Next, the user is prompted to stare at the marker point 2 in the bottom area, and the image is rotated with the marker point 1 in the top area as the rotation center until the yaw angle of the user's line of sight is equal to 0, and the angle compensation parameter is determined according to the rotation angle of the image, and then the horizontal displacement compensation parameter and the angle compensation parameter are determined as the rotation compensation parameter.

[0118] It should be noted that, when the target image is a virtual reality image of a VST scene displayed by an XR device, the process of determining the rotation compensation parameters is the same as that described above. Figure 6The process of (B) in the corresponding example is the same. Since the vertical consistency of the marker points cannot be guaranteed when selecting the marker points in the VST scene, after the virtual reality image is horizontally moved and rotated, the two marker points in the virtual reality image will be on a vertical line. Therefore, after determining the rotation compensation parameters, the rotation of the virtual reality image can be inversely compensated based on the rotation compensation parameters to restore the virtual reality image to its original or expected orientation.

[0119] In this way, by moving and rotating the test image so that the yaw angle of the user's line of sight is 0 degrees when the user looks at the marker point in the test image, the marker point in the test image can be adjusted to a position directly opposite the user's line of sight by rotating the test image, thereby correcting the rotational deviation of the image display, thereby eliminating stereoscopic parallax and improving the display effect of the XR device.

[0120] In an embodiment of the present application, by monitoring the yaw angle of the user's line of sight and horizontally adjusting and rotating the image, the XR device can ensure that the image is precisely aligned with the user's line of sight, thereby improving the accuracy of the image.

[0121] In some embodiments of the present application, the M marker points include a third marker point and a fourth marker point symmetrically distributed in the vertical direction relative to the center point of the target image, the image correction parameters include size compensation parameters, and the azimuth angle information includes a pitch angle.

[0122] Exemplarily, the above step 201 may include the following steps 201d1 and 201d2:

[0123] Step 201d1: the XR device scales the target image according to the first pitch angle of the user's line of sight when the user gazes at the fourth marker point in the target image, until the first pitch angle is equal to the second pitch angle.

[0124] The second pitch angle is the pitch angle of the user's line of sight when the user looks at the third marker point.

[0125] Step 201d2: The XR device determines a size compensation parameter according to the scaling ratio of the target image.

[0126] In some embodiments of the present application, the third marker point may be located in a top area of ​​the target image, and the fourth marker point may be located in a bottom area of ​​the target image.

[0127] In some embodiments of the present application, the XR device may scale the target image according to the pitch angle of the user's line of sight, obtain the scaling ratio of the target image, and then determine the size compensation parameter according to the scaling ratio.

[0128] Exemplarily, when the user looks at the third mark point of the image of the left eye through the XR device, the XR device obtains and records the pitch angle of the user's line of sight, and then when the user looks at the fourth mark point of the image of the left eye through the XR device, the built-in sensor of the XR device detects the pitch angle of the user's left eye line of sight in real time, and according to the detected pitch angle, scales the test image until the pitch angle of the left eye line of sight is the same as the pitch angle when looking at the third mark point, and then determines the size compensation parameters corresponding to the left eye according to the scaling ratio of the test image. Similarly, the XR device can determine the size compensation parameters corresponding to the right eye according to the above method, thereby obtaining the size compensation parameters of the binoculars.

[0129] It should be noted that the size compensation parameter in the embodiment of the present application is the scaling ratio.

[0130] In some embodiments of the present application, the XR device may scale the target image based on the pitch angle of the user's line of sight, taking the third marker point as an anchor point.

[0131] It should be noted that scaling the target image with the third marker point as the anchor point means that during the scaling process, the ratio of the third marker point to other parts of the image remains unchanged.

[0132] The process of determining the vertical displacement compensation parameters of the left eye or the right eye is exemplarily described below with reference to the accompanying drawings.

[0133] For example, in combination with the above Figure 6 B. The XR device can prompt the user to stare at marker point 1 on the test image, obtain and record the pitch angle of the user's line of sight, and then prompt the user to stare at marker point 2 on the test image, scale the test image with marker point 1 as the anchor point, and monitor the pitch angle of the user's line of sight in real time during the scaling process until the pitch angle of the user's current line of sight is equal to the pitch angle when staring at marker point 1, and then determine the image scaling ratio as the size compensation parameter.

[0134] It should be noted that since marker point 1 and marker point 2 are symmetrical about the center point of the image in the vertical direction, if the user's line of sight is horizontal (i.e., not tilted), then due to symmetry, the pitch angles of marker point 1 and marker point 2 are theoretically equal. During the zooming process, keeping the relative position of marker point 1 unchanged can indirectly affect the position of marker point 2, causing the relative position between marker point 2 and marker point 1 to change. Therefore, during the zooming process, by detecting the pitch angle of the user's line of sight when looking at marker point 2, the pitch angle is made the same as the pitch angle when looking at marker point 1, to ensure that the image has been zoomed to a specific ratio, so that from the user's perspective, the two points have the same degree of inclination in the vertical direction, thereby maintaining the symmetry of the two points.

[0135] In this way, by scaling the test image, the marker points in the test image can be adjusted to a position that matches the user's viewing angle, thereby correcting the size deviation of the image display, thereby eliminating stereoscopic parallax and improving the display effect of the XR device.

[0136] The following uses an XR device as MR glasses as an example to exemplify the image correction method provided in the embodiment of the present application.

[0137] In some examples, the above image correction method may include the following steps:

[0138] Step 11: Start the binocular parallax calibration procedure

[0139] Step 12: Present the test image to the left eye and the right eye.

[0140] Exemplarily, the test image presentation methods include: simultaneous presentation to both eyes and independent presentation to the left and right eyes.

[0141] Step 13, obtain the user's line of sight axis data when gazing at each landmark point through the gaze point system of the MR glasses: left eye: (pitch_L, yaw_L)_i and right eye: (pitch_R, yaw_R)_i, where i represents the line of sight data of the i-th point.

[0142] Step 14, taking the test image including two marker points located at the top and bottom areas of the image as an example, first adjust the upper marker point to the horizontal center, prompt the user to stare at the upper marker point, move the horizontal position of the image, and monitor the line of sight data in real time until yaw_1=yaw_3=0, and obtain the binocular compensation coefficients x1_L and x1_R.

[0143] Among them, yaw_1 is the yaw angle of the point located in the top area of ​​the test image for the left eye, yaw_3 is the yaw angle of the point located in the top area of ​​the test image for the right eye, x1_L is the horizontal displacement compensation parameter for the left eye, and x1_R is the horizontal displacement compensation parameter for the right eye.

[0144] Step 15: Adjust the position of the lower mark point to the horizontal center, prompt the user to stare at the lower mark point, and rotate the image with the upper mark point as the rotation center until yaw_2=yaw_4=0, and obtain the binocular compensation coefficients x2_L and x2_R.

[0145] Among them, yaw_2 is the yaw angle of the point located in the bottom area in the test image of the left eye, yaw_4 is the yaw angle of the point located in the bottom area in the test image of the right eye, x2_L is the angle compensation parameter of the left eye, and x2_R is the angle compensation parameter of the right eye.

[0146] Step 16: adjust the size of the test image. Taking the size of the left eye image as the target, prompt the user to stare at the lower mark point, use the upper mark point as the anchor point, and scale the right eye test image until pitch_3 = pitch_4 to obtain the binocular compensation coefficients x3_L and x3_R.

[0147] Among them, pitch_3 is the pitch angle of the point located in the top area of ​​the right test image, pitch_4 is the pitch angle of the point located in the bottom area of ​​the right eye test image, x3_L is the size compensation parameter of the left eye, and x3_R is the size compensation parameter of the right eye.

[0148] Step 17: The MR rendering system reads the parallax compensation parameters, displaces and renders the binocular images, and realizes binocular parallax correction.

[0149] In an embodiment of the present application, the deviation of the binocular eye is calibrated by identifying the gaze point information of the user's binocular eyes. Compared with the traditional production line binocular eye deviation calibration solution, calibration can be performed at any time during the user's use, thereby improving the problem of binocular image non-fusion caused by structural aging, deformation and wearing status, and improving the user experience.

[0150] In some examples, the above image correction method may include the following steps:

[0151] Step 21: Start the VST binocular disparity calibration procedure.

[0152] Step 22: Determine the landmark points in the VST scene.

[0153] For example, a marker point in a VST scene may be selected through an interactive method such as a handle.

[0154] For example, the VST scene has a square and a diamond object, and two points of them are selected using a handle or gesture.

[0155] Step 23, using the gaze point system of the MR glasses to obtain the line of sight axis data of the user when gazing at each identification point: left eye: (pitch_L, yaw_L)_i and right eye: (pitch_R, yaw_R)_i, where i represents the line of sight data of the i-th point.

[0156] Step 24: Compensate for binocular parallax according to the values ​​of (pitch_L, yaw_L)_i and (pitch_R, yaw_R)_i.

[0157] It should be noted that the process of step 24 may refer to steps 14 to 16 in the above example.

[0158] Furthermore, since the vertical consistency of the marker points cannot be guaranteed when selecting marker points in the VST scene, after operations such as translation or rotation of the image, the two marker points will become on a vertical line. A step needs to be added to compensate the image to restore the original display state of the image.

[0159] Step 25: The MR rendering system reads the parallax compensation parameters, and performs displacement, rotation, and scaling operations on the VST binocular image to achieve parallax correction for the VST binocular.

[0160] In an embodiment of the present application, the deviation of the binocular eye is calibrated by identifying the gaze point information of the user's binocular eyes. Compared with the traditional calibration scheme of the binocular eye deviation of the production line, calibration can be performed at any time during the user's use, thereby improving the binocular image infusion caused by aging, deformation and wearing status of structural parts and the binocular image infusion caused by the fusion deviation of the binocular camera and the binocular eye, thereby improving the user experience.

[0161] The above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or, under the premise that there is no contradiction, can also be executed in combination with each other. The specific implementation can be determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0162] The image correction method provided in the embodiment of the present application can be executed by an image correction device. In the embodiment of the present application, the image correction device provided in the embodiment of the present application is described by taking the image correction device executing the image correction method as an example.

[0163] Figure 7 A schematic diagram of the structure of the image correction device provided in the embodiment of the present application is shown in FIG. Figure 7 As shown, the image correction device 700 may include a processing module 701, and the processing module 701 is used to determine image correction parameters according to the azimuth information of the user's line of sight when he looks at M marker points in the target image; the target image is a test image displayed by the optical machine of the XR device, or a virtual reality image displayed by the XR device, and M is a positive integer; the processing module 701 is also used to perform image correction on the first image displayed by the XR device according to the image correction parameters, and output a second image.

[0164] In some embodiments of the present application, the processing module is specifically used to perform a geometric transformation on the first image according to the image correction parameters and output a second image.

[0165] In some embodiments of the present application, the M marker points include one marker point; the image correction parameters include horizontal displacement compensation parameters, and the azimuth information includes a yaw angle; the processing module is specifically used to horizontally move the target image according to the yaw angle of the line of sight when the user looks at the marker point in the target image until the yaw angle of the line of sight is 0°; the processing module is specifically used to determine the horizontal displacement compensation parameter according to the distance the target image moves in the horizontal direction.

[0166] In some embodiments of the present application, the M marker points include one marker point; the image correction parameters include vertical displacement compensation parameters, and the azimuth information includes a pitch angle; the processing module is specifically used to move the target image in the vertical direction according to the pitch angle of the line of sight when the user looks at the marker point in the target image, until the pitch angle of the line of sight is 0°; the processing module is specifically used to determine the vertical displacement compensation parameter according to the distance the target image moves in the vertical direction.

[0167] In some embodiments of the present application, the M marker points include a first marker point and a second marker point symmetrically distributed in the vertical direction relative to the center point of the target image, the image correction parameters include rotation compensation parameters, and the azimuth information includes a yaw angle; the processing module is specifically used to horizontally move the target image according to the first yaw angle of the line of sight when the user looks at the first marker point in the target image, until the first yaw angle is 0°; the processing module is specifically used to determine the horizontal displacement compensation parameter according to the distance the target image moves in the horizontal direction; the processing module is specifically used to rotate the target image with the first marker point as the rotation center according to the second yaw angle of the line of sight when the user looks at the second marker point in the target image, until the second yaw angle is 0; the processing module is specifically used to determine the angle compensation parameter according to the rotation angle of the target image; the processing module is specifically used to determine the rotation compensation parameter according to the horizontal displacement compensation parameter and the angle compensation parameter.

[0168] In some embodiments of the present application, the M marker points include a third marker point and a fourth marker point symmetrically distributed in the vertical direction relative to the center point of the target image, the image correction parameters include size compensation parameters, and the azimuth information includes a pitch angle; the processing module is specifically used to scale the target image according to a first pitch angle of a line of sight when the user is looking at the fourth marker point in the target image, until the first pitch angle is equal to the second pitch angle; wherein the second pitch angle is the pitch angle of the line of sight when the user is looking at the third marker point.

[0169] The image correction device provided in the embodiment of the present application determines the image correction parameters according to the azimuth information of the line of sight when the user is gazing at M mark points in the target image, the target image is a test image displayed by the optical machine of the XR device, or is a virtual reality image displayed by the XR device, M is a positive integer, and the XR device performs image correction on the first image displayed by the XR device according to the image correction parameters, and outputs the second image. Through this method, the XR device can accurately determine the image correction parameters according to the azimuth information of the line of sight when the user is gazing at the binocular image, so that when the optical machine position changes due to factors such as pupil distance adjustment, wearing pressure or aging of structural parts during the use of the XR device, the image correction parameters can be determined in time and accurately to correct the image. In addition, since the azimuth information of the user's line of sight can more accurately reflect the user's perspective, correcting the image based on the azimuth information of the user's line of sight can ensure that the user always sees a clear and accurate image, so that the stereoscopic parallax of the image is matched, thereby improving the display effect of the XR device.

[0170] The image correction device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than the terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a palmtop computer, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0171] The image correction device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0172] The image correction device provided in the embodiment of the present application can implement each process implemented in the above-mentioned image correction method embodiment, and will not be described again here to avoid repetition.

[0173] Alternatively, if Figure 8 As shown, an embodiment of the present application further provides an electronic device 800, including a processor 801 and a memory 802, wherein the memory 802 stores a program or instruction that can be executed on the processor 801, and when the program or instruction is executed by the processor 801, each step of the above-mentioned image correction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0174] It should be noted that the electronic device in the embodiment of the present application includes the mobile electronic device and non-mobile electronic device mentioned above, and the electronic device in the embodiment of the present application may be an XR device.

[0175] Fig. 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0176] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.

[0177] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0178] Among them, the above-mentioned processor 110 is used to determine the image correction parameters according to the azimuth information of the user's line of sight when he looks at M mark points in the target image; the above-mentioned target image is a test image displayed by the optical machine of the above-mentioned electronic device, or is a virtual reality image displayed by the electronic device, and M is a positive integer; the above-mentioned processor 110 is also used to perform image correction on the first image displayed by the electronic device according to the above-mentioned image correction parameters, and output a second image.

[0179] In some embodiments of the present application, the processor 110 is specifically configured to perform a geometric transformation on the first image according to the image correction parameters and output a second image.

[0180] In some embodiments of the present application, the M marker points include one marker point; the image correction parameters include horizontal displacement compensation parameters, and the azimuth information includes a yaw angle; the processor 110 is specifically used to horizontally move the target image according to the yaw angle of the line of sight when the user looks at the marker point in the target image until the yaw angle of the line of sight is 0°; the processor 110 is specifically used to determine the horizontal displacement compensation parameter according to the distance the target image moves in the horizontal direction.

[0181] In some embodiments of the present application, the M marker points include one marker point; the image correction parameters include vertical displacement compensation parameters, and the azimuth information includes a pitch angle; the processor 110 is specifically used to move the target image in the vertical direction according to the pitch angle of the line of sight when the user looks at the marker point in the target image, until the pitch angle of the line of sight is 0°; the processor 110 is specifically used to determine the vertical displacement compensation parameter according to the distance the target image moves in the vertical direction.

[0182] In some embodiments of the present application, the M marker points include a first marker point and a second marker point symmetrically distributed in the vertical direction relative to the center point of the target image, the image correction parameters include rotation compensation parameters, and the azimuth information includes a yaw angle; the processor 110 is specifically used to horizontally move the target image according to a first yaw angle of a line of sight when the user looks at the first marker point in the target image until the first yaw angle is 0°; the processor 110 is specifically used to determine a horizontal displacement compensation parameter according to a distance the target image moves in the horizontal direction; the processor 110 is specifically used to rotate the target image with the first marker point as the rotation center according to a second yaw angle of a line of sight when the user looks at the second marker point in the target image until the second yaw angle is 0; the processor 110 is specifically used to determine an angle compensation parameter according to the rotation angle of the target image; the processor 110 is specifically used to determine a rotation compensation parameter according to the horizontal displacement compensation parameter and the angle compensation parameter.

[0183] In some embodiments of the present application, the M marker points include a third marker point and a fourth marker point that are symmetrically distributed in the vertical direction relative to the center point of the target image, the image correction parameters include size compensation parameters, and the azimuth information includes a pitch angle; the processor 110 is specifically used to scale the target image according to a first pitch angle of a line of sight when the user looks at the fourth marker point in the target image, until the first pitch angle is equal to the second pitch angle; wherein the second pitch angle is the pitch angle of the line of sight when the user looks at the third marker point.

[0184] The electronic device provided in the embodiment of the present application determines the image correction parameters according to the azimuth information of the sight line when the user is gazing at M mark points in the target image, the target image is a test image displayed by the electronic device's optical machine, or is a virtual reality image displayed by the electronic device, M is a positive integer, and the electronic device performs image correction on the first image displayed by the electronic device according to the image correction parameters, and outputs the second image. Through this method, the electronic device can accurately determine the image correction parameters according to the azimuth information of the sight line when the user is gazing at the binocular image, so that when the optical machine position changes during the use of the electronic device due to factors such as pupil distance adjustment, wearing pressure or aging of structural parts, the image correction parameters can be determined in time and accurately to correct the image. In addition, since the azimuth information of the user's sight line can more accurately reflect the user's perspective, correcting the image based on the azimuth information of the user's sight line can ensure that the user always sees a clear and accurate image, so that the stereoscopic parallax of the image matches, thereby improving the display effect of the electronic device.

[0185] It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0186] The memory 109 can be used to store software programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0187] The processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 110.

[0188] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image correction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0189] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0190] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image correction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0191] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0192] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image correction method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0193] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0194] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0195] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image correction method, applied to an extended reality XR device, characterized in that: The method comprises: Determine the image correction parameters according to the azimuth information of the line of sight when the user gazes at M marker points in the target image; the target image is a test image displayed by the optical machine of the XR device, or a virtual reality image displayed by the XR device, and M is a positive integer; According to the image correction parameters, image correction is performed on the first image displayed by the XR device to output a second image.

2. The method according to claim 1, characterized in that: The step of performing image correction on the first image displayed by the XR device according to the image correction parameter and outputting a second image includes: According to the image correction parameters, a geometric transformation is performed on the first image to output a second image.

3. The method according to claim 1 or 2, characterized in that: The M marker points include one marker point; the image correction parameter includes a horizontal displacement compensation parameter, and the azimuth information includes a yaw angle; The method of determining the image correction parameters according to the azimuth information of the sight line when the user gazes at the M marker points in the target image includes: According to the yaw angle of the sight line when the user looks at the mark point in the target image, the target image is horizontally moved until the yaw angle of the sight line is 0°; The horizontal displacement compensation parameter is determined according to the distance that the target image moves in the horizontal direction.

4. The method according to claim 1 or 2, characterized in that: The M marker points include one marker point; the image correction parameter includes a vertical displacement compensation parameter, and the azimuth angle information includes a pitch angle; The method of determining the image correction parameters according to the azimuth information of the sight line when the user gazes at the M marker points in the target image includes: According to the pitch angle of the sight line when the user looks at the mark point in the target image, move the target image in the vertical direction until the pitch angle of the sight line is 0°; The vertical displacement compensation parameter is determined according to the distance that the target image moves in the vertical direction.

5. The method according to claim 1 or 2, characterized in that: The M marker points include a first marker point and a second marker point that are symmetrically distributed in a vertical direction relative to a center point of the target image, the image correction parameter includes a rotation compensation parameter, and the azimuth angle information includes a yaw angle; The method of determining the image correction parameters according to the azimuth information of the sight line when the user gazes at the M marker points in the target image includes: According to a first yaw angle of the sight line of the user when gazing at the first mark point in the target image, horizontally move the target image until the first yaw angle is 0°; Determining a horizontal displacement compensation parameter according to a distance that the target image moves in the horizontal direction; According to a second yaw angle of the sight line of the user when gazing at the second mark point in the target image, rotating the target image with the first mark point as the rotation center until the second yaw angle is 0; Determining an angle compensation parameter according to the rotation angle of the target image; The rotation compensation parameter is determined according to the horizontal displacement compensation parameter and the angle compensation parameter.

6. The method according to claim 1 or 2, characterized in that: The M marker points include a third marker point and a fourth marker point that are symmetrically distributed in the vertical direction relative to the center point of the target image, the image correction parameter includes a size compensation parameter, and the azimuth angle information includes a pitch angle; The method of determining the image correction parameters according to the azimuth information of the sight line when the user gazes at the M marker points in the target image includes: According to a first pitch angle of a user's sight when gazing at the fourth mark point in the target image, scaling the target image until the first pitch angle is equal to the second pitch angle; Determining a size compensation parameter according to the scaling ratio of the target image; The second pitch angle is the pitch angle of the user's sight when he or she looks at the third marker point.

7. An image correction device, characterized in that: The device comprises: a processing module; The processing module is used to determine the image correction parameters according to the azimuth information of the sight line when the user looks at M marker points in the target image; the target image is a test image displayed by the optical machine of the XR device, or a virtual reality image displayed by the XR device, and M is a positive integer; The processing module is further used to perform image correction on the first image displayed by the XR device according to the image correction parameters, and output a second image.

8. The device according to claim 7, characterized in that The processing module is specifically used to perform geometric transformation on the first image according to the image correction parameters and output a second image.

9. The device according to claim 7 or 8, characterized in that The M marker points include one marker point; the image correction parameter includes a horizontal displacement compensation parameter, and the azimuth information includes a yaw angle; The processing module is specifically used to horizontally move the target image according to the yaw angle of the sight line when the user looks at the mark point in the target image, until the yaw angle of the sight line is 0°; The processing module is specifically used to determine the horizontal displacement compensation parameter according to the distance that the target image moves in the horizontal direction.

10. The device according to claim 7 or 8, characterized in that The M marker points include one marker point; the image correction parameter includes a vertical displacement compensation parameter, and the azimuth angle information includes a pitch angle; The processing module is specifically used to move the target image in the vertical direction according to the pitch angle of the sight line when the user looks at the mark point in the target image, until the pitch angle of the sight line is 0°; The processing module is specifically used to determine the vertical displacement compensation parameter according to the distance that the target image moves in the vertical direction.

11. The device according to claim 7 or 8, characterized in that The M marker points include a first marker point and a second marker point that are symmetrically distributed in a vertical direction relative to a center point of the target image, the image correction parameter includes a rotation compensation parameter, and the azimuth angle information includes a yaw angle; The processing module is specifically configured to horizontally move the target image according to a first yaw angle of the user's sight when the user is gazing at the first mark point in the target image, until the first yaw angle is 0°; The processing module is specifically used to determine the horizontal displacement compensation parameter according to the distance that the target image moves in the horizontal direction; The processing module is specifically configured to rotate the target image with the first mark point as the rotation center according to the second yaw angle of the sight line when the user looks at the second mark point in the target image, until the second yaw angle is 0; The processing module is specifically used to determine an angle compensation parameter according to the rotation angle of the target image; The processing module is specifically used to determine the rotation compensation parameter according to the horizontal displacement compensation parameter and the angle compensation parameter.

12. The device according to claim 7 or 8, characterized in that The M marker points include a third marker point and a fourth marker point that are symmetrically distributed in the vertical direction relative to the center point of the target image, the image correction parameter includes a size compensation parameter, and the azimuth angle information includes a pitch angle; The processing module is specifically configured to perform scaling processing on the target image according to a first pitch angle of a line of sight when the user is gazing at the fourth mark point in the target image, until the first pitch angle is equal to the second pitch angle; The second pitch angle is the pitch angle of the user's sight when he or she looks at the third marker point.

13. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image correction method according to any one of claims 1 to 6 are implemented.

14. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image correction method according to any one of claims 1 to 6 are implemented.

Images