Method and device for calculating convergence adjustment amount, electronic equipment and medium

Abstract

The invention provides a method and device for calculating convergence adjustment amount, electronic equipment and a medium. According to the specific implementation scheme, a first image shot by a first camera and a second image shot by a second camera are obtained, wherein the first image and the second image are calibration images displayed through a first display and a second display respectively; and determining a first pixel coordinate of the preset feature point in the coordinate system of the first image and a second pixel coordinate of the preset feature point in the coordinate system of the second image. Acquiring a first screen coordinate of the preset feature point in the coordinate system of the first display and a second screen coordinate of the preset feature point in the coordinate system of the second display; and determining a first convergence adjustment amount corresponding to the first display and a second convergence adjustment amount corresponding to the second display according to the first pixel coordinate, the second pixel coordinate, the first screen coordinate, the second screen coordinate and a preset convergence plane.

Description

Technical Field

[0001] The present disclosure relates to the technical field of wearable devices, and in particular to a method, device, electronic device, and medium for calculating a convergence adjustment amount. Background Art

[0002] At present, wearable display devices such as virtual reality and augmented reality are gradually entering various fields or scenarios. Wearable display devices usually use the principle of binocular vision to bring a stereoscopic image viewing experience to users. In binocular vision, the accuracy of binocular image synthesis affects the user's experience of perceiving three-dimensional objects. Summary of the Invention

[0003] The purpose of the present disclosure is to provide a method, device, electronic device, and medium for calculating a convergence adjustment amount.

[0004] In a first aspect, an embodiment of the present disclosure provides a method for calculating a convergence adjustment amount, including: obtaining a first image captured by a first camera and a second image captured by a second camera, where the first image is a calibration image displayed through a first display, the second image is a calibration image displayed through a second display, and the calibration image includes preset feature points; determining a first pixel coordinate of the preset feature points in the coordinate system of the first image and a second pixel coordinate of the preset feature points in the coordinate system of the second image; obtaining a first screen coordinate of the preset feature points in the coordinate system of the first display and a second screen coordinate of the preset feature points in the coordinate system of the second display; determining a first convergence adjustment amount corresponding to the first display and a second convergence adjustment amount corresponding to the second display according to the first pixel coordinate, the second pixel coordinate, the first screen coordinate, the second screen coordinate, and a preset convergence plane; where the preset convergence plane is at a preset distance from between the first camera and the second camera.

[0005] In a second aspect, an embodiment of the present disclosure provides a device for calculating a convergence adjustment amount, including: a first obtaining module, configured to obtain a first image captured by a first camera and a second image captured by a second camera, where the first image is a calibration image displayed through a first display, the second image is a calibration image displayed through a second display, and the calibration image includes preset feature points; a first processing module, configured to determine a first pixel coordinate of the preset feature points in the coordinate system of the first image and a second pixel coordinate of the preset feature points in the coordinate system of the second image; a second obtaining module, configured to obtain a first screen coordinate of the preset feature points in the coordinate system of the first display and a second screen coordinate of the preset feature points in the coordinate system of the second display; a second processing module, configured to determine a first convergence adjustment amount corresponding to the first display and a second convergence adjustment amount corresponding to the second display according to the first pixel coordinate, the second pixel coordinate, the first screen coordinate, the second screen coordinate, and a preset convergence plane; where the preset convergence plane is at a preset distance from between the first camera and the second camera.

[0006] In a third aspect, embodiments of the present disclosure provide a computer-readable storage medium storing a computer program for executing the method for calculating the convergence adjustment amount provided in any of the above embodiments of the present disclosure.

[0007] In a fourth aspect, embodiments of the present disclosure provide an electronic device, including: a processor; a memory for storing executable instructions of the processor; and the processor for reading the executable instructions from the memory and executing the instructions to implement the method for calculating the convergence adjustment amount provided in any of the above embodiments of the present disclosure.

[0008] In a fifth aspect, embodiments of the present disclosure provide a head-mounted display device, including:

[0009] a processor; a memory for storing executable instructions of the processor; and the processor for reading the executable instructions from the memory and executing the instructions to implement the method for calculating the convergence adjustment amount provided in any of the above embodiments of the present disclosure.

[0010] In a sixth aspect, embodiments of the present disclosure provide a computer program product including computer program instructions that, when run on a processor, cause the processor to execute the method for calculating the convergence adjustment amount provided in any of the above embodiments.

[0011] The technical solutions of the present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] By describing the embodiments of the present disclosure in more detail with reference to the accompanying drawings, the above and other objects, features, and advantages of the present disclosure will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification, together with the embodiments of the present disclosure, to explain the present disclosure, and do not constitute a limitation to the present disclosure. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0013] Figure 1 is an exemplary system architecture to which the method or apparatus for calculating the convergence adjustment amount according to the present disclosure can be applied;

[0014] Figure 2 is a flowchart of the method for calculating the convergence adjustment amount provided in an exemplary embodiment of the present disclosure;

[0015] Figure 3 is a flowchart of determining the pixel coordinates of preset feature points provided in an exemplary embodiment of the present disclosure;

[0016] Figure 4It is a schematic flowchart of determining the convergence adjustment amount provided by an exemplary embodiment of the present disclosure;

[0017] Figure 5 It is a schematic flowchart of determining the convergence adjustment amount provided by another exemplary embodiment of the present disclosure;

[0018] Figure 6 It is a schematic diagram of the principle of constructing the objective function provided by an exemplary embodiment of the present disclosure;

[0019] Figure 7 It is a schematic diagram of the principle of constructing the objective function provided by another exemplary embodiment of the present disclosure;

[0020] Figure 8 It is a schematic flowchart of the method for calculating the convergence adjustment amount provided by another exemplary embodiment of the present disclosure;

[0021] Figure 9 It is a schematic structural diagram of the device for calculating the convergence adjustment amount provided by an exemplary embodiment of the present disclosure;

[0022] Figure 10 It is a schematic structural diagram of the device for calculating the convergence adjustment amount provided by another exemplary embodiment of the present disclosure;

[0023] Figure 11 It is a schematic structural diagram of an application embodiment of the electronic device of the present disclosure. Detailed implementation manners

[0024] Next, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure. It should be understood that the present disclosure is not limited by the exemplary embodiments described herein.

[0025] It should be noted that: Unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions and values set forth in these embodiments do not limit the scope of the present disclosure.

[0026] Those skilled in the art can understand that terms such as "first", "second", etc. in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and do not represent any specific technical meaning nor indicate an inevitable logical order between them.

[0027] It should also be understood that in the embodiments of the present disclosure, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0028] It should also be understood that the present disclosure emphasizes the differences between the various embodiments. The same or similar parts can be referred to each other. For the sake of brevity, they will not be described one by one.

[0029] Embodiments of the present disclosure can be applied to electronic devices such as terminal devices and servers, which can operate together with many other general-purpose or special-purpose computing system environments or configurations. Electronic devices such as terminal devices and servers can be described in the general context of computer system-executable instructions (such as program modules) executed by a computer system. Generally, program modules can include routines, programs, target programs, components, logics, data structures, etc., which perform specific tasks or implement specific abstract data types. The computer system / server can be implemented in a distributed cloud computing environment, where tasks are executed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0030] The following describes in detail the specific embodiments of the present disclosure with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustrating and explaining the present disclosure, and are not used to limit the present disclosure.

[0031] Figure 1 is an exemplary system architecture of an embodiment of a method or device for calculating a convergence adjustment amount that can be applied to the present disclosure. The system architecture can include a first camera 1, a second camera 2, a fixing fixture 3, a head-mounted display device 4, and a terminal device 5. The fixing fixture 3 is used to fix the first camera 1, the second camera 2, and the head-mounted display device 4. The first camera 1 and the second camera 2 can be connected to the terminal device 5 to send the captured images to the terminal device 5.

[0032] The first camera 1 and the second camera 2 are used to capture the display screen of the head-mounted display device 4. The display screen of the head-mounted display device 4 can be displayed through a display. For example, a calibration image is displayed through the display for a specific application. In some examples, the calibration image displayed by the display can be directly captured by the first camera 1 and the second camera 2. In some examples, the head-mounted display device 4 includes a display and an optical element, and the image displayed by the display can be projected to the outside through the optical element, and the first camera 1 and the second camera 2 can also capture the projected image through the optical element.

[0033] Optionally, the first camera 1 is used to capture the calibration image displayed through the first display to obtain a first image. The second camera 2 is used to capture the calibration image displayed through the second display to obtain a second image.

[0034] The head-mounted display device 4 can be an electronic device with image display function and audio playback function. Users can view flat or stereoscopic pictures or videos through the head-mounted display device. The head-mounted display device includes, but is not limited to, Augmented Reality (AR) devices, Virtual Reality (VR) devices, Mixed Reality (MR) devices, etc.

[0035] The first camera 1 and the second camera 2 can be fixed on the fixing jig 3 all the time, and the first camera 1 and the second camera 2 can be calibrated to obtain the calibration parameters of the first camera 1 and the second camera 2. The calibration parameters can include parameters such as camera extrinsic parameters and camera intrinsic parameters. Each head-mounted display device to be calculated for the convergence adjustment amount is fixed on the fixing jig 3 together with the first camera 1 and the second camera 2 in turn, ensuring that the shooting positions and directions of the first camera 1 and the second camera 2 are approximately equivalent to the eye positions of a human eye wearing the head-mounted display device.

[0036] The terminal device 5 can be a mobile phone, a computer, a tablet, a server, etc. The terminal device 5 can be used to calculate the convergence adjustment amount. For example, the terminal device 5 acquires a first image captured by the first camera and a second image captured by the second camera. The first image is a calibration image displayed through the first display, and the second image is a calibration image displayed through the second display. The calibration image can include preset feature points. The terminal device 5 can determine the first pixel coordinates of the preset feature points in the coordinate system of the first image and the second pixel coordinates of the preset feature points in the coordinate system of the second image. The terminal device 5 can also acquire the first screen coordinates of the preset feature points in the coordinate system of the first display and the second screen coordinates of the preset feature points in the coordinate system of the second display. According to the first pixel coordinates, the second pixel coordinates, the first screen coordinates, the second screen coordinates, and the preset convergence plane, the first convergence adjustment amount corresponding to the first display and the second convergence adjustment amount corresponding to the second display are determined.

[0037] Understandably, a head-mounted display device usually has two displays corresponding to the user's two eyes respectively. It is necessary for the light rays emitted from the points at the same position of a certain picture displayed on the two displays to converge in order to provide a better visual experience for the user. According to the design of the product, the light rays emitted from the points at the same position of a certain picture displayed on the two displays should theoretically converge at the designed position. That is, theoretically, the head-mounted display device has a preset convergence position. Due to reasons such as production, assembly, software configuration, etc., multiple head-mounted display devices produced according to the same design may not have the same convergence position. In the related art, there may be a solution for calculating the actual optimal convergence positions of one or more head-mounted display devices, and then using the results calculated by this solution to calibrate the one or more head-mounted display devices. In the above solution, the preset convergence position of some head-mounted display devices may be very different from the designed convergence position, or after calibration, different head-mounted display devices may have different convergence positions. This may result in poor consistency of head-mounted display devices of the same model, and also because the head-mounted display devices do not have a consistent convergence position, it is impossible to conduct a unified inspection of good products for the convergence positions of the head-mounted display devices subsequently.

[0038] In the solution of the embodiment of the present disclosure, a preset convergence plane is set. The preset convergence plane can be understood as a plane that is designed in advance and is at a preset distance from the head-mounted display device. Through the head-mounted display device, the user can view a display picture that is at a preset distance from the head-mounted display device. Since the camera can always be fixed on the fixed fixture and is also at a fixed value distance from the position for fixing the head-mounted display device, it can also be considered that there is also a certain preset distance between the preset convergence plane and the camera.

[0039] Exemplary method

[0040] Figure 2 It is a schematic flowchart of a method for calculating a convergence adjustment amount provided by an exemplary embodiment of the present disclosure. The embodiment of the present disclosure can be applied to an electronic device, such as Figure 2 As shown, the method includes the following steps:

[0041] Step 210, obtain a first image captured by a first camera and a second image captured by a second camera.

[0042] In some optional embodiments of the present disclosure, the first image is a calibration image displayed through a first display. The second image is a calibration image displayed through a second display. The calibration image includes preset feature points.

[0043] In some optional embodiments of the present disclosure, the first camera and the second camera can be understood as constituting a binocular camera.

[0044] In some alternative embodiments of the present disclosure, the first display and the second display are the displays of a head-mounted display device. For example, the display of an augmented reality glasses.

[0045] In some alternative embodiments of the present disclosure, the calibration image may be an image similar to a calibration board. For example, the calibration image may be a checkerboard dot matrix image, a dot matrix image of circles, a dot matrix image with feature markers (such as digital markers), and so on. The calibration image includes at least one preset feature point.

[0046] In some alternative embodiments of the present disclosure, the calibration images displayed on the first display and the second display are respectively projected onto the first camera and the second camera through their respective corresponding optical modules, so that the first camera captures a first image and the second camera captures a second image. For example, the first display and the second display display the same calibration image containing at least one preset feature point.

[0047] The execution subject of the method for calculating the convergence adjustment amount (which may be a terminal device, such as a mobile phone, a tablet, a server, etc.) obtains the first image captured by the first camera and the second image captured by the second camera.

[0048] Step 220, determine the first pixel coordinates of the preset feature point in the coordinate system of the first image and the second pixel coordinates of the preset feature point in the coordinate system of the second image.

[0049] In some alternative embodiments of the present disclosure, the first pixel coordinates and the second pixel coordinates of the preset feature point can be obtained by performing feature point detection and feature matching on the first image and the second image.

[0050] In some alternative embodiments of the present disclosure, the number of preset feature points is at least one. When the number of preset feature points is multiple, the first pixel coordinates and the second pixel coordinates respectively corresponding to each preset feature point can be determined.

[0051] Step 230, obtain the first screen coordinates of the preset feature point in the coordinate system of the first display and the second screen coordinates of the preset feature point in the coordinate system of the second display.

[0052] In some alternative embodiments of the present disclosure, the first screen coordinates and the second screen coordinates of the preset feature point can be obtained in advance. For example, according to the position of the preset feature point in the calibration image and the relationship between the calibration image and the coordinate systems of the first display and the second display, the first screen coordinates and the second screen coordinates of the preset feature point are obtained.

[0053] For example, taking the case where the same calibration image is used for display on the first display and the second display, and the calibration image includes a preset feature point A. If the calibration image is a preset image, the position of point A in the preset image is known. By controlling the first display and the second display to display the preset image, the display positions of point A on the first display and the second display can be obtained, that is, the first screen coordinates and the second screen coordinates are obtained. After obtaining the first image captured by the first camera and the second image captured by the second camera, the positions of point A in the first image and the second image can be obtained, that is, the first pixel coordinates and the second pixel coordinates are obtained.

[0054] Step 240, determine a first convergence adjustment amount corresponding to the first display and a second convergence adjustment amount corresponding to the second display according to the first pixel coordinates, the second pixel coordinates, the first screen coordinates, the second screen coordinates, and the preset convergence plane.

[0055] In some alternative embodiments of the present disclosure, the preset convergence plane is at a preset distance from between the first camera and the second camera. It can be understood that the distance between the designed eye position of the left eye or the right eye of the head-mounted display device and the preset convergence plane can also be the above preset distance, or a distance that differs from the above preset distance by a certain value. When the user wears the head-mounted display device, the user's both eyes can view the picture displayed at the position of the preset convergence plane.

[0056] In some alternative embodiments of the present disclosure, the preset distance can be set according to actual needs. For example, the preset distance can be 3 meters, 4 meters, 6 meters, 8 meters or other distances suitable for viewing the display picture. For example, if it is desired to view the display picture through the head-mounted display device at a distance of 6 meters from the user's both eyes, the preset distance between the preset convergence plane and the first camera and the second camera can be set to 6 meters.

[0057] In some alternative embodiments of the present disclosure, through calibration, the conversion relationship between the coordinate system of the first camera and the coordinate system of the first image, and the conversion relationship between the coordinate system of the second camera and the coordinate system of the second image can be obtained. Combining the above conversion relationships and the first pixel coordinates, the second pixel coordinates, the first screen coordinates and the second screen coordinates, a mapping relationship between the coordinate system of the first display, the coordinate system of the first camera and the coordinate system of the preset convergence plane can be established, and a mapping relationship between the coordinate system of the second display, the coordinate system of the second camera and the coordinate system of the preset convergence plane can be established. Based on each mapping relationship, the conversion between coordinate systems can be realized.

[0058] In some alternative embodiments of the present disclosure, the first convergence adjustment amount and the second convergence adjustment amount can be obtained based on the principle that the adjusted screen coordinates of the preset feature points can converge on the preset convergence plane. For example, the first convergence adjustment amount and the second convergence adjustment amount can be initialized, and with the goal of making the adjusted screen coordinates of the preset feature points converge on the preset convergence plane, the first convergence adjustment amount and the second convergence adjustment amount can be continuously updated and optimized to obtain the optimal first convergence adjustment amount and the second convergence adjustment amount.

[0059] For example, for the first convergence adjustment amount and the second convergence adjustment amount obtained for each update, the first screen coordinates of the preset feature points can be adjusted according to the first convergence adjustment amount to obtain the first adjusted screen coordinates. The second screen coordinates of the preset feature points can be adjusted according to the second convergence adjustment amount to obtain the second adjusted screen coordinates. Based on the error between the projected coordinates of the first adjusted screen coordinates on the preset convergence plane and the projected coordinates of the second adjusted screen coordinates on the preset convergence plane (which can be simply referred to as the error on the convergence plane), the first convergence adjustment amount and the second convergence adjustment amount can be updated to obtain the updated first convergence adjustment amount and the second convergence adjustment amount. Then, for the first convergence adjustment amount and the second convergence adjustment amount obtained in this update, the above process can be repeatedly executed until the update end condition is met, and the obtained first convergence adjustment amount and the second convergence adjustment amount can be used as the first convergence adjustment amount corresponding to the first display and the second convergence adjustment amount corresponding to the second display.

[0060] As another example, after obtaining the first adjusted screen coordinates and the second adjusted screen coordinates, the first convergence adjustment amount can also be updated based on the error between the projected coordinates of the expected convergence point of the preset feature points on the first display and the first adjusted screen coordinates (which can be simply referred to as the error on the display) to obtain the updated first convergence adjustment amount. The second convergence adjustment amount can be updated based on the error between the projected coordinates of the expected convergence point of the second display and the second adjusted screen coordinates to obtain the updated second convergence adjustment amount. By repeatedly executing the above process, the first convergence adjustment amount corresponding to the first display and the second convergence adjustment amount corresponding to the second display can be obtained.

[0061] The method for calculating the convergence adjustment amount provided by the embodiments of the present disclosure is such that the first camera captures a calibration image displayed on the first display, and the second camera captures a calibration image displayed on the second display. The first pixel coordinates of a preset feature point on the calibration image in the coordinate system of the first image and the second pixel coordinates in the coordinate system of the second image can be determined, and the first screen coordinates of the preset feature point on the calibration image in the coordinate system of the first display and the second screen coordinates in the coordinate system of the second display can be obtained. With the goal of converging the preset feature point on a preset convergence plane, the first convergence adjustment amount corresponding to the first display and the second convergence adjustment amount corresponding to the second display are determined, so that the first convergence adjustment amount and the second convergence adjustment amount for adjusting the first display and the second display respectively can be obtained. The first display and the second display are adjusted using the obtained first convergence adjustment amount and second convergence adjustment amount, so that the images displayed on the two displays can converge on the preset convergence plane. Using this solution, the head-mounted display device can have a convergence position consistent with the design. If multiple head-mounted display devices are adjusted using this solution, it helps to improve the product consistency.

[0062] Figure 3 It is a schematic flowchart of determining the pixel coordinates of a preset feature point provided by an exemplary embodiment of the present disclosure.

[0063] In some alternative embodiments of the present disclosure, step 220 may include step 2210, step 2220, step 2230, and step 2240.

[0064] In step 2210, feature points are extracted from the first image to obtain the first feature points in the first image.

[0065] In step 2220, feature points are extracted from the second image to obtain the second feature points in the second image.

[0066] In some alternative embodiments of the present disclosure, any feasible feature extraction method may be adopted for extracting feature points from the first image and the second image.

[0067] In some alternative embodiments of the present disclosure, the first feature points in the first image are the pixel points of the preset feature point in the first image. The second feature points are similar to the first feature points and will not be elaborated here.

[0068] In step 2230, the first feature points and the second feature points are feature-matched to obtain the matching relationship between the first feature points and the second feature points.

[0069] In some alternative embodiments of the present disclosure, feature matching can be implemented by any feasible feature matching method. For example, a feature matching algorithm based on SIFT, a feature matching algorithm based on ORB, etc.

[0070] Step 2240: Determine the first pixel coordinates of the preset feature point in the first image and the second pixel coordinates of the preset feature point in the second image according to the pixel coordinates of the first feature point in the first image, the pixel coordinates of the second feature point in the second image, and the matching relationship between the first feature point and the second feature point.

[0071] In some optional embodiments of the present disclosure, according to the matching relationship between the first feature point and the second feature point, the corresponding relationship between the first feature point in the first image and the second feature point in the second image can be obtained. That is, the first feature point and the second feature point in the first image that belong to the same preset feature point as in the second image are obtained. Based on this, combining the pixel coordinates of the first feature point in the first image and the pixel coordinates of the second feature point in the second image, the first pixel coordinates of the preset feature point in the first image and the second pixel coordinates of the preset feature point in the second image can be obtained.

[0072] In this embodiment, by performing feature point extraction and matching on the first image and the second image, the first pixel coordinates of the preset feature point in the first image and the second pixel coordinates of the preset feature point in the second image can be obtained, which is convenient for determining the first convergence adjustment amount and the second convergence adjustment amount.

[0073] Figure 4 It is a schematic flowchart of a process for determining the convergence adjustment amount provided by an exemplary embodiment of the present disclosure.

[0074] In some optional embodiments of the present disclosure, step 240 may include step 2410 and step 2420.

[0075] Step 2410: Establish a first mapping relationship between the coordinate system of the first display and the coordinate system of the first camera and a second mapping relationship between the coordinate system of the second display and the coordinate system of the second camera according to the first pixel coordinates, the second pixel coordinates, the first screen coordinates, the second screen coordinates, the first conversion relationship between the coordinate system of the first image and the coordinate system of the first camera, and the second conversion relationship between the coordinate system of the second image and the coordinate system of the second camera.

[0076] In some optional embodiments of the present disclosure, the first conversion relationship between the coordinate system of the first image and the coordinate system of the first camera and the second conversion relationship between the coordinate system of the second image and the coordinate system of the second camera can be obtained in advance by calibrating the first camera and the second camera. For example, the first conversion relationship may include the internal parameters of the first camera, and the second conversion relationship may include the internal parameters of the second camera.

[0077] In some alternative embodiments of the present disclosure, according to the first conversion relationship between the coordinate system of the first image and the coordinate system of the first camera, the first pixel coordinates can be converted to the coordinate system of the first camera to obtain the first coordinates of the preset feature points in the coordinate system of the first camera. According to the first coordinates and the first screen coordinates, a first mapping relationship between the coordinate system of the first display and the coordinate system of the first camera can be established. For example, the first mapping relationship can be expressed as F l , for the first screen coordinate p screen,l on the first display, cam,l the mapping to the coordinate system of the first camera can be expressed as p l = F screen,l (p

[0078] r In some alternative embodiments of the present disclosure, according to the second conversion relationship between the coordinate system of the second image and the coordinate system of the second camera, the second pixel coordinates can be converted to the coordinate system of the second camera to obtain the second coordinates of the preset feature points in the coordinate system of the second camera. According to the second coordinates and the second screen coordinates, a second mapping relationship between the coordinate system of the second display and the coordinate system of the second camera can be established. For example, the second mapping relationship can be expressed as F screen,r , for the second screen coordinate p cam,r on the second display, r the mapping to the coordinate system of the second camera can be expressed as p screen,r = F

[0079]

[0080] Step 2420, determine the first convergence adjustment amount and the second convergence adjustment amount according to the first screen coordinates, the second screen coordinates, the first mapping relationship, the second mapping relationship, the third mapping relationship between the coordinate system of the first camera and the coordinate system of the preset convergence plane, and the fourth mapping relationship between the coordinate system of the second camera and the coordinate system of the preset convergence plane.

[0081] In some alternative embodiments of the present disclosure, the third mapping relationship can be obtained based on the calibration parameters of the first camera and the spatial information of the preset convergence plane. The fourth mapping relationship can be obtained based on the calibration parameters of the second camera and the spatial information of the preset convergence plane.

[0082] In some alternative embodiments of the present disclosure, the third mapping relationship and the fourth mapping relationship can be obtained and stored in advance. Then, the third mapping relationship and the fourth mapping relationship can be obtained from the corresponding storage space.In some alternative embodiments of the present disclosure, by combining the first mapping relationship, the second mapping relationship, the third mapping relationship, and the fourth mapping relationship, the coordinate system conversion between the coordinate systems of the display (including the coordinate system of the first display and the coordinate system of the second display), the image coordinate system (including the coordinate system of the first image and the coordinate system of the second image), and the coordinate system of the preset convergence plane can be realized, which is convenient for the coordinate conversion during the update and optimization process of the convergence adjustment amount (including the first convergence adjustment amount and the second convergence adjustment amount), so as to construct an objective function based on the convergence error and realize the update and optimization of the convergence adjustment amount. The convergence error can be the error in the convergence plane or the error in the display. The error in the convergence plane and the error in the display can be referred to the foregoing embodiments.

[0083] In this embodiment, by establishing the first mapping relationship between the coordinate system of the first display and the coordinate system of the first camera, and the second mapping relationship between the coordinate system of the second display and the coordinate system of the second camera, it is convenient to combine the third mapping relationship between the coordinate system of the first camera and the coordinate system of the preset convergence plane, and the fourth mapping relationship between the coordinate system of the second camera and the coordinate system of the preset convergence plane, so as to realize the coordinate conversion between the coordinate system of the first display and the coordinate system of the preset convergence plane, and the coordinate conversion between the coordinate system of the second display and the coordinate system of the preset convergence plane, thereby contributing to calculating the convergence error of the adjusted screen coordinates and facilitating the update and optimization of the convergence adjustment amount.

[0084] Figure 5 It is a schematic flowchart of determining the convergence adjustment amount provided by another exemplary embodiment of the present disclosure.

[0085] In some alternative embodiments of the present disclosure, step 2420 may include 24210 and 24220.

[0086] Step 24210, construct an objective function regarding the first convergence adjustment amount parameter and the second convergence adjustment amount parameter according to the first screen coordinate, the first convergence adjustment amount parameter, the first mapping relationship and the third mapping relationship, the second screen coordinate, the second convergence adjustment amount parameter, the second mapping relationship and the fourth mapping relationship.

[0087] In some alternative embodiments of the present disclosure, the first convergence adjustment amount parameter represents the first convergence adjustment amount to be solved. The second convergence adjustment amount parameter represents the second convergence adjustment amount to be solved. That is, the first convergence adjustment amount parameter and the second convergence adjustment amount parameter are used as parameters to be optimized to construct an objective function. For example, the first convergence adjustment amount parameter can be expressed as (Δx1, Δy1), and the second convergence adjustment amount parameter can be expressed as (Δx2, Δy2). Wherein, Δx1 represents the horizontal convergence adjustment amount parameter corresponding to the first display, Δy1 represents the vertical convergence adjustment amount parameter corresponding to the first display, Δx2 represents the horizontal convergence adjustment amount parameter corresponding to the second display, and Δy2 represents the vertical convergence adjustment amount parameter corresponding to the second display.

[0088] In some alternative embodiments of the present disclosure, the objective function can be constructed based on any one of the errors of the preset feature points in the convergence plane and the errors in the display.

[0089] Step 24220: Solve the first convergence adjustment amount parameter and the second convergence adjustment amount parameter that satisfy the preset conditions of the objective function to obtain the first convergence adjustment amount and the second convergence adjustment amount.

[0090] In some alternative embodiments of the present disclosure, any implementable solution method can be used to solve the first convergence adjustment amount parameter and the second convergence adjustment amount parameter that satisfy the preset conditions of the objective function to obtain the first convergence adjustment amount and the second convergence adjustment amount. For example, any one of the gradient descent method, Newton method, Gauss-Newton method, LM (Levenberg–Marquardt) algorithm, etc. can be used to solve the first convergence adjustment amount parameter and the second convergence adjustment amount parameter that satisfy the preset conditions of the objective function to obtain the first convergence adjustment amount and the second convergence adjustment amount.

[0091] In this embodiment, by constructing an objective function regarding the first convergence adjustment amount parameter and the second convergence adjustment amount parameter, it is convenient to obtain the first convergence adjustment amount and the second convergence adjustment amount through a solution method, which helps to improve the accuracy and effectiveness of the convergence adjustment amount.

[0092] In some alternative embodiments of the present disclosure, the first convergence adjustment amount and the second convergence adjustment amount are obtained by solving the following objective function:

[0093]

[0094] Wherein, argminJ represents solving (Δx k , Δy k ) that minimizes J. k = 1, 2, (Δx1, Δy1) represents the first convergence adjustment amount parameter to be solved. (Δx2, Δy2) represents the second convergence adjustment amount parameter to be solved. n represents the number of preset feature points. w iRepresents the weight of the i-th preset feature point. |||| Represents taking the absolute value. p proj,1,i Represents the first convergence point projection coordinates on the first display of the expected convergence point of the i-th preset feature point on the preset convergence plane. p prij,2,i Represents the second convergence point projection coordinates on the second display of the expected convergence point of the i-th preset feature point. p′ screen,1,i (Δx1, Δy1) represents the screen coordinates of the i-th preset feature point adjusted based on the first convergence adjustment parameter. p′ screen,2,i (Δx2, Δy2) represents the screen coordinates of the i-th preset feature point adjusted based on the second convergence adjustment parameter.

[0095] In some alternative embodiments of the present disclosure, w i can be a preset weight or can be continuously adjusted and optimized during the optimization process.

[0096] In some alternative embodiments of the present disclosure, the weight w i can be set according to the distance between the preset feature point and the center of the calibration image. For example, the closer the preset feature point is to the center of the calibration image, the greater the weight, and the farther away from the center, the smaller the weight.

[0097] In some alternative embodiments of the present disclosure, the calibration image can also be divided into regions, different regions are set with different weights, and each region corresponds to a weight. The specific setting of the weight is not limited.

[0098] In some alternative embodiments of the present disclosure, for the expected convergence point of any preset feature point, the expected convergence point of the preset feature can be determined according to the first projection coordinates of the first screen coordinates of the preset feature point on the preset convergence plane and the second projection coordinates of the second screen coordinates of the preset feature point on the preset convergence plane. For example, calculate the midpoint between the first projection coordinates and the second projection coordinates and use the midpoint as the expected convergence point. For another example, use any one of the first projection coordinates and the second projection coordinates as the expected convergence point. Other points between the first projection coordinates and the second projection coordinates can also be used as the expected convergence point. The specific is not limited.

[0099] In some alternative embodiments of the present disclosure, according to the conversion relationship between the coordinate system of the first display and the coordinate system of the preset convergence plane, the expected convergence point of the i-th preset feature point on the preset convergence plane can be converted to the coordinate system of the first display to obtain the first convergence point projection coordinates p proj,1,i . According to the conversion relationship between the coordinate system of the second display and the coordinate system of the preset convergence plane, the expected convergence point of the i-th preset feature point on the preset convergence plane can be converted to the coordinate system of the second display to obtain the second convergence point projection coordinates p proj,2,i .

[0100] In some alternative embodiments of the present disclosure, the first screen coordinates of the i-th preset feature point may be adjusted based on the first convergence adjustment parameter (Δx1, Δy1) to obtain the adjusted screen coordinates p' of the i-th preset feature point. screen,1,i (Δx1, Δy1). The second screen coordinates of the i-th preset feature point are adjusted based on the second convergence adjustment parameter (Δx2, Δy2) to obtain the adjusted screen coordinates p' of the i-th preset feature point. screen,2,i (Δx2, Δy2).

[0101] In some alternative embodiments of the present disclosure, the first screen coordinates may be superimposed with the first convergence adjustment parameter to obtain the adjusted screen coordinates corresponding to the first screen coordinates. The second screen coordinates are superimposed with the second convergence adjustment parameter to obtain the adjusted screen coordinates corresponding to the second screen coordinates. For example, if the first screen coordinates are represented as (x1, y1), the adjusted screen coordinates may be represented as (x1 + Δx1, y1 + Δy1).

[0102] In some alternative embodiments of the present disclosure, Figure 6 is a schematic diagram of the principle of constructing an objective function provided by an exemplary embodiment of the present disclosure. As Figure 6 shown, a preset convergence plane is at a preset distance from the first camera and the second camera. For any preset feature point A, the coordinates of the preset feature point A in the coordinate system of the first display are the first screen coordinates, and the coordinates of A in the coordinate system of the second display are the second screen coordinates. If it is desired to make the preset feature point A converge on the preset convergence plane, assuming the desired convergence point is as Figure 6 shown, then the desired convergence point is projected onto the first display to obtain the first convergence point projection coordinates, and the desired convergence point is projected onto the second display to obtain the second convergence point projection coordinates. The desired convergence point may represent the convergence point of the adjusted screen coordinates of A on the preset convergence plane. If the first screen coordinates and the second screen coordinates of A are adjusted respectively based on the first convergence adjustment parameter and the second convergence adjustment parameter, such that the adjusted screen coordinates corresponding to the first screen coordinates and the adjusted screen coordinates corresponding to the second screen coordinates can converge at the desired convergence point, then the first convergence point projection coordinates should coincide with the adjusted screen coordinates of the first screen coordinates, and the second convergence point projection coordinates should coincide with the adjusted screen coordinates of the second screen coordinates. Based on this, the objective function of the above formula (1) is constructed, and by solving the first convergence adjustment parameter and the second convergence adjustment parameter that minimize the error between the convergence point projection coordinates and the adjusted screen coordinates of each preset feature point, the first convergence adjustment amount and the second convergence adjustment amount can be obtained.

[0103] In this embodiment, the objective function is constructed based on the error between the adjusted screen coordinates of the preset feature points and the projected coordinates of the desired convergence point on the convergence point of the display, and the first convergence adjustment amount and the second convergence adjustment amount are obtained by solving, realizing the effective determination of the convergence adjustment amount, which helps to improve the accuracy and effectiveness of the convergence adjustment amount.

[0104] In some alternative embodiments of the present disclosure, the first convergence adjustment amount and the second convergence adjustment amount are obtained by solving the following objective function:

[0105]

[0106] where argminJ represents solving for (Δx l , Δy l ) and (Δx r , Δy r ) that minimize J. (Δx l , Δu l ) represents the first convergence adjustment amount parameter. (Δx r , Δu r ) represents the second convergence adjustment amount parameter. n represents the number of preset feature points. w i represents the weight of the i-th preset feature point. |||| represents taking the absolute value. p plane,l,i (Δx l , Δy l ) represents the projected coordinates on the preset convergence plane of the screen coordinates of the i-th preset feature point adjusted based on the first convergence adjustment amount parameter. p plane,r,i (Δx r , Δy r ) represents the projected coordinates on the preset convergence plane of the screen coordinates of the i-th preset feature point adjusted based on the second convergence adjustment amount parameter.

[0107] In some alternative embodiments of the present disclosure, the weight w i is similar to the weight in the aforementioned formula (1) and will not be elaborated here.

[0108] In some alternative embodiments of the present disclosure, the first screen coordinates (x l , y l ) of the i-th preset feature point can be adjusted based on the first convergence adjustment amount parameter (Δx l,i , Δy l,i ) to obtain the adjusted screen coordinates (x' l,i , y' l,i ) corresponding to the first screen coordinates. The second screen coordinates (x r , y r ) of the i-th preset feature point are adjusted based on the second convergence adjustment amount parameter (Δx r,i , Δy r,i)Adjust to obtain the adjusted screen coordinates (x′ r,i , y′ r,i ) corresponding to the second screen coordinates.

[0109] In some alternative embodiments of the present disclosure, according to the conversion relationship between the coordinate system of the first display and the coordinate system of the preset convergence plane, (x′ l,i , y′ l,i ) is converted to the coordinate system of the preset convergence plane to obtain p plane,l,i (Δx l , Δy l ). According to the conversion relationship between the coordinate system of the second display and the coordinate system of the preset convergence plane, (x ′ r,i , y ′ r,i ) is converted to the coordinate system of the preset convergence plane to obtain p plane,r,i (Δx r , Δy r ).

[0110] In some alternative embodiments of the present disclosure, Figure 7 is a schematic diagram of the principle of constructing the objective function provided by another exemplary embodiment of the present disclosure. As Figure 7 shown, for any preset feature point A, the screen coordinates of A in the coordinate system of the first display are the first screen coordinates, and the screen coordinates of A in the coordinate system of the second display are the second screen coordinates. The projection point coordinates of the first screen coordinates of A on the preset convergence plane are the first projection coordinates. The projection point coordinates of the second screen coordinates of A on the preset convergence plane are the second projection coordinates. The screen coordinates obtained by adjusting the first screen coordinates based on the first convergence adjustment parameter are the first adjusted screen coordinates. The screen coordinates obtained by adjusting the second screen coordinates based on the second convergence adjustment parameter are the second adjusted screen coordinates. The first adjusted screen coordinates are projected onto the preset convergence plane to obtain the first adjusted projection coordinates. The second adjusted screen coordinates are projected onto the preset convergence plane to obtain the second adjusted projection coordinates. If the adjusted preset feature point A can converge on the preset convergence plane, the first adjusted projection coordinates and the second adjusted projection coordinates should coincide. Based on this, the objective function of the above formula (2) is constructed, and by solving the first convergence adjustment parameter and the second convergence adjustment parameter that minimize the error between the first adjusted projection coordinates and the second adjusted projection coordinates of each preset feature point, the first convergence adjustment amount and the second convergence adjustment amount are obtained.

[0111] In this embodiment, a target function is constructed based on the convergence error of the screen coordinates adjusted by the preset feature points on the convergence plane, and the first convergence adjustment amount and the second convergence adjustment amount are obtained by solving, realizing the effective determination of the convergence adjustment amount, which helps to improve the accuracy and effectiveness of the convergence adjustment amount, so that the imaging pictures of the first display and the second display after adjustment can converge on the preset convergence plane.

[0112] In some alternative embodiments of the present disclosure, the solution in step 24220 to obtain the first convergence adjustment amount parameter and the second convergence adjustment amount parameter that satisfy the preset conditions, and obtain the first convergence adjustment amount and the second convergence adjustment amount, includes: determining the first current convergence adjustment amount of the first convergence adjustment amount parameter and the second current convergence adjustment amount of the second convergence adjustment amount parameter. The first convergence adjustment amount parameter is initially the first initial convergence adjustment amount. The second convergence adjustment amount parameter is initially the second initial convergence adjustment amount. According to the first current convergence adjustment amount, the first screen coordinates are adjusted to obtain the first adjusted screen coordinates of the preset feature points. According to the first mapping relationship and the third mapping relationship, the first adjusted screen coordinates are mapped to the coordinate system of the preset convergence plane to obtain the first convergence coordinates of the preset feature points. According to the second current convergence adjustment amount, the second screen coordinates are adjusted to obtain the second adjusted screen coordinates of the preset feature points. According to the second mapping relationship and the fourth mapping relationship, the second adjusted screen coordinates are mapped to the coordinate system of the preset convergence plane to obtain the second convergence coordinates of the preset feature points. According to the first convergence coordinates and the second convergence coordinates of the preset feature points, the current target function value is determined. Based on the current target function value, the first current convergence adjustment amount and the second current convergence adjustment amount are updated to obtain the first updated convergence adjustment amount and the second updated convergence adjustment amount. In response to meeting the iteration end condition, the first updated convergence adjustment amount is used as the first convergence adjustment amount, and the second updated convergence adjustment amount is used as the second convergence adjustment amount. In response to not meeting the iteration end condition, the first updated convergence adjustment amount is used as the first current convergence adjustment amount, and the second updated convergence adjustment amount is used as the second current convergence adjustment amount, and the step of adjusting the first screen coordinates of the preset feature points according to the first current convergence adjustment amount to obtain the first adjusted screen coordinates of the preset feature points is repeated.

[0113] In some alternative embodiments of the present disclosure, the first current convergence adjustment amount and the second current convergence adjustment amount, which are the convergence adjustment amounts, may be the first convergence adjustment amount and the second convergence adjustment amount after the previous update. The first current convergence adjustment amount is initially the initialization result of the first convergence adjustment amount parameter. The second current convergence adjustment amount is initially the initialization result of the second convergence adjustment amount parameter. For example, the first convergence adjustment amount parameter and the second convergence adjustment amount parameter may be initialized according to a preset initialization method as the initial first current convergence adjustment amount and the second current convergence adjustment amount, and then enter the iterative optimization process. The preset initialization method may be, for example, random initialization, preset value initialization (i.e., initializing the first convergence adjustment amount parameter and the second convergence adjustment amount parameter to preset values), etc. In the first iteration process, according to the initial first current convergence adjustment amount, the first screen coordinates are adjusted to obtain the first adjusted screen coordinates of the preset feature points. According to the first mapping relationship and the third mapping relationship, the first adjusted screen coordinates are mapped to the coordinate system of the preset convergence plane to obtain the first convergence coordinates of the preset feature points (which can be understood by referring to the first adjusted projection coordinates in Figure 7 . According to the second current convergence adjustment amount, the second screen coordinates are adjusted to obtain the second adjusted screen coordinates of the preset feature points. According to the second mapping relationship and the fourth mapping relationship, the second adjusted screen coordinates are mapped to the coordinate system of the preset convergence plane to obtain the second convergence coordinates of the preset feature points (which can be understood by referring to the second adjusted projection coordinates in Figure 7 . According to the first convergence coordinates and the second convergence coordinates of the preset feature points, the current objective function value is determined. Based on the current objective function value, the first current convergence adjustment amount and the second current convergence adjustment amount are updated to obtain the first updated convergence adjustment amount and the second updated convergence adjustment amount. Enter the second iteration process, determine the first current convergence adjustment amount obtained in the first iteration as the first current convergence adjustment amount in the second iteration process, and determine the second updated convergence adjustment amount obtained in the first iteration as the second current convergence adjustment amount in the second iteration process. Repeat the above process until the iteration end condition is met. Then, take the first updated convergence adjustment amount obtained in the last iteration as the first convergence adjustment amount corresponding to the first display, and take the second updated convergence adjustment amount obtained in the last iteration as the second convergence adjustment amount corresponding to the second display.

[0114] In some alternative embodiments of the present disclosure, the current objective function value is determined according to the first convergence coordinates, the second convergence coordinates, and the objective function of the above formula (2).

[0115] In some alternative embodiments of the present disclosure, after each iterative update, it may be determined whether the iteration end condition is met. If the iteration end condition is not met, enter the next iteration. If the iteration end condition is met, end the iteration to obtain the first convergence adjustment amount and the second convergence adjustment amount.

[0116] In some alternative embodiments of the present disclosure, updating the first current convergence adjustment amount and the second current convergence adjustment amount based on the current objective function value can be achieved by the Newton method, the Gauss-Newton method, etc. For example, based on the current objective function value, the iteration step size can be calculated by any algorithm, and the first current convergence adjustment amount and the second current convergence adjustment amount can be updated according to the iteration step size.

[0117] In this embodiment, by using the error of the convergence coordinates of the preset feature points after adjustment in the preset convergence plane on the screen coordinates (i.e., the error in the convergence plane), the convergence adjustment amount that minimizes the error is solved, realizing the effective determination of the convergence adjustment amount, which helps to improve the accuracy and effectiveness of the first convergence adjustment amount and the second convergence adjustment amount.

[0118] In some alternative embodiments of the present disclosure, constructing the objective function regarding the first convergence adjustment amount parameter and the second convergence adjustment amount parameter in step 24210 according to the first screen coordinate, the first convergence adjustment amount parameter, the first mapping relationship and the third mapping relationship, the second screen coordinate, the second convergence adjustment amount parameter, the second mapping relationship and the fourth mapping relationship includes: determining the target convergence coordinates of the preset feature points in the coordinate system of the preset convergence plane according to the first screen coordinate, the second screen coordinate, the first mapping relationship, the second mapping relationship, the third mapping relationship and the fourth mapping relationship. Mapping the target convergence coordinates to the first display to obtain the first convergence point projection coordinates of the preset feature points. Mapping the target convergence coordinates to the second display to obtain the second convergence point projection coordinates of the preset feature points. Constructing the first objective function regarding the first convergence adjustment amount parameter according to the first convergence point projection coordinates, the first screen coordinate, the first convergence adjustment amount parameter, the first mapping relationship and the third mapping relationship. Constructing the second objective function regarding the second convergence adjustment amount parameter according to the second convergence point projection coordinates, the second screen coordinate, the second convergence adjustment amount parameter, the second mapping relationship and the fourth mapping relationship.

[0119] In some alternative embodiments of the present disclosure, reference may be made to Figure 6 , the first screen coordinate can be mapped to the preset convergence plane according to the first mapping relationship and the third mapping relationship to obtain the first projection coordinate corresponding to the first screen coordinate in the preset convergence plane. According to the second mapping relationship and the fourth mapping relationship, the second screen coordinate is mapped to the preset convergence plane to obtain the second projection coordinate corresponding to the second screen coordinate in the preset convergence plane. According to the first projection coordinate and the second projection coordinate, the target convergence coordinates are determined (which can be understood by referring to Figure 6 for the desired convergence point).

[0120] In some alternative embodiments of the present disclosure, the target convergence coordinates can be mapped to the first display according to the conversion relationship between the coordinate system of the first display and the coordinate system of the preset convergence plane, to obtain the first convergence point projection coordinates. According to the conversion relationship between the coordinate system of the second display and the coordinate system of the preset convergence plane, the target convergence coordinates are mapped to the second display to obtain the second convergence point projection coordinates.

[0121] In some alternative embodiments of the present disclosure, the first objective function is the objective function when k = 1 in the above formula (1). The second objective function is the objective function when k = 2 in the above formula (1), and details are not described herein.

[0122] In this embodiment, by determining the target convergence coordinates of the preset feature points on the preset convergence plane and mapping the target convergence coordinates to the coordinate systems of the first display and the second display, it is convenient to construct the first objective function corresponding to the first display and the second objective function corresponding to the second display, so that the first convergence adjustment amount and the second convergence adjustment amount can be obtained by separate solutions, enabling the adjusted screen coordinates to converge at the target convergence coordinates and improving the accuracy and effectiveness of the convergence adjustment amount.

[0123] In some alternative embodiments of the present disclosure, the solution of step 24220 to obtain the first convergence adjustment amount parameter and the second convergence adjustment amount parameter that satisfy the preset conditions for the objective function, and obtain the first convergence adjustment amount and the second convergence adjustment amount, includes: solving the first convergence adjustment amount parameter that satisfies the first preset condition for the first objective function to obtain the first convergence adjustment amount. Solving the second convergence adjustment amount parameter that satisfies the second preset condition for the second objective function to obtain the second convergence adjustment amount.

[0124] In some alternative embodiments of the present disclosure, the specific solution methods for the first convergence adjustment amount and the second convergence adjustment amount can be implemented by any one of the Newton method, the Gauss-Newton method, etc.

[0125] In this embodiment, through the first objective function and the second objective function, the first convergence adjustment amount and the second convergence adjustment amount can be obtained by separate solutions, enabling the adjusted screen coordinates to converge at the target convergence coordinates, reducing the convergence error, and improving the convergence accuracy.

[0126] In some alternative embodiments of the present disclosure, determining the target convergence coordinates of a preset feature point in the coordinate system of a preset convergence plane according to a first screen coordinate, a second screen coordinate, a first mapping relationship, a second mapping relationship, a third mapping relationship, and a fourth mapping relationship includes: mapping the first screen coordinate to the coordinate system of the preset convergence plane according to the first mapping relationship and the third mapping relationship to obtain a first projection coordinate of the preset feature point. Mapping the second screen coordinate to the coordinate system of the preset convergence plane according to the second mapping relationship and the fourth mapping relationship to obtain a second projection coordinate of the preset feature point. Determining the target convergence coordinates of the preset feature point in the coordinate system of the preset convergence plane according to the first projection coordinate and the second projection coordinate.

[0127] In some alternative embodiments of the present disclosure, the first mapping relationship is a mapping relationship between the coordinate system of a first display and the coordinate system of a first camera. Therefore, the first screen coordinate can be mapped to the coordinate system of the first camera according to the first mapping relationship to obtain a first coordinate in the coordinate system of the first camera. The third mapping relationship is a mapping relationship between the coordinate system of the first camera and the coordinate system of the preset convergence plane. Therefore, the first coordinate can be mapped to the preset convergence plane according to the third mapping relationship to obtain the first projection coordinate. Similarly, the second screen coordinate can be mapped to the coordinate system of a second camera according to the second mapping relationship to obtain a second coordinate in the coordinate system of the second camera. The second coordinate is mapped to the preset convergence plane according to the fourth mapping relationship to obtain the second projection coordinate.

[0128] In some alternative embodiments of the present disclosure, either the first projection coordinate or the second projection coordinate can be used as the target convergence coordinate.

[0129] In some alternative embodiments of the present disclosure, the coordinate of any point located between the first projection coordinate and the second projection coordinate can be used as the target convergence coordinate. For example, the coordinate of the midpoint between the first projection coordinate and the second projection coordinate is used as the target convergence coordinate.

[0130] In this embodiment, the target convergence coordinates are determined through the projection coordinates of the first screen coordinate and the second screen coordinate of the preset feature point in the preset convergence plane, which can guide the optimization of the convergence adjustment amount during the optimization process of the convergence adjustment amount, so that the adjusted screen coordinates can converge at the target convergence coordinates.

[0131] Figure 8 It is a schematic flowchart of a method for calculating a convergence adjustment amount provided by another exemplary embodiment of the present disclosure.

[0132] In some alternative embodiments of the present disclosure, the method of the embodiments of the present disclosure further includes step 310 and step 320.

[0133] Step 310: Establish a third mapping relationship between the coordinate system of the first camera and the coordinate system of the preset convergence plane according to the calibration parameters of the first camera and the spatial information of the preset convergence plane.

[0134] In some alternative embodiments of the present disclosure, the calibration parameters of the first camera may include the internal parameters, external parameters, and other related parameters of the first camera.

[0135] For example, the external parameters may represent the pose information of the first camera in the reference coordinate system. The reference coordinate system may be a three-dimensional space coordinate system with a preset position in space as the origin. The spatial information of the preset convergence plane may include the pose information of the preset convergence plane in the reference coordinate system. For example, the spatial information of the preset convergence plane may include the coordinates of at least one point on the preset convergence plane and the direction vector of the preset convergence plane. The specific representation method of the spatial information is not limited.

[0136] In some alternative embodiments of the present disclosure, according to the calibration parameters of the first camera and the spatial information of the preset convergence plane, the pose information of the preset convergence plane in the coordinate system of the first camera may be calculated. According to the pose information of the preset convergence plane in the coordinate system of the first camera, a third mapping relationship between the coordinate system of the first camera and the coordinate system of the preset convergence plane is established. For example, the third mapping relationship may be expressed as f l , for a point p cam,l in the coordinate system of the first camera, plane,l mapping to the preset convergence plane may be expressed as p 2,l = f cam,l (p

[0137] Step 320: Establish a fourth mapping relationship between the coordinate system of the second camera and the coordinate system of the preset convergence plane according to the calibration parameters of the second camera and the spatial information of the preset convergence plane.

[0138] In some alternative embodiments of the present disclosure, the fourth mapping relationship is similar to the above-mentioned third mapping relationship, and will not be elaborated here. For example, the fourth mapping relationship may be expressed as f r , for a point p cam,r in the coordinate system of the second camera, plane,r mapping to the preset convergence plane may be expressed as p 2,r = f cam,r (p

[0139] In this embodiment, by establishing a third mapping relationship between the coordinate system of the first camera and the coordinate system of the preset convergence plane and a fourth mapping relationship between the coordinate system of the second camera and the coordinate system of the preset convergence plane, it is convenient to realize the conversion between the camera coordinate system and the coordinate system of the preset convergence plane during the determination of the convergence adjustment amount, which helps to provide an effective coordinate system conversion relationship for calculating the convergence adjustment amount.

[0140] In some alternative embodiments of the present disclosure, determining the first convergence adjustment amount corresponding to the first display and the second convergence adjustment amount corresponding to the second display according to the first pixel coordinates, the second pixel coordinates, the first screen coordinates, the second screen coordinates, and the preset convergence plane in step 240 includes: determining some preset feature points in a preset area from the preset feature points as target feature points. Determining the first convergence adjustment amount and the second convergence adjustment amount according to the first pixel coordinates, the second pixel coordinates, the first screen coordinates, the second screen coordinates, and the preset convergence plane of the target feature points.

[0141] In some alternative embodiments of the present disclosure, when the number of preset feature points is multiple, some preset feature points in a preset area can be determined from the preset feature points as target feature points. The target feature points are used to determine the first convergence adjustment amount and the second convergence adjustment amount. The specific operations of determining the first convergence adjustment amount and the second convergence adjustment amount according to the target feature points can refer to the determination of the first convergence adjustment amount and the second convergence adjustment amount according to the preset feature points mentioned in the foregoing embodiments, and will not be elaborated herein.

[0142] In some alternative embodiments of the present disclosure, the preset area can be set according to actual needs. For example, the area where the user often gazes can be used as the preset area.

[0143] In some alternative embodiments of the present disclosure, after obtaining the first convergence adjustment amount and the second convergence adjustment amount, the display can be adjusted based on the first convergence adjustment amount and the second convergence adjustment amount. The adjustment method can include any one of adjusting the positions of the first display and the second display, adjusting the imaging parameters of the display, etc., so that after adjustment, the imaging pictures of the first display and the second display converge on the preset convergence plane.

[0144] In the related art, when calculating the convergence error, usually only the theoretically optimal convergence point is calculated, and the convergence error of the display is calculated using the optimal convergence point, without making specific depth restrictions. For example, calculating the difference in the longitudinal direction between the first screen coordinate of the same feature point in the first display and the second screen coordinate of the same feature point in the second display, and the optimization goal is to minimize this difference so that the feature points can converge. For example, reference can be made to Figure 6 where the dotted line where the first screen coordinate and the first projection coordinate are located converges with the dotted line where the second screen coordinate and the second projection coordinate are located at the intersection point, but there is no limitation on the distance at which they converge.

[0145] In the method of the embodiments of the present disclosure, in order to ensure that the content seen by the user converges at a comfortable distance, corresponding depth constraints are specified in advance. Based on the depth constraints, a preset convergence plane is set. The goal of calculating the first convergence adjustment amount and the second convergence adjustment amount is to enable the imaging screens of the adjusted first display and second display to converge on the preset convergence plane. On the one hand, it helps to improve the viewing comfort of the user, and on the other hand, it can also improve the product consistency of the head-mounted display device.

[0146] Optionally, the calibration image can be a dot matrix image, so that the first convergence adjustment amount and the second convergence adjustment amount can be calculated by comprehensively considering the convergence errors at different positions of the display.

[0147] Optionally, the first camera and the second camera are binocular cameras, and the parameters of the binocular cameras are calibrated. During the calculation process, the calibrated external parameters are used to make the calculated convergence adjustment amount more accurate. In the related art, it is usually assumed that the two cameras that capture the two images corresponding to the left and right eyes are parallel to each other, and the error depends on the actual parallel degree of the two cameras.

[0148] By calculating the first convergence adjustment amount corresponding to the first display and the second convergence adjustment amount corresponding to the second display, and subsequently cooperating with the adjustment of the display or the adjustment of the display screen, the imaging screens of the displays of each head-mounted display device with consistent designs can be presented on the preset convergence plane, which helps to improve the product consistency and facilitates subsequent detection and evaluation of each head-mounted display device using a unified evaluation standard.

[0149] Each of the above embodiments or optional examples of the present disclosure can be implemented alone or in any combination without conflict, which can be specifically set according to actual needs, and the present disclosure does not make any limitations.

[0150] Any method for calculating the convergence adjustment amount provided by the embodiments of the present disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers, etc. Alternatively, any method for calculating the convergence adjustment amount provided by the embodiments of the present disclosure can be executed by a processor. For example, the processor executes any method for calculating the convergence adjustment amount mentioned in the embodiments of the present disclosure by calling the corresponding instructions stored in the memory. This will not be elaborated further below.

[0151] Exemplary device

[0152] Figure 9 It is a schematic structural diagram of a device for calculating the convergence adjustment amount provided by an exemplary embodiment of the present disclosure. The device for calculating the convergence adjustment amount in this embodiment can be used to implement the corresponding method embodiment for calculating the convergence adjustment amount of the present disclosure, such as Figure 9The device shown includes: a first acquisition module 410, a first processing module 420, a second acquisition module 430, and a second processing module 440.

[0153] The first acquisition module 410 is configured to acquire a first image captured by a first camera and a second image captured by a second camera. The first image is a calibration image displayed on a first display, and the second image is a calibration image displayed on a second display. The calibration image includes preset feature points.

[0154] The first processing module 420 is configured to determine a first pixel coordinate of the preset feature point in the coordinate system of the first image and a second pixel coordinate of the preset feature point in the coordinate system of the second image.

[0155] The second acquisition module 430 is configured to acquire a first screen coordinate of the preset feature point in the coordinate system of the first display and a second screen coordinate of the preset feature point in the coordinate system of the second display.

[0156] The second processing module 440 is configured to determine a first convergence adjustment amount corresponding to the first display and a second convergence adjustment amount corresponding to the second display according to the first pixel coordinate, the second pixel coordinate, the first screen coordinate, the second screen coordinate, and a preset convergence plane. The preset convergence plane is at a preset distance from between the first camera and the second camera.

[0157] In some alternative embodiments of the present disclosure, the second processing module 440 is specifically configured to: establish a first mapping relationship between the coordinate system of the first display and the coordinate system of the first camera, and a second mapping relationship between the coordinate system of the second display and the coordinate system of the second camera according to the first pixel coordinate, the second pixel coordinate, the first screen coordinate, the second screen coordinate, a first conversion relationship between the coordinate system of the first image and the coordinate system of the first camera, and a second conversion relationship between the coordinate system of the second image and the coordinate system of the second camera. Determine the first convergence adjustment amount and the second convergence adjustment amount according to the first screen coordinate, the second screen coordinate, the first mapping relationship, the second mapping relationship, a third mapping relationship between the coordinate system of the first camera and the coordinate system of the preset convergence plane, and a fourth mapping relationship between the coordinate system of the second camera and the coordinate system of the preset convergence plane.

[0158] In some alternative embodiments of the present disclosure, the second processing module 440 is specifically configured to: construct an objective function regarding a first convergence adjustment amount parameter and a second convergence adjustment amount parameter according to the first screen coordinate, the first convergence adjustment amount parameter, the first mapping relationship and the third mapping relationship, the second screen coordinate, the second convergence adjustment amount parameter, the second mapping relationship and the fourth mapping relationship. Solve for the first convergence adjustment amount parameter and the second convergence adjustment amount parameter that satisfy the preset conditions of the objective function, and obtain the first convergence adjustment amount and the second convergence adjustment amount.

[0159] In some alternative embodiments of the present disclosure, the second processing module 440 is specifically configured to: obtain a first convergence adjustment amount and a second convergence adjustment amount by solving the following objective function:

[0160]

[0161] where argmin J represents solving for (Δx k , Δy k ) that minimizes J. k = 1, 2, (Δx1, Δy1) represents the parameter of the first convergence adjustment amount to be solved. (Δx2, Δy2) represents the parameter of the second convergence adjustment amount to be solved. n represents the number of preset feature points. w i represents the weight of the i-th preset feature point. |||| represents taking the absolute value. p proj,1,i represents the projection coordinate of the expected convergence point of the i-th preset feature point on the preset convergence plane at the first convergence point of the first display. p proj,2,i represents the projection coordinate of the expected convergence point of the i-th preset feature point at the second convergence point of the second display. p′ screen,1,i (Δx1, Δy1) represents the screen coordinate of the i-th preset feature point adjusted based on the parameter of the first convergence adjustment amount. p′ screen,2,i (Δx2, Δy2) represents the screen coordinate of the i-th preset feature point adjusted based on the parameter of the second convergence adjustment amount.

[0162] In some alternative embodiments of the present disclosure, the second processing module 440 is specifically configured to: obtain a first convergence adjustment amount and a second convergence adjustment amount by solving the following objective function:

[0163]

[0164] where argmin J represents solving for (Δx l , Δy l ) and (Δx r , Δy r ) that minimizes J, (Δx l , Δy l ) represents the parameter of the first convergence adjustment amount. (Δx r , Δt r ) represents the parameter of the second convergence adjustment amount. n represents the number of preset feature points, w i represents the weight of the i-th preset feature point. |||| represents taking the absolute value. p plane,l,i (Δx l , Δy l ) represents the projection coordinate on the preset convergence plane of the screen coordinate of the i-th preset feature point adjusted based on the parameter of the first convergence adjustment amount. p plane,r,i (Δx r , Δy r) represents the projection coordinates of the screen coordinates of the i-th preset feature point adjusted based on the second convergence adjustment parameter on the preset convergence plane.

[0165] In some alternative embodiments of the present disclosure, the second processing module 440 is specifically configured to: determine a first current convergence adjustment amount of the first convergence adjustment parameter and a second current convergence adjustment amount of the second convergence adjustment parameter. The first convergence adjustment parameter is initially the first initial convergence adjustment amount. The second convergence adjustment parameter is initially the second initial convergence adjustment amount. Adjust the first screen coordinates according to the first current convergence adjustment amount to obtain the first adjusted screen coordinates of the preset feature point. Map the first adjusted screen coordinates to the coordinate system of the preset convergence plane according to the first mapping relationship and the third mapping relationship to obtain the first convergence coordinates of the preset feature point. Adjust the second screen coordinates according to the second current convergence adjustment amount to obtain the second adjusted screen coordinates of the preset feature point. Map the second adjusted screen coordinates to the coordinate system of the preset convergence plane according to the second mapping relationship and the fourth mapping relationship to obtain the second convergence coordinates of the preset feature point. Determine the current objective function value according to the first convergence coordinates and the second convergence coordinates of the preset feature point. Update the first current convergence adjustment amount and the second current convergence adjustment amount based on the current objective function value to obtain a first updated convergence adjustment amount and a second updated convergence adjustment amount. In response to satisfying the iteration end condition, use the first updated convergence adjustment amount as the first convergence adjustment amount and the second updated convergence adjustment amount as the second convergence adjustment amount. In response to not satisfying the iteration end condition, use the first updated convergence adjustment amount as the first current convergence adjustment amount and the second updated convergence adjustment amount as the second current convergence adjustment amount, and repeat the step of adjusting the first screen coordinates of the preset feature point according to the first current convergence adjustment amount to obtain the first adjusted screen coordinates of the preset feature point.

[0166] In some alternative embodiments of the present disclosure, the second processing module 440 is specifically configured to: determine the target convergence coordinates of the preset feature point in the coordinate system of the preset convergence plane according to the first screen coordinates, the second screen coordinates, the first mapping relationship, the second mapping relationship, the third mapping relationship, and the fourth mapping relationship. Map the target convergence coordinates to the first display to obtain the first convergence point projection coordinates of the preset feature point. Map the target convergence coordinates to the second display to obtain the second convergence point projection coordinates of the preset feature point. Construct a first objective function regarding the first convergence adjustment parameter according to the first convergence point projection coordinates, the first screen coordinates, the first convergence adjustment parameter, the first mapping relationship, and the third mapping relationship. Construct a second objective function regarding the second convergence adjustment parameter according to the second convergence point projection coordinates, the second screen coordinates, the second convergence adjustment parameter, the second mapping relationship, and the fourth mapping relationship.

[0167] In some alternative embodiments of the present disclosure, the second processing module 440 is specifically configured to: solve a first convergence adjustment parameter that satisfies a first preset condition for a first objective function to obtain a first convergence adjustment amount. Solve a second convergence adjustment parameter that satisfies a second preset condition for a second objective function to obtain a second convergence adjustment amount.

[0168] In some alternative embodiments of the present disclosure, the second processing module 440 is specifically configured to: map the first screen coordinates to the coordinate system of a preset convergence plane according to a first mapping relationship and a third mapping relationship to obtain first projection coordinates of a preset feature point. Map the second screen coordinates to the coordinate system of the preset convergence plane according to a second mapping relationship and a fourth mapping relationship to obtain second projection coordinates of the preset feature point. Determine target convergence coordinates of the preset feature point in the coordinate system of the preset convergence plane according to the first projection coordinates and the second projection coordinates.

[0169] Figure 10 FIG. is a schematic structural diagram of a device for calculating a convergence adjustment amount provided by another exemplary embodiment of the present disclosure.

[0170] In some alternative embodiments of the present disclosure, the device of the embodiments of the present disclosure further includes: a third processing module 510 and a fourth processing module 520.

[0171] The third processing module 510 is configured to establish a third mapping relationship between the coordinate system of the first camera and the coordinate system of the preset convergence plane according to the calibration parameters of the first camera and the spatial information of the preset convergence plane.

[0172] The fourth processing module 520 is configured to establish a fourth mapping relationship between the coordinate system of the second camera and the coordinate system of the preset convergence plane according to the calibration parameters of the second camera and the spatial information of the preset convergence plane.

[0173] In some alternative embodiments of the present disclosure, the first processing module 420 is specifically configured to: extract feature points from the first image to obtain first feature points in the first image. Extract feature points from the second image to obtain second feature points in the second image. Perform feature matching on the first feature points and the second feature points to obtain a matching relationship between the first feature points and the second feature points. Determine first pixel coordinates of the preset feature point in the first image and second pixel coordinates of the preset feature point in the second image according to the pixel coordinates of the first feature point in the first image, the pixel coordinates of the second feature point in the second image, and the matching relationship between the first feature points and the second feature points.

[0174] In some alternative embodiments of the present disclosure, the second processing module 440 is specifically configured to: determine some preset feature points of a preset region from the preset feature points as target feature points. Determine a first convergence adjustment amount and a second convergence adjustment amount according to the first pixel coordinates, the second pixel coordinates, the first screen coordinates, the second screen coordinates of the target feature points, and the preset convergence plane.

[0175] It should be noted that the specific implementation manner of the device for calculating the convergence adjustment amount in the embodiments of the present disclosure is similar to the specific implementation manner of the method for calculating the convergence adjustment amount in the embodiments of the present disclosure. For details, refer to the part of the method for calculating the convergence adjustment amount. To reduce redundancy, it will not be elaborated here.

[0176] Exemplary electronic device

[0177] The embodiments of the present disclosure further provide an electronic device, including: a processor and a memory for storing processor-executable instructions.

[0178] The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the method for calculating the convergence adjustment amount described in any of the above embodiments of the present disclosure.

[0179] Figure 11 FIG. 16 is a schematic structural diagram of an application embodiment of the electronic device of the present disclosure. In this embodiment, the electronic device 100 includes one or more processors 110 and a memory 120.

[0180] The processor 110 may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.

[0181] The memory 120 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage media, and the processor 110 may run the program instructions to implement the methods of the various embodiments of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage media.

[0182] In one example, the electronic device 100 may further include: an input device 130 and an output device 140, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0183] For example, the input device 130 may be the above-mentioned microphone or microphone array for capturing the input signal of the sound source.

[0184] In addition, the input device 130 may further include, for example, a keyboard, a mouse, and so on.

[0185] The output device 140 may output various information to the outside, including the determined distance information, direction information, etc. The output device 140 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, and so on.

[0186] Of course, for simplicity, Figure 11 only some of the components related to the present disclosure in the electronic device 100 are shown, and components such as a bus, an input / output interface, and so on are omitted. In addition, according to specific application scenarios, the electronic device 100 may further include any other appropriate components.

[0187] Exemplary computer program product and computer-readable storage medium

[0188] In addition to the above methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions. When the computer program instructions are run by a processor, the processor is caused to execute the steps in the methods according to various embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0189] The computer program product may be written in any combination of one or more programming languages to write program code for performing the operations of the embodiments of the present disclosure. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, executed as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0190] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are run by a processor, the processor is caused to execute the steps in the methods according to various embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0191] The computer-readable storage medium may adopt any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0192] It should also be noted that in the devices, equipment, and methods of the present disclosure, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent solutions of the present disclosure.

[0193] The foregoing description has been given for purposes of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although several example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for calculating a convergence adjustment amount, comprising: Obtaining a first image captured by a first camera and a second image captured by a second camera, wherein the first image is a calibration image displayed by a first display, the second image is the calibration image displayed by a second display, and the calibration image includes preset feature points; Determining a first pixel coordinate of the preset feature point in the coordinate system of the first image and a second pixel coordinate of the preset feature point in the coordinate system of the second image; Obtaining a first screen coordinate of the preset feature point in the coordinate system of the first display and a second screen coordinate of the preset feature point in the coordinate system of the second display; Determining a first convergence adjustment amount corresponding to the first display and a second convergence adjustment amount corresponding to the second display according to the first pixel coordinate, the second pixel coordinate, the first screen coordinate, the second screen coordinate, and a preset convergence plane; wherein the preset convergence plane is at a preset distance from between the first camera and the second camera.

2. The method according to claim 1, wherein, The determining the first convergence adjustment amount corresponding to the first display and the second convergence adjustment amount corresponding to the second display according to the first pixel coordinate, the second pixel coordinate, the first screen coordinate, the second screen coordinate, and the preset convergence plane includes: Establishing a first mapping relationship between the coordinate system of the first display and the coordinate system of the first camera, and a second mapping relationship between the coordinate system of the second display and the coordinate system of the second camera according to the first pixel coordinate, the second pixel coordinate, the first screen coordinate, the second screen coordinate, a first conversion relationship between the coordinate system of the first image and the coordinate system of the first camera, and a second conversion relationship between the coordinate system of the second image and the coordinate system of the second camera; Determining the first convergence adjustment amount and the second convergence adjustment amount according to the first screen coordinate, the second screen coordinate, the first mapping relationship, the second mapping relationship, a third mapping relationship between the coordinate system of the first camera and the coordinate system of the preset convergence plane, and a fourth mapping relationship between the coordinate system of the second camera and the coordinate system of the preset convergence plane.

3. The method according to claim 2, wherein, The determining the first convergence adjustment amount and the second convergence adjustment amount according to the first screen coordinate, the second screen coordinate, the first mapping relationship, the second mapping relationship, the third mapping relationship between the coordinate system of the first camera and the coordinate system of the preset convergence plane, and the fourth mapping relationship between the coordinate system of the second camera and the coordinate system of the preset convergence plane includes: Constructing an objective function for the first convergence adjustment amount parameter and the second convergence adjustment amount parameter according to the first screen coordinate, the first convergence adjustment amount parameter, the first mapping relationship and the third mapping relationship, the second screen coordinate, the second convergence adjustment amount parameter, the second mapping relationship and the fourth mapping relationship; Solve for the first convergence adjustment parameter and the second convergence adjustment parameter that satisfy the preset conditions for the target function, and obtain the first convergence adjustment and the second convergence adjustment.

4. The method according to claim 2, wherein, The first convergence adjustment and the second convergence adjustment are obtained by solving the following target function: Among them, argminJ represents solving for (Δx k , Δγ k ) that minimizes J, where k = 1, 2, (Δx1, Δy1) represents the first convergence adjustment parameter to be solved, and (Δx2, Δy2) represents the second convergence adjustment parameter to be solved; n represents the number of preset feature points, and w i represents the weight of the i-th preset feature point; || || represents taking the absolute value, and p proj，1，i represents the projection coordinates of the expected convergence point of the i-th preset feature point on the preset convergence plane at the first convergence point of the first display; p proj，2，i represents the projection coordinates of the expected convergence point of the i-th preset feature point at the second convergence point of the second display; p′ screen，1，i (Δx1, Δy1) represents the screen coordinates of the i-th preset feature point after being adjusted based on the first convergence adjustment parameter; p′ screen，2，i (Δx2, Δy2) represents the screen coordinates of the i-th preset feature point after being adjusted based on the second convergence adjustment parameter.

5. The method according to claim 2, wherein, The first convergence adjustment and the second convergence adjustment are obtained by solving the following target function: Among them, argminJ represents solving for (Δx l , Δy l ) and (Δx r , Δy r ) that minimize J. (Δx l , Δy l ) represents the first convergence adjustment parameter, and (Δx r , Δy r ) represents the second convergence adjustment parameter; n represents the number of preset feature points, w i represents the weight of the i-th preset feature point; || || represents taking the absolute value; p plane，l，i (Δx l , Δy l ) represents the projection coordinates of the screen coordinates of the i-th preset feature point adjusted based on the first convergence adjustment parameter on the preset convergence plane; p plane，r，i (Δx r , Δy r ) represents the projection coordinates of the screen coordinates of the i-th preset feature point adjusted based on the second convergence adjustment parameter on the preset convergence plane.

6. The method according to claim 3, wherein The step of solving for the first convergence adjustment parameter and the second convergence adjustment parameter that satisfy the preset conditions for the target function, and obtaining the first convergence adjustment and the second convergence adjustment includes: Determine the first current convergence adjustment of the first convergence adjustment parameter and the second current convergence adjustment of the second convergence adjustment parameter; the first convergence adjustment parameter is initially the first initial convergence adjustment; the second convergence adjustment parameter is initially the second initial convergence adjustment; According to the first current convergence adjustment, adjust the first screen coordinates to obtain the first adjusted screen coordinates of the preset feature point; according to the first mapping relationship and the third mapping relationship, map the first adjusted screen coordinates to the coordinate system of the preset convergence plane to obtain the first convergence coordinates of the preset feature point; According to the second current convergence adjustment, adjust the second screen coordinates to obtain the second adjusted screen coordinates of the preset feature point; according to the second mapping relationship and the fourth mapping relationship, map the second adjusted screen coordinates to the coordinate system of the preset convergence plane to obtain the second convergence coordinates of the preset feature point; Determine the current target function value according to the first convergence coordinates and the second convergence coordinates of the preset feature point; Based on the current target function value, update the first current convergence adjustment and the second current convergence adjustment to obtain a first updated convergence adjustment and a second updated convergence adjustment; In response to satisfying the iteration end condition, use the first updated convergence adjustment as the first convergence adjustment and the second updated convergence adjustment as the second convergence adjustment; In response to not satisfying the iteration end condition, use the first updated convergence adjustment as the first current convergence adjustment and the second updated convergence adjustment as the second current convergence adjustment, and repeat the step of adjusting the first screen coordinates of the preset feature point according to the first current convergence adjustment to obtain the first adjusted screen coordinates of the preset feature point.

7. The method according to claim 3, wherein The step of constructing a target function for the first convergence adjustment parameter and the second convergence adjustment parameter according to the first screen coordinates, the first convergence adjustment parameter, the first mapping relationship and the third mapping relationship, the second screen coordinates, the second convergence adjustment parameter, the second mapping relationship and the fourth mapping relationship includes: Determine the target convergence coordinates of the preset feature point in the coordinate system of the preset convergence plane according to the first screen coordinate, the second screen coordinate, the first mapping relationship, the second mapping relationship, the third mapping relationship, and the fourth mapping relationship; Map the target convergence coordinates to the first display to obtain the first convergence point projection coordinates of the preset feature point; Map the target convergence coordinates to the second display to obtain the second convergence point projection coordinates of the preset feature point; Construct a first objective function regarding the first convergence adjustment parameter according to the first convergence point projection coordinates, the first screen coordinate, the first convergence adjustment parameter, the first mapping relationship, and the third mapping relationship; Construct a second objective function regarding the second convergence adjustment parameter according to the second convergence point projection coordinates, the second screen coordinate, the second convergence adjustment parameter, the second mapping relationship, and the fourth mapping relationship.

8. The method according to claim 7, wherein The solving for the first convergence adjustment parameter and the second convergence adjustment parameter that satisfy the preset conditions for the objective function to obtain the first convergence adjustment amount and the second convergence adjustment amount includes: Solve for the first convergence adjustment parameter that satisfies the first preset condition for the first objective function to obtain the first convergence adjustment amount; Solve for the second convergence adjustment parameter that satisfies the second preset condition for the second objective function to obtain the second convergence adjustment amount.

9. The method according to claim 7, wherein, The determining the target convergence coordinates of the preset feature point in the coordinate system of the preset convergence plane according to the first screen coordinate, the second screen coordinate, the first mapping relationship, the second mapping relationship, the third mapping relationship, and the fourth mapping relationship includes: Map the first screen coordinate to the coordinate system of the preset convergence plane according to the first mapping relationship and the third mapping relationship to obtain the first projection coordinates of the preset feature point; Map the second screen coordinate to the coordinate system of the preset convergence plane according to the second mapping relationship and the fourth mapping relationship to obtain the second projection coordinates of the preset feature point; Determine the target convergence coordinates of the preset feature point in the coordinate system of the preset convergence plane according to the first projection coordinates and the second projection coordinates.

10. The method according to any one of claims 2-9, wherein, Further includes: Establish the third mapping relationship between the coordinate system of the first camera and the coordinate system of the preset convergence plane according to the calibration parameters of the first camera and the spatial information of the preset convergence plane; Establish the fourth mapping relationship between the coordinate system of the second camera and the coordinate system of the preset convergence plane according to the calibration parameters of the second camera and the spatial information of the preset convergence plane.

11. The method according to any one of claims 1-9, wherein The determining the first pixel coordinates of the preset feature point in the coordinate system of the first image and the second pixel coordinates of the preset feature point in the coordinate system of the second image includes: Extract feature points from the first image to obtain the first feature points in the first image; Extract feature points from the second image to obtain the second feature points in the second image; Perform feature matching on the first feature point and the second feature point to obtain the matching relationship between the first feature point and the second feature point; Determine the first pixel coordinates of the preset feature point in the first image and the second pixel coordinates of the preset feature point in the second image according to the pixel coordinates of the first feature point in the first image, the pixel coordinates of the second feature point in the second image, and the matching relationship between the first feature point and the second feature point.

12. The method according to claim 11, wherein, The determining the first convergence adjustment amount corresponding to the first display and the second convergence adjustment amount corresponding to the second display according to the first pixel coordinates, the second pixel coordinates, the first screen coordinates, the second screen coordinates, and a preset convergence plane includes: Determine partial preset feature points in a preset area from the preset feature points as target feature points; Determine the first convergence adjustment amount and the second convergence adjustment amount according to the first pixel coordinates, the second pixel coordinates, the first screen coordinates, the second screen coordinates, and the preset convergence plane of the target feature points.

13. A device for calculating a convergence adjustment amount, comprising: A first acquisition module, configured to acquire a first image captured by a first camera and a second image captured by a second camera, where the first image is a calibration image displayed by a first display, the second image is the calibration image displayed by a second display, and the calibration image includes preset feature points; A first processing module, configured to determine the first pixel coordinates of the preset feature point in the coordinate system of the first image and the second pixel coordinates of the preset feature point in the coordinate system of the second image; A second acquisition module, configured to acquire the first screen coordinates of the preset feature point in the coordinate system of the first display and the second screen coordinates of the preset feature point in the coordinate system of the second display; A second processing module, configured to determine the first convergence adjustment amount corresponding to the first display and the second convergence adjustment amount corresponding to the second display according to the first pixel coordinates, the second pixel coordinates, the first screen coordinates, the second screen coordinates, and a preset convergence plane; where the preset convergence plane is at a preset distance from between the first camera and the second camera.

14. A computer-readable storage medium, where the storage medium stores a computer program, and the computer program is used to execute the method for calculating a convergence adjustment amount according to any one of claims 1-12 above.

15. An electronic device, the electronic device includes: A processor; A memory for storing executable instructions of the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method for calculating a convergence adjustment amount according to any one of claims 1-12 above.

Images