A method, device and shooting system for adaptive adjustment of a 3D camera pose

Abstract

This application discloses a method, apparatus, and shooting system for adaptive adjustment of 3D camera pose. The method includes: acquiring the offset angle of each eye; calculating the distance between the eyes based on the offset angle; using the offset angle of each eye as the rotation angle of each camera lens, and calculating the target distance of each camera lens based on the distance between the eyes and a preset target distance formula; controlling each camera lens to rotate the offset angle in the offset direction of the eyes, and controlling each camera lens to move so that the distance of each camera lens from the center between the two camera lenses is the target distance. This embodiment achieves adaptive pose adjustment, improving the visual experience.

Description

Technical Field

[0001] This application relates to imaging technology, and more particularly to a method, apparatus and shooting system for adaptive adjustment of 3D camera pose. Background Technology

[0002] With the continuous development of imaging technology, people are no longer satisfied with 2D video applications for a better experience, but are constantly exploring 3D video applications. This is evident in applications such as 3D movies in the film industry, VR (virtual reality) games in the gaming industry, and holographic projection. Therefore, the traditional video surveillance field also needs to develop towards 3D video.

[0003] Currently, the encoding end in the video surveillance field commonly uses monocular cameras, while the decoding end typically uses large-screen displays, displaying 2D images. To support 3D video input, the encoding end must first use a binocular camera to capture the left and right images for the viewer's eyes. Then, the decoding end can use VR glasses to allow the left and right images to enter the viewer's left and right eyes respectively. However, the relative positions and postures of each person's left and right eyes differ, and current 3D cameras cannot achieve adaptive posture adjustment, thus failing to achieve a better visual experience. Summary of the Invention

[0004] This application provides a method, apparatus, and shooting system for adaptive adjustment of 3D camera pose, which can achieve adaptive adjustment of pose and improve the visual experience.

[0005] This application provides a method for adaptive adjustment of the pose of a 3D camera, wherein the 3D camera may include two camera lenses; the method may include:

[0006] Obtain the offset angle of both eyes;

[0007] Calculate the distance between the two eyes based on the offset angle;

[0008] The offset angle of the two eyes is used as the rotation angle of each camera lens, and the target distance of each camera lens is calculated based on the distance between the two eyes and the preset target distance formula.

[0009] Control each camera lens to rotate the offset angle in the direction of the eye offset, and control each camera lens to move so that the distance of each camera lens from the center between the two camera lenses is the target distance.

[0010] In an exemplary embodiment of this application, obtaining the offset angle of both eyes may include:

[0011] After the virtual reality (VR) glasses connected to the 3D camera are worn on a person's eyes and the angle is adapted to the person's eyes, the angle between the two lenses of the VR glasses is collected by the angle sensor set on the VR glasses.

[0012] The angle between the two lenses is taken as the angle between the two eyes; wherein the angle between the two eyes is twice the offset angle.

[0013] In an exemplary embodiment of this application, calculating the distance between the eyes based on the offset angle may include:

[0014] The system retrieves pre-set images from the two lenses of a VR headset connected to a 3D camera after the headset is worn and adapted to the human eye. The images include a first image and a second image. The first image and the second image are initially in the same plane.

[0015] The first acquisition screen and the second acquisition screen are controlled to rotate by the offset angle respectively based on the initial state; wherein, the included angle between the first acquisition screen and the second acquisition screen after rotation is twice the offset angle;

[0016] Obtain the first coordinate position of the first eyeball in the first acquisition screen, and obtain the second coordinate position of the second eyeball in the second acquisition screen;

[0017] Based on the angle between the first captured image and the second captured image, and the distance between the first coordinate position and the second coordinate position, the distance between the first eyeball and the second eyeball is calculated as the distance between the two eyes.

[0018] In an exemplary embodiment of this application, obtaining the first coordinate position of the first eyeball in the first acquisition frame and obtaining the second coordinate position of the second eyeball in the second acquisition frame may include:

[0019] In the first acquisition screen, a first coordinate system is established with the first center point of the first acquisition screen as the origin, the horizontal central axis of the first acquisition screen as the X-axis, the vertical central axis of the first acquisition screen as the Z-axis, and the perpendicular line of the first acquisition screen passing through the first center point as the Y-axis.

[0020] Determine the first coordinate position of the first eyeball in the first coordinate system;

[0021] In the second acquisition screen, a second coordinate system is established with the second center point of the second acquisition screen as the origin, the horizontal central axis of the second acquisition screen as the X-axis, the vertical central axis of the second acquisition screen as the Z-axis, and the perpendicular line of the second acquisition screen passing through the second center point as the Y-axis.

[0022] The second coordinate position of the second eyeball in the second coordinate system is determined.

[0023] In an exemplary embodiment of this application, before calculating the distance between the first eyeball and the second eyeball, the method may further include: pre-calculating the distance in the X-axis direction between the origin of the first coordinate system and the origin of the second coordinate system;

[0024] The pre-calculation of the distance between the origin of the first coordinate system and the origin of the second coordinate system in the X-axis direction includes:

[0025] Before the VR glasses are worn, rectangular images corresponding to the two lenses are captured respectively; wherein, the length of the rectangular image is the field of view width of the VR glasses, and the width of the rectangular image is the field of view height of the VR glasses;

[0026] The rectangular images corresponding to the two lenses are respectively used as the first calibration image and the second calibration image;

[0027] The distance between the center point of the first calibration screen and the center point of the second calibration screen is calculated as the distance between the origin of the first coordinate system and the origin of the second coordinate system in the X-axis direction.

[0028] In an exemplary embodiment of this application, calculating the distance between the first eyeball and the second eyeball based on the angle between the first captured image and the second captured image and the distance between the first coordinate position and the second coordinate position may include:

[0029] The relationship between the first coordinate system and the second coordinate system is obtained based on the included angle and the distance between the origin of the first coordinate system and the origin of the second coordinate system in the X-axis direction.

[0030] Based on the relationship between the first coordinate system and the second coordinate system, the second coordinate position is transformed into the first coordinate system to obtain the position of the second coordinate position in the first coordinate system, which is recorded as the third coordinate position;

[0031] The distance between the first eyeball and the second eyeball is calculated based on the first coordinate position and the third coordinate position.

[0032] In an exemplary embodiment of this application, obtaining the relationship between the first coordinate system and the second coordinate system based on the included angle and the distance between the origin of the first coordinate system and the origin of the second coordinate system in the X-axis direction may include:

[0033] Establish a relational equation between the first coordinate system and the second coordinate system, including one or more unknown parameters;

[0034] Obtain multiple known coordinates in the second coordinate system;

[0035] The coordinates of the plurality of known coordinates in the first coordinate system are calculated based on the included angle and the distance between the origin of the first coordinate system and the origin of the second coordinate system in the X-axis direction.

[0036] Substitute the plurality of known coordinates and their corresponding coordinates in the first coordinate system into the relational equation to solve for one or more parameters.

[0037] Substitute the solved parameters into the relational equation to obtain the relational equations for the first coordinate system and the second coordinate system, which contain the known parameters.

[0038] In an exemplary embodiment of this application, the target distance calculation formula may include:

[0039]

[0040] Wherein, d2 is the target distance, d1 is the distance between the two eyes, and a is the offset angle.

[0041] This application embodiment also provides a 3D camera pose adaptive adjustment device, which may include: a processor and a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and when the instructions are executed by the processor, the method for adaptive adjustment of 3D camera pose is implemented.

[0042] This application also provides a shooting system, which may include: the 3D camera pose adaptive adjustment device.

[0043] Compared with related technologies, the embodiments of this application may include: obtaining the offset angle of both eyes; calculating the distance between the two eyes based on the offset angle; using the offset angle of both eyes as the rotation angle of each camera lens, and calculating the target distance of each camera lens based on the distance between the two eyes and a preset target distance formula; controlling each camera lens to rotate the offset angle in the offset direction of the eyes, and controlling each camera lens to move so that the distance of each camera lens from the center between the two camera lenses is the target distance. Through this embodiment, adaptive pose adjustment is achieved, improving the visual experience.

[0044] Other features and advantages of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the application. Other advantages of this application can be realized and obtained by means of the solutions described in the description and the accompanying drawings. Attached Figure Description

[0045] The accompanying drawings are used to provide an understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0046] Figure 1 This is a flowchart of a 3D camera pose adaptive adjustment method according to an embodiment of this application;

[0047] Figure 2 This is a schematic diagram of the eye offset in an embodiment of this application;

[0048] Figure 3 This is a schematic diagram illustrating the image deviation caused by the eye shift in an embodiment of this application;

[0049] Figure 4 This is a schematic diagram of the offset angle in an embodiment of this application;

[0050] Figure 5 This is a flowchart of a method for obtaining the offset angle of both eyes according to an embodiment of this application;

[0051] Figure 6 This is a flowchart illustrating a method for calculating the distance between the eyes based on an offset angle, according to an embodiment of this application.

[0052] Figure 7 This is a schematic diagram of the calibration screen in an embodiment of this application;

[0053] Figure 8 This is a schematic diagram showing the position of the human eyeball in the captured image according to an embodiment of this application;

[0054] Figure 9 This is a three-dimensional schematic diagram of the first and second acquisition screens according to an embodiment of this application;

[0055] Figure 10 This is a flowchart illustrating a method for calculating the distance between the first eyeball and the second eyeball according to an embodiment of this application.

[0056] Figure 11 This is a schematic diagram of the pose adjustment method according to an embodiment of this application;

[0057] Figure 12 This is a schematic diagram of a 3D camera lens after pose adjustment according to an embodiment of this application;

[0058] Figure 13 This is a block diagram of the 3D camera pose adaptive adjustment device according to an embodiment of this application;

[0059] Figure 14 This is a block diagram of the imaging system according to an embodiment of this application. Detailed Implementation

[0060] This application describes several embodiments, but these descriptions are exemplary and not restrictive, and it will be apparent to those skilled in the art that many more embodiments and implementations are possible within the scope of the embodiments described herein. Although many possible combinations of features are shown in the drawings and discussed in the detailed description, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with, or may replace, any feature or element of any other embodiment.

[0061] This application includes and contemplates combinations of features and elements known to those skilled in the art. The embodiments, features, and elements disclosed in this application may also be combined with any conventional features or elements to form a unique inventive scheme as defined by the claims. Any feature or element of any embodiment may also be combined with features or elements from other inventive schemes to form another unique inventive scheme as defined by the claims. Therefore, it should be understood that any feature shown and / or discussed in this application may be implemented individually or in any suitable combination. Therefore, the embodiments are not limited except by the limitations imposed by the appended claims and their equivalents. Furthermore, various modifications and changes may be made within the scope of the appended claims.

[0062] Furthermore, in describing representative embodiments, the specification may have presented methods and / or processes as a specific sequence of steps. However, the method or process should not be limited to the specific order of steps described herein, to the extent that it does not depend on such a specific order. As will be understood by those skilled in the art, other sequences of steps are also possible. Therefore, the specific order of steps set forth in the specification should not be construed as a limitation of the claims. Moreover, the claims concerning the method and / or process should not be limited to the steps performed in the written order, and those skilled in the art will readily understand that these orders can be varied and still remain within the spirit and scope of the embodiments of this application.

[0063] This application provides a method for adaptive pose adjustment of a 3D camera, wherein the 3D camera may include: two camera lenses; as shown in the example. Figure 1 As shown, the method may include steps S101-S104:

[0064] S101, Obtain the offset angle of both eyes;

[0065] S102. Calculate the distance between the two eyes based on the offset angle;

[0066] S103. The offset angle of the two eyes is used as the rotation angle of each camera lens, and the target distance of each camera lens is calculated according to the distance between the two eyes and the preset target distance calculation formula.

[0067] S104. Control each camera lens to rotate the offset angle in the direction of the offset of the eyes, and control each camera lens to move so that the distance of each camera lens from the center between the two camera lenses is the target distance.

[0068] In the exemplary embodiments of this application, in current 3D camera control schemes, the relative distance and orientation between most binocular camera lenses are fixed. While there are methods to adjust the offset distance of the input image to the left and right eyes based on the distance between the two eyes, these methods do not consider that in addition to distance offset, there is also angular offset between the two eyes. Figure 2 As shown, since the human face is not a standard flat plane, but has a curvature, the two eyes of a person will also have a certain curvature along with the curvature of the face. That is, there will be a certain angle between the human eye and the ideal face plane (i.e., the offset angle mentioned above). This angle varies from person to person, and is generally between 4° and 6° (for example, usually around 5°). If only the distance between the two eyes is considered when controlling the 3D camera, without considering the offset angle between the two eyes, there will be a deviation of about 2.7% in the line of sight.

[0069] In exemplary embodiments of this application, as Figure 3 As shown, the field of view of a typical 3D camera is approximately 45°. Assuming the object distance is d3 and the field of view width is w1, a 5° eye angle shift causes a gap between the center points of the image S1 before the eye angle shift and the image S2 after the eye angle shift, as shown... Figure 3 The distance d4 shown; the distance offset error can be calculated to be approximately 10.6% using the following formula:

[0070]

[0071] In the exemplary embodiments of this application, as can be seen from the above analysis, calculating the relative offset of the two images (S1 and S2) based solely on the distance between the two eyes will result in an angular deviation of approximately 2.7% and a distance deviation of approximately 10.6%.

[0072] In an exemplary embodiment of this application, the present application proposes a method for adaptive adjustment of 3D camera pose. This method can adjust the relative pose between the binocular lenses of a 3D camera based on the relative pose between a person's eyes, making the binocular lenses (i.e., the two camera lenses) of the 3D camera more in line with the characteristics of the video viewer's eyes. This reduces the deviation mentioned above when calculating the relative offset of the two images (S1 and S2) based solely on the distance between the two eyes, thereby improving the viewer's visual experience.

[0073] In an exemplary embodiment of this application, the distance between the left and right eyes, and the tilt angle of the eyes on both sides of the face (i.e., the aforementioned offset angle), cause the images captured by the left and right eyes to form a certain angle, such as... Figure 4 The image shown is a top-down view, where the two rectangles represent the two eyes, A represents the standard facial plane, and a is the offset angle. Based on these two points, the relative pose of the human eyes can be identified by the following two parameters: the distance d1 between the two eyes and the offset angle a of the human eyes.

[0074] In an exemplary embodiment of this application, adjusting the relative pose of the 3D camera based on the relative pose of the eyes first requires capturing the relative pose of the human eyes. Playing 3D video resources, similar to VR glasses, involves displaying the left and right images captured by the 3D camera on the left and right sides of the VR glasses to provide a 3D visual effect. When the 3D camera's live video stream (i.e., real-time playback) is displayed on a VR headset, the platform can control the live stream to prevent others from intercepting it, allowing only the user of that VR headset to control it. Through this VR headset, partial pose information of the wearer's eyes can be captured, such as the eye offset angle.

[0075] In exemplary embodiments of this application, as Figure 5As shown, obtaining the offset angle of both eyes may include steps S201-S202:

[0076] S201. After the virtual reality (VR) glasses connected to the 3D camera are worn on the human eye and the angle is adapted according to the human eye, the angle between the two lenses of the VR glasses is collected according to the angle sensor set on the VR glasses.

[0077] S202, The angle between the two lenses is taken as the angle between the eyes; wherein the angle between the eyes is twice the offset angle.

[0078] In an exemplary embodiment of this application, an angle sensor can be added between the left and right lenses (i.e., the two lenses mentioned above) of the VR glasses. When the VR glasses are not worn, the angle sensor detects an angle of 0 between the two lenses. When a user wears and fastens the VR glasses, the left and right lenses adapt to the current user's eye, adjusting their positions. At this time, the two lenses shift with the current user's eye, and the two lenses form an angle of β between them. Figure 4 Therefore, β = 2a.

[0079] In an exemplary embodiment of this application, orientation sensors can be added to the left and right lenses of the VR glasses. When the VR glasses are not worn, the orientation sensors detect no offset direction of the two lenses (in the initial state, the two lenses can be considered to be basically in the same plane). When the user wears and fastens the VR glasses, the left and right lenses will adapt to the current user's eyes and adjust the position between the two lenses. At this time, the two lenses will offset with the current user's eyes, and the two lenses will change from the original state of being in the same plane to offset with the offset direction of the eyes. At this time, the orientation sensors can record the offset direction as the offset direction of the eyes, so that when adjusting the pose of the camera lens later, it can be rotated in the offset direction of the eyes by a certain offset angle.

[0080] In exemplary embodiments of this application, as Figure 6 As shown, calculating the distance between the two eyes based on the offset angle may include steps S301-S304:

[0081] S301. Retrieve the pre-set capture images corresponding to the two lenses of the VR glasses connected to the 3D camera; the capture images include a first capture image and a second capture image; the first capture image and the second capture image are initially in the same plane.

[0082] In an exemplary embodiment of this application, miniature cameras can be configured inside the left and right lenses of the VR glasses to collect human eye image information. Based on the human eye image information, the relative position of the human eyeball and the left and right lenses can be obtained. S302, control the first acquisition screen and the second acquisition screen to rotate by the offset angle respectively based on the initial state; wherein, the included angle between the first acquisition screen and the second acquisition screen after rotation is twice the offset angle.

[0083] In an exemplary embodiment of this application, in order to make the first and second acquisition screens match the scene at that time during calculation, the first and second acquisition screens can be rotated according to the offset angle, so that there is an included angle β between the first and second acquisition screens, where β = 2a.

[0084] S303. Obtain the first coordinate position of the first eyeball in the first acquisition screen, and obtain the second coordinate position of the second eyeball in the second acquisition screen.

[0085] In an exemplary embodiment of this application, as is known from the foregoing, miniature cameras can be respectively configured inside the left and right lenses of the VR glasses to collect human eye image information. Therefore, based on the collected human eye image information, the position of the first eyeball in the first captured image and the position of the second eyeball in the second captured image can be determined, such as... Figure 8 As shown, M represents the first eyeball, N represents the second eyeball, O represents the center point of the first captured image, and O' represents the center point of the second captured image.

[0086] In the exemplary embodiment of this application, considering that the first acquisition screen and the second acquisition screen are not in the same plane at this time, and there is an included angle β between the first acquisition screen and the second acquisition screen, the three-dimensional schematic diagram of the first acquisition screen and the second acquisition screen is as follows. Figure 9 As shown.

[0087] In an exemplary embodiment of this application, obtaining the first coordinate position of the first eyeball in the first acquisition frame and obtaining the second coordinate position of the second eyeball in the second acquisition frame may include:

[0088] In the first acquisition screen, a first coordinate system is established with the first center point of the first acquisition screen as the origin, the horizontal central axis of the first acquisition screen as the X-axis, the vertical central axis of the first acquisition screen as the Z-axis, and the perpendicular line of the first acquisition screen passing through the first center point as the Y-axis.

[0089] Determine the first coordinate position of the first eyeball in the first coordinate system;

[0090] In the second acquisition screen, a second coordinate system is established with the second center point of the second acquisition screen as the origin, the horizontal central axis of the second acquisition screen as the X-axis, the vertical central axis of the second acquisition screen as the Z-axis, and the perpendicular line of the second acquisition screen passing through the second center point as the Y-axis.

[0091] The second coordinate position of the second eyeball in the second coordinate system is determined.

[0092] In exemplary embodiments of this application, as Figure 9 As shown, a first coordinate system (xyz) can be established with the center point O of the left image (i.e., the first captured image) as the origin, the horizontal central axis of the left image as the x-axis, the vertical central axis of the left image as the z-axis, and the direction perpendicular to the image from front to back on the left image as the y-axis. Assuming the center point of the right image (i.e., the second captured image) is O', a second coordinate system (x'y'z') can be established with the center point O' of the right image (i.e., the second captured image) as the origin, the horizontal central axis of the right image as the x-axis, the vertical central axis of the right image as the z-axis, and the direction perpendicular to the image from front to back on the right image as the y-axis. Here, the center point of the first eyeball in the left image is point M, and the center point of the second eyeball in the right image is point N.

[0093] S304. Based on the angle between the first captured image and the second captured image and the distance between the first coordinate position and the second coordinate position, calculate the distance between the first eyeball and the second eyeball, and use it as the distance between the two eyes.

[0094] In exemplary embodiments of this application, as Figure 10 As shown, calculating the distance between the first eyeball and the second eyeball based on the angle between the first and second captured images and the distance between the first and second coordinate positions may include steps S401-S403:

[0095] S401. Obtain the relationship between the first coordinate system and the second coordinate system based on the included angle and the distance between the origin of the first coordinate system and the origin of the second coordinate system in the X-axis direction.

[0096] In an exemplary embodiment of this application, obtaining the relationship between the first coordinate system and the second coordinate system based on the included angle and the distance between the origin of the first coordinate system and the origin of the second coordinate system in the X-axis direction may include:

[0097] Establish a relational equation between the first coordinate system and the second coordinate system, including one or more unknown parameters;

[0098] Obtain multiple known coordinates in the second coordinate system;

[0099] The coordinates of the plurality of known coordinates in the first coordinate system are calculated based on the included angle and the distance between the origin of the first coordinate system and the origin of the second coordinate system in the X-axis direction.

[0100] Substitute the known coordinates and their corresponding coordinates into the relational equation to solve for one or more parameters;

[0101] Substitute the solved parameters into the relational equation to obtain the relational equations for the first coordinate system and the second coordinate system, which contain the known parameters.

[0102] In an exemplary embodiment of this application, before calculating the distance between the first eyeball and the second eyeball, the method may further include: pre-calculating the distance in the X-axis direction between the origin of the first coordinate system and the origin of the second coordinate system;

[0103] The pre-calculation of the distance between the origin of the first coordinate system and the origin of the second coordinate system in the X-axis direction may include:

[0104] Before the VR glasses are worn, rectangular images corresponding to the two lenses are captured respectively; wherein, the length of the rectangular image is the field of view width of the VR glasses, and the width of the rectangular image is the field of view height of the VR glasses;

[0105] The rectangular images corresponding to the two lenses are respectively used as the first calibration image and the second calibration image;

[0106] The distance between the center point of the first calibration screen and the center point of the second calibration screen is calculated as the distance between the origin of the first coordinate system and the origin of the second coordinate system in the X-axis direction.

[0107] In an exemplary embodiment of this application, the left and right lenses of the VR glasses can be pre-calibrated before use, such as... Figure 7 The image shown is the image captured by the two internal miniature cameras on the left and right sides of the VR glasses during calibration, which is used as the calibration image. Among them, S3 is the first calibration image, which can be the image captured by the miniature camera in the left lens of the VR glasses, and S4 is the second calibration image, which can be the image captured by the miniature camera in the right lens of the VR glasses.

[0108] In an exemplary embodiment of this application, during calibration, the first calibration screen and the second calibration screen can be kept in the same plane, and the distance L between the center point of the first calibration screen and the center point of the second calibration screen can be obtained, that is, the distance between the origin of the first coordinate system and the origin of the second coordinate system in the X-axis direction.

[0109] In an exemplary embodiment of this application, the physical length w and physical width h represented by a single pixel in a single frame of the first calibration frame and the second calibration frame can also be obtained; wherein, it is assumed that the two parameters, physical length w and physical width h represented by a single pixel in the two frames, are the same, and the actual physical length can be calculated in subsequent calculations based on the calculated pixel length (e.g., how many pixels a certain length is).

[0110] In an exemplary embodiment of this application, a detailed scheme for obtaining the relationship between the first coordinate system and the second coordinate system is described below through an embodiment.

[0111] In an exemplary embodiment of this application, it is assumed that the following relationship exists between the first coordinate system xyz and the second coordinate system x'y'z':

[0112]

[0113] In an exemplary embodiment of this application, when the VR glasses are not worn, the origins of the two coordinate systems differ by a distance L along the x-axis (i.e., the distance between the center points of the two initially calibrated images). When the VR glasses are worn, the coordinates of point O' in the xyz coordinate system are... so:

[0114]

[0115] In an exemplary embodiment of this application, coordinate points (1,0,0), (0,1,0), and (0,0,1) are obtained in the second coordinate system x'y'z'. Then, the angle β between the first acquisition screen and the second acquisition screen, and the coordinates of the origin of the second coordinate system x'y'z' in the first coordinate system xyz, can be used to calculate the coordinates of coordinate points (1,0,0), (0,1,0), and (0,0,1) in the first coordinate system xyz: (cosβ,sinβ,0), (-cosβ,sinβ,0), and (0,0,1), respectively.

[0116] In an exemplary embodiment of this application, the coordinates (1,0,0), (0,1,0), (0,0,1) obtained in the second coordinate system x'y'z' and the corresponding coordinates (cosβ,sinβ,0), (-cosβ,sinβ,0), (0,0,1) in the first coordinate system xyz are respectively substituted into Then, the transformation relationship between the first coordinate system and the second coordinate system can be obtained as follows:

[0117]

[0118] S402. Based on the relationship between the first coordinate system and the second coordinate system, transform the second coordinate position to the first coordinate system, obtain the position of the second coordinate position in the first coordinate system, and record it as the third coordinate position.

[0119] In an exemplary embodiment of this application, it is assumed that the pixel coordinates of the first eyeball (point M) in the first coordinate system xyz are (x m ,0,z m The pixel coordinates of the second eyeball (point N) in the second coordinate system x'y'z' are (x n ',0,z n Based on the calibrated parameters, the position coordinates of point M in the xyz coordinate system are obtained as (wx). m ,0,hz m The position coordinates of point N in the x'y'z' coordinate system are (wx) n ',0,hz n '). The position coordinates (wx) of point N in the x'y'z' coordinate system. n ',0,hz n Substitute The coordinates in the first coordinate system (xyz) can be obtained, which is the third coordinate position.

[0120] S403. Calculate the distance between the first eyeball and the second eyeball based on the first coordinate position and the third coordinate position.

[0121] In an exemplary embodiment of this application, the distance d1 between the first eyeball and the second eyeball is equal to the first coordinate position (x). m y m , z m ) and the third coordinate position (x) n y n , z n The distance between them, that is,

[0122] In an exemplary embodiment of this application, the distance d1 between the eyes is thus obtained.

[0123] In an exemplary embodiment of this application, the acquisition of the offset direction and offset angle of the two eyes, as well as the calculation of the distance between the two eyes based on the offset angle, can all be completed in the VR glasses. The VR glasses then send the acquired offset direction, offset angle, and distance between the two eyes to the 3D camera, which adjusts the pose of the camera lens based on these data.

[0124] In an exemplary embodiment of this application, before the 3D camera adjusts the pose of the camera lenses based on this data, a target distance for each camera lens can be calculated based on the distance between the two eyes and a preset target distance formula. This target distance is the distance of each camera lens from the center between the two camera lenses.

[0125] In an exemplary embodiment of this application, the target distance calculation formula may include:

[0126]

[0127] Wherein, d2 is the target distance, d1 is the distance between the two eyes, and a is the offset angle.

[0128] In an exemplary embodiment of this application, the 3D camera may include: a first camera lens and a second camera lens; a central pivot may be provided between the first camera lens and the second camera lens; a first connecting rod and a second connecting rod may be connected to the central pivot; the first connecting rod and the second connecting rod are rotatable around the central pivot; the first camera lens is disposed on the first connecting rod and is movable on the first connecting rod, and the second camera lens is disposed on the second connecting rod and is movable on the first connecting rod. When adjusting the pose of the first camera lens and the second camera lens, the offset angle can be adjusted by rotating the first connecting rod and the second connecting rod (rotating by angle a respectively). By moving the first camera lens on the first connecting rod so that the distance between the first camera lens and the central pivot is d2, and by moving the second camera lens on the second connecting rod so that the distance between the second camera lens and the central pivot is d2, the target distance can be adjusted by following the distance between the eyes, such as... Figure 11 As shown.

[0129] In an exemplary embodiment of this application, after the operations described above, the pose of the front-end 3D camera is adjusted according to the facial features of the VR glasses user, such as... Figure 12 As shown, this can provide 3D video viewers with a better visual experience.

[0130] This application embodiment also provides a 3D camera pose adaptive adjustment device 1, such as... Figure 13 As shown, it may include: a processor 11 and a computer-readable storage medium 12, wherein the computer-readable storage medium 12 stores instructions that, when executed by the processor 11, implement the method for adaptive adjustment of 3D camera pose.

[0131] In the exemplary embodiments of this application, any of the embodiments in the foregoing method embodiments are applicable to the device embodiments, and will not be described in detail here.

[0132] This application also provides a shooting system 2, such as... Figure 14 As shown, it may include: the 3D camera pose adaptive adjustment device 1.

[0133] In the exemplary embodiments of this application, any of the embodiments in the foregoing method embodiments are applicable to the imaging system embodiments, and will not be described in detail here.

[0134] In an exemplary embodiment of this application, the shooting system 2 may include a 3D camera and virtual reality (VR) glasses connected to the 3D camera. The 3D camera pose adaptive adjustment device 1 may be partially disposed on the 3D camera and partially disposed on the VR glasses, so that the 3D camera and the VR glasses can cooperate to adjust the pose.

[0135] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

Claims

1. A method for 3D camera pose adaptive adjustment, characterized in that, The 3D camera includes two camera lenses; the method includes: Obtain the offset angle between the two eyes; the offset angle is half the angle between the two lenses when the user is wearing VR glasses; Calculate the distance between the two eyes based on the offset angle; The offset angle of the two eyes is used as the rotation angle of each camera lens, and the target distance of each camera lens is calculated based on the distance between the two eyes and the preset target distance formula; the target distance is the distance of each camera lens from the central axis of rotation between the two camera lenses; Control each camera lens to rotate by the offset angle in the direction of the eye offset, and control each camera lens to move so that the distance of each camera lens from the central axis of rotation between the two camera lenses is the target distance; The step of calculating the distance between the two eyes based on the offset angle includes: The system retrieves pre-set images from the two lenses of a VR headset connected to a 3D camera after the headset is worn and adapted to the human eye. The images include a first image and a second image. The first image and the second image are initially in the same plane. The first acquisition screen and the second acquisition screen are controlled to rotate by the offset angle respectively based on the initial state; wherein, the included angle between the first acquisition screen and the second acquisition screen after rotation is twice the offset angle; Obtain the first coordinate position of the first eyeball in the first acquisition screen, and obtain the second coordinate position of the second eyeball in the second acquisition screen; Based on the angle between the first captured image and the second captured image, and the distance between the first coordinate position and the second coordinate position, the distance between the first eyeball and the second eyeball is calculated as the distance between the two eyes.

2. The method for 3D camera pose self-adaptive adjustment according to claim 1, characterized in that, The process of obtaining the offset angle of both eyes includes: After the virtual reality (VR) glasses connected to the 3D camera are worn on a person's eyes and the angle is adapted to the person's eyes, the angle between the two lenses of the VR glasses is collected by the angle sensor set on the VR glasses. The angle between the two lenses is taken as the angle between the two eyes; wherein the angle between the two eyes is twice the offset angle.

3. The method for 3D camera pose self-adaptive adjustment according to claim 1, characterized in that, The step of obtaining the first coordinate position of the first eyeball in the first acquisition frame and obtaining the second coordinate position of the second eyeball in the second acquisition frame includes: In the first acquisition screen, a first coordinate system is established with the first center point of the first acquisition screen as the origin, the horizontal central axis of the first acquisition screen as the X-axis, the vertical central axis of the first acquisition screen as the Z-axis, and the perpendicular line of the first acquisition screen passing through the first center point as the Y-axis. Determine the first coordinate position of the first eyeball in the first coordinate system; In the second acquisition screen, a second coordinate system is established with the second center point of the second acquisition screen as the origin, the horizontal central axis of the second acquisition screen as the X-axis, the vertical central axis of the second acquisition screen as the Z-axis, and the perpendicular line of the second acquisition screen passing through the second center point as the Y-axis. The second coordinate position of the second eyeball in the second coordinate system is determined.

4. The method for 3D camera pose self-adaptive adjustment according to claim 3, characterized in that, Before calculating the distance between the first eyeball and the second eyeball, the method further includes: pre-calculating the distance in the X-axis direction between the origin of the first coordinate system and the origin of the second coordinate system; The pre-calculation of the distance between the origin of the first coordinate system and the origin of the second coordinate system in the X-axis direction includes: Before the VR glasses are worn, rectangular images corresponding to the two lenses are captured respectively; wherein, the length of the rectangular image is the field of view width of the VR glasses, and the width of the rectangular image is the field of view height of the VR glasses; The rectangular images corresponding to the two lenses are respectively used as the first calibration image and the second calibration image; The distance between the center point of the first calibration screen and the center point of the second calibration screen is calculated as the distance between the origin of the first coordinate system and the origin of the second coordinate system in the X-axis direction.

5. The method for 3D camera pose self-adaptive adjustment according to claim 4, characterized in that, The step of calculating the distance between the first eyeball and the second eyeball based on the angle between the first captured image and the second captured image and the distance between the first coordinate position and the second coordinate position includes: The relationship between the first coordinate system and the second coordinate system is obtained based on the included angle and the distance between the origin of the first coordinate system and the origin of the second coordinate system in the X-axis direction. Based on the relationship between the first coordinate system and the second coordinate system, the second coordinate position is transformed into the first coordinate system to obtain the position of the second coordinate position in the first coordinate system, which is recorded as the third coordinate position; The distance between the first eyeball and the second eyeball is calculated based on the first coordinate position and the third coordinate position.

6. The method for 3D camera pose self-adaptive adjustment according to claim 5, characterized in that, The step of obtaining the relationship between the first coordinate system and the second coordinate system based on the included angle and the distance between the origin of the first coordinate system and the origin of the second coordinate system in the X-axis direction includes: Establish a relational equation between the first coordinate system and the second coordinate system, including one or more unknown parameters; Obtain multiple known coordinates in the second coordinate system; The coordinates of the plurality of known coordinates in the first coordinate system are calculated based on the included angle and the distance between the origin of the first coordinate system and the origin of the second coordinate system in the X-axis direction. Substitute the plurality of known coordinates and their corresponding coordinates in the first coordinate system into the relational equation to solve for one or more parameters; Substitute the solved parameters into the relational equation to obtain the relational equations for the first coordinate system and the second coordinate system, which contain the known parameters.

7. The method for 3D camera pose self-adaptive adjustment according to claim 1, characterized in that, The target distance calculation formula includes: ; Wherein, d2 is the target distance, d1 is the distance between the two eyes, and a is the offset angle.

8. A 3D camera pose adaptive adjustment apparatus, characterized in that, include: A processor and a computer-readable storage medium storing instructions that, when executed by the processor, implement the method for adaptive adjustment of 3D camera pose as described in any one of claims 1-7.

9. A photographing system characterized by comprising: include: The 3D camera pose adaptive adjustment device as described in claim 8.

Images