Calibration parameter generation method of virtual camera and electronic equipment

Abstract

The invention relates to the technical field of camera calibration, and discloses a calibration parameter generation method of a virtual camera and electronic equipment. The calibration parameter generation method comprises the steps that a cut image cut by a virtual camera in a shot image and a size parameter of the cut image are acquired, the shot image is shot by a preset real camera, the cut image comprises a target image area corresponding to a target object, a target point cloud of the target image area under a preset world coordinate system is acquired, and the size parameter of the cut image is obtained; and determining a virtual external parameter of the virtual camera according to a real external parameter of the real camera and the target point cloud, determining a virtual internal parameter of the virtual camera according to the size parameter, the virtual external parameter and the target point cloud, and forming calibration parameters by the virtual internal parameter and the virtual external parameter. According to the embodiment of the invention, the virtual camera is constructed for the cut image, and the camera parameters of the cut image are solved by calibrating the virtual camera, so that the missing camera parameters of the cut image can be made up, and the cut image can be used as input image data to be applied to deep learning.

Description

Technical Field

[0001] The present invention relates to the technical field of camera calibration, and in particular to a method for generating calibration parameters of a virtual camera and an electronic device. Background Art

[0002] The original image taken by the camera contains camera parameters, namely intrinsic parameters and extrinsic parameters. Intrinsic parameters are parameters related to the camera's own characteristics, such as the camera's focal length, pixel size, etc., and extrinsic parameters are parameters in the world coordinate system, such as the camera's position, rotation direction, etc.

[0003] The original image with camera parameters can be used as image input data for neural networks in deep learning applications to test, train, and infer models built from "image-camera parameters". However, currently, the cropped image cut from the original image lacks camera parameters, which limits the application of cropped images in deep learning. Summary of the invention

[0004] An object of an embodiment of the present invention is to provide a method for generating calibration parameters of a virtual camera and an electronic device, which can solve the technical problem in the prior art that a cropped image cropped from an original image lacks camera parameters.

[0005] In a first aspect, an embodiment of the present invention provides a method for generating calibration parameters of a virtual camera, comprising:

[0006] Acquire a cropped image cropped by a virtual camera from a captured image and size parameters of the cropped image, wherein the captured image is captured by a preset real camera, and the cropped image includes a target image area corresponding to a target object;

[0007] Acquire a target point cloud of the target image area in a preset world coordinate system;

[0008] Determining virtual extrinsic parameters of the virtual camera according to real extrinsic parameters of the real camera and the target point cloud;

[0009] The virtual internal parameters of the virtual camera are determined according to the size parameters, the virtual external parameters and the target point cloud, and the virtual internal parameters and the virtual external parameters constitute calibration parameters.

[0010] Optionally, determining the virtual extrinsic parameters of the virtual camera according to the real extrinsic parameters of the real camera and the target point cloud comprises:

[0011] Determine a first position of the target object according to the target point cloud, where the first position is a position of the target object in the preset world coordinate system;

[0012] According to the real external parameters of the real camera, the first position is converted into a real camera coordinate system of the real camera to obtain a second position of the target object, where the second position is the position of the target object in the real camera coordinate system;

[0013] According to the second position, a virtual external parameter of the virtual camera is determined.

[0014] Optionally, determining a virtual external parameter of the virtual camera according to the second position includes:

[0015] Determine a position vector between the second position and the origin of the real camera coordinate system;

[0016] Determine a transformation matrix for transforming the real camera coordinate system into the virtual camera coordinate system according to the position vector and the Z-axis vector of the real camera coordinate system;

[0017] The virtual extrinsic parameters of the virtual camera are determined according to the transformation matrix and the real extrinsic parameters of the real camera.

[0018] Optionally, determining a position vector between the second position and an origin of a target camera coordinate system comprises:

[0019] Determine a position vector between the second position and the origin of the real camera coordinate system according to the second position and the origin of the real camera coordinate system;

[0020] The position vector is normalized to obtain a position vector between the second position and the origin of the real camera coordinate system.

[0021] Optionally, determining the first position of the target object according to the target point cloud includes:

[0022] Generating a plurality of two-dimensional depth maps according to the target point cloud;

[0023] Extracting key points of the target object according to the plurality of two-dimensional depth maps;

[0024] A first position of the target object is determined according to the key point.

[0025] Optionally, determining the virtual internal parameter of the virtual camera according to the size parameter, the virtual external parameter and the target point cloud includes:

[0026] Determining virtual optical center point parameters of the virtual camera according to the size parameters;

[0027] Determine edge points according to virtual external parameters of the virtual camera and the target point cloud, where the edge points are points where the target object is located at the edge of the cropped image;

[0028] A virtual focal length parameter of the virtual camera is determined according to the edge point, the virtual optical center point and the size parameter, and the virtual focal length parameter and the virtual optical center point parameter constitute a virtual internal parameter of the virtual camera.

[0029] Optionally, determining edge points according to virtual external parameters of the virtual camera and the target point cloud includes:

[0030] According to the virtual external parameters of the virtual camera, the target point cloud is converted into a virtual camera coordinate system to obtain a converted point cloud;

[0031] The point where the specified axis of the transformed point cloud in the virtual camera coordinate system takes the maximum value or the minimum value is determined as an edge point.

[0032] Optionally, determining the virtual focal length parameter of the virtual camera according to the edge point, the virtual optical center point parameter and the size parameter includes:

[0033] Determining that the edge point falls within a target edge of the cropped image;

[0034] Determining a target pixel parameter of the edge point in a virtual pixel coordinate system of the virtual camera according to the target edge and the size parameter;

[0035] A virtual focal length parameter of the virtual camera is determined according to the virtual optical center point parameter, the edge point and the target pixel parameter.

[0036] In a second aspect, an embodiment of the present invention provides an electronic device, including:

[0037] at least one processor; and,

[0038] a memory communicatively connected to the at least one processor; wherein,

[0039] The memory stores commands that can be executed by the at least one processor, and the commands are executed by the at least one processor so that the at least one processor can execute the calibration parameter generation method for the virtual camera as described above.

[0040] In a third aspect, an embodiment of the present invention provides a non-volatile computer storage medium, wherein the non-volatile computer storage medium stores computer executable commands, and the computer executable commands are used to enable an electronic device to execute the calibration parameter generation method for a virtual camera as described above.

[0041] In the calibration parameter generation method of the virtual camera of the embodiment of the present invention, it includes: obtaining a cropped image and size parameters of the cropped image cropped by the virtual camera in the captured image, the captured image is captured by a preset real camera, the cropped image includes a target image area corresponding to the target object, obtaining a target point cloud of the target image area in a preset world coordinate system, determining the virtual external parameters of the virtual camera according to the real external parameters and the target point cloud of the real camera, determining the virtual internal parameters of the virtual camera according to the size parameters, the virtual external parameters and the target point cloud, and the virtual internal parameters and the virtual external parameters constitute the calibration parameters. Therefore, this embodiment can make up for the missing camera parameters of the cropped image by constructing a virtual camera for the cropped image and solving the camera parameters of the cropped image by calibrating the virtual camera, so that the cropped image can be used as input image data in deep learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] One or more embodiments are illustrated only as examples in the corresponding drawings, and these examples do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0043] Figure 1 is a schematic diagram of the structure of a virtual camera calibration system provided by an embodiment of the present invention;

[0044] Figure 2 is a schematic diagram of a real camera coordinate system provided by an embodiment of the present invention;

[0045] Figure 3 It is a flow chart of a method for generating calibration parameters of a virtual camera provided by an embodiment of the present invention;

[0046] Figure 4 yes Figure 3 The schematic diagram of the process of S33 shown;

[0047] Figure 5 is a schematic diagram of a target point cloud provided by an embodiment of the present invention;

[0048] Figure 6 yes Figure 3 The schematic diagram of the process of S34 shown;

[0049] Figure 7 is a schematic diagram of a virtual camera coordinate system provided by an embodiment of the present invention;

[0050] Figure 8 It is a structural schematic diagram of a calibration parameter generation device for a virtual camera provided by an embodiment of the present invention;

[0051] Fig. 9 yes Figure 8The structural diagram of the first determination module 83 shown;

[0052] Fig.10 yes Figure 8 The structural diagram of the second determination module 84 shown;

[0053] Fig.11 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0055] It should be noted that, if there is no conflict, the various features in the embodiments of the present invention can be combined with each other, all within the scope of protection of the present invention. In addition, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order from the module division in the device or the flow chart. Furthermore, the words "first", "second", "third", etc. used in the present invention do not limit the data and execution order, but only distinguish the same items or similar items with basically the same functions and effects.

[0056] The embodiment of the present invention provides a virtual camera calibration system. Figure 1 , a virtual camera calibration system 100 includes a real camera 10 and an electronic device 20.

[0057] The real camera 10 can be placed at any position in the environment to shoot any object and obtain a captured image. Since the real camera 10 can be placed at any position in the environment, it is necessary to select a reference coordinate system in the environment to describe the position of the real camera 10 and use it to describe the position of any object in the environment. This coordinate system is called the world coordinate system. Figure 2 As shown, the world coordinate system is O w -X w -Y w -Z w Indicates that in the world coordinate system, the position of any object in the environment is expressed by coordinates (X w , Y w , Z w ) indicates that the coordinate (X w , Y w , Z w) is the three-dimensional coordinate of the object in the physical world.

[0058] The rectangular coordinate system established with the optical center of the real camera 11 as the origin and the optical axis as the Z axis is the real camera coordinate system, such as Figure 2 As shown, the real camera coordinate system is O c -X c -Y c -Z c express.

[0059] When determining the position of a point in the real camera coordinate system in the physical world, it is necessary to transform the world coordinate system to the real camera coordinate system. The relationship between the real camera coordinate system and the world coordinate system can be described by a rotation and translation matrix.

[0060] The point in the real camera coordinate system is expressed by the formula:

[0061]

[0062] Wherein, [R|t] represents the rotation and translation matrix of the world coordinate system converted to the real camera coordinate system, which is also called the real external parameter of the real camera 10.

[0063] The coordinate system representing the captured image is the image coordinate system. The image coordinate system is a two-dimensional coordinate system with the center point of the image as the origin. The center point of the image is the intersection of the optical axis of the real camera 10 and the imaging plane, such as Figure 2 As shown, the image coordinate system is represented by Oxy.

[0064] The transformation relationship from the real camera coordinate system to the image coordinate system belongs to the perspective projection relationship from three-dimensional to two-dimensional, which can be expressed by the following matrix:

[0065]

[0066] Wherein, f is the real focal length of the real camera 10.

[0067] The point in the image coordinate system is expressed by the formula:

[0068]

[0069] The captured images captured by the real camera 10 can be stored in the computer as an array, and each element in the array is a pixel, that is, the captured images are composed of pixels. The position of the pixel in the captured image (i.e., the pixel coordinate) is represented by the pixel coordinate system, such as Figure 2 As shown, the pixel coordinate system is a coordinate system with the upper left corner of the image as the origin, the horizontal rightward direction as the u axis, and the vertical downward direction as the v axis.

[0070] The transformation relationship from the image coordinate system to the pixel coordinate system can be expressed by the following matrix:

[0071]

[0072] The matrix is ​​also called the real intrinsic parameter of the real camera 10, f x1 and f y1 is the real focal length parameter of the real camera 10, c x1 and c y1 is the real optical center point parameter of the real camera 10.

[0073] The point in the pixel coordinate system is expressed by the formula:

[0074]

[0075] According to the above formula, we can get the following formula 1:

[0076]

[0077] The real camera 10 includes but is not limited to any suitable type of camera such as a SLR camera, a micro-single-lens camera, a mirrorless camera, a dual-lens reflex camera, a compact camera, etc. In some embodiments, the real camera 10 captures an object in the environment to obtain a captured image, and also obtains the three-dimensional spatial coordinates of each point in the captured image, thereby obtaining a point cloud of the object in the captured image, wherein the three-dimensional spatial coordinates are the coordinates of each point in the captured image in the world coordinate system.

[0078] The electronic device 20 may be a mobile phone, a tablet computer, a vehicle-mounted device, a laptop computer, a wearable device, an augmented reality (AR) device, a virtual reality (VR) device, an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a personal digital assistant (PDA), a smart watch, a smart bracelet, and other electronic devices. This embodiment does not impose any restrictions on the specific type of the terminal device 12. The electronic device 20, as the core of the virtual camera calibration system 100, is used to execute the virtual camera calibration parameter generation method as described below.

[0079] See also Figure 3 The method for generating calibration parameters of a virtual camera provided by an embodiment of the present invention includes:

[0080] S31, obtaining a cropped image cropped by the virtual camera from a captured image and size parameters of the cropped image, wherein the captured image is captured by a preset real camera, and the cropped image includes a target image area corresponding to the target object;

[0081] In this step, the virtual camera is a camera that captures the cropped image, and the cropped image can be cropped from the captured image. For example, the captured image is an image of a human body captured by a real camera, and the cropped image is an image obtained by cropping the human hand in the captured image.

[0082] It is understandable that the virtual camera is not a real camera, but a virtual camera constructed in order to solve the camera parameters of the cropped image.

[0083] The cropped image includes a target image area, which is an image area corresponding to the target object photographed in the cropped image. For example, if the target object is a human hand, then the target image area is an image area including the human hand.

[0084] The size parameters of the cropped image include the width image_w and height image_h of the cropped image.

[0085] S32, obtaining a target point cloud of the target image area in a preset world coordinate system;

[0086] In this step, the target point cloud includes multiple data points, each of which is the three-dimensional spatial coordinate of the target object corresponding to the target image area in the preset world coordinate system. In order to better describe the position of the real camera in the physical world, the preset world coordinate system definition is not fixed and can be set according to actual needs.

[0087] S33, determining virtual extrinsic parameters of the virtual camera according to real extrinsic parameters of the real camera and the target point cloud;

[0088] In this step, the virtual external parameters of the virtual camera are used to describe the conversion relationship from the world coordinate system to the virtual camera coordinate system of the virtual camera. The virtual camera coordinate system is a rectangular coordinate system with the virtual optical center of the virtual camera as the origin and the virtual optical axis of the virtual camera as the Z axis.

[0089] S34, determining the virtual internal parameters of the virtual camera according to the size parameters, the virtual external parameters and the target point cloud, and the virtual internal parameters and the virtual external parameters constitute the calibration parameters.

[0090] Therefore, this embodiment constructs a virtual camera for the cropped image and solves the camera parameters of the cropped image by calibrating the virtual camera, so as to make up for the missing camera parameters of the cropped image, so that the cropped image can be used as input image data in deep learning.

[0091] Since the cropped image contains camera parameters, the cropped image containing camera parameters can also be used as image input data for neural networks in deep learning applications. The model can be built through "image-camera parameters" and the constructed model can be tested, trained and inferred, thereby enriching the application of cropped images in deep learning.

[0092] In some embodiments, see Figure 4 , S33 includes:

[0093] S331, determining a first position of the target object according to the target point cloud, where the first position is a position of the target object in a preset world coordinate system;

[0094] In this step, since the target point cloud includes multiple data points corresponding to the target object, and the target point cloud may include data points irrelevant to the target object, that is, noise, the electronic device needs to filter out the noise that may exist in the target point cloud in order to more accurately determine the first position of the target object.

[0095] In some embodiments, the electronic device generates multiple two-dimensional depth maps based on the target point cloud, extracts key points of the target object based on the multiple two-dimensional depth maps, and determines the first position of the target object based on the key points.

[0096] For example, Figure 5 As shown, the target object is a human hand, and the target point cloud is a three-dimensional point cloud formed by multiple data points corresponding to the hand. The electronic device projects the three-dimensional point cloud in different directions to obtain multiple two-dimensional depth maps, each of which contains position information for describing the target object. The electronic device extracts key points for describing the position information of the target object from each two-dimensional depth map, and determines the key point with the largest position and the key point with the smallest position from the extracted key points, and averages the key point with the largest position and the key point with the smallest position to obtain the center position of the target object, that is, the first position of the target object.

[0097] Therefore, this embodiment can avoid obtaining the first position of the target object based on the noise of the target point cloud, which is conducive to more accurately determining the first position of the target object.

[0098] S332, converting the first position to a real camera coordinate system of the real camera according to the real external parameters of the real camera, to obtain a second position of the target object, where the second position is the position of the target object in the real camera coordinate system;

[0099] S333. Determine virtual external parameters of the virtual camera according to the second position.

[0100] In some embodiments, the electronic device determines a position vector between the second position and the origin of the real camera coordinate system, determines a transformation matrix for transforming the real camera coordinate system into the virtual camera coordinate system based on the position vector and the Z-axis vector of the real camera coordinate system, and determines virtual external parameters of the virtual camera based on the transformation matrix and the real external parameters of the real camera.

[0101] For example, the electronic device connects the second position with the origin of the real camera coordinate system to obtain a position vector between the second position and the origin of the real camera coordinate system, and the electronic device normalizes the position vector by placing the position vector at the modulus of the position vector to obtain a position vector between the second position and the origin of the real camera coordinate system. It can be understood that the position vector between the second position and the origin of the real camera coordinate system is a unit vector.

[0102] The Z-axis vector of the real camera coordinate system is (0, 0, 1), which is also a unit vector. Through the Rodriguez formula, the angle between the two vectors can be calculated based on the position vector between the second position and the origin of the real camera coordinate system and the Z-axis vector of the real camera coordinate system.

[0103] The Rodriguez formula is a formula that describes the rotation of a vector in space. It can calculate the new vector obtained by converting a vector around the rotation axis at a given angle in three-dimensional space. Since the position vector and the Z-axis vector are both known vectors, the electronic device only needs to input the two vectors into the Rodriguez formula to calculate the angle between the two vectors.

[0104] The angle between the position vector and the Z-axis vector is the rotation matrix of the real camera coordinate system to the virtual camera coordinate system. In the present embodiment, since both the real camera coordinate system and the virtual camera coordinate system are basic coordinate systems, the translation vector of the real camera coordinate system to the virtual camera coordinate system is 0. The electronic device can directly determine the transformation matrix of the real camera coordinate system to the virtual camera coordinate system according to the angle between the position vector and the Z-axis vector.

[0105] Since the real extrinsic parameters of the real camera, that is, the transformation matrix from the preset world coordinates to the real camera coordinates, are known, the electronic device multiplies the transformation matrix from the preset world coordinates to the real camera coordinates with the transformation matrix from the real camera coordinate system to the virtual camera coordinate system to determine the transformation matrix from the preset world coordinate system to the virtual camera coordinate system, that is, the virtual extrinsic parameters of the virtual camera.

[0106] In some embodiments, see Figure 6 , S34 includes:

[0107] S341. Determine the virtual optical center point parameters of the virtual camera according to the size parameters. In this step, similar to a real camera, the internal parameters of the virtual camera are expressed by the following formula:

[0108]

[0109] Among them, f x2 and f y2 is the virtual focal length parameter of the virtual camera, c x2and c y2 It is the virtual optical center point parameter of the virtual camera.

[0110] In some embodiments, the electronic device determines the virtual optical center point parameters of the virtual camera based on the pinhole imaging principle according to the following formula:

[0111]

[0112]

[0113] Among them, image_w is the width of the cropped image, and image_h is the height of the cropped image.

[0114] S342, determining edge points according to the virtual external parameters of the virtual camera and the target point cloud, where the edge points are points where the target object is located at the edge of the cropped image;

[0115] In some embodiments, the electronic device converts the target point cloud to the virtual camera coordinate system according to the virtual external parameters of the virtual camera to obtain the converted point cloud, and determines that the point where the converted point cloud takes the maximum or minimum value of the specified axis in the virtual camera coordinate system is the edge point. The specified axis in the virtual camera coordinate system is the X axis or the Y axis.

[0116] For example, see Figure 7 In the converted point cloud, the points falling into the edge of the cropped image include point A, point B, point C and point D, where point A is the point where the X-axis of the converted point cloud in the virtual camera coordinate system takes the minimum value, point B is the point where the X-axis of the converted point cloud in the virtual camera coordinate system takes the maximum value, point C is the point where the Y-axis of the converted point cloud in the virtual camera coordinate system takes the maximum value, and point D is the point where the Y-axis of the converted point cloud in the virtual camera coordinate system takes the minimum value.

[0117] S343. Determine the virtual focal length parameters of the virtual camera according to the edge points, the virtual optical center point and the size parameters. The virtual focal length parameters and the virtual optical center point parameters constitute the virtual internal parameters of the virtual camera.

[0118] In some embodiments, the electronic device determines that the edge point falls within the target edge of the cropped image, determines the target pixel parameters of the edge point in the virtual pixel coordinate system of the virtual camera based on the target edge and size parameters, and determines the virtual focal length parameters of the virtual camera based on the virtual optical center point parameters, the edge point and the target pixel parameters.

[0119] For example, Figure 7As shown, the virtual pixel coordinate system of the virtual camera is a coordinate system with the upper left corner of the cropped image as the origin, the horizontal rightward direction as the u axis, and the vertical downward direction as the v axis. The cropped image includes a left edge, a right edge, an upper edge, and a lower edge. As mentioned above, the electronic device determines that the target edge where point A falls into the cropped image is the left edge, the target edge where point B falls into the cropped image is the right edge, the target edge where point C falls into the cropped image is the upper edge, and the target edge where point D falls into the cropped image is the lower edge.

[0120] Since the size parameters of the cropped image, i.e., the width image_w and height image_h of the cropped image are known, the electronic device determines that the target pixel parameter of point A in the virtual pixel coordinate system is 0, the target pixel parameter of point B in the virtual pixel coordinate system is image_w, the target pixel parameter of point C in the virtual pixel coordinate system is image_h, and the target pixel parameter of point D in the virtual pixel coordinate system is 0.

[0121] Similar to a real camera, the points in the virtual pixel coordinate system are expressed by the following formula:

[0122]

[0123]

[0124] Assume that the coordinates of point A in the virtual camera coordinate system are (X c1 , Y c1 , Z c1 ), the coordinates of point B in the virtual camera coordinate system are (X c2 , Y c2 , Z c2 ), the coordinates of point C in the virtual camera coordinate system are (X c3 , Y c3 , Z c3 ), the coordinates of point D in the virtual camera coordinate system are (X c4 , Y c4 , Z c4 ).

[0125] For point A, the electronic device can determine the virtual focal length parameter f according to the following formula x2 :

[0126]

[0127]

[0128] For point B, the electronic device can determine the virtual focal length parameter f according to the following formula x2 :

[0129]

[0130]

[0131] For point C, the electronic device can determine the virtual focal length parameter f according to the following formula y2 :

[0132]

[0133]

[0134] For point D, the electronic device can determine the virtual focal length parameter f according to the following formula y2 :

[0135]

[0136]

[0137] It can be understood that the electronic device can determine the virtual focal length parameter f by simply using the point falling on the left edge / right edge and the upper edge / lower edge of the cropped image. x2 and f y2 , for example, point A and point C, or point A and point D, or point B and point C, or point B and point D.

[0138] At this point, the electronic device has generated calibration parameters of the virtual camera, including virtual external parameters and virtual internal parameters of the virtual camera, thereby realizing the calibration of the virtual camera.

[0139] The embodiment of the present invention provides a device for generating calibration parameters of a virtual camera. Figure 8 The calibration parameter generating device 800 includes a first acquisition module 81 , a second acquisition module 82 , a first determination module 83 and a second determination module 84 .

[0140] The first acquisition module 81 is used to obtain the cropped image and size parameters of the cropped image cropped by the virtual camera in the captured image, the captured image is captured by a preset real camera, and the cropped image includes a target image area corresponding to the target object. The second acquisition module 82 is used to obtain the target point cloud of the target image area in a preset world coordinate system. The first determination module 83 is used to determine the virtual external parameters of the virtual camera according to the real external parameters and the target point cloud of the real camera. The second determination module 84 determines the virtual internal parameters of the virtual camera according to the size parameters, the virtual external parameters and the target point cloud. The virtual internal parameters and the virtual external parameters constitute the calibration parameters.

[0141] Therefore, this embodiment constructs a virtual camera for the cropped image and solves the camera parameters of the cropped image by calibrating the virtual camera, so as to make up for the missing camera parameters of the cropped image, so that the cropped image can be used as input image data in deep learning.

[0142] In some embodiments, see Fig. 9 The first determination module 83 includes a first determination unit 831 , a conversion unit 832 and a second determination unit 833 .

[0143] The first determination unit 831 is used to determine the first position of the target object according to the target point cloud, where the first position is the position of the target object in a preset world coordinate system. The conversion unit 832 is used to convert the first position to the real camera coordinate system of the real camera according to the real external parameters of the real camera to obtain the second position of the target object, where the second position is the position of the target object in the real camera coordinate system. The second determination unit 833 is used to determine the virtual external parameters of the virtual camera according to the second position.

[0144] In some embodiments, the second determination unit 833 is specifically used to: determine the position vector between the second position and the origin of the real camera coordinate system, determine the transformation matrix of the real camera coordinate system to transform the virtual camera coordinate system according to the position vector and the Z-axis vector of the real camera coordinate system, and determine the virtual external parameters of the virtual camera according to the transformation matrix and the real external parameters of the real camera.

[0145] In some embodiments, the first determination unit 831 is specifically used to: generate multiple two-dimensional depth maps according to the target point cloud, extract key points of the target object according to the multiple two-dimensional depth maps, and determine the first position of the target object according to the key points.

[0146] In some embodiments, see Fig.10 The second determining module 84 includes a third determining unit 841 , a fourth determining unit 842 and a fifth determining unit 843 .

[0147] The third determination unit 841 is used to determine the virtual optical center point parameters of the virtual camera according to the size parameters. The fourth determination unit 842 is used to determine the edge points according to the virtual external parameters of the virtual camera and the target point cloud. The edge points are the points of the target object located at the edge of the cropped image. The fifth determination unit 843 is used to determine the virtual focal length parameters of the virtual camera according to the edge points, the virtual optical center point and the size parameters. The virtual focal length parameters and the virtual optical center point parameters constitute the virtual internal parameters of the virtual camera.

[0148] In some embodiments, the fourth determination unit 842 is specifically used to: convert the target point cloud to the virtual camera coordinate system according to the virtual external parameters of the virtual camera to obtain the converted point cloud, and determine the point where the converted point cloud takes the maximum or minimum value of the specified axis in the virtual camera coordinate system as the edge point.

[0149] In some embodiments, the fifth determination unit 843 is specifically used to: determine whether the edge point falls within the target edge of the cropped image, determine the target pixel parameters of the edge point in the virtual pixel coordinate system of the virtual camera based on the target edge and size parameters, and determine the virtual focal length parameters of the virtual camera based on the virtual optical center point parameters, the edge point and the target pixel parameters.

[0150] It should be noted that the calibration parameter generation device of the virtual camera can execute the calibration parameter generation method of the virtual camera provided in the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in the embodiment of the input device of the text content, please refer to the calibration parameter generation method of the virtual camera provided in the embodiment of the present invention.

[0151] See also Fig.11 , Fig.11 The hardware structure diagram of an electronic device provided by an embodiment of the present invention is shown in FIG. Fig.11 As shown, the electronic device 20 includes one or more processors 21 and a memory 22. Fig.11 A processor 21 is taken as an example.

[0152] The processor 21 and the memory 22 may be connected via a bus or other means. Fig.11 The example of connecting through bus is taken in the following.

[0153] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the calibration parameter generation method of the virtual camera in the embodiment of the present invention. The processor 21 executes various functional applications and data processing of the calibration parameter generation device of the virtual camera by running the non-volatile software programs, instructions and modules stored in the memory 22, that is, realizes the calibration parameter generation method of the virtual camera provided by the above method embodiment and the functions of each module or unit of the above device embodiment.

[0154] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0155] The program instructions / modules are stored in the memory 22 , and when executed by the one or more processors 21 , the calibration parameter generation method for a virtual camera in any of the above method embodiments is executed.

[0156] An embodiment of the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors, such as Fig.11 A processor 21 in the embodiment may enable the one or more processors to execute the calibration parameter generation method of the virtual camera in any of the above method embodiments.

[0157] An embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by an electronic device, the electronic device executes any one of the methods for generating calibration parameters of a virtual camera.

[0158] The above described device or equipment embodiments are merely illustrative, wherein the unit modules described as separate components may or may not be physically separated, and the components displayed as module units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network module units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0159] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0160] Finally, it should be noted that the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. These embodiments are not intended to be additional limitations on the content of the present invention. The purpose of providing these implementations is to make the understanding of the disclosure of the present invention more thorough and comprehensive. In addition, under the idea of ​​the present invention, the above-mentioned technical features continue to be combined with each other, and there are many other changes in different aspects of the present invention as described above, which are all considered to be within the scope of the present invention specification; further, for ordinary technicians in this field, they can be improved or transformed according to the above description, and all these improvements and transformations should belong to the scope of protection of the claims attached to the present invention.

Claims

1. A method for generating calibration parameters of a virtual camera, characterized in that: include: Acquire a cropped image cropped by a virtual camera from a captured image and size parameters of the cropped image, wherein the captured image is captured by a preset real camera, and the cropped image includes a target image area corresponding to a target object; Acquire a target point cloud of the target image area in a preset world coordinate system; Determining virtual extrinsic parameters of the virtual camera according to real extrinsic parameters of the real camera and the target point cloud; The virtual internal parameters of the virtual camera are determined according to the size parameters, the virtual external parameters and the target point cloud, and the virtual internal parameters and the virtual external parameters constitute calibration parameters.

2. The method according to claim 1, characterized in that: Determining the virtual extrinsic parameters of the virtual camera according to the real extrinsic parameters of the real camera and the target point cloud includes: Determine a first position of the target object according to the target point cloud, where the first position is a position of the target object in the preset world coordinate system; According to the real external parameters of the real camera, the first position is converted into a real camera coordinate system of the real camera to obtain a second position of the target object, where the second position is the position of the target object in the real camera coordinate system; According to the second position, a virtual external parameter of the virtual camera is determined.

3. The method according to claim 2, characterized in that Determining the virtual external parameters of the virtual camera according to the second position includes: Determine a position vector between the second position and the origin of the real camera coordinate system; Determine a transformation matrix for transforming the real camera coordinate system into the virtual camera coordinate system according to the position vector and the Z-axis vector of the real camera coordinate system; The virtual extrinsic parameters of the virtual camera are determined according to the transformation matrix and the real extrinsic parameters of the real camera.

4. The method according to claim 3, characterized in that Determining the position vector between the second position and the origin of the target camera coordinate system includes: Determine a position vector between the second position and the origin of the real camera coordinate system according to the second position and the origin of the real camera coordinate system; The position vector is normalized to obtain a position vector between the second position and the origin of the real camera coordinate system.

5. The method according to claim 2, characterized in that: Determining the first position of the target object according to the target point cloud includes: Generating a plurality of two-dimensional depth maps according to the target point cloud; Extracting key points of the target object according to the plurality of two-dimensional depth maps; A first position of the target object is determined according to the key point.

6. The method according to claim 1, characterized in that The step of determining the virtual internal parameters of the virtual camera according to the size parameter, the virtual external parameters and the target point cloud comprises: Determining virtual optical center point parameters of the virtual camera according to the size parameters; Determine edge points according to virtual external parameters of the virtual camera and the target point cloud, where the edge points are points where the target object is located at the edge of the cropped image; A virtual focal length parameter of the virtual camera is determined according to the edge point, the virtual optical center point and the size parameter, and the virtual focal length parameter and the virtual optical center point parameter constitute a virtual internal parameter of the virtual camera.

7. The method according to claim 6, characterized in that Determining edge points according to the virtual external parameters of the virtual camera and the target point cloud includes: According to the virtual external parameters of the virtual camera, the target point cloud is converted into a virtual camera coordinate system to obtain a converted point cloud; The point where the specified axis of the transformed point cloud in the virtual camera coordinate system takes the maximum value or the minimum value is determined as an edge point.

8. The method according to claim 6, characterized in that Determining the virtual focal length parameter of the virtual camera according to the edge point, the virtual optical center point parameter and the size parameter includes: Determining that the edge point falls within a target edge of the cropped image; Determining a target pixel parameter of the edge point in a virtual pixel coordinate system of the virtual camera according to the target edge and the size parameter; A virtual focal length parameter of the virtual camera is determined according to the virtual optical center point parameter, the edge point and the target pixel parameter.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores commands that can be executed by the at least one processor, and the commands are executed by the at least one processor so that the at least one processor can execute the calibration parameter generation method for a virtual camera as described in any one of claims 1 to 8.

10. A non-volatile computing storage medium, characterized in that: The non-volatile computer storage medium stores computer executable commands, and the computer executable commands are used to enable an electronic device to execute the method for generating calibration parameters of a virtual camera as described in any one of claims 1 to 8.

Images