A method and device for generating virtual-real fusion images, electronic equipment, and storage medium
By connecting ordinary cameras with binocular three-dimensional devices and using depth information rendering, virtual and real fusion images are directly generated, which solves the problems of obtaining efficiency and quality of virtual and real fusion images in the existing technology, and achieves efficient and low-cost virtual and real fusion images generation.
Patent Information
- Application Number
- CN202411119018.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-15
AI Technical Summary
It is difficult to quickly obtain high-quality virtual and real fusion images in the prior art. AR cameras have problems such as lens shaking and low video resolution, while ordinary cameras require manual production in the later stage, resulting in high cost and low efficiency.
By rigidly connecting ordinary cameras with binocular stereo equipment, images of ordinary cameras and binocular stereo equipment are obtained, and images of binocular stereo equipment are rendered using depth information to directly generate virtual and real fusion images to avoid post-human production.
It realizes efficient and low-cost acquisition of high-quality virtual and real fusion images, avoids the quality problems of AR camera equipment, and reduces the post-production cost of ordinary cameras.
Smart Images

Figure CN118644408B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to a method and apparatus for generating virtual-real fusion images, an electronic device, and a storage medium. Background Art
[0002] Virtual-real fusion images refer to images obtained by fusing virtual objects with real-world scenes. As the demand for virtual-real fusion images increases, it is crucial to acquire high-quality virtual-real fusion images quickly and cost-effectively.
[0003] Currently, there are two main methods for obtaining fused virtual-real images. The first is to directly capture images using specialized AR camera equipment, which combines virtual objects with the actual scene to directly obtain a fused virtual-real image. The second method is to capture scene images using ordinary high-definition cameras. This involves using cameras that do not have position tracking and depth acquisition capabilities but can capture high-quality images, first capturing high-quality scene images, and then manually blending the virtual objects with the actual scene in the captured images in post-production.
[0004] However, the first method suffers from issues like camera shake and poor video resolution and clarity during recording, making it impossible to guarantee high-quality fusion images. While the second method can ensure image quality, it is costly and time-consuming to produce, and because it requires post-production, it cannot be previewed in a timely manner, resulting in fusion images that may not meet expectations. Summary of the Invention
[0005] In view of the shortcomings of the prior art, this application provides a method and apparatus for generating virtual-real fusion images, an electronic device, and a storage medium to solve the problem that the prior art cannot quickly obtain high-quality virtual-real fusion images.
[0006] In order to achieve the above objectives, this application provides the following technical solutions:
[0007] The first aspect of this application provides a method for generating virtual-real fusion images, including:
[0008] The system acquires images of the current scene from a standard camera and a stereoscopic device, respectively. The standard camera and the stereoscopic device are rigidly connected and fixed together. The standard camera refers to a camera that does not have position tracking and depth acquisition functions.
[0009] The ordinary camera image is drawn onto a pre-established virtual space plane to obtain an initial virtual-real fusion image;
[0010] If there are dynamic objects in the current scene, then the pixels on the stereoscopic device image corresponding to each pixel on the ordinary camera image are matched.
[0011] The depth information of the pixels in the corresponding stereoscopic device image is determined as the depth information of the pixels in the ordinary camera image;
[0012] Using the depth information of each pixel in the ordinary camera image, the initial virtual-real fusion image is rendered with depth information to obtain the final virtual-real fusion image;
[0013] If there are no dynamic objects in the current scene, the depth of the pre-built object model of the current scene is rendered in the virtual space to obtain the depth information of the objects in the current scene in the ordinary camera image.
[0014] Using the depth information of objects in the current scene from the ordinary camera image, the initial virtual-real fusion image is rendered with depth information to obtain the final virtual-real fusion image.
[0015] Optionally, in the virtual-real fusion image generation method provided above, before acquiring the ordinary camera image and the stereoscopic device image of the current scene respectively acquired by the ordinary camera and the stereoscopic device, the method further includes:
[0016] The position of the stereoscopic device is bound to the position of the virtual camera, and the orientation of the stereoscopic device is bound to the virtual camera, so as to drive the position and orientation of the virtual camera through the real-time position and orientation of the stereoscopic device.
[0017] Construct a virtual space plane as a child node of the virtual camera.
[0018] Optionally, in the virtual-real fusion image generation method provided above, after acquiring the ordinary camera image and the stereoscopic device image of the current scene respectively, the method further includes:
[0019] Based on the current position and orientation information of the binocular stereo device, the position and orientation of the virtual camera are adjusted so as to adjust the virtual space plane according to the position and orientation of the virtual camera.
[0020] Optionally, in the virtual-real fusion image generation method provided above, the step of matching the pixels on the stereoscopic device image corresponding to each pixel on the ordinary camera image includes:
[0021] Multiple salient feature points were identified on the ordinary camera image and multiple salient feature points were identified on the binocular stereo device image;
[0022] Based on the positional relationship between each salient feature point on the ordinary camera image and the corresponding salient feature point on the stereoscopic device image, the depth transformation matrix of the pixels between the ordinary camera image and the stereoscopic device image is calculated; wherein, the depth transformation matrix includes a translation vector and a rotation matrix;
[0023] Based on the coordinates of each pixel in the ordinary camera image and the depth transformation matrix, the coordinates of the corresponding pixel in the stereo device image are calculated.
[0024] Optionally, in the virtual-real fusion image generation method provided above, the step of calculating the depth transformation matrix of pixels between the ordinary camera image and the stereoscopic device image based on the positional relationship between each salient feature point on the ordinary camera image and the corresponding salient feature points on the stereoscopic device image includes:
[0025] The mean coordinates of each salient feature point on the ordinary camera image are calculated to obtain the average position of the salient feature points in the ordinary camera image, and the mean coordinates of each salient feature point on the binocular stereo device image are calculated to obtain the average position of the salient feature points in the binocular stereo device image.
[0026] The translation vector is obtained by calculating the difference between the average position of salient feature points in the stereo image and the average position of salient feature points in the ordinary camera image.
[0027] Based on the difference between each salient feature point on the ordinary camera image and the average position of the salient feature points on the ordinary camera image, and the difference between each salient feature point on the stereoscopic device image and the average position of the salient feature points on the stereoscopic device image, the rotation matrix is calculated using a least squares optimization algorithm.
[0028] The translation vector and the rotation matrix are combined to form the depth transformation matrix.
[0029] Optionally, in the virtual-real fusion image generation method provided above, the rotation matrix is calculated using a least squares optimization algorithm based on the difference between the salient feature points on the ordinary camera image and the average position of the salient feature points on the ordinary camera image, and the difference between the salient feature points on the stereoscopic device image and the average position of the salient feature points on the stereoscopic device image. This includes:
[0030] A covariance matrix is generated by combining a matrix composed of the differences between the salient feature points on the ordinary camera image and the average position of the salient feature points on the ordinary camera image, and a matrix composed of the differences between the salient feature points on the stereoscopic device image and the average position of the salient feature points on the stereoscopic device image.
[0031] Singular value analysis was performed on the covariance matrix to obtain two orthogonal matrices and one diagonal matrix;
[0032] The rotation matrix is calculated using the two orthogonal matrices.
[0033] A second aspect of this application provides a virtual-real fusion image generation apparatus, comprising:
[0034] An image acquisition unit is used to acquire images of the current scene captured by a conventional camera and images of the stereoscopic device, respectively; wherein, the conventional camera and the stereoscopic device are rigidly connected and fixed together; the conventional camera refers to a camera that does not have position tracking and depth acquisition functions;
[0035] The drawing unit is used to draw the ordinary camera image onto a pre-established virtual space plane to obtain an initial virtual-real fusion image;
[0036] The matching unit is used to match the pixels on the stereoscopic device image corresponding to each pixel on the ordinary camera image when there are dynamic objects in the current scene.
[0037] The information determination unit is used to determine the depth information of the pixels on the corresponding binocular stereo device image as the depth information of the pixels on the ordinary camera image;
[0038] The first rendering unit is used to render the initial virtual-real fusion image by utilizing the depth information of each pixel on the ordinary camera image to obtain the final virtual-real fusion image.
[0039] The depth rendering unit is used to render the depth of a pre-built object model of the current scene in virtual space when there are no dynamic objects in the current scene, so as to obtain the depth information of the objects in the current scene in the ordinary camera image.
[0040] The second rendering unit is used to render the initial virtual-real fusion image by utilizing the depth information of objects in the current scene in the ordinary camera image, so as to obtain the final virtual-real fusion image.
[0041] Optionally, the virtual-real fusion image generation apparatus provided above further includes:
[0042] A binding unit is used to bind the position of the stereoscopic device to the position of the virtual camera, and to bind the orientation of the stereoscopic device to the virtual camera, so as to drive the position and orientation of the virtual camera through the real-time position and real-time orientation of the stereoscopic device.
[0043] A screen construction unit is used to construct a virtual space plane as a child node of the virtual camera.
[0044] Optionally, the virtual-real fusion image generation apparatus provided above further includes:
[0045] The position adjustment unit is used to adjust the position and orientation of the virtual camera according to the current position information and current orientation information of the binocular stereo device, so as to adjust the virtual space plane according to the position and orientation of the virtual camera.
[0046] Optionally, in the virtual-real fusion image generation apparatus provided above, the matching unit includes:
[0047] The calibration unit is used to calibrate multiple salient feature points on the ordinary camera image and multiple salient feature points on the binocular stereo device image;
[0048] The transformation matrix calculation unit is used to calculate the depth transformation matrix of pixels between the ordinary camera image and the stereo device image based on the positional relationship between each salient feature point on the ordinary camera image and each salient feature point on the corresponding stereo device image; wherein, the depth transformation matrix includes a translation vector and a rotation matrix;
[0049] The matching calculation unit is used to calculate the coordinates of the pixel in the stereo device image corresponding to each pixel in the ordinary camera image based on the coordinates of each pixel in the ordinary camera image and the depth transformation matrix.
[0050] Optionally, in the virtual-real fusion image generation apparatus provided above, the transformation matrix calculation unit includes:
[0051] The average position calculation unit is used to calculate the average coordinates of each salient feature point on the ordinary camera image to obtain the average position of the salient feature points in the ordinary camera image, and to calculate the average coordinates of each salient feature point on the stereoscopic device image to obtain the average position of the salient feature points in the stereoscopic device image.
[0052] The translation vector calculation unit is used to calculate the difference between the average position of the salient feature points in the stereoscopic device image and the average position of the salient feature points in the ordinary camera image, and obtain the translation vector.
[0053] The rotation matrix calculation unit is used to calculate the rotation matrix based on the difference between each salient feature point on the ordinary camera image and the average position of the salient feature points on the ordinary camera image, and the difference between each salient feature point on the stereoscopic device image and the average position of the salient feature points on the stereoscopic device image, using a least squares optimization algorithm.
[0054] A combination unit is used to combine the translation vector and the rotation matrix to form the depth transformation matrix.
[0055] Optionally, in the virtual-real fusion image generation apparatus provided above, the rotation matrix calculation unit includes:
[0056] The matrix generation unit is used to generate a covariance matrix by using a matrix composed of the difference between each salient feature point on the ordinary camera image and the average position of the salient feature points on the ordinary camera image, and a matrix composed of the difference between each salient feature point on the stereoscopic device image and the average position of the salient feature points on the stereoscopic device image.
[0057] The decomposition unit is used to perform singular value analysis on the covariance matrix to obtain two orthogonal matrices and one diagonal matrix;
[0058] A matrix calculation unit is used to calculate the rotation matrix using two orthogonal matrices.
[0059] A third aspect of this application provides an electronic device, comprising:
[0060] Memory and processor;
[0061] The memory is used to store programs;
[0062] The processor is used to execute the program, which, when executed, is specifically used to implement the virtual-real fusion image generation method as described in any of the above.
[0063] A fourth aspect of this application provides a computer storage medium for storing a computer program, which, when executed, is used to implement the virtual-real fusion image generation method as described in any of the preceding claims.
[0064] This application provides a method for generating virtual-real fusion images. A regular camera and a stereoscopic device are rigidly connected and fixed together to capture high-quality images using the regular camera, avoiding problems associated with AR cameras and ensuring the accuracy of the obtained images. Furthermore, the stereoscopic device compensates for the lack of position tracking and depth acquisition capabilities of the regular camera. Specifically, during image generation, the regular camera image and the stereoscopic device image of the current scene are acquired separately. Then, the regular camera image is drawn onto a pre-established virtual space plane to obtain an initial virtual-real fusion image. If dynamic objects exist in the current scene, the corresponding pixels in the stereoscopic device image are matched to the pixels in the regular camera image, and the depth information of the corresponding pixels in the stereoscopic device image is used as the depth information of the pixels in the regular camera image. Finally, the depth information of the pixels in the regular camera image is used to render the initial virtual-real fusion image to obtain the final virtual-real fusion image. If no dynamic objects exist in the current scene, the depth of a pre-constructed object model of the current scene is rendered in the virtual space to obtain the depth information of the objects in the current scene from the regular camera image. By utilizing the depth information of objects in the current scene from ordinary camera images, depth information rendering is performed on the initial virtual-real fusion image to obtain the final virtual-real fusion image. Therefore, it achieves the combination of information acquired by ordinary cameras and stereoscopic devices, directly rendering virtual-real fusion images without the need for post-processing, thus efficiently obtaining high-quality virtual-real fusion images. Attached Figure Description
[0065] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.
[0066] Figure 1 A flowchart illustrating a virtual-real fusion image generation method provided in this application embodiment;
[0067] Figure 2 A schematic diagram illustrating the connection between a conventional camera and a binocular stereo device, provided as an embodiment of this application;
[0068] Figure 3 A flowchart illustrating a method for matching pixels in a conventional camera image and a stereoscopic device image, provided in an embodiment of this application;
[0069] Figure 4A schematic diagram illustrating the annotation of significant feature points on ordinary camera images and stereoscopic device images provided in this application embodiment;
[0070] Figure 5 A flowchart illustrating a method for calculating a depth transformation matrix provided in this application embodiment;
[0071] Figure 6 A flowchart illustrating a method for calculating a rotation matrix provided in an embodiment of this application;
[0072] Figure 7 This is a schematic diagram of the architecture of a virtual-real fusion image generation device provided in an embodiment of this application;
[0073] Figure 8 This is a schematic diagram of the architecture of an electronic device provided in an embodiment of this application. DETAILED DESCRIPTION
[0074] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0075] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0076] This application provides a method for generating virtual-real fusion images, such as... Figure 1 As shown, the following steps are included:
[0077] S101. Acquire the images of the current scene captured by the ordinary camera and the stereoscopic device, respectively.
[0078] In this context, a regular camera refers to a camera used for capturing high-definition images, while an AR camera refers to a camera that does not have position tracking and depth acquisition capabilities. Therefore, in this embodiment, a virtual-real fusion image can be obtained through a regular camera, thereby avoiding the problem of low image quality caused by AR cameras.
[0079] Since ordinary cameras lack position tracking and depth acquisition capabilities, and generating fused virtual-real images requires adjusting the viewing angle based on position and determining occlusion relationships between objects based on depth information, this embodiment further includes a binocular stereo device for position tracking and depth information acquisition. Therefore, image acquisition requires simultaneous use of both an ordinary camera and a binocular stereo device. Images acquired by the ordinary camera are defined as ordinary camera images, while images acquired by the binocular stereo device are defined as binocular stereo device images.
[0080] To determine the depth information of each pixel in the image captured by the ordinary camera and the stereoscopic device based on the pixel correspondence between images captured by the ordinary camera and the stereoscopic device, the positions of the ordinary camera and the stereoscopic device must be fixed to determine the pixel correspondence between the images delivered by the two. Therefore, in this application, the ordinary camera and the stereoscopic device are rigidly connected and fixed together. Of course, the connection between the ordinary camera and the stereoscopic device can be removed when no longer in use. However, before removal, the connection must be kept fixed to ensure the quality of the resulting fused image.
[0081] Specifically, the connection methods between ordinary cameras and stereoscopic devices can be as follows: Figure 2 As shown. Specifically, the requirements for fixing the stereoscopic device and the ordinary camera vary depending on the matching scenario of the set information. Since there are dynamic objects in the scene that require depth fusion, their depth information is constantly changing. Therefore, it is necessary to obtain the depth information based on the mapping relationship between the ordinary camera and the stereoscopic device. Thus, for the connection between the ordinary camera and the stereoscopic device, the left camera of the stereoscopic device and the ordinary camera need to be overlapped as much as possible to reduce the ineffective depth area. If the objects in the scene that need depth fusion are all static objects, there are no special requirements for the relative position of the stereoscopic device and the ordinary camera; they only need to be rigidly connected together.
[0082] In order to adjust the position information of the virtual camera and update the virtual-real fusion image accordingly, optionally, in another embodiment of this application, before performing step S101, the following can be further performed:
[0083] The position of the stereoscopic device is bound to the position of the virtual camera, and the orientation of the stereoscopic device is bound to the virtual camera, so as to drive the position and orientation of the virtual camera through the real-time position and orientation of the stereoscopic device.
[0084] Construct a virtual space plane as a child node of the virtual camera.
[0085] Specifically, in this embodiment, the position information of the stereoscopic device is used to determine the virtual camera; that is, the position information of the generated virtual-real fusion image is adjusted using the position information of the stereoscopic device. The position information of the stereoscopic device can be obtained through the SDK provided by the stereoscopic device, and its position information specifically includes 3 degrees of freedom position and 3 degrees of freedom orientation, which are then bound to the position and orientation of the virtual camera, respectively. Then, a virtual space plane is constructed as a child node of the virtual camera, so that the position and orientation of the virtual plane will continuously change as the position and orientation of the virtual camera move and rotate.
[0086] Therefore, in the embodiments of this application, after executing step S101, the following can be further executed:
[0087] Based on the current position and orientation information of the binocular stereo device, adjust the position and orientation of the virtual camera to adjust the virtual space plane according to the position and orientation of the virtual camera.
[0088] S102. Draw the ordinary camera image onto the pre-established virtual space plane to obtain the initial virtual-real fusion image.
[0089] Specifically, by creating dynamic textures, ordinary camera images can be drawn onto a pre-established virtual space plane to obtain an initial image blended with reality. It should be noted that since the depth information of the ordinary camera image is not yet determined at this stage, it is usually drawn with an initial depth information first, and then further adjustments are made after the accurate depth information is determined later.
[0090] It should be noted that virtual objects can be set up in the virtual space plane as needed, so as to initially merge them with the real images corresponding to ordinary camera images.
[0091] S103. Determine whether there are dynamic objects in the current scene.
[0092] It should be noted that whether there are dynamic objects in the current scene refers to whether there are dynamic objects in the current scene that need to be deep-blended with virtual objects. Since the depth information of dynamic objects changes continuously as they move, while the depth information of static objects is fixed, the methods for obtaining depth information differ between dynamic and static objects. Therefore, it is necessary to first determine whether there are dynamic objects in the current scene that need to be blended with virtual objects. If dynamic objects are found to exist in the current scene, proceed to step S104. If no dynamic objects are found in the current scene, meaning that all objects requiring depth blending in the current scene are static objects, proceed to step S107.
[0093] Optionally, two modes can be set, one for scenes with moving objects and one for scenes without moving objects. Users can select the appropriate mode based on the current scene they are shooting, thus determining the presence of moving objects based on the set mode. Alternatively, it can be determined by checking for changes in depth information acquired by the stereoscopic device.
[0094] S104. Match the pixels on the stereo device image to the pixels on the ordinary camera image.
[0095] It should be noted that when a stereoscopic device acquires images, it can capture the depth information of the pixels in the stereoscopic image, while a regular camera cannot. However, since the stereoscopic device and the regular camera capture the same scene, i.e., the same image, their images are identical. Therefore, there is a correspondence between the pixels in the regular camera image and the stereoscopic device image. Thus, it is possible to match the corresponding pixels in the stereoscopic device image for each pixel in the regular camera image, and use the depth information of the pixels in the stereoscopic device image as the depth information of the corresponding pixels in the regular camera image.
[0096] Optionally, in another embodiment of this application, one specific implementation of step S104 is as follows: Figure 3 As shown, the following steps are included:
[0097] S301, calibrates multiple salient feature points on ordinary camera images and multiple salient feature points on binocular stereo images.
[0098] Among them, salient feature points refer to pixels that are clearly distinguishable from other surrounding feature points. Specifically, pixels can be identified based on variations in their color values, etc. For example, ... Figure 4As shown, five salient feature points are marked on ordinary camera images and stereo images respectively. These five salient features are all very prominent in the images, thus ensuring that the pixel at the same location is marked as the salient feature point in both types of images.
[0099] Because of the special existence of salient feature points, they will not be confused with other pixels. Therefore, the salient feature points on ordinary camera images and the corresponding salient feature points on stereoscopic device images can be accurately determined. By using the depth correspondence between the salient features on ordinary camera images and stereoscopic device images, the depth correspondence between ordinary camera images and stereoscopic device images can be determined. Based on this correspondence, the corresponding pixel on the stereoscopic device image can be determined for each pixel on the ordinary camera image.
[0100] It should be noted that since the ordinary camera and the stereo device are fixedly connected, only a small number of significant feature points need to be marked to obtain the correspondence between the pixels on the ordinary camera image and the pixels on the stereo device image.
[0101] S302. Based on the positional relationship between each salient feature point on the ordinary camera image and each salient feature point on the corresponding stereo device image, calculate the depth transformation matrix of the pixels between the ordinary camera image and the stereo device image.
[0102] The depth transformation matrix includes a translation vector and a rotation matrix, which represent the correspondence between the two images. Optionally, the depth transformation matrix can be represented as M(T, R), where T is the translation vector and R is the rotation matrix.
[0103] Under the influence of M, salient pixels P in ordinary camera images can be... i ' (x i ',y i ') The salient pixel P transformed onto the image of the stereoscopic device i (x i ,y i ).
[0104] Optionally, in another embodiment of this application, one specific implementation of step S302 is as follows: Figure 5 As shown, the following steps are included:
[0105] S501. Calculate the mean coordinates of each salient feature point on the ordinary camera image to obtain the average position of the salient feature points in the ordinary camera image, and calculate the mean coordinates of each salient feature point on the stereoscopic device image to obtain the average position of the salient feature points in the stereoscopic device image.
[0106] Specifically, the average position of salient feature points in a typical camera image can be calculated as follows:
[0107]
[0108] The average position of salient feature points in a binocular stereo image can be calculated as follows:
[0109]
[0110] S502. Calculate the difference between the average position of salient feature points in the stereo image and the average position of salient feature points in the ordinary camera image to obtain the translation vector.
[0111] Specifically, the translation vector is:
[0112]
[0113] S503. Based on the difference between the average position of each salient feature point on the ordinary camera image and the average position of the salient feature points on the ordinary camera image, and the difference between the average position of each salient feature point on the stereoscopic device image and the average position of the salient feature points on the stereoscopic device image, the rotation matrix is calculated using the least squares optimization algorithm.
[0114] It should be noted that the conversion between images from a regular camera and images from a stereoscopic device should be performed with minimal error. Optionally, the depth transformation error can be defined as:
[0115]
[0116] After calculating the rotation matrix using various methods, it was determined that the least squares optimization algorithm could obtain the minimum error. Therefore, in this embodiment of the application, the least squares optimization algorithm is used to calculate the rotation matrix.
[0117] Optionally, in another embodiment of this application, one specific implementation of step S503 is as follows: Figure 6 As shown, the following steps are included:
[0118] S601. Generate a covariance matrix by combining the matrix formed by the difference between the salient feature points on the ordinary camera image and the average position of the salient feature points on the ordinary camera image, and the matrix formed by the difference between the salient feature points on the stereoscopic device image and the average position of the salient feature points on the stereoscopic device image.
[0119] Specifically, the generated covariance matrix can be represented as:
[0120]
[0121] S602. Perform singular value analysis on the covariance matrix to obtain two orthogonal matrices and one diagonal matrix.
[0122] Specifically, singular value analysis of the covariance matrix can be expressed as:
[0123]
[0124] Here, S and V are two orthogonal matrices. Σ is a diagonal matrix.
[0125] S603. Using two orthogonal matrices, calculate the rotation matrix.
[0126] Specifically, the rotation matrix can be represented as:
[0127]
[0128] S504. Combine the translation vector and rotation matrix to form the depth transformation matrix.
[0129] S303. Based on the coordinates of each pixel in the ordinary camera image and the depth transformation matrix, calculate the coordinates of the corresponding pixel in the stereo device image.
[0130] Specifically, after obtaining the depth transformation matrix, each pixel in the ordinary camera image can be transformed using the depth transformation matrix to become a pixel in the stereo image. Specifically, for any point in the ordinary camera image, the corresponding pixel in the stereo image can be obtained using the inverse transformation of M(R,T), which can be represented as:
[0131]
[0132] S105. Determine the depth information of the pixels on the corresponding stereo device image as the depth information of the pixels on the ordinary camera image.
[0133] Since the pixels in a regular camera image and the corresponding pixels in a stereoscopic device image are at the same location, the depth information of the pixel in the stereoscopic device image can be directly used as the depth information of the corresponding pixel in the camera image.
[0134] Alternatively, depth information of pixels in the stereo image can be obtained from the SDK provided by the stereo device.
[0135] S106. Using the depth information of each pixel in the ordinary camera image, perform depth information rendering on the initial virtual-real fusion image to obtain the final virtual-real fusion image.
[0136] After obtaining the depth information of each pixel in a regular camera image, the depth information can be used to adjust the depth information of each pixel in the initial virtual-real fusion image, thereby obtaining the final virtual-real fusion image.
[0137] S107. Render the depth of the pre-built object model of the current scene in the virtual space to obtain the depth information of the objects in the current scene in the ordinary camera image.
[0138] Since all objects in the current scene that need to be deep-blended are static objects, we can pre-model the objects that need to be occluded in the current scene, i.e. the objects that need to be deep-blended. Then we can directly perform depth rendering on the model in the virtual space, that is, only render the depth and not its color information, so as to obtain the depth information of the object.
[0139] It should also be noted that since the position and orientation of the virtual camera are changing, it is necessary to render the current position information of the virtual camera accordingly to obtain the depth information of the corresponding position, and then execute step S108.
[0140] S108. Using the depth information of objects in the current scene from ordinary camera images, perform depth information rendering on the initial virtual-real fusion image to obtain the final virtual-real fusion image.
[0141] This application provides a method for generating virtual-real fusion images. A regular camera and a stereoscopic device are rigidly connected and fixed together to capture high-quality images using the regular camera, avoiding problems associated with AR cameras and ensuring the accuracy of the obtained images. Furthermore, the stereoscopic device compensates for the lack of position tracking and depth acquisition capabilities of the regular camera. Specifically, during image generation, the regular camera image and the stereoscopic device image of the current scene are acquired separately. Then, the regular camera image is drawn onto a pre-established virtual space plane to obtain an initial virtual-real fusion image. If dynamic objects exist in the current scene, the corresponding pixels in the stereoscopic device image are matched to the pixels in the regular camera image, and the depth information of the corresponding pixels in the stereoscopic device image is used as the depth information of the pixels in the regular camera image. Finally, the depth information of the pixels in the regular camera image is used to render the initial virtual-real fusion image to obtain the final virtual-real fusion image. If no dynamic objects exist in the current scene, the depth of a pre-constructed object model of the current scene is rendered in the virtual space to obtain the depth information of the objects in the current scene from the regular camera image. By utilizing the depth information of objects in the current scene from ordinary camera images, depth information rendering is performed on the initial virtual-real fusion image to obtain the final virtual-real fusion image. Therefore, it achieves the combination of information acquired by ordinary cameras and stereoscopic devices, directly rendering virtual-real fusion images without the need for post-processing, thus efficiently obtaining high-quality virtual-real fusion images.
[0142] Another embodiment of this application provides a virtual-real fusion image generation apparatus, such as... Figure 7 As shown, it includes the following units:
[0143] Image acquisition unit 701 is used to acquire images of the current scene from both a conventional camera and a stereoscopic device, respectively. The conventional camera and the stereoscopic device are rigidly connected and fixed together. A conventional camera refers to a camera that does not have position tracking or depth acquisition capabilities.
[0144] The drawing unit 702 is used to draw ordinary camera images onto a pre-established virtual space plane to obtain an initial virtual-real fusion image.
[0145] The matching unit 703 is used to match the pixels on the stereo device image corresponding to each pixel on the ordinary camera image when there are dynamic objects in the current scene.
[0146] The information determination unit 704 is used to determine the depth information of the pixels on the corresponding stereo device image as the depth information of the pixels on the ordinary camera image.
[0147] The first rendering unit 705 is used to render the initial virtual-real fusion image by utilizing the depth information of each pixel on the ordinary camera image to obtain the final virtual-real fusion image.
[0148] The depth rendering unit 706 is used to render the depth of a pre-built object model of the current scene in virtual space when there are no dynamic objects in the current scene, so as to obtain the depth information of the objects in the current scene in the ordinary camera image.
[0149] The second rendering unit 707 is used to render the initial virtual-real fusion image by utilizing the depth information of objects in the current scene in the ordinary camera image, so as to obtain the final virtual-real fusion image.
[0150] Optionally, in another embodiment of the virtual-real fusion image generation apparatus provided in this application, the apparatus further includes:
[0151] The binding unit is used to bind the position of the stereoscopic device to the position of the virtual camera, and to bind the orientation of the stereoscopic device to the virtual camera, so as to drive the position and orientation of the virtual camera through the real-time position and orientation of the stereoscopic device.
[0152] Screen building blocks are used to construct a virtual space plane, which serves as a child node of the virtual camera.
[0153] Optionally, in another embodiment of the virtual-real fusion image generation apparatus provided in this application, the apparatus further includes:
[0154] The position adjustment unit is used to adjust the position and orientation of the virtual camera based on the current position and orientation information of the binocular stereo device, so as to adjust the virtual space plane according to the position and orientation of the virtual camera.
[0155] Optionally, in another embodiment of the virtual-real fusion image generation apparatus provided in this application, the matching unit includes:
[0156] The calibration unit is used to calibrate multiple salient feature points on ordinary camera images and multiple salient feature points on binocular stereo images.
[0157] The transformation matrix calculation unit is used to calculate the depth transformation matrix of pixels between the ordinary camera image and the stereo device image based on the positional relationship between each salient feature point on the ordinary camera image and each salient feature point on the corresponding stereo device image. The depth transformation matrix includes a translation vector and a rotation matrix.
[0158] The matching calculation unit is used to calculate the coordinates of the corresponding pixel in the stereo device image based on the coordinates of each pixel in the ordinary camera image and the depth transformation matrix.
[0159] Optionally, in another embodiment of the virtual-real fusion image generation apparatus provided in this application, the transformation matrix calculation unit includes:
[0160] The average position calculation unit is used to calculate the average coordinates of each salient feature point on the ordinary camera image to obtain the average position of the salient feature points in the ordinary camera image, and to calculate the average coordinates of each salient feature point on the stereoscopic device image to obtain the average position of the salient feature points in the stereoscopic device image.
[0161] The translation vector calculation unit is used to calculate the difference between the average position of salient feature points in the stereo image and the average position of salient feature points in the ordinary camera image, and obtain the translation vector.
[0162] The rotation matrix calculation unit is used to calculate the rotation matrix based on the difference between the average position of each salient feature point on the ordinary camera image and the average position of the salient feature points on the stereo image and the stereo image, using a least squares optimization algorithm.
[0163] Combining units are used to combine translation vectors and rotation matrices to form a depth transformation matrix.
[0164] Optionally, in another embodiment of the virtual-real fusion image generation apparatus provided in this application, the rotation matrix calculation unit includes:
[0165] The matrix generation unit is used to generate a covariance matrix by combining a matrix composed of the difference between the salient feature points on the ordinary camera image and the average position of the salient feature points on the ordinary camera image, and a matrix composed of the difference between the salient feature points on the stereo image and the average position of the salient feature points on the stereo image.
[0166] The decomposition unit is used to perform singular value analysis on the covariance matrix, resulting in two orthogonal matrices and a diagonal matrix.
[0167] The matrix calculation unit is used to calculate the rotation matrix using two orthogonal matrices.
[0168] It should be noted that the specific working process of each unit provided in the above embodiments of this application can be referred to the implementation process of the corresponding steps in the above method embodiments, and will not be repeated here.
[0169] Another embodiment of this application provides an electronic device, such as... Figure 8 Shown, including:
[0170] Memory 801 and processor 802.
[0171] Among them, memory 801 is used to store programs.
[0172] The processor 802 is used to execute the program stored in the memory 801. When the program is executed, it is specifically used to implement the virtual-real fusion image generation method provided in any of the above embodiments.
[0173] Another embodiment of this application provides a computer storage medium for storing a computer program, which, when executed, is used to implement the virtual-real fusion image generation method as described above.
[0174] Computer storage media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0175] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0176] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for generating a virtual-real fusion image, characterized in that: include: Obtaining an ordinary camera image and a binocular stereo device image of the current scene captured by an ordinary camera and a binocular stereo device respectively; wherein the ordinary camera and the binocular stereo device are fixed together by a rigid connection; the ordinary camera refers to a camera that does not have position tracking and depth acquisition functions; Drawing the ordinary camera image onto a pre-established virtual space plane to obtain an initial virtual-real fusion picture; wherein the position and direction of the virtual space plane are adjusted based on the real-time position and real-time direction of the binocular stereo device; If there is a dynamic object in the current scene, the pixel points on the binocular stereo device image corresponding to each pixel point on the ordinary camera image are matched; wherein the dynamic object refers to a dynamic object that needs to be deeply integrated with the virtual object; Determine the depth information of the corresponding pixel point on the binocular stereo device image as the depth information of the pixel point on the ordinary camera image; Using the depth information of each pixel point on the ordinary camera image, the depth information of the initial virtual-real fusion picture is rendered to obtain a final virtual-real fusion picture; If there is no dynamic object in the current scene, rendering the depth of the pre-built object model of the current scene in the virtual space to obtain the depth information of the object of the current scene in the ordinary camera image; The depth information of the object in the current scene in the ordinary camera image is used to render the initial virtual-reality fusion picture to obtain a final virtual-reality fusion picture.
2. The method according to claim 1, characterized in that Before respectively acquiring the ordinary camera image and the binocular stereo device image of the current scene captured by the ordinary camera and the binocular stereo device, the method further includes: Binding the position of the binocular stereo device to the position of the virtual camera, and binding the direction of the binocular stereo device to the virtual camera, so as to drive the position and direction of the virtual camera through the real-time position and real-time direction of the binocular stereo device; Construct a virtual space plane as a child node of the virtual camera.
3. The method according to claim 2, characterized in that After respectively acquiring the ordinary camera image and the binocular stereo device image of the current scene captured by the ordinary camera and the binocular stereo device, the method further includes: The position and direction of the virtual camera are adjusted according to the current position information and the current direction information of the binocular stereo device, so as to adjust the virtual space plane according to the position and direction of the virtual camera.
4. The method according to claim 1, characterized in that: The matching of the pixel points on the binocular stereo device image corresponding to each pixel point on the ordinary camera image includes: Calibrate a plurality of significant feature points on the ordinary camera image, and calibrate a plurality of significant feature points on the binocular stereo device image; Based on the positional relationship between each significant feature point on the ordinary camera image and each significant feature point on the corresponding binocular stereo device image, a depth transformation matrix of pixel points between the ordinary camera image and the binocular stereo device image is calculated; wherein the depth transformation matrix includes a translation vector and a rotation matrix; Based on the coordinates of each pixel point on the ordinary camera image and the depth transformation matrix, the coordinates of the pixel point on the binocular stereo device image corresponding to each pixel point on the ordinary camera image are calculated.
5. The method according to claim 4, characterized in that The method of calculating a depth transformation matrix of pixel points between the ordinary camera image and the binocular stereo device image based on a positional relationship between each significant feature point on the ordinary camera image and each significant feature point on the corresponding binocular stereo device image comprises: Calculating the mean of the coordinates of each significant feature point on the ordinary camera image to obtain the average position of the significant feature points of the ordinary camera image, and calculating the mean of the coordinates of each significant feature point on the binocular stereo device image to obtain the average position of the significant feature points of the binocular stereo device image; Calculate the difference between the average position of the significant feature points of the binocular stereo device image and the average position of the significant feature points of the ordinary camera image to obtain the translation vector; The rotation matrix is calculated by a least squares optimization algorithm based on the difference between each significant feature point on the ordinary camera image and the average position of the significant feature points of the ordinary camera image, and the difference between each significant feature point on the binocular stereo device image and the average position of the significant feature points of the binocular stereo device image; The translation vector and the rotation matrix are combined to form the depth transformation matrix.
6. The method according to claim 5, characterized in that The rotation matrix is calculated based on the difference between each significant feature point on the ordinary camera image and the average position of the significant feature points of the ordinary camera image, and each significant feature point on the binocular stereo device image and the average position of the significant feature points of the binocular stereo device image, by a least squares optimization algorithm, including: Generate a covariance matrix using a matrix consisting of differences between each significant feature point on the ordinary camera image and an average position of the significant feature points of the ordinary camera image, and a matrix consisting of differences between each significant feature point on the binocular stereo device image and an average position of the significant feature points of the binocular stereo device image; Performing singular value analysis on the covariance matrix to obtain two orthogonal matrices and a diagonal matrix; The rotation matrix is calculated using the two orthogonal matrices.
7. A virtual-real fusion image generation device, characterized in that: include: An image acquisition unit, used to acquire an ordinary camera image and a binocular stereo device image of a current scene respectively acquired by an ordinary camera and a binocular stereo device; wherein the ordinary camera and the binocular stereo device are fixed together by a rigid connection; the ordinary camera refers to a camera without position tracking and depth acquisition functions; A drawing unit, used for drawing the ordinary camera image onto a pre-established virtual space plane to obtain an initial virtual-real fusion picture; wherein the position and direction of the virtual space plane are adjusted based on the real-time position and real-time direction of the binocular stereo device; A matching unit, configured to match the pixel points on the binocular stereo device image corresponding to each pixel point on the ordinary camera image when there is a dynamic object in the current scene; wherein the dynamic object refers to a dynamic object that needs to be deeply fused with the virtual object; An information determining unit, used to determine the depth information of the corresponding pixel point on the binocular stereo device image as the depth information of the pixel point on the ordinary camera image; A first rendering unit is used to perform depth information rendering on the initial virtual-real fusion picture by using the depth information of each pixel point on the ordinary camera image to obtain a final virtual-real fusion picture; A depth rendering unit, configured to render the depth of a pre-built object model of the current scene in a virtual space when no dynamic object exists in the current scene, so as to obtain depth information of the object of the current scene in the ordinary camera image; The second rendering unit is used to use the depth information of the object in the current scene in the ordinary camera image to perform depth information rendering on the initial virtual-real fusion picture to obtain a final virtual-real fusion picture.
8. The device according to claim 7, characterized in that Also includes: A binding unit, used to bind the position of the binocular stereo device with the position of the virtual camera, and to bind the direction of the binocular stereo device with the virtual camera, so as to drive the position and direction of the virtual camera through the real-time position and real-time direction of the binocular stereo device; The screen construction unit is used to construct a virtual space plane as a child node of the virtual camera.
9. An electronic device, characterized in that: include: Memory and processor; Wherein, the memory is used to store programs; The processor is used to execute the program, and when the program is executed, it is specifically used to implement the virtual-real fusion image generation method as described in any one of claims 1 to 6.
10. A computer storage medium, characterized in that: Used to store a computer program, which, when executed, is used to implement the method for generating a virtual-real fusion image as described in any one of claims 1 to 6.
Citation Information
Patent Citations
Method, system and device for synchronously realizing three-dimensional reconstruction and AR virtual-real registration
CN112070884A
VR image processing method and device, electronic equipment and readable storage medium
CN116450002A
Video generation method and device based on virtual reality, equipment and medium
CN116527863A
Method and system for generating three-dimensional virtual real scene in fixed space
CN118037956A
Machine learning based model localization system
US20180189974A1
