View angle image synthesis method and device, field bridge, storage medium and program product
By constructing a three-dimensional model of the target scene and synthesizing multi-view images, the problems of high training data acquisition cost and insufficient model robustness in the prior art are solved, and efficient and accurate training data generation and model training effects are achieved.
Patent Information
- Application Number
- CN202411944536.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-26
- Publication Date
- 2025-04-25
AI Technical Summary
The prior art is difficult to generate multi-view training data efficiently and accurately, resulting in high training cost for the object detection network model and insufficient model robustness and generalization capabilities.
By using the camera to obtain the real scene map of the target scene at multiple different shooting angles, combining the monocular depth estimation model, corner deviation network model and texture feature extractor, a three-dimensional model of the target scene is constructed, and the images of the target scene at the target perspective are synthesized based on the three-dimensional model.
It reduces the acquisition cost of training images, improves the accuracy and robustness of the trained model, and is more practical.
Smart Images
Figure CN119888402A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing, and in particular to a perspective image synthesis method, device, field bridge, storage medium and program product. Background Art
[0002] In the field of computer vision and image processing, object detection is a key technology that is widely used in many scenarios such as autonomous driving, intelligent monitoring, large vehicle collision avoidance, and large vehicle deviation correction. With the advancement of artificial intelligence and deep learning technologies, the performance of object detection network models has been significantly improved.
[0003] However, the training of target detection network models usually relies on a large amount of labeled data, which usually needs to cover a variety of scenarios and perspectives to ensure the robustness and generalization ability of the model. Traditionally, obtaining this training data relies on field photography and manual labeling, which not only requires a lot of manpower, material resources, and financial resources, but may also be limited by the site, weather, and equipment, making it impossible to obtain training data accurately and quickly.
[0004] Therefore, how to efficiently and accurately generate multi-view training data becomes a problem that needs to be solved urgently. Summary of the invention
[0005] The embodiments of the present application provide a perspective image synthesis method, device, field bridge, storage medium and program product to achieve technical effects such as reducing the acquisition cost of training images and improving the accuracy and robustness of the trained model.
[0006] In a first aspect, an embodiment of the present application provides a method for synthesizing a viewing angle image, the method comprising:
[0007] Use a camera to obtain real scene images of the target scene under multiple different shooting angles;
[0008] The real scene image is sequentially input into a preset monocular depth estimation model, a preset corner deviation network model, and a preset texture feature extractor for processing, and a depth map of the real scene image, a corner deviation corresponding to each pixel of the target scene, and texture feature information of the target scene are obtained respectively;
[0009] Constructing a three-dimensional model of the target scene according to the depth map, the corner point deviation and the texture feature information;
[0010] Based on the three-dimensional model, a target image of the target scene under a target perspective is synthesized; wherein the target image is used to train a target detection network model.
[0011] In a possible implementation, constructing the three-dimensional model of the target scene according to the depth map, the corner point deviation, and the texture feature information includes:
[0012] Constructing a three-dimensional grid of the target scene according to the depth map and the corner point deviations;
[0013] Connecting grid corner points in the three-dimensional grid according to a preset rule to obtain a grid surface of the target scene;
[0014] The texture feature information is mapped onto the mesh surface to obtain the three-dimensional model.
[0015] In a possible implementation, constructing a three-dimensional grid of the target scene according to the depth map and the corner point deviation includes:
[0016] Based on the depth map, determining the three-dimensional coordinates corresponding to each pixel of the target scene in the real scene map;
[0017] Based on the three-dimensional coordinates and the corner point deviations, the coordinates of the grid corner points corresponding to each pixel of the target scene are determined to construct a three-dimensional grid of the target scene.
[0018] In a possible implementation, synthesizing a target image of the target scene at a target perspective based on the three-dimensional model includes:
[0019] After controlling the three-dimensional model to rotate to a target viewing angle, the three-dimensional model is mapped to generate a target image of the target scene at the target viewing angle.
[0020] In a possible implementation, before using a camera to obtain real scene images of the target scene at multiple different shooting angles, the method further includes:
[0021] Calibrate the camera to determine the camera parameters of the camera;
[0022] After using the camera to obtain real scene images of the target scene at multiple different shooting angles, the method further includes:
[0023] According to the camera parameters, distortion correction is performed on the real scene image to obtain a corrected real scene image.
[0024] In a possible implementation manner, calibrating the camera to determine the camera parameters of the camera includes:
[0025] Use a camera to obtain calibration images of the calibration plate at multiple different shooting angles;
[0026] Extracting feature points of the calibration plate in the calibration image, and determining a correspondence between the feature points of the calibration plate in the calibration image and feature points on the physical calibration plate;
[0027] The camera parameters of the camera are determined according to the corresponding relationship and the known geometric information of the calibration plate.
[0028] In a second aspect, an embodiment of the present application provides a perspective image synthesis device, the device comprising:
[0029] An acquisition unit, used for acquiring a real scene image of a target scene under multiple different shooting angles using a camera;
[0030] A processing unit, used to sequentially input the real scene image into a preset monocular depth estimation model, a preset corner deviation network model, and a preset texture feature extractor for processing, to obtain a depth map of the real scene image, a corner deviation corresponding to each pixel of the target scene, and texture feature information of the target scene;
[0031] A construction unit, configured to construct a three-dimensional model of the target scene according to the depth map, the corner point deviation and the texture feature information;
[0032] A synthesis unit is used to synthesize a target image of the target scene under a target perspective based on the three-dimensional model; wherein the target image is used to train a target detection network model.
[0033] In a third aspect, an embodiment of the present application provides a field bridge, which includes the perspective image synthesis device as described in the second aspect above.
[0034] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.
[0035] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.
[0036] In the perspective image synthesis method, device, field bridge, storage medium and program product provided by the embodiment of the present application, the method includes: using a camera to obtain a real scene image of a target scene under multiple different shooting perspectives; inputting the real scene image into a preset monocular depth estimation model, a preset corner deviation network model and a preset texture feature extractor for processing, respectively obtaining a depth map of the real scene image, a corner deviation corresponding to each pixel of the target scene and texture feature information of the target scene; constructing a three-dimensional model of the target scene according to the depth map, the corner deviation and the texture feature information; synthesizing a target image of the target scene under the target perspective according to the three-dimensional model; wherein the target image is used to train the target detection network model. First, a three-dimensional model of the target scene is constructed through a real scene image, and then based on the three-dimensional model, an image of the target scene under the target perspective is synthesized for training the target detection model. Based on this, only one real scene image needs to be collected. After the three-dimensional model of the target scene is generated, the perspective image under any angle can be synthesized, which greatly reduces the acquisition cost of the training image, and also improves the accuracy and robustness of the trained model, and is more practical. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0038] Figure 1 A schematic diagram of a framework for perspective image synthesis provided in an embodiment of the present application;
[0039] Figure 2 A schematic diagram of a flow chart of a method for synthesizing a viewing angle image provided in an embodiment of the present application;
[0040] Figure 3 A schematic diagram of a flow chart of another method for synthesizing perspective images provided in an embodiment of the present application;
[0041] Figure 4 A schematic diagram of the structure of a viewing angle image synthesis device provided in an embodiment of the present application;
[0042] Figure 5 A schematic diagram of the structure of a perspective image synthesis device provided in an embodiment of the present application.
[0043] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0044] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0045] In the field of computer vision and image processing, object detection is a key technology, and object detection network models are also widely used in various scenarios such as autonomous driving, intelligent monitoring, large vehicle collision avoidance, and large vehicle deviation correction. However, training a high-performance object detection network model usually requires a large amount of labeled data, which needs to cover different scenarios and perspectives to ensure the robustness and generalization ability of the model.
[0046] Traditionally, obtaining these training data relies on field photography and manual annotation. However, if the models of visual sensors used in field photography are different, or the installation positions on the equipment (such as vehicles, field bridge equipment, brackets, etc.) are different, or the project scenarios are different, etc., the visual image field imaging will be different. Therefore, the parameters of the model trained by using the images collected by the camera on any model as training data cannot be directly migrated to other devices for use, and the cost of directly collecting real scene images as training data is too high and the cycle is long. Therefore, how to quickly and accurately obtain new perspective images for use in target detection network models in various usage scenarios has become an urgent problem to be solved.
[0047] In some examples, a Neural Radiance Fields Model (NERF) is used to synthesize new perspective images. However, the generalization performance of the current NERF model is poor, and multiple NERF models need to be trained to synthesize relatively good new perspective images, which is not practical enough.
[0048] In order to solve the above problems, the embodiment of the present application provides a method for synthesizing a perspective image, which first constructs a three-dimensional model of the target scene through a real scene image, and then synthesizes the image of the target scene under the target perspective based on the three-dimensional model, so as to be used for the training of various target detection models. Based on the solution of the present application, it is only necessary to collect a real scene image once, and after generating a three-dimensional model of the target scene, the perspective image under any angle can be synthesized, which greatly reduces the acquisition cost of the training image, and also improves the accuracy and robustness of the trained model, and is more practical.
[0049] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0050] For example, Figure 1 A schematic diagram of a perspective image synthesis framework provided in an embodiment of the present application. Figure 1 As shown, by processing the real scene image captured by the camera, a depth map of the real scene image, the corner point deviation corresponding to each pixel in the real scene image, and texture feature information can be obtained; based on the depth map of each real scene image, the corresponding corner point deviation and texture feature information, a three-dimensional model can be constructed; based on the three-dimensional model, a synthetic image can be obtained.
[0051] Figure 2 A flowchart of a method for synthesizing a perspective image is provided in an embodiment of the present application. For example, the execution subject of the embodiment of the present application may be a perspective image synthesis device, which may be located on an electronic device, and the electronic device may be a mobile terminal such as a mobile phone, a tablet, a laptop computer, a computer, etc., which is not limited in the embodiment of the present application.
[0052] like Figure 2 As shown, the perspective image synthesis method provided in the embodiment of the present application may include:
[0053] S201: Use a camera to obtain real scene images of a target scene under multiple different shooting perspectives.
[0054] For example, the embodiment of the present application aims to generate images of the target scene under multiple perspectives through three-dimensional reconstruction and image synthesis methods, so as to train the target detection network model as training data, thereby reducing the data acquisition cost and improving the accuracy and robustness of the model. To this end, a three-dimensional model of the target scene can be constructed first.
[0055] The target scene may be any object or environment, such as the appearance of a building, a natural landscape, etc. In addition, the target scene may also be various postures of a specific device in a specific scene, such as various possible postures of a field bridge device during use.
[0056] After determining the target scene, select the appropriate camera equipment and adjust the camera parameter settings as needed, such as resolution, focal length, aperture, ISO sensitivity, etc., to ensure that a clear image can be captured. Furthermore, according to the characteristics of the target scene, multiple different shooting perspectives can be planned. These perspectives should cover all important parts of the target scene so that complete scene information can be obtained in subsequent processing. Among them, the choice of perspective can include different heights, angles, and distances, so that the camera can obtain the real scene map of the target scene under multiple different shooting perspectives. For example, the same set of visual sensors can be used for data collection on different field bridge equipment, and used together to build a three-dimensional model.
[0057] Optionally, during or after the shooting process, the image quality of the acquired real scene graph can also be checked to ensure that there are no problems such as blur, underexposure or overexposure. If problems are found, the image can be re-shot or deleted to ensure the quality of the real scene graph, thereby providing a more accurate data basis for subsequent image processing and analysis.
[0058] S202, inputting the real scene image into a preset monocular depth estimation model, a preset corner deviation network model, and a preset texture feature extractor for processing in sequence, and obtaining a depth map of the real scene image, a corner deviation corresponding to each pixel of the target scene, and texture feature information of the target scene respectively.
[0059] Exemplarily, the preset monocular depth estimation model is a pre-trained neural network model that can infer the depth information of the scene in the image from a single two-dimensional image. The embodiment of the present application does not limit the specific structure of the preset monocular depth estimation model. For example, the preset monocular depth estimation model can be constructed and trained based on the basic model of the open source Depth Anything V2. The embodiment of the present application inputs the acquired real scene graph into the preset monocular depth estimation model. The model analyzes the visual clues in the image (such as shadows, texture changes, perspective effects, etc.) to estimate the depth of each pixel, thereby outputting a depth map of the real scene graph.
[0060] The preset corner deviation network model is used to predict the corner deviation corresponding to each pixel in the target scene. The embodiment of the present application does not limit the specific structure of the preset corner deviation network model. For example, the preset corner deviation network model can be constructed and trained based on the open source ResNet (Residual Network) series model. The real scene graph is input into the preset corner deviation network model, and the model calculates the corner deviation value V (x, y) of each pixel of the target scene based on the built-in prediction formula.
[0061] Optionally, the preset corner deviation network model can use the following formulas (1) to (3) for calculation when predicting corner deviation. Wherein, formulas (1) to (3) are as follows:
[0062]
[0063] z w,h =g(W3q w,h +b3) (3)
[0064] Among them, in formulas (1) to (3), is the lateral corner deviation, is the longitudinal corner deviation, z w,h is the scaled depth value; W1, W2, and W3 are preset weights; q w,h is the feature vector of the real scene graph; b1, b2, b3 are the model parameters to be trained; W m is the preset width of the 3D grid; H m is the preset height of the 3D grid; g(.) is a nonlinear function used to scale the predicted value of the depth estimate.
[0065] The preset corner deviation network model is based on the above formulas (1) to (3), and combined with the prediction function of the model, it can output the corner deviation of each pixel of the target scene in the real scene image.
[0066] The preset texture feature extractor is an algorithm or model for extracting texture feature information from an image. The specific structure of the preset texture feature extractor is not limited in the embodiments of the present application. The real scene image is input into the preset texture feature extractor for feature extraction processing, and the preset texture feature extractor outputs the texture feature information of the real scene image. Among them, the texture feature information may include RGB information of the image, pattern size, shape of the texture, repeatability and other features, usually represented in the form of feature vectors or feature maps, which are used to describe the texture properties of the image.
[0067] This application is directed to each real scene image under different shooting angles, and each image is input into a preset monocular depth estimation model, a preset corner deviation network model, and a preset texture feature extractor for processing, so as to obtain a depth map corresponding to each real scene image, a corner deviation corresponding to each pixel of the target scene in the image, and texture feature information of the target scene for subsequent three-dimensional reconstruction. It can be understood that the higher the model accuracy of the preset monocular depth estimation model, the preset corner deviation network model, and the preset texture feature extractor, the more accurate the output result, and the more accurate the target three-dimensional model subsequently established in the embodiment of this application will also be.
[0068] S203: construct a three-dimensional model of the target scene according to the depth map, corner point deviation and texture feature information.
[0069] Exemplarily, a three-dimensional model of the target scene can be constructed based on the depth map, corner point deviation, and texture feature information output by each model. Specifically, the depth map can be converted into a three-dimensional point cloud, and the camera's internal and external parameters are used to perform coordinate transformation to determine the three-dimensional spatial coordinates of each point cloud; then, the three-dimensional spatial coordinates of each point cloud are processed using the corner point deviation to improve the accuracy of the point cloud and ensure the accurate positioning of key features in the three-dimensional space; then, the point cloud is converted into a three-dimensional mesh, and the extracted texture feature information is mapped to the three-dimensional mesh to enhance the visual effect of the model, and a three-dimensional model of the target scene can be obtained.
[0070] Optionally, in a possible embodiment, constructing a three-dimensional model of a target scene according to the depth map, corner point deviation, and texture feature information may include:
[0071] S1, construct a 3D mesh of the target scene based on the depth map and corner point deviation;
[0072] S2, connecting the grid corner points in the three-dimensional grid according to a preset rule to obtain a grid surface of the target scene;
[0073] S3. Mapping the texture feature information onto the mesh surface to obtain a three-dimensional model.
[0074] Exemplarily, according to the corner point deviation corresponding to each pixel of the target scene in the depth map of the real scene map and the real scene map, the precise spatial positioning of each pixel of the target scene can be determined, and more accurate point cloud data of the target scene can be obtained. Then, the point cloud can be converted into a three-dimensional grid using a triangulation algorithm to construct a three-dimensional grid of the target scene. Further, the grid corner points in the three-dimensional grid are connected according to preset rules. For example, according to the fineness requirements, one corner point can be connected to 4 or 8 corner points. After the corner points in the grid are connected along the edge in sequence, the grid surface of the target scene can be obtained. Finally, the extracted texture feature information is mapped to the grid surface, so that the three-dimensional model has the corresponding color and pattern feature information visually, and the three-dimensional model of the target scene can be obtained.
[0075] Among them, the depth map provides basic three-dimensional structural information, the corner deviation ensures the accurate positioning of key pixels, and the texture feature information provides rich visual details for the model. The embodiment of the present application can construct a detailed and accurate three-dimensional model by combining the depth map, corner deviation and texture feature information, based on which a more accurate new perspective image can be synthesized.
[0076] Optionally, in a possible embodiment, constructing a three-dimensional grid of the target scene according to the depth map and the corner point deviation may include:
[0077] S11, based on the depth map, determining the three-dimensional coordinates corresponding to each pixel of the target scene in the real scene map;
[0078] S12. Based on the three-dimensional coordinates and the corner point deviations, the coordinates of the grid corner points corresponding to each pixel of the target scene are determined to construct a three-dimensional grid of the target scene.
[0079] Exemplarily, when constructing a three-dimensional grid of the target scene based on the depth map and corner point deviations, the depth map can be converted into a three-dimensional point cloud. Specifically, by combining the depth value of each pixel in the depth map and the actual position of each pixel in the real scene map, and then using the camera's intrinsic parameters (such as focal length, principal point coordinates, etc.) and extrinsic parameters (such as the camera's rotation and translation matrices, etc.) to perform coordinate transformation, the three-dimensional coordinates corresponding to each pixel can be determined. Among them, the embodiment of the present application does not limit how to perform coordinate transformation to obtain the three-dimensional coordinates of each pixel in the depth map. It can be calculated according to any feasible conversion formula in the prior art, which will not be repeated here.
[0080] After obtaining the three-dimensional coordinates of each pixel, the three-dimensional coordinates of each pixel can be corrected based on the corner point deviation to adjust the three-dimensional coordinates of each pixel to obtain more accurate coordinates. Optionally, the coordinates of the grid corner points corresponding to each pixel of the target scene can be determined by the following formula (4). Wherein, formula (4) is as follows:
[0081]
[0082] In formula (4), θ F It is the field of view of the camera when capturing the real scene image; is the horizontal coordinate corresponding to the pixel (the size range is -1 to 1); is the vertical coordinate of the pixel (the range is -1 to 1); V w,h are the coordinates of the grid corner points corresponding to the pixels in the target scene.
[0083] According to formula (4), the coordinates of the grid corner points corresponding to each pixel of the target scene can be calculated. Furthermore, a triangulation algorithm can be used to connect the midpoints of the corrected three-dimensional point cloud into a preset shape such as a triangle to form a grid structure, thereby obtaining a three-dimensional grid of the target scene.
[0084] The embodiment of the present application utilizes the distance information provided by the depth map and the precise positioning information provided by the corner point deviation to construct a more accurate three-dimensional grid of the target scene, so that a more detailed and accurate three-dimensional model can be subsequently constructed to facilitate the subsequent synthesis of a more accurate new perspective image.
[0085] Optionally, in some possible embodiments, the generated three-dimensional model can also be optimized, for example, by smoothing and detail enhancement, etc., to improve the visual effect and accuracy of the model; of course, the constructed three-dimensional model can also be compared with the real target scene to verify the accuracy of the model, and the three-dimensional model can be adjusted according to the real scene to further improve the accuracy of the three-dimensional model, which is not limited to the embodiments of the present application.
[0086] S204. Synthesize a target image of the target scene from a target perspective based on the three-dimensional model; wherein the target image is used to train a target detection network model.
[0087] Exemplarily, when it is necessary to use the image of the target scene under the target perspective as training data to train the target detection network model, the target image of the target scene under the target perspective can be synthesized according to the three-dimensional model for training the target detection network model. Optionally, the target detection network model can be any model that may be trained with the target scene image, for example, it can be a large vehicle collision avoidance model, a large vehicle deviation correction model, etc., which is not limited in the embodiments of the present application.
[0088] Optionally, in a possible embodiment, synthesizing a target image of a target scene under a target perspective according to the three-dimensional model may include:
[0089] After controlling the three-dimensional model to rotate to the target viewing angle, the three-dimensional model is mapped to generate a target image of the target scene at the target viewing angle.
[0090] Exemplarily, the target viewing angle refers to the shooting angle of the target scene in the real scene, which determines the angle and distance from which the three-dimensional model is observed, which can be determined by the camera position, orientation, field of view (FOV), etc. After determining the target viewing angle, the three-dimensional model can be controlled to rotate to the target viewing angle. Then, a suitable rendering engine is selected to map the three-dimensional model, and the three-dimensional model can be converted into a two-dimensional image to obtain a target image of the target scene under the target viewing angle.
[0091] Based on this application, images from the target perspective can be generated from the three-dimensional model, and these images can be used to train the target detection network model. The synthesized images not only increase the diversity of training data, but also can accurately control the scene and perspective, thereby improving the generalization ability of the model.
[0092] The perspective image synthesis method provided in the embodiment of the present application includes: using a camera to obtain a real scene image of a target scene under multiple different shooting perspectives; inputting the real scene image into a preset monocular depth estimation model, a preset corner deviation network model, and a preset texture feature extractor for processing, respectively obtaining a depth map of the real scene image, a corner deviation corresponding to each pixel of the target scene, and texture feature information of the target scene; constructing a three-dimensional model of the target scene according to the depth map, corner deviation, and texture feature information; synthesizing a target image of the target scene under the target perspective according to the three-dimensional model; wherein the target image is used to train the target detection network model. The embodiment of the present application first constructs a three-dimensional model of the target scene through a real scene image, and then synthesizes an image of the target scene under the target perspective based on the three-dimensional model for training the target detection network model. Based on this, only one real scene image needs to be collected. After generating the three-dimensional model of the target scene, the perspective image under any angle can be synthesized, which greatly reduces the acquisition cost of the training image, and also improves the accuracy and robustness of the trained model, and is more practical.
[0093] Figure 3 A flowchart of another method for synthesizing perspective images provided in an embodiment of the present application is shown below. Figure 3 As shown, the perspective image synthesis method provided in the embodiment of the present application may include:
[0094] S301: Calibrate the camera to determine the camera parameters.
[0095] Exemplarily, in order to improve the accuracy of the constructed three-dimensional model of the target scene, before obtaining the real scene images of the target scene under multiple shooting angles through the camera, the camera may be calibrated to determine the camera parameters.
[0096] Among them, in the embodiment of the present application, the camera parameters of the camera mainly include the intrinsic parameters and extrinsic parameters of the camera. The intrinsic parameter matrix of the camera is a 3×3 matrix used to describe the optical characteristics of the camera, which includes the focal length (fx, fy), the principal point coordinates (cx, cy) and the lens distortion parameters (such as radial distortion and tangential distortion). Based on the camera intrinsic parameters, the three-dimensional world coordinates can be converted into two-dimensional image coordinates. The camera extrinsic parameters mainly include the rotation and translation matrices of the camera relative to the calibration plate. When shooting real scene images at multiple different shooting angles, each real scene image has corresponding extrinsic parameters to describe the position and orientation of the camera when shooting each image.
[0097] Optionally, in a possible embodiment, calibrating a camera to determine camera parameters of the camera may include:
[0098] S10, using a camera to obtain calibration images of the calibration plate at multiple different shooting angles;
[0099] S20, extracting feature points of the calibration plate in the calibration image, and determining the correspondence between the feature points of the calibration plate in the calibration image and the feature points on the physical calibration plate;
[0100] S30: Determine the camera parameters of the camera according to the corresponding relationship and the known geometric information of the calibration plate.
[0101] For example, in practical applications, a checkerboard or dot array is usually used as a calibration plate, because these patterns have obvious feature points, which are easy to detect and match. During the shooting process, the camera can be used to shoot the calibration plate from multiple different angles and positions, and the calibration plate can be controlled to have different plane orientations and tilt angles during shooting to ensure that the image covers the camera's field of view as much as possible, and to ensure that the calibration plate has sufficient viewing angle changes in each image, so as to use a clearly visible image without motion blur and overexposure as the calibration image for camera calibration.
[0102] After the calibration images are captured, in each calibration image, feature points on the calibration plate are extracted using image processing algorithms (such as Harris corner detection or sub-pixel corner detection, etc.). For example, on a chessboard, feature points are usually the intersections of the chessboard. Usually, the geometry of the calibration plate is known (such as the size and spacing of the chessboard), which makes it possible to match feature points in the image with actual points on the calibration plate, thereby determining the correspondence between feature points in the calibration image and feature points on the physical calibration plate.
[0103] Furthermore, based on the one-to-one correspondence between the feature points in the calibration image and the feature points on the physical calibration plate, combined with the known geometric information of the calibration plate, the camera intrinsic parameters can be estimated using the least squares optimization technique. Similarly, the camera extrinsic parameters corresponding to each calibration image can also be calculated.
[0104] Among them, in the camera calibration process, the known geometric information of the calibration plate refers to the known physical layout and size of the feature points on the calibration plate. This information is the basis for the camera calibration algorithm to calculate the camera intrinsic parameters. Specifically, the geometric information of the calibration plate based on the checkerboard pattern mainly includes the physical size of each square, the number of rows and columns of the checkerboard, etc.; the geometric information of the calibration plate based on the dot array mainly includes the physical distance between the centers of adjacent dots, the number of rows and columns of the dot array, etc. The embodiment of the present application does not limit the pattern of the calibration plate. Based on a flat calibration plate and accurate calibration plate geometric information, accurate calibration results can be obtained.
[0105] Optionally, in some possible embodiments, a nonlinear optimization method can be used to optimize the intrinsic parameters and the extrinsic parameters to minimize the reprojection error. The reprojection error refers to the distance between the points on the physical calibration plate and the actual detected image feature points after being projected onto the image plane. By optimizing the intrinsic parameters and the extrinsic parameters, more accurate camera parameters can be obtained.
[0106] The embodiment of the present application uses a calibration plate to calibrate the camera, which can more accurately determine the intrinsic and extrinsic parameters of the camera, so that the camera can accurately map points in the three-dimensional world to the two-dimensional image plane, thereby improving the reconstruction effect of the subsequent three-dimensional model.
[0107] S302: Use a camera to obtain real scene images of the target scene under multiple different shooting angles.
[0108] It should be noted that the specific implementation of step S302 can refer to the description of step S201 and will not be repeated here.
[0109] S303: Perform distortion correction on the real scene image according to the camera parameters to obtain a corrected real scene image.
[0110] For example, the correction mapping matrix can be calculated using the camera intrinsic parameters and the distortion coefficients, and the real scene image can be remapped using the calculated mapping matrix to obtain a corrected real scene image.
[0111] It is understandable that the higher the accuracy of camera calibration, the better the distortion correction effect. Based on the corrected real scene image, a more accurate 3D model can be constructed, thereby improving the realism of the synthesized image.
[0112] S304, inputting the corrected real scene image into a preset monocular depth estimation model, a preset corner deviation network model, and a preset texture feature extractor for processing in sequence, to obtain a depth map of the real scene image, a corner deviation corresponding to each pixel of the target scene, and texture feature information of the target scene, respectively.
[0113] S305: construct a three-dimensional mesh of the target scene according to the depth map and the corner point deviations.
[0114] S306: Connect the grid corner points in the three-dimensional grid according to a preset rule to obtain a grid surface of the target scene.
[0115] S307, mapping the texture feature information onto the mesh surface to obtain a three-dimensional model.
[0116] S308 , after controlling the three-dimensional model to rotate to the target viewing angle, mapping the three-dimensional model to generate a target image of the target scene at the target viewing angle.
[0117] It should be noted that the specific implementation of steps S304 to S308 can refer to the description of other embodiments, which will not be described in detail here. In actual perspective image synthesis, some or all of the steps of the embodiment of the present application may be included, and the embodiment of the present application does not limit this.
[0118] The perspective image synthesis method provided by the embodiment of the present application includes: calibrating a camera to determine the camera parameters of the camera, performing distortion correction on a real scene image according to the camera parameters to obtain a corrected real scene image, and using the camera to obtain a real scene image of a target scene under multiple different shooting perspectives; inputting the real scene image into a preset monocular depth estimation model, a preset corner point deviation network model, and a preset texture feature extractor in sequence for processing, and respectively obtaining a depth map of the real scene image, a corner point deviation corresponding to each pixel of the target scene, and texture feature information of the target scene; constructing a three-dimensional grid of the target scene according to the depth map and the corner point deviation; connecting grid corner points in the three-dimensional grid according to preset rules to obtain a grid surface of the target scene; mapping the texture feature information onto the grid surface to obtain a three-dimensional model; and after controlling the three-dimensional model to rotate to the target perspective, mapping the three-dimensional model to generate a target image of the target scene under the target perspective. The method of the embodiment of the present application calibrates the camera before using the camera to capture the real scene image, and uses the calibration result to perform distortion correction on the captured real scene image, thereby further improving the three-dimensional reconstruction effect. The reconstructed three-dimensional model of the target scene is more accurate, and thus the synthesized perspective image is also more accurate. While reducing the acquisition cost of the training image, the accuracy and robustness of the trained model are further improved.
[0119] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.
[0120] Figure 4 Schematic diagram of the structure of a viewing angle image synthesis device provided in an embodiment of the present application. Figure 4 As shown, the perspective image synthesis device 40 provided in the embodiment of the present application includes an acquisition unit 401, a processing unit 402, a construction unit 403 and a synthesis unit 404.
[0121] The acquisition unit 401 is used to use a camera to acquire a real scene image of a target scene under multiple different shooting angles;
[0122] The processing unit 402 is used to sequentially input the real scene image into a preset monocular depth estimation model, a preset corner deviation network model, and a preset texture feature extractor for processing, and obtain a depth map of the real scene image, a corner deviation corresponding to each pixel of the target scene, and texture feature information of the target scene;
[0123] A construction unit 403 is used to construct a three-dimensional model of the target scene according to the depth map, corner point deviation and texture feature information;
[0124] The synthesis unit 404 is used to synthesize a target image of the target scene under the target perspective according to the three-dimensional model; wherein the target image is used to train the target detection network model.
[0125] The device provided in this embodiment can be used to execute the method of the above embodiment. Its implementation principle and technical effect are similar and will not be described in detail here.
[0126] Based on the above device embodiment, in a possible implementation, the construction unit 403 is specifically configured to:
[0127] Construct a 3D mesh of the target scene based on the depth map and corner point deviations;
[0128] Connect the grid corner points in the three-dimensional grid according to preset rules to obtain the grid surface of the target scene;
[0129] The texture feature information is mapped onto the mesh surface to obtain a three-dimensional model.
[0130] Based on the above device embodiment, in a possible implementation, the construction unit 403 is specifically configured to:
[0131] Based on the depth map, determine the three-dimensional coordinates corresponding to each pixel of the target scene in the real scene map;
[0132] Based on the three-dimensional coordinates and corner point deviations, the coordinates of the grid corner points corresponding to each pixel of the target scene are determined, and a three-dimensional grid of the target scene is constructed.
[0133] Based on the above device embodiment, in a possible implementation, the synthesis unit 404 is specifically configured to:
[0134] After controlling the three-dimensional model to rotate to the target viewing angle, the three-dimensional model is mapped to generate a target image of the target scene at the target viewing angle.
[0135] Based on the above device embodiment, in a possible implementation manner, the acquisition unit 401 is further configured to:
[0136] Before using the camera to obtain the real scene images of the target scene under multiple different shooting angles, the camera is calibrated to determine the camera parameters;
[0137] After using a camera to obtain a real scene image of a target scene under multiple different shooting angles, the real scene image is distorted and corrected according to camera parameters to obtain a corrected real scene image.
[0138] Based on the above device embodiment, in a possible implementation manner, the acquisition unit 401 is further configured to:
[0139] Use a camera to obtain calibration images of the calibration plate at multiple different shooting angles;
[0140] Extracting feature points of the calibration plate in the calibration image, and determining the correspondence between the feature points of the calibration plate in the calibration image and the feature points on the physical calibration plate;
[0141] The camera parameters of the camera are determined according to the correspondence and the known geometric information of the calibration plate.
[0142] The perspective image synthesis device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.
[0143] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. In addition, it can also be stored in the memory of the above device in the form of program code, and the functions of the above data processing modules can be called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.
[0144] Figure 5 This is a schematic diagram of the structure of a viewing angle image synthesis device provided in an embodiment of the present application. Figure 5 As shown, the perspective image synthesis device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus 504.
[0145] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that at least one processor 501 executes the above method.
[0146] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.
[0147] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.
[0148] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.
[0149] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.
[0150] The present application also provides a field bridge, which includes any of the aforementioned perspective image synthesis devices.
[0151] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.
[0152] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.
[0153] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.
[0154] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.
[0155] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.
[0156] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0157] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0158] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0159] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.
[0160] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.
Claims
1. A method for synthesizing a perspective image, characterized in that: The method comprises: Use a camera to obtain real scene images of the target scene under multiple different shooting angles; The real scene image is sequentially input into a preset monocular depth estimation model, a preset corner deviation network model, and a preset texture feature extractor for processing, and a depth map of the real scene image, a corner deviation corresponding to each pixel of the target scene, and texture feature information of the target scene are obtained respectively; Constructing a three-dimensional model of the target scene according to the depth map, the corner point deviation and the texture feature information; Based on the three-dimensional model, a target image of the target scene under a target perspective is synthesized; wherein the target image is used to train a target detection network model.
2. The method according to claim 1, characterized in that The constructing the three-dimensional model of the target scene according to the depth map, the corner point deviation and the texture feature information includes: Constructing a three-dimensional grid of the target scene according to the depth map and the corner point deviations; Connecting grid corner points in the three-dimensional grid according to a preset rule to obtain a grid surface of the target scene; The texture feature information is mapped onto the mesh surface to obtain the three-dimensional model.
3. The method according to claim 2, characterized in that The step of constructing a three-dimensional grid of the target scene according to the depth map and the corner point deviations includes: Based on the depth map, determining the three-dimensional coordinates corresponding to each pixel of the target scene in the real scene map; Based on the three-dimensional coordinates and the corner point deviations, the coordinates of the grid corner points corresponding to each pixel of the target scene are determined to construct a three-dimensional grid of the target scene.
4. The method according to claim 1, characterized in that: The step of synthesizing a target image of the target scene from a target perspective according to the three-dimensional model includes: After controlling the three-dimensional model to rotate to a target viewing angle, the three-dimensional model is mapped to generate a target image of the target scene at the target viewing angle.
5. The method according to any one of claims 1 to 4, characterized in that Before using a camera to obtain a real scene image of the target scene under a plurality of different shooting angles, the method further includes: Calibrate the camera to determine the camera parameters of the camera; After using the camera to obtain real scene images of the target scene at multiple different shooting angles, the method further includes: According to the camera parameters, distortion correction is performed on the real scene image to obtain a corrected real scene image.
6. The method according to claim 5, characterized in that The step of calibrating the camera to determine the camera parameters of the camera includes: Use a camera to obtain calibration images of the calibration plate at multiple different shooting angles; Extracting feature points of the calibration plate in the calibration image, and determining a correspondence between the feature points of the calibration plate in the calibration image and feature points on the physical calibration plate; The camera parameters of the camera are determined according to the corresponding relationship and the known geometric information of the calibration plate.
7. A perspective image synthesis device, characterized in that: The device comprises: An acquisition unit, used for acquiring a real scene image of a target scene under multiple different shooting angles using a camera; A processing unit, used to sequentially input the real scene image into a preset monocular depth estimation model, a preset corner deviation network model, and a preset texture feature extractor for processing, to obtain a depth map of the real scene image, a corner deviation corresponding to each pixel of the target scene, and texture feature information of the target scene; A construction unit, configured to construct a three-dimensional model of the target scene according to the depth map, the corner point deviation and the texture feature information; A synthesis unit is used to synthesize a target image of the target scene under a target perspective based on the three-dimensional model; wherein the target image is used to train a target detection network model.
8. A field bridge, characterized in that: The field bridge includes the perspective image synthesis device as claimed in claim 7.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.
10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.