Virtual camera projection method and system, electronic equipment and medium

Abstract

The invention provides a virtual camera projection method and system, electronic equipment and a medium, and belongs to the technical field of target detection, and the method comprises the steps: constructing a training data set according to sampling points in a three-dimensional space; modeling projection and back projection between the real camera view and the virtual camera view to obtain a two-stage multi-layer perceptron model; training a two-stage multi-layer perceptron model according to the training data set to obtain a trained model; and obtaining a pixel mapping table between the virtual camera view and the real camera view based on the trained model. According to the method, the reversible pixel mapping relation between the real camera and the virtual camera is constructed by adopting the two-stage multi-layer perceptron model, and the pictures under any group of camera parameters are uniformly converted to be under the virtual camera parameters, so that high-quality label data are efficiently generated through the automatic labeling model; the data utilization rate and the generalization ability of the automatic labeling model are remarkably improved, so that the model can directly process multi-source heterogeneous data, and the model adaptation cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a virtual camera projection method, system, electronic device, and medium. Background Technology

[0002] Visual object detection is a core component of autonomous driving technology. Its main objective is to locate and identify various road targets by analyzing image data captured by onboard cameras, providing necessary information support for driver assistance systems to achieve advanced driving functions such as driverless driving and automatic parking. In recent years, deep learning-based methods have dominated this field. To ensure the diversity and consistency of training datasets while reducing the high manpower and time costs associated with dataset annotation, large-scale fleet data collection and automatic annotation models have been widely used.

[0003] However, in practical applications, the performance of automatic annotation models is often significantly constrained by the heterogeneity of acquisition devices. Differences in parameters between multiple cameras lead to problems such as low data utilization and decreased generalization performance of automatic annotation models. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art, and proposes a virtual camera projection method, system, electronic device and medium.

[0005] In a first aspect, embodiments of the present invention provide a virtual camera projection method, comprising:

[0006] A training dataset is constructed based on sampling points in three-dimensional space;

[0007] Modeling the projection and back projection between the real camera view and the virtual camera view yields a two-stage multilayer perceptron model.

[0008] The two-stage multilayer perceptron model is trained based on the training dataset to obtain the trained model.

[0009] The trained model is used to obtain a pixel mapping table between the virtual camera view and the real camera view.

[0010] In some embodiments, constructing a training dataset based on sampling points in three-dimensional space includes:

[0011] The sampling points are obtained by uniformly sampling points in three-dimensional space.

[0012] The sampling points are projected into a two-dimensional space and filtered to obtain effective two-dimensional pixels and projection angles.

[0013] The effective two-dimensional pixels are normalized to obtain normalized effective two-dimensional pixels;

[0014] The training dataset is constructed based on the normalized effective two-dimensional pixels and the projection angle.

[0015] In some embodiments, projecting the sampling points into a two-dimensional space and performing filtering to obtain effective two-dimensional pixels and projection angles includes:

[0016] The sampling points are projected into a two-dimensional space based on the intrinsic and extrinsic parameters of a real camera;

[0017] Valid two-dimensional pixels and their corresponding three-dimensional spatial points are selected based on the resolution of the actual camera.

[0018] Based on the preset intrinsic and extrinsic parameters of the virtual camera, the selected three-dimensional spatial points are projected onto a virtual two-dimensional space, and the projection angle is calculated.

[0019] In some embodiments, the method further includes:

[0020] Calculate the distance from a point in three-dimensional space to the origin of the coordinate system;

[0021] The distance is used as a weight for loss calculation during the training of the two-stage multilayer perceptron model.

[0022] In some embodiments, the two-stage multilayer perceptron model includes a first-stage multilayer perceptron model and a second-stage multilayer perceptron model; wherein...

[0023] The first-stage multilayer perceptron model is used to map pixels from the real camera to the virtual camera;

[0024] The second-stage multilayer perceptron model is used to map pixels from the virtual camera to the real camera.

[0025] In some embodiments, training the two-stage multilayer perceptron model based on the training dataset to obtain the trained model includes:

[0026] Based on the two-stage multilayer perceptron model, the first-stage multilayer perceptron model and the second-stage multilayer perceptron model are obtained.

[0027] Input the training dataset into the first-stage multilayer perceptron model and output the normalized pixel coordinates in the virtual camera view;

[0028] A weighted projection angle error loss term is constructed to constrain the training of the first-stage multilayer perceptron model;

[0029] Input the normalized pixel coordinates in the virtual camera view into the second-stage multilayer perceptron model, and output the predicted normalized pixel coordinates in the real camera view;

[0030] An absolute pixel error loss term is constructed to constrain the training of the second-stage multilayer perceptron model;

[0031] The two-stage multilayer perceptron model is optimized based on the weighted projection angle error loss term and the absolute pixel error loss term to obtain the trained model.

[0032] In some embodiments, optimizing the two-stage multilayer perceptron model based on the weighted projection angle error loss term and the absolute pixel error loss term to obtain the trained model includes:

[0033] The target loss function is set based on a weighted linear combination of the weighted projection angle error loss term and the absolute pixel error loss term;

[0034] The first-stage multilayer perceptron model and the second-stage multilayer perceptron model are optimized according to the objective loss function to obtain the trained model.

[0035] In a second aspect, embodiments of the present invention provide a virtual camera projection system, comprising:

[0036] The sampling module is used to construct a training dataset based on sampling points in three-dimensional space.

[0037] The model building module is used to model the projection and back projection between the real camera view and the virtual camera view to obtain a two-stage multilayer perceptron model.

[0038] The model training module is used to train the two-stage multilayer perceptron model based on the training dataset to obtain the trained model.

[0039] The pixel mapping module is used to obtain a pixel mapping table between the virtual camera view and the real camera view based on the trained model.

[0040] Thirdly, embodiments of the present invention provide an electronic device, including:

[0041] One or more processors;

[0042] Memory, used to store one or more programs;

[0043] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the methods described above.

[0044] Fourthly, embodiments of the present invention provide a computer-readable medium on which a computer program is stored, the computer program being executed by a processor to implement the steps of any of the methods described above.

[0045] The virtual camera projection method provided by this invention includes: constructing a training dataset based on sampling points in three-dimensional space; modeling the projection and back-projection between the real camera view and the virtual camera view to obtain a two-stage multilayer perceptron model; training the two-stage multilayer perceptron model based on the training dataset to obtain a trained model; and obtaining a pixel mapping table between the virtual camera view and the real camera view based on the trained model. This invention, by employing a two-stage multilayer perceptron model to construct a reversible pixel mapping relationship between the real camera and the virtual camera, can uniformly convert images under any set of camera parameters to virtual camera parameters. Furthermore, it efficiently generates high-quality labeled data through an automatic annotation model, significantly improving data utilization and the generalization ability of the automatic annotation model. This allows the model to directly process multi-source heterogeneous data and reduces model adaptation costs. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating a virtual camera projection method provided in an embodiment of the present invention.

[0047] Figure 2 This is a schematic diagram of the implementation steps involved in the embodiments of the present invention;

[0048] Figure 3 This is a schematic diagram of the process of training a two-stage multilayer perceptron model in an embodiment of the present invention;

[0049] Figure 4 A structural block diagram of a virtual camera projection system provided in an embodiment of the present invention;

[0050] Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0051] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0052] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0053] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0054] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0055] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0056] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0057] The key terms involved in this invention are defined as follows:

[0058] MLP: Multilayer Perceptron.

[0059] In related technologies, pure visual object detection is a core component of autonomous driving technology. In recent years, deep learning-based methods have dominated this field, gradually achieving accurate detection of multi-scale and multi-morphological targets by constructing complex convolutional neural network architectures. To ensure the diversity and consistency of training datasets while reducing the high manpower and time costs associated with dataset annotation, large-scale fleet data collection and automatic annotation models are widely used.

[0060] However, in practical applications, the performance of automatic annotation models is often significantly constrained by the heterogeneity of the acquisition devices. Because cameras on different acquisition vehicles vary in focal length, mounting pose, and imaging parameters, the raw image data often exhibits differences in features such as viewpoint shift, scale inconsistency, and geometric deformation. This misalignment in data distribution leads to problems such as decreased detection accuracy and unstable annotation results in cross-device scenarios. To improve model robustness, traditional solutions typically rely on data cleaning for imaging results from different cameras or independently training multiple device-specific models. However, this not only reduces data utilization and increases model maintenance costs but also makes it difficult to adapt to the complex scenarios of large-scale, dynamic data acquisition by convoys. When the number of acquisition vehicles increases or devices are upgraded, the system faces unacceptable adaptation pressure. Therefore, how to achieve geometrically consistent alignment of data across devices while ensuring annotation quality and model scalability has become a core problem that urgently needs to be addressed in the practical application of pure vision-based automatic annotation models.

[0061] The limitations of existing image projection methods are analyzed as follows: On the one hand, one image processing method includes: acquiring a first target image captured by a target camera; performing a viewpoint transformation on the first target image based on the camera parameters of the target camera to obtain a second target image from the target viewpoint; inputting the second target image into a pre-generated image processing model for processing; the image processing model is a model pre-trained based on sample images from the target viewpoint. This allows for viewpoint transformation of the captured image before image processing, unifying different camera viewpoints to the target viewpoint. The image processing model only needs to process images from this target viewpoint, and during the training phase, it can be trained using only sample images from this target viewpoint, reducing the workload of sample acquisition and improving the accuracy and reliability of the trained model. However, the viewpoint transformation in this scheme is based on camera intrinsic and extrinsic parameters or a viewpoint transformation matrix, mapping two-dimensional pixels to three-dimensional space and then converting them back to two-dimensional space to obtain the correspondence between real and virtual pixels. The entire process explicitly assumes uniform depth information, which is unsuitable for scenarios with varying target depths in autonomous driving scenarios.

[0062] On the other hand, one object detection method involves: responding to vehicle startup, controlling a camera to capture an image of the road where the vehicle is currently located; this image serves as the initial image. With a virtual camera enabled, pixel mapping of the initial image is performed based on a homography matrix to obtain a virtual camera image. This virtual camera image is then input into a pre-trained first detection model for detection processing, yielding the first object detection result. However, the image pixel mapping process in this implementation has limited applicability. This scheme uses a homography matrix to map pixels, relying on a planar assumption. However, in actual autonomous driving, the road environment contains non-planar objects (such as vehicles, pedestrians, and traffic lights), causing deformation errors in the mapped target. Furthermore, this scheme performs homography matrix calculations for different vehicles during the vehicle startup phase. As the data acquisition scale increases, the time consumed in this phase is not reduced, and the calculation of high-precision homography matrices also consumes considerable time.

[0063] To address at least one of the technical problems existing in the aforementioned related technologies, the present invention provides a virtual camera projection method. Figure 1 This is a flowchart illustrating a virtual camera projection method provided in an embodiment of the present invention.

[0064] As an embodiment of the present invention, such as Figure 1 As shown, the virtual camera projection method includes:

[0065] Step S1: Construct a training dataset based on sampling points in three-dimensional space;

[0066] Step S2: Model the projection and back projection between the real camera view and the virtual camera view to obtain a two-stage multilayer perceptron model;

[0067] Step S3: Train the two-stage multilayer perceptron model based on the training dataset to obtain the trained model;

[0068] Step S4: Obtain a pixel mapping table between the virtual camera view and the real camera view based on the trained model.

[0069] It should be noted that the execution subject in this embodiment can be an electronic device, which can be a computer device with data processing function, or other devices that can achieve the same or similar functions. This embodiment does not limit this. In this embodiment, the execution subject is a computer device as an example for explanation.

[0070] Specifically, to address the issues of low data utilization and decreased generalization performance of automatic annotation models caused by differences in multi-camera parameters, this embodiment proposes a virtual camera projection method. This method is based on a two-stage multilayer perceptron, and its core lies in constructing a reversible pixel mapping relationship between the real camera and the virtual camera. The virtual camera is a set of sensors set by the user, and its intrinsic and extrinsic parameters can be arbitrarily set according to requirements. Due to its high flexibility, the view from the real camera can be uniformly transformed to the view from the virtual camera, thus achieving geometric consistency alignment of data across devices. This embodiment proposes to use a two-stage multilayer perceptron model to construct the reversible pixel mapping. The following describes the specific steps.

[0071] In some embodiments, constructing a training dataset based on sampling points in three-dimensional space includes: uniformly sampling three-dimensional spatial points in three-dimensional space to obtain sampling points; projecting the sampling points onto two-dimensional space and performing filtering processing to obtain effective two-dimensional pixels and projection angles; normalizing the effective two-dimensional pixels to obtain normalized effective two-dimensional pixels; and constructing a training dataset based on the normalized effective two-dimensional pixels and projection angles.

[0072] In some embodiments, the sampling points are projected onto a two-dimensional space and filtered to obtain valid two-dimensional pixels and projection angles, including: projecting the sampling points onto a two-dimensional space based on the intrinsic and extrinsic parameters of a real camera; filtering out valid two-dimensional pixels and corresponding three-dimensional spatial points based on the resolution of the real camera; projecting the filtered three-dimensional spatial points onto a virtual two-dimensional space based on the intrinsic and extrinsic parameters of a preset virtual camera, and calculating the projection angle.

[0073] In some embodiments, the method further includes: calculating the distance from a point in three-dimensional space to the origin of the coordinate system; and using the distance as a weight for loss calculation during the training of a two-stage multilayer perceptron model.

[0074] Specifically, such as Figure 2 As shown, the training dataset is created as follows: A large number of 3D spatial points are uniformly sampled within a specified range of 3D space. These sampling points are projected onto a 2D space using the intrinsic and extrinsic parameters of a real camera. Based on the camera resolution, valid 2D pixels are selected to correspond to corresponding 3D spatial points, ensuring these valid 2D pixels cover as many pixel locations as possible. The selected 3D spatial points are then projected onto a virtual 2D space using the given virtual camera intrinsic and extrinsic parameters, and the projection angle is calculated. The normalized valid 2D pixels and their projection angles constitute the training dataset. Simultaneously, the distance from each 3D spatial point to the origin of the coordinate system is calculated as a weight in the loss calculation.

[0075] For example, within a defined 3D spatial region, many points are selected according to certain rules (e.g., uniform distribution), and a large number of 3D spatial points are uniformly sampled. These points will be used as input data for training. Using the intrinsic and extrinsic parameters of a real camera, these 3D spatial points are transformed onto a 2D image plane through perspective projection, mapping the 3D points to corresponding 2D pixels. Using the intrinsic and extrinsic parameters of the real camera (intrinsic parameters include focal length, imaging plane size, etc., extrinsic parameters include position and orientation, etc.), the points in 3D space are geometrically transformed and mapped onto the 2D image plane. Based on the camera's resolution, pixels that can be mapped onto the camera's imaging plane are selected, i.e., those points within the camera's field of view and not exceeding the boundaries of the imaging plane. When selecting effective 2D pixels, the goal is to make these points cover the entire imaging plane as uniformly as possible, so that the trained model can handle any position within the camera's field of view. Using the intrinsic and extrinsic parameters of a virtual camera, the selected 3D spatial points are projected again, this time in a virtual 2D space. Simultaneously, the angles of these points during the projection process are calculated, which is important for understanding the mapping relationship from 3D space to a 2D image. The selected valid two-dimensional pixels and their projection angles are normalized and then combined into a training dataset. Normalization helps to accelerate the training process and improve the model's generalization ability.

[0076] It should be noted that, in order to give different levels of importance to different sample points during the training process, the distance from each three-dimensional space point to the origin of the coordinate system is calculated and used as the weight when calculating the loss function.

[0077] In some embodiments, the two-stage multilayer perceptron model includes a first-stage multilayer perceptron model and a second-stage multilayer perceptron model; wherein, the first-stage multilayer perceptron model is used to map pixels in a real camera to a virtual camera; and the second-stage multilayer perceptron model is used to map pixels in a virtual camera to a real camera.

[0078] Specifically, a multilayer perceptron (MLP) is a type of feedforward neural network that can consist of multiple layers (including an input layer, hidden layers, and an output layer), connected by activation functions. In this embodiment, the MLP is used to establish a mapping relationship from real camera pixels to virtual camera pixels.

[0079] In this embodiment, a multilayer perceptron model is proposed to model the mapping relationship from real camera pixels to virtual camera pixels, implicitly representing the depth information of the target, and a weighted projection angle error loss constraint network is constructed for training. Through the multilayer perceptron model, the mapping relationship from real camera pixels to virtual camera pixels can be learned. Since this mapping relationship includes the projection process from 3D space to 2D space, it implicitly contains the depth information of the target. Although the model itself does not directly output depth values, the mapping relationship learned by the model can be used to infer the depth information of points in 3D space.

[0080] In some embodiments, training the two-stage multilayer perceptron model based on the training dataset to obtain the trained model includes: obtaining a first-stage multilayer perceptron model and a second-stage multilayer perceptron model based on the two-stage multilayer perceptron model; inputting the training dataset into the first-stage multilayer perceptron model and outputting normalized pixel coordinates in the virtual camera view; constructing a weighted projection angle error loss term to constrain the training of the first-stage multilayer perceptron model; inputting the normalized pixel coordinates in the virtual camera view into the second-stage multilayer perceptron model and outputting the predicted normalized pixel coordinates in the real camera view; constructing an absolute pixel error loss term to constrain the training of the second-stage multilayer perceptron model; and optimizing the two-stage multilayer perceptron model based on the weighted projection angle error loss term and the absolute pixel error loss term to obtain the trained model.

[0081] In some embodiments, optimizing the two-stage multilayer perceptron model based on the weighted projection angle error loss term and the absolute pixel error loss term to obtain a trained model includes: setting a target loss function based on a weighted linear combination of the weighted projection angle error loss term and the absolute pixel error loss term; and optimizing the first-stage multilayer perceptron model and the second-stage multilayer perceptron model based on the target loss function to obtain a trained model.

[0082] Specifically, such as Figure 2 As shown, a two-stage multilayer perceptron model is constructed and trained. Figure 3As shown, the first-stage model (first-stage MLP model) is responsible for mapping pixels from the real camera to the virtual camera. The input information is the normalized coordinates of valid 2D pixels selected during the training dataset creation process. The direct output of the first-stage model is the normalized pixel coordinates in the virtual camera view. Further, a new projection angle is calculated based on the virtual camera's intrinsic and extrinsic parameters, and the absolute error is calculated using the label information. This is combined with weights to obtain a weighted projection angle error loss term. The second-stage model (second-stage MLP model) is responsible for mapping pixels from the virtual camera to the real camera. The input information is the output of the first-stage model. The second-stage model predicts the normalized pixel coordinates in the real camera view and constructs an absolute pixel error to constrain the consistency between the predicted pixels and the real pixels. Finally, the loss function is set as a weighted linear combination of the two loss terms to simultaneously optimize the models in both stages.

[0083] It is understood that, in addition to absolute error, this embodiment may also use squared error or other commonly used error functions in regression tasks. This embodiment does not impose any restrictions on this.

[0084] For example, the first-stage model takes the selected, normalized, valid 2D pixel coordinates (normalized dense 2D pixel coordinates) as input. The output is the normalized pixel coordinates in the virtual camera view, which are the pixel positions predicted by the first-stage model on the virtual camera's imaging plane. The loss function calculates the absolute error between the model-predicted pixel coordinates and the actual pixel coordinates under the virtual camera's intrinsic and extrinsic parameters. Additionally, the absolute error between the projection angle corresponding to the model-predicted pixel coordinates and the actual projection angle is calculated and weighted to obtain a weighted projection angle error loss term. These loss terms are combined to evaluate the model's performance in the first stage.

[0085] For example, the second-stage model takes the output of the first-stage model as input, which is the normalized pixel coordinates in the virtual camera view. The output is the normalized pixel coordinates in the real camera view (predicted normalized 2D pixel coordinates), which are the pixel positions predicted by the model on the real camera's imaging plane. The loss function is constructed as the absolute pixel error, the absolute error between the model's predicted pixel coordinates and the real pixel coordinates. This error is used to evaluate the model's performance in the second stage, ensuring that the model can correctly map pixels from the virtual camera view back to the real camera view.

[0086] It should be noted that in this embodiment, the loss terms of the first and second stages are weighted and linearly combined to form a composite loss function. Optimization algorithms such as gradient descent can be used to simultaneously optimize the model parameters of both stages, minimizing the composite loss function and achieving high accuracy for the model on both stages of the mapping task.

[0087] Understandably, to more accurately guide network optimization, this embodiment constructs a weighted projection angle error to ensure the stability of the image structure for the first-stage projection model (first-stage MLP model). For the second-stage back-projection model (second-stage MLP model), this embodiment sets a reprojection pixel error between the actual camera views to ensure the consistency between the reconstructed image and the real image. Finally, a composite loss function is formed to achieve end-to-end synchronous optimization of the two-stage models. Based on the pixel mapping relationship given by the back-projection model, images under any set of camera parameters can be uniformly converted to virtual camera parameters, thereby efficiently generating high-quality labeled data through the automatic annotation model. This embodiment significantly improves data utilization and the generalization ability of the automatic annotation model, enabling the model to directly process multi-source heterogeneous data and reducing model adaptation costs.

[0088] In some embodiments, a pixel mapping table between the virtual camera view and the real camera view is obtained based on the trained model.

[0089] Specifically, such as Figure 2 As shown, the correspondence between pixels in the virtual view and pixels in the real view is obtained based on the second-stage model. For each pixel in the virtual view, its pixel coordinates are normalized and then fed into the trained second-stage model (the trained second-stage MLP model) for forward inference to obtain the normalized pixel coordinates in the real view. These coordinates are then restored to the image scale and rounded according to the image resolution to obtain the pixel mapping table. For each pixel in the virtual view, if the corresponding real view pixel is within the image range, its pixel value is the same as the corresponding pixel value in the real view; otherwise, it is set to 0.

[0090] For example, a reversible mapping from the real camera to the virtual camera and back is achieved through a trained model (a trained two-stage multilayer perceptron model). This mapping (pixel mapping table) is highly beneficial for geometric consistency alignment of data across devices, allowing image data from different camera settings to be compared and analyzed in a unified virtual camera coordinate system.

[0091] This embodiment proposes a virtual camera projection method based on a two-stage multilayer perceptron. Given the intrinsic and extrinsic parameters of the virtual camera, this method can transform an image from any real camera view to the virtual camera view. This method improves the generalization ability of the automatic annotation model under different camera parameters, and enhances data utilization and model robustness.

[0092] It should be noted that this embodiment designs a pixel mapping method through steps of creating a training dataset and constructing and training a two-stage multilayer perceptron model. It utilizes two multilayer perceptron models to model the projection and back-projection between the real camera view and the virtual camera view. A training dataset is constructed using densely sampled points in 3D space. A joint loss function is built to simultaneously optimize the two models, resulting in a high-precision nonlinear mapping relationship (pixel mapping table). By obtaining the correspondence between pixels in the virtual view and pixels in the real view based on the second-stage model, a pixel mapping table between the virtual and real camera views can be obtained before device operation, thereby improving the efficiency of multi-camera view conversion during actual operation.

[0093] The virtual camera projection method provided in this embodiment includes: constructing a training dataset based on sampling points in three-dimensional space; modeling the projection and back-projection between the real camera view and the virtual camera view to obtain a two-stage multilayer perceptron model; training the two-stage multilayer perceptron model based on the training dataset to obtain a trained model; and obtaining a pixel mapping table between the virtual camera view and the real camera view based on the trained model. This embodiment, by employing a two-stage multilayer perceptron model to construct a reversible pixel mapping relationship between the real camera and the virtual camera, can uniformly convert images under any set of camera parameters to virtual camera parameters. Furthermore, it efficiently generates high-quality labeled data through an automatic annotation model, significantly improving data utilization and the generalization ability of the automatic annotation model. This allows the model to directly process multi-source heterogeneous data, reducing model adaptation costs.

[0094] Reference Figure 4 , Figure 4 This is a structural block diagram of an embodiment of the virtual camera projection system of the present invention. Figure 4 As shown, the virtual camera projection system includes:

[0095] Sampling module 10 is used to construct a training dataset based on sampling points in three-dimensional space;

[0096] Model building module 20 is used to model the projection and back projection between the real camera view and the virtual camera view to obtain a two-stage multilayer perceptron model.

[0097] Model training module 30 is used to train the two-stage multilayer perceptron model based on the training dataset to obtain the trained model;

[0098] The pixel mapping module 40 is used to obtain a pixel mapping table between the virtual camera view and the real camera view based on the trained model.

[0099] The virtual camera projection system provided in this embodiment constructs a reversible pixel mapping relationship between the real camera and the virtual camera using a two-stage multilayer perceptron model. This allows images under any set of camera parameters to be uniformly converted to virtual camera parameters. Furthermore, the system efficiently generates high-quality labeled data through an automatic labeling model, significantly improving data utilization and the generalization ability of the automatic labeling model. This enables the model to directly process multi-source heterogeneous data and reduces model adaptation costs.

[0100] In addition, for technical details not described in detail in this virtual camera projection system embodiment, please refer to the virtual camera projection method provided in any embodiment of the present invention, which will not be repeated here.

[0101] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Figure 5 As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement any of the virtual camera projection methods described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0102] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0103] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0104] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0105] This invention also provides a computer-readable medium. The computer-readable medium stores a computer program, which, when executed by a processor, implements the steps of any of the virtual camera projection methods described in the above embodiments. The computer-readable storage medium may be volatile or non-volatile.

[0106] This invention also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described virtual camera projection method.

[0107] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0108] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0109] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0110] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0111] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0112] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0113] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0114] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0115] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0116] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A virtual camera projection method, characterized in that, include: A training dataset is constructed based on sampling points in three-dimensional space; Modeling the projection and back projection between the real camera view and the virtual camera view yields a two-stage multilayer perceptron model. The two-stage multilayer perceptron model is trained based on the training dataset to obtain the trained model. The trained model is used to obtain a pixel mapping table between the virtual camera view and the real camera view.

2. The method according to claim 1, characterized in that, The construction of the training dataset based on sampling points in three-dimensional space includes: The sampling points are obtained by uniformly sampling points in three-dimensional space. The sampling points are projected into a two-dimensional space and filtered to obtain effective two-dimensional pixels and projection angles. The effective two-dimensional pixels are normalized to obtain normalized effective two-dimensional pixels; The training dataset is constructed based on the normalized effective two-dimensional pixels and the projection angle.

3. The method according to claim 2, characterized in that, The step of projecting the sampling points into a two-dimensional space and performing filtering to obtain effective two-dimensional pixels and projection angles includes: The sampling points are projected into a two-dimensional space based on the intrinsic and extrinsic parameters of a real camera; Valid two-dimensional pixels and their corresponding three-dimensional spatial points are selected based on the resolution of the actual camera. Based on the preset intrinsic and extrinsic parameters of the virtual camera, the selected three-dimensional spatial points are projected onto a virtual two-dimensional space, and the projection angle is calculated.

4. The method according to claim 3, characterized in that, The method further includes: Calculate the distance from a point in three-dimensional space to the origin of the coordinate system; The distance is used as a weight for loss calculation during the training of the two-stage multilayer perceptron model.

5. The method according to claim 1, characterized in that, The two-stage multilayer perceptron model includes a first-stage multilayer perceptron model and a second-stage multilayer perceptron model; wherein... The first-stage multilayer perceptron model is used to map pixels from the real camera to the virtual camera; The second-stage multilayer perceptron model is used to map pixels from the virtual camera to the real camera.

6. The method according to claim 1, characterized in that, The step of training the two-stage multilayer perceptron model based on the training dataset to obtain the trained model includes: Based on the two-stage multilayer perceptron model, the first-stage multilayer perceptron model and the second-stage multilayer perceptron model are obtained. Input the training dataset into the first-stage multilayer perceptron model and output the normalized pixel coordinates in the virtual camera view; A weighted projection angle error loss term is constructed to constrain the training of the first-stage multilayer perceptron model; Input the normalized pixel coordinates in the virtual camera view into the second-stage multilayer perceptron model, and output the predicted normalized pixel coordinates in the real camera view; An absolute pixel error loss term is constructed to constrain the training of the second-stage multilayer perceptron model; The two-stage multilayer perceptron model is optimized based on the weighted projection angle error loss term and the absolute pixel error loss term to obtain the trained model.

7. The method according to claim 6, characterized in that, The step of optimizing the two-stage multilayer perceptron model based on the weighted projection angle error loss term and the absolute pixel error loss term to obtain the trained model includes: The target loss function is set based on a weighted linear combination of the weighted projection angle error loss term and the absolute pixel error loss term; The first-stage multilayer perceptron model and the second-stage multilayer perceptron model are optimized according to the objective loss function to obtain the trained model.

8. A virtual camera projection system, characterized in that, include: The sampling module is used to construct a training dataset based on sampling points in three-dimensional space. The model building module is used to model the projection and back projection between the real camera view and the virtual camera view to obtain a two-stage multilayer perceptron model. The model training module is used to train the two-stage multilayer perceptron model based on the training dataset to obtain the trained model. The pixel mapping module is used to obtain a pixel mapping table between the virtual camera view and the real camera view based on the trained model.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images