A method, device and equipment for automatically labeling a virtual image

Abstract

Embodiments of the present application provide a method, device and equipment for automatic labeling of virtual images. A virtual scene containing a calibration board and a virtual camera is set. Since the virtual scene is pre-set, the spatial positions of feature points in the calibration board in the virtual scene can be determined. In the process of determining a corrected image according to a target parameter, i.e. the process of light ray tracing rendering, a photo-level calibration board image of the real camera distortion effect can be obtained, and automatic labeling of the feature points can be realized. Training samples are generated based on the target image and the spatial positions of the feature points in the virtual scene to train an original camera calibration model, i.e. the photo-level image and the labeling information are jointly applied to model training. Manual labeling processing of the collected images is not required, the manual calibration cost is saved, the efficiency of virtual image labeling is improved, a large number of training samples are generated without consuming manpower and resources, and the accuracy of the camera calibration model is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence application technology, and in particular to a method, apparatus and device for automatic annotation of virtual images. Background Technology

[0002] In some applications, we need to determine the locations of corner points on an image calibration board. For example, a neural network needs to be trained. During training, the input includes labeled images and training samples containing the spatial locations of the calibration board corner points. The output is the camera's intrinsic and extrinsic parameters. To accurately train a neural network, we need a large number of training samples. In current technology, images are usually labeled manually to obtain training samples. However, manual image labeling is inefficient, resulting in low efficiency in generating training samples. Summary of the Invention

[0003] The purpose of this application is to provide a method, apparatus, and device for automatic virtual image annotation, so as to improve the efficiency of virtual image annotation and generate a large number of training samples without consuming manpower and resources. The specific technical solution is as follows:

[0004] In a first aspect, embodiments of this application provide a method for automatic annotation of virtual images, the method comprising:

[0005] Obtain the target parameters of a virtual camera set in a virtual scene, wherein a calibration board is also set in the virtual scene;

[0006] Based on the target parameters, the virtual image formed by the light emitted from the calibration plate by the virtual camera is determined without lens distortion.

[0007] Based on the first coordinate transformation relationship between the first coordinate system and the second coordinate system, the virtual image is transformed to obtain a corrected image; wherein, the first coordinate system is the image coordinate system of the virtual camera without lens distortion, and the second coordinate system is the image coordinate system of the virtual camera with lens distortion.

[0008] Based on the spatial position of each feature point in the calibration board in the virtual scene, the image position of each feature point in the calibration board is marked in the corrected image to obtain the target image;

[0009] Training samples are generated based on the target image and the spatial positions of each feature point in the calibration board in the virtual scene, so that the original camera calibration model can be trained based on the training samples.

[0010] In one possible embodiment, the step of marking the image positions of each feature point in the calibration board in the corrected image according to the spatial positions of each feature point in the calibration board in the virtual scene to obtain the target image includes:

[0011] Based on the spatial coordinates of each feature point in the calibration board in the virtual scene, the second coordinate transformation relationship, and the first coordinate transformation relationship, the image position of each feature point in the calibration board is marked in the corrected image to obtain the target image; wherein, the second coordinate transformation relationship is the coordinate transformation relationship between the spatial coordinate system and the first coordinate system.

[0012] In one possible embodiment, the method further includes:

[0013] The first coordinate transformation relationship is determined based on the radial distortion and distortion radius of the virtual camera lens;

[0014] Determining the first coordinate transformation relationship based on the radial distortion and distortion radius of the virtual camera lens includes:

[0015] The x-coordinate and y-coordinate of the distorted point are calculated using the following formula:

[0016]

[0017]

[0018] Where, x d and y d Let x be the x-coordinate and y-coordinate of the distorted point. p and y p Let x be the x-coordinate and y-coordinate of the imaging point in the camera coordinate system, k1, k2, k3...k6 be the radial distortion of the virtual camera lens, p1 and p2 be the tangential distortion of the virtual camera lens, and r be the distortion radius.

[0019] In one possible embodiment, determining the virtual image formed by the light emitted from the calibration plate by the virtual camera, based on the target parameters without lens distortion, includes:

[0020] Based on the target parameters, without lens distortion, and under different light source conditions, the virtual images formed by the virtual camera sensing the light emitted by the calibration plate are determined, wherein the different light source conditions include strong light sources.

[0021] In one possible embodiment, determining the virtual image formed by the light emitted from the calibration plate by the virtual camera, based on the target parameters without lens distortion, includes:

[0022] Based on the target parameters, without lens distortion and when the calibration plate is in different states, the virtual images formed by the light emitted by the calibration plate and sensed by the virtual camera are determined. The different states of the calibration plate include one of the following: calibration plate wrinkles, damage, and stains.

[0023] In one possible embodiment, the virtual scene further includes a car model, and the virtual camera is a fisheye camera mounted on the car model.

[0024] Secondly, embodiments of this application provide a virtual image automatic annotation device, the device comprising:

[0025] A camera parameter acquisition module is used to acquire target parameters of a virtual camera set in a virtual scene, wherein a calibration board is also set in the virtual scene;

[0026] The virtual image determination module is used to determine, based on the target parameters, the virtual image formed by the light emitted by the calibration plate sensed by the virtual camera, without lens distortion;

[0027] The image coordinate transformation module is used to perform coordinate transformation on the virtual image according to the first coordinate transformation relationship between the first coordinate system and the second coordinate system to obtain a corrected image; wherein, the first coordinate system is the image coordinate system of the virtual camera without lens distortion, and the second coordinate system is the image coordinate system of the virtual camera with lens distortion.

[0028] The image location annotation module is used to annotate the image location of each feature point in the calibration board in the corrected image according to the spatial location of each feature point in the calibration board in the virtual scene, so as to obtain the target image;

[0029] The calibration model training module is used to generate training samples based on the target image and the spatial positions of each feature point in the calibration board in the virtual scene, so as to train the original camera calibration model based on the training samples.

[0030] In one possible embodiment, the image location annotation module includes:

[0031] The image location annotation submodule annotates the image location of each feature point in the calibration board in the virtual scene according to the spatial coordinates of each feature point in the calibration board, the second coordinate transformation relationship, and the first coordinate transformation relationship, thereby obtaining the target image; wherein, the second coordinate transformation relationship is the coordinate transformation relationship between the spatial coordinate system and the first coordinate system;

[0032] In one possible embodiment, the device further includes:

[0033] The transformation relationship determination module is used to determine the first coordinate transformation relationship based on the radial distortion and distortion radius of the virtual camera lens;

[0034] The conversion relationship determination module includes:

[0035] The transformation relationship determination submodule is used to calculate the x-coordinate and y-coordinate of the distorted point according to the following formula:

[0036]

[0037]

[0038] Where, x d and y d Let x be the x-coordinate and y-coordinate of the distorted point. p and y p Let x be the x-coordinate and y-coordinate of the imaging point in the camera coordinate system, k1, k2, k3...k6 be the radial distortion of the virtual camera lens, p1 and p2 be the tangential distortion of the virtual camera lens, and r be the distortion radius.

[0039] In one possible embodiment, the virtual image determination module includes:

[0040] The virtual image determination first submodule is used to determine, based on the target parameters, various virtual images formed by the light emitted by the calibration plate sensed by the virtual camera under different light source conditions, without lens distortion, and for different light source conditions, wherein the different light source conditions include strong light sources;

[0041] In one possible embodiment, the virtual image determination module includes:

[0042] The virtual image determination second submodule is used to determine, based on the target parameters, various virtual images formed by the light emitted by the calibration plate by the virtual camera when no lens distortion occurs and the calibration plate is in different states, wherein the different states of the calibration plate include one of calibration plate wrinkles, damage and stains;

[0043] In one possible embodiment, the virtual scene further includes a car model, and the virtual camera is a fisheye camera mounted on the car model.

[0044] Thirdly, an electronic device is provided, including a processor and a machine-readable storage medium storing machine-executable instructions that can be executed by the processor, the processor being prompted by the machine-executable instructions to implement any of the above-described methods for automatic annotation of virtual images.

[0045] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements any of the above-described methods for automatic annotation of virtual images.

[0046] Beneficial effects of the embodiments in this application:

[0047] This application provides a method for automatic virtual image annotation. The method involves acquiring target parameters of a virtual camera set in a virtual scene, where a calibration board is also provided. Based on the target parameters, a virtual image formed by the light emitted from the calibration board by the virtual camera is determined without lens distortion. The virtual image is then transformed according to a first coordinate transformation relationship between a first coordinate system and a second coordinate system to obtain a corrected image. The first coordinate system includes the image coordinate system of the virtual camera without lens distortion, and the second coordinate system includes the image coordinate system of the virtual camera with lens distortion. Based on the spatial positions of each feature point on the calibration board in the virtual scene, the image positions of each feature point on the calibration board are marked in the corrected image to obtain a target image. Training samples are generated based on the target image and the spatial positions of each feature point on the calibration board in the virtual scene, so that an original camera calibration model can be trained based on the training samples. By applying the scheme of this application, a virtual scene and a virtual camera within the virtual scene are set up. The virtual scene also includes a calibration board. Because the virtual scene is pre-set, the spatial position of feature points on the calibration board within the virtual scene can be clearly obtained. During the process of determining the corrected image based on the target parameters, i.e., during ray tracing rendering, a photorealistic calibration board image with realistic camera distortion effects can be obtained. The transformation relationship between the virtual spatial position and the image coordinate system in the camera can also be obtained. Therefore, the coordinate position of the feature point in the image can be directly calculated based on the spatial coordinates of the feature point, i.e., converting three-dimensional coordinates into two-dimensional coordinates, thus achieving automated annotation of feature points. The original camera calibration model is trained by generating training samples based on the spatial positions of each feature point in the target image and calibration board in the virtual scene. This means that photorealistic images and annotation information are used together in model training, eliminating the need for manual annotation of the acquired images, saving manual calibration costs, improving the efficiency of virtual image annotation, generating a large number of training samples without consuming manpower and resources, and reducing the errors caused by manual image annotation compared with manually annotated images, thereby reducing the mislabeling rate and improving the accuracy of the camera calibration model.

[0048] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0050] Figure 1a A schematic diagram of a correctly stitched panoramic bird's-eye view provided for an embodiment of this application;

[0051] Figure 1b An incorrectly stitched panoramic bird's-eye view provided for an embodiment of this application;

[0052] Figure 2 A flowchart illustrating an automatic virtual image annotation method provided in this application embodiment;

[0053] Figure 3a A schematic diagram of a car model provided for an embodiment of this application;

[0054] Figure 3b A schematic diagram of camera parameters provided for an embodiment of this application;

[0055] Figure 3c A schematic diagram of a calibration board model provided in an embodiment of this application;

[0056] Figure 3d A schematic diagram illustrating the placement of a car model as provided in an embodiment of this application;

[0057] Figure 4a This is a first schematic diagram of a virtual image provided in an embodiment of this application;

[0058] Figure 4b This is a second schematic diagram of a virtual image provided in an embodiment of this application;

[0059] Figure 4c This is a third schematic diagram of a virtual image provided in an embodiment of this application;

[0060] Figure 5a This is a fourth schematic diagram of a virtual image provided in an embodiment of this application;

[0061] Figure 5b A schematic diagram of a corrected image provided in an embodiment of this application;

[0062] Figure 6a This is a first schematic diagram of a target image provided in an embodiment of this application;

[0063] Figure 6b This is a second schematic diagram of a target image provided in an embodiment of this application;

[0064] Figure 7 A schematic diagram illustrating the steps of an automatic virtual image annotation method provided in an embodiment of this application;

[0065] Figure 8A schematic diagram of the structure of a device for automatic virtual image annotation provided in an embodiment of this application;

[0066] Figure 9 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0067] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0068] To more clearly illustrate the method for automatic virtual image annotation provided in this application, the following will provide an exemplary description of a possible application scenario of the method. It is understood that the following example is only one possible application scenario of the method for automatic virtual image annotation provided in this application. In other possible embodiments, the method for automatic virtual image annotation provided in this application can also be applied to other possible embodiments. The following example does not impose any limitations on this.

[0069] To achieve autonomous or assisted driving capabilities, vehicles equipped with these functions need to accurately identify traffic markings on the surrounding ground, such as zebra crossings and turn arrows. Therefore, four surround-view cameras can be installed in the front, rear, left, and right rearview mirrors. During vehicle movement, each camera collects image data, and a bird's-eye view is generated based on this data. These bird's-eye views are then stitched together according to the intrinsic and extrinsic parameters of each camera to obtain a final image. Figure 1a The panoramic bird's-eye view shown below Figure 1a This is a schematic diagram of a correctly stitched panoramic bird's-eye view provided for an embodiment of this application.

[0070] However, if the intrinsic and extrinsic parameters of each camera cannot be accurately determined, it will be impossible to accurately stitch the various bird's-eye view images into a panoramic bird's-eye view. There may be some offset at the stitching points between the images, resulting in an image like... Figure 1b The panoramic bird's-eye view shown below Figure 1b This is a poorly stitched panoramic bird's-eye view provided for an embodiment of this application. Figure 1b As can be seen, the zebra crossing in the stitched panoramic view is broken. This prevents vehicles from accurately recognizing traffic signs on the ground, making it difficult or even impossible to effectively achieve autonomous or assisted driving functions.

[0071] To accurately determine the intrinsic and extrinsic parameters of each camera, one possible approach is to use a trained neural network. The input to this network consists of training samples containing labeled images and the spatial locations of calibration board corner points, and the output is the camera's intrinsic and extrinsic parameters. Training such a neural network requires a large number of training samples labeled with the camera's true intrinsic and extrinsic parameters. However, manually labeling corner points is inefficient, resulting in low efficiency in generating training samples.

[0072] Based on this, this application provides a method for automatic annotation of virtual images, such as... Figure 2 As shown, Figure 2 A flowchart of a virtual image automatic annotation method provided in this application embodiment includes:

[0073] S201, Obtain the target parameters of the virtual camera set in the virtual scene, where a calibration board is also set.

[0074] S202, Based on the target parameters, determine the virtual image formed by the light emitted by the virtual camera sensor calibration board without lens distortion.

[0075] S203, according to the first coordinate transformation relationship between the first coordinate system and the second coordinate system, the virtual image is transformed to obtain the corrected image; wherein, the first coordinate system is the image coordinate system of the virtual camera when no lens distortion occurs, and the second coordinate system is the image coordinate system of the virtual camera when lens distortion occurs.

[0076] S204. Based on the spatial position of each feature point in the calibration board in the virtual scene, mark the image position of each feature point in the calibration board in the corrected image to obtain the target image.

[0077] S205, generate training samples based on the target image and the spatial positions of each feature point in the calibration board in the virtual scene, so as to train the original camera calibration model based on the training samples.

[0078] In this embodiment, a virtual scene and a virtual camera within that scene are set up. The virtual scene also includes a calibration board. Because the virtual scene is pre-set, the spatial positions of feature points on the calibration board within the virtual scene can be clearly determined. During the process of determining the corrected image based on the target parameters, i.e., during ray tracing rendering, a photorealistic calibration board image with realistic camera distortion effects can be obtained. The transformation relationship between the virtual spatial position and the image coordinate system in the camera can also be obtained. Therefore, the coordinate position of the feature point in the image can be directly calculated based on its spatial coordinates, i.e., converting three-dimensional coordinates into two-dimensional coordinates, thus achieving automated annotation of feature points. The original camera calibration model is trained by generating training samples based on the spatial positions of each feature point in the target image and calibration board in the virtual scene. This means that photorealistic images and annotation information are used together in model training, eliminating the need for manual annotation of the acquired images, saving manual calibration costs, improving the efficiency of virtual image annotation, generating a large number of training samples without consuming manpower and resources, and reducing the errors caused by manual image annotation compared with manually annotated images, thereby reducing the mislabeling rate and improving the accuracy of the camera calibration model.

[0079] The following will provide a detailed explanation of the aforementioned S201-S205:

[0080] For ease of understanding, the coordinate systems appearing in this application will be explained below. In this application, the world coordinate system in the virtual scene is referred to as the spatial coordinate system, the image coordinate system of the virtual camera without lens distortion is referred to as the first coordinate system, and the image coordinate system of the virtual camera with lens distortion is referred to as the second coordinate system.

[0081] In S201, in one embodiment, a car model is set in the virtual scene, and the virtual camera is a fisheye camera set on the car model, respectively positioned in the front, rear, left, and right rearview mirrors of the car model. The car model can be virtualized by modeling it using any software with 3D modeling capabilities, such as... Figure 3a As shown, Figure 3a This is a schematic diagram of a car model provided in an embodiment of this application.

[0082] The target parameters of a virtual camera include two types: intrinsic and extrinsic parameters, such as... Figure 3b As shown, Figure 3b This is a schematic diagram of camera parameters provided for this application. The extrinsic parameters of the camera refer to its position (x, y, z) and orientation (represented by Euler angles: pitch, yaw, roll) in the virtual scene. The intrinsic parameters of the camera refer to the internal parameters of the camera itself caused by lens distortion and the imaging system, including focal length (F... x Fy The offset of the camera's optical axis in the image coordinate system (C) x C y ), radial distortion of the lens (k1, k2, k3, k4, k5, k6), tangential distortion of the lens (p1, p2).

[0083] The virtual scene can also include other objects besides the car model and virtual camera to make it more realistic. For example, taking the fisheye camera set on the car model as an example, to make the virtual scene captured by the fisheye camera more realistic, rooms, ground, lighting, environment maps, lane lines, obstacles, etc., can also be added to the virtual scene.

[0084] The virtual scene includes calibration boards, which can be checkerboard patterns or circular grids, but are not limited to these. Different styles of calibration boards can be 3D modeled according to the requirements of different manufacturers and algorithms, and placed around the car model, maintaining the same dimensions as the real thing. Figure 3c As shown, Figure 3c This is a schematic diagram of a calibration plate model provided in an embodiment of this application. Figure 3c In the diagram, assuming the car model is 5400mm long and 2500mm wide, the area within the solid-lined frame in the left half of the image represents the area where the car model will be placed. In the right half of the image, the blank area in the middle, measuring 540cm long and 240cm wide, represents the area where the car model will be placed.

[0085] After creating the virtual scene, including the car model and calibration board model, place the car model in the center of the calibration board, such as... Figure 3d As shown, Figure 3d This is a schematic diagram illustrating the placement of a car model provided in this application.

[0086] The virtual scene projection reconstruction technology described above can be used to realize images with various virtual perspectives, including top view, front and rear corner view, forward or reverse view, left and right side view, and any virtual viewpoint view.

[0087] By using the above embodiments, the real annotation process can be simulated in a virtual scene, thereby saving manpower and material resources and improving annotation efficiency.

[0088] In S202, the light emitted by the calibration plate mentioned above is a simulation of light emitted by the calibration plate, not light actually emitted by the calibration plate. Rather, it refers to a simulation of light emitted by a light source in the virtual scene reflected by the calibration plate. The virtual image mentioned above is not an image actually captured by the virtual camera, but rather an image that the virtual camera could theoretically capture, obtained through calculation and simulation.

[0089] In one possible embodiment, the light emitted by the calibration plate can be determined based on target parameters to form various virtual images by incident on the virtual camera under different light source conditions without lens distortion, wherein the different light source conditions include strong light sources.

[0090] In another possible embodiment, it can also be based on target parameters to determine the various virtual images formed by the light emitted by the calibration plate incident on the virtual camera when there is no lens distortion and the calibration plate is in different states, wherein the different states of the calibration plate include one of calibration plate wrinkles, damage and stains.

[0091] To simulate various situations that might occur in real-world scenarios, special effects can be added during the process of determining the virtual image based on the virtual camera's intrinsic and extrinsic parameters. This allows for the creation of different virtual images formed when light emitted from the calibration board is incident on the virtual camera under different lighting conditions and / or when the calibration board is in different states, without lens distortion. The virtual scene can be positioned under different lighting conditions by adjusting the parameters of the lights within the virtual scene.

[0092] In practical applications, different light source conditions can include strong light sources. Under strong light source conditions, the resulting virtual image can achieve effects such as overexposure and halo effects, for example... Figure 4a As shown, Figure 4a This is a first schematic diagram of a virtual image provided in this application, showing a virtual image obtained after achieving overexposure and halo effects.

[0093] The different states of the calibration plate refer to various conditions that affect the reflected light from the calibration plate, such as wrinkles, damage, and stains. In addition, obstacles can be arbitrarily placed by inputting a 3D model, such as... Figure 4b , Figure 4c As shown, Figure 4b This is a second schematic diagram of a virtual image provided in this application, showing a virtual image obtained after placing obstacles. Figure 4c This is a third schematic diagram of a virtual image provided in an embodiment of this application. The figure is a schematic diagram of a virtual image obtained when the calibration plate has wrinkles.

[0094] Virtual images can also be obtained under different light source conditions and when the calibration plate is in different states. For example, a virtual image can be obtained under strong light source conditions and when the calibration plate is in a wrinkled state, a virtual image can be obtained under strong light source conditions and when the calibration plate is in a damaged state, and a virtual image can be obtained under strong light source conditions and when the calibration plate is in a stained state.

[0095] By using the above embodiments, various special and abnormal effects are added to the virtual scene, so that the final rendered virtual image can contain various situations that can be encountered in the real scene, thus making the final target image richer. Using rich target images to train the calibration model, the calibration model can learn how to accurately calibrate the camera's intrinsic and extrinsic parameters in various application scenarios during the training process, making the trained model more applicable.

[0096] In S203, it is understandable that images captured by a camera typically exhibit distortion, causing lines that were originally straight to appear noticeably curved in the captured image, such as... Figure 5a As shown, Figure 5a This is a fourth schematic diagram of a virtual image provided in an embodiment of this application. The figure shows a virtual image of a calibration board captured by a camera after distortion. The distortion of fisheye cameras is particularly severe. In the aforementioned autonomous driving scenario, most of the cameras used to capture the area around the vehicle are fisheye cameras. Therefore, in order to make the captured image closer to the real image, distortion correction of the camera is necessary.

[0097] In one possible embodiment, the first coordinate transformation relationship can be determined based on the radial distortion and distortion radius of the virtual camera lens. Specifically, the abscissa and ordinate of the distortion point can be calculated according to the following formula (1):

[0098]

[0099]

[0100] Where, x d and y d Let x be the x-coordinate and y-coordinate of the distorted point. p and y p Let x be the x-coordinate and y-coordinate of the imaging point in the camera coordinate system, k1, k2, k3...k6 be the radial distortion of the virtual camera lens, p1 and p2 be the tangential distortion of the virtual camera lens, and r be the distortion radius.

[0101] In practical applications, the coordinates of an imaging point on the camera film in the camera coordinate system can be [x...]. p y p f0] T The coordinates of the corresponding distortion point can be [x d y d f0] TWhere f0 is the camera's focal length, and T is the translation of the camera coordinate system origin relative to the spatial coordinate system origin. After calculating the x-coordinate and y-coordinate of the corresponding distortion point, a mapping relationship between the two can be established, thus obtaining the first coordinate transformation relationship. In practical applications, the virtual image can be transformed according to the first coordinate transformation relationship, that is, the pixel value of an imaging point on the camera film in the camera coordinate system is assigned as the pixel value of the corresponding distortion point, resulting in a corrected image. This realizes the simulation of camera distortion. After rendering, the image obtained after distortion correction is as follows: Figure 5b As shown, Figure 5b This is a schematic diagram of a corrected image provided in an embodiment of this application.

[0102] By selecting this embodiment, distortion correction of the camera is performed to obtain an image of real lens distortion, making the corrected image closer to the real image. This results in the final target image being closer to the image with labeled corner information in the real scene, thus making the trained calibration model more accurate.

[0103] In S104, in one possible embodiment, the target image can be an image whose content is consistent with the corrected image, the only difference between the target image and the corrected image being that the target image is an annotated image.

[0104] In another possible embodiment, the target image can also be obtained after the corrected image has been annotated and then processed, such as denoising the image. In this case, the difference between the target image and the corrected image is not only that the target image is an annotated image, but also that the content of the target image and the corrected image are different.

[0105] In another possible embodiment, based on the spatial position of each feature point in the calibration board in the virtual scene, the image position of each feature point in the calibration board is marked in the corrected image to obtain the target image, including:

[0106] Based on the spatial coordinates of each feature point in the calibration board within the virtual scene, the second coordinate transformation relationship, and the first coordinate transformation relationship, the image positions of each feature point in the calibration board are marked in the corrected image to obtain the target image. The second coordinate transformation relationship refers to the coordinate transformation between the spatial coordinate system and the first coordinate system.

[0107] The above steps involve obtaining the coordinates of the feature points in the first coordinate system based on their spatial coordinates and the second coordinate transformation relationship. The spatial coordinates represent the position of the feature points in the spatial coordinate system. Then, the virtual image is transformed according to the first coordinate transformation relationship, thereby marking the image positions of each feature point in the corrected image, achieving automatic annotation. The resulting labeled target image is shown below. Figure 6a As shown, Figure 6a This is a first schematic diagram of a target image provided in an embodiment of this application. Figure 6a In the example shown, the feature points are corner points, and each corner point is pre-assigned a number. The number in the figure is the number of each corner point, and the position of the number is the position of the corner point identified by that number.

[0108] The images captured by the four fisheye cameras are stitched together to obtain a complete calibration plate corner point map, such as... Figure 6b As shown, Figure 6b A second schematic diagram of a target image provided in this application includes a front view, a rear view, a left view, and a right view.

[0109] By using this embodiment, the three-dimensional coordinates of feature points can be transformed into two-dimensional coordinates, enabling automated annotation of feature points. Compared with manual annotation, the method is simpler and easier to operate, reduces the consumption of manpower and material resources, and improves the annotation efficiency of virtual images.

[0110] In S205, the target image is labeled with the image positions of each feature point in the calibration board. Training samples generated based on the spatial positions of the feature points in the target image and calibration board within the virtual scene reflect both the image position and the spatial position of each feature point in the calibration board. Therefore, the original camera calibration model can determine the transformation relationship based on the image position and spatial position, and then predict the intrinsic and extrinsic parameters of the virtual camera, i.e., the estimated parameters, based on the transformation relationship. Specifically, a loss function can be constructed based on the difference between the estimated parameters and the target parameters. Based on this loss function, the model parameters are adjusted to train the original camera calibration model.

[0111] In one possible embodiment, the training samples may also include data other than the spatial locations of the feature points in the target image and calibration board within the virtual scene. For example, this could be mask information reflecting the occlusion status of the calibration board in the virtual image.

[0112] By using the above embodiments, the spatial positions of the target image and each feature point in the calibration board in the virtual scene are input into the original camera calibration model. There is no need to manually annotate the acquired images, saving the cost of manual calibration and improving the efficiency of virtual image annotation. A large number of training samples are generated without consuming manpower and resources. Compared with manually annotating images to generate training samples, it can reduce the error caused by manually annotated images, thereby reducing the mislabeling rate. Furthermore, since there are many target images generated and the types are more comprehensive and realistic, the accuracy of the trained camera calibration model is further improved.

[0113] In the above method, it is necessary to obtain the target parameters of the virtual camera in the virtual scene. Therefore, a virtual scene needs to be constructed in advance, and the intrinsic and extrinsic parameters of the virtual camera need to be set. Before this, it is necessary to model the car / calibration board / scene and set the camera's intrinsic / extrinsic parameters. After completing the virtual scene modeling, ray tracing rendering is performed on the virtual scene, which is equivalent to steps S201 and S202 above. After the ray tracing rendering is completed, it is necessary to determine the position of feature points in the virtual image, which can be regarded as calibrating feature points in the image, i.e., automatic calibration, equivalent to steps S203 and S204 above. Therefore, the solution of this application can be simply summarized into four steps: First, car / calibration board / scene modeling; second, setting camera intrinsic / extrinsic parameters; third, ray tracing rendering; fourth, automatic target calibration. Figure 7 As shown, Figure 7 This is a schematic diagram illustrating the steps of an automatic virtual image annotation method provided in an embodiment of this application.

[0114] To more clearly illustrate the automatic virtual image annotation method provided in this application, the following will provide an exemplary description in conjunction with a specific application scenario, taking the aforementioned surround-view camera in an autonomous driving scenario as an example:

[0115] Suppose we have a car and we need to calibrate the surround-view cameras installed in the vehicle. This application can pre-build a calibration model. The input of the calibration model is a sample image containing calibrated corner information and training samples of the spatial positions of the corners. The output is the intrinsic and extrinsic parameters of the surround-view cameras.

[0116] To obtain a large number of training samples without expending manpower and resources, we can pre-create a virtual scene, including a car model, a virtual camera, a calibration board model, etc. We then obtain the target parameters of the virtual camera, i.e., the true values ​​of its intrinsic and extrinsic parameters. Based on these parameters, we perform ray tracing rendering on the virtual scene to obtain a distortion-free virtual image.

[0117] In one possible embodiment, to enhance the applicability and accuracy of the trained calibration model, special or unusual effects can be added during the ray tracing rendering process. Special effects could include setting the lighting conditions to a strong light source, causing the rendered virtual image to overexpose or exhibit halos. Unusual effects could include making the calibration board appear wrinkled, damaged, or stained.

[0118] To make the stitched virtual image closer to a bird's-eye view captured by a camera in a real scene, distortion correction is required. This is achieved by controlling the distortion function, specifically by using the formula (1) above, to obtain a corrected image. After obtaining the corrected image, the coordinates of the feature points in the first coordinate system are obtained using the spatial coordinates of each corner point on the calibration board and the second coordinate transformation relationship. Then, the virtual image is transformed according to the first coordinate transformation relationship, converting the three-dimensional coordinates to two-dimensional coordinates. This marks the image positions of each corner point in the corrected image, resulting in the target image, i.e., the sample image, thus achieving automatic annotation. Figure 6a A portion of the target image with corner information labeled.

[0119] By stitching together the target images captured by the virtual camera, a complete calibration board corner point map can be obtained, such as... Figure 6b As shown, the sample images of the calibrated corner points and their spatial locations are input into the calibration model to train it, thereby obtaining a more applicable and accurate camera calibration model.

[0120] After obtaining the camera calibration model, a calibration board is placed within the scene where the car is located, and images captured by the car's surround-view cameras are acquired. The positions of the corner points on the calibration board are marked in the images, resulting in an annotated image. The annotated image and the spatial positions of the corner points in the scene are input into the obtained target camera calibration model to obtain the intrinsic and extrinsic parameters output by the target camera calibration model. These intrinsic and extrinsic parameters are the intrinsic and extrinsic parameters of the vehicle's surround-view cameras. Based on these intrinsic and extrinsic parameters, the images captured by the various surround-view cameras of the vehicle can be accurately stitched together to obtain, as shown below. Figure 1a The bird's-eye view shown.

[0121] Corresponding to the above method embodiments, this application provides an apparatus for automatic annotation of virtual images, such as... Figure 8 As shown, Figure 8 This is a schematic diagram of the structure of a device for automatic virtual image annotation provided in an embodiment of this application. The device may include the following modules:

[0122] The camera parameter acquisition module 801 is used to acquire the target parameters of the virtual camera set in the virtual scene, and the virtual scene is also equipped with a calibration plate;

[0123] The virtual image determination module 802 is used to determine, based on the target parameters, the virtual image formed by the light emitted by the calibration plate sensed by the virtual camera, without lens distortion;

[0124] The image coordinate transformation module 803 is used to perform coordinate transformation on the virtual image according to the first coordinate transformation relationship between the first coordinate system and the second coordinate system to obtain a corrected image; wherein, the first coordinate system is the image coordinate system of the virtual camera when no lens distortion occurs, and the second coordinate system is the image coordinate system of the virtual camera when lens distortion occurs;

[0125] The image position annotation module 804 is used to annotate the image position of each feature point in the calibration board in the corrected image according to the spatial position of each feature point in the calibration board in the virtual scene, so as to obtain the target image;

[0126] The calibration model training module 805 is used to generate training samples based on the target image and the spatial positions of each feature point in the calibration board in the virtual scene, so as to train the original camera calibration model based on the training samples.

[0127] In an embodiment of this application, target parameters of a virtual camera set in a virtual scene are obtained, and a calibration board is also set in the virtual scene. Based on the target parameters, a virtual image formed by the light emitted by the calibration board and sensed by the virtual camera is determined without lens distortion. The virtual image is then transformed according to a first coordinate transformation relationship between a first coordinate system and a second coordinate system to obtain a corrected image. The first coordinate system includes the image coordinate system of the virtual camera without lens distortion, and the second coordinate system includes the image coordinate system of the virtual camera with lens distortion. Based on the spatial positions of each feature point on the calibration board in the virtual scene, the image positions of each feature point on the calibration board are marked in the corrected image to obtain a target image. Training samples are generated based on the target image and the spatial positions of each feature point on the calibration board in the virtual scene, so that the original camera calibration model can be trained based on the training samples. By applying the scheme of this application, a virtual scene and a virtual camera within the virtual scene are set up. The virtual scene also includes a calibration board. Because the virtual scene is pre-set, the spatial position of feature points on the calibration board within the virtual scene can be clearly obtained. During the process of determining the corrected image based on the target parameters, i.e., during ray tracing rendering, a photorealistic calibration board image with realistic camera distortion effects can be obtained. The transformation relationship between the virtual spatial position and the image coordinate system in the camera can also be obtained. Therefore, the coordinate position of the feature point in the image can be directly calculated based on the spatial coordinates of the feature point, i.e., converting three-dimensional coordinates into two-dimensional coordinates, thus achieving automated annotation of feature points. The original camera calibration model is trained by generating training samples based on the spatial positions of each feature point in the target image and calibration board in the virtual scene. This means that photorealistic images and annotation information are used together in model training, eliminating the need for manual annotation of the acquired images, saving manual calibration costs, improving the efficiency of virtual image annotation, generating a large number of training samples without consuming manpower and resources, and reducing the errors caused by manual image annotation compared with manually annotated images, thereby reducing the mislabeling rate and improving the accuracy of the camera calibration model.

[0128] In one possible embodiment, the image location annotation module includes:

[0129] The image location annotation submodule annotates the image location of each feature point in the calibration board in the virtual scene according to the spatial coordinates of each feature point in the calibration board, the second coordinate transformation relationship, and the first coordinate transformation relationship, thereby obtaining the target image; wherein, the second coordinate transformation relationship is the coordinate transformation relationship between the spatial coordinate system and the first coordinate system;

[0130] The device further includes:

[0131] The transformation relationship determination module is used to determine the first coordinate transformation relationship based on the radial distortion and distortion radius of the virtual camera lens;

[0132] The conversion relationship determination module includes:

[0133] The transformation relationship determination submodule is used to calculate the x-coordinate and y-coordinate of the distorted point according to the following formula:

[0134]

[0135]

[0136] Where, x d and y d Let x be the x-coordinate and y-coordinate of the distorted point. p and y p Let x be the x-coordinate and y-coordinate of the imaging point in the camera coordinate system, k1, k2, k3...k6 be the radial distortion of the virtual camera lens, p1 and p2 be the tangential distortion of the virtual camera lens, and r be the distortion radius.

[0137] The virtual image determination module includes:

[0138] The virtual image determination first submodule is used to determine, based on the target parameters, various virtual images formed by the light emitted by the calibration plate sensed by the virtual camera under different light source conditions, without lens distortion, and for different light source conditions, wherein the different light source conditions include strong light sources;

[0139] The virtual image determination module includes:

[0140] The virtual image determination second submodule is used to determine, based on the target parameters, various virtual images formed by the light emitted by the calibration plate by the virtual camera when no lens distortion occurs and the calibration plate is in different states, wherein the different states of the calibration plate include one of calibration plate wrinkles, damage and stains;

[0141] The virtual scene also includes a car model, and the virtual camera is a fisheye camera mounted on the car model.

[0142] Corresponding to the above method embodiments, this application also provides an electronic device, such as... Figure 9 As shown, it includes a memory 901 and a processor 902. The memory 901 is used to store computer programs; the processor 902 is used to execute the computer programs stored in the memory to implement the following steps:

[0143] Obtain the target parameters of a virtual camera set in a virtual scene, wherein a calibration board is also set in the virtual scene;

[0144] Based on the target parameters, the virtual image formed by the light emitted from the calibration plate by the virtual camera is determined without lens distortion.

[0145] Based on the first coordinate transformation relationship between the first coordinate system and the second coordinate system, the virtual image is transformed to obtain a corrected image; wherein, the first coordinate system is the image coordinate system of the virtual camera without lens distortion, and the second coordinate system is the image coordinate system of the virtual camera with lens distortion.

[0146] Based on the spatial position of each feature point in the calibration board in the virtual scene, the image position of each feature point in the calibration board is marked in the corrected image to obtain the target image;

[0147] Training samples are generated based on the target image and the spatial positions of each feature point in the calibration board in the virtual scene, so that the original camera calibration model can be trained based on the training samples.

[0148] The aforementioned memory may include RAM (Random Access Memory) or NVM (Non-volatile Memory), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0149] The processors mentioned above can be general-purpose processors, including CPUs (Central Processing Units), NPs (Network Processors), etc.; they can also be DSPs (Digital Signal Processors), ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0150] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described methods for automatic annotation of virtual images.

[0151] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the virtual image automatic annotation methods described in the above embodiments.

[0152] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0153] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0154] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0155] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for automatic annotation of virtual images, characterized in that, The method comprises the following steps: acquiring target parameters of a virtual camera arranged in a virtual scene, the virtual scene being further arranged with a calibration board; determining a virtual image formed by light rays emitted by the calibration board and sensed by the virtual camera under the condition that no lens distortion occurs according to the target parameters; performing coordinate conversion on the virtual image according to a first coordinate conversion relationship between a first coordinate system and a second coordinate system to obtain a corrected image, wherein the first coordinate system is an image coordinate system of the virtual camera under the condition that no lens distortion occurs, and the second coordinate system is an image coordinate system of the virtual camera under the condition that lens distortion occurs; labeling image positions of each feature point in the calibration board in the corrected image according to spatial positions of each feature point in the calibration board in the virtual scene to obtain a target image; generating a training sample based on the target image and the spatial positions of each feature point in the calibration board in the virtual scene, so as to train an original camera calibration model based on the training sample; The method further comprises the following steps: determining the first coordinate conversion relationship according to radial distortion and distortion radius of a lens of the virtual camera; The step of determining the first coordinate conversion relationship according to radial distortion and distortion radius of a lens of the virtual camera comprises the following steps: calculating the horizontal coordinate and the vertical coordinate of a distortion point according to the following formula: ， ， wherein and are the horizontal and vertical coordinates of the distortion point, and are the horizontal and vertical coordinates of the imaging point in the camera coordinate system, , , ... is the radial distortion of the virtual camera lens, and are the tangential distortion of the virtual camera lens, r is the distortion radius.

2. The method of claim 1, wherein, The step of labeling image positions of each feature point in the calibration board in the corrected image according to spatial positions of each feature point in the calibration board in the virtual scene to obtain a target image comprises the following step: labeling image positions of each feature point in the calibration board in the corrected image according to spatial position coordinates of each feature point in the calibration board in the virtual scene, a second coordinate conversion relationship and the first coordinate conversion relationship to obtain a target image, wherein the second coordinate conversion relationship is a coordinate conversion relationship between a spatial coordinate system and the first coordinate system.

3. The method of claim 1, wherein, The step of determining a virtual image formed by light rays emitted by the calibration board and sensed by the virtual camera under the condition that no lens distortion occurs according to the target parameters comprises the following steps: determining each virtual image formed by light rays emitted by the calibration board and sensed by the virtual camera under the condition that no lens distortion occurs according to the target parameters and different light source conditions, wherein the different light source conditions include a strong light source.

4. The method of claim 1, wherein, The step of determining a virtual image formed by light rays emitted by the calibration board and sensed by the virtual camera under the condition that no lens distortion occurs according to the target parameters comprises the following steps: determining each virtual image formed by light rays emitted by the calibration board and sensed by the virtual camera under the condition that no lens distortion occurs according to the target parameters and different states of the calibration board, wherein the different states of the calibration board include one of a wrinkle, a damage and a stain of the calibration board.

5. The method of claim 1, wherein, The virtual scene is further arranged with a car model, and the virtual camera is a fisheye camera arranged on the car model.

6. A device for automatic annotation of virtual images, characterized in that, The method comprises the following steps: a camera parameter acquisition module is configured to acquire target parameters of a virtual camera arranged in a virtual scene, the virtual scene being further arranged with a calibration board; a virtual image determining module configured to determine a virtual image formed by light rays emitted by the calibration board and sensed by the virtual camera under the condition that no lens distortion occurs, according to the target parameters; an image coordinate conversion module configured to perform coordinate conversion on the virtual image to obtain a corrected image, according to a first coordinate conversion relationship between a first coordinate system and a second coordinate system, wherein the first coordinate system is an image coordinate system of the virtual camera under the condition that no lens distortion occurs, and the second coordinate system is an image coordinate system of the virtual camera under the condition that lens distortion occurs; an image position marking module configured to mark image positions of the feature points in the calibration board in the corrected image, according to spatial positions of the feature points in the calibration board in the virtual scene, to obtain a target image; a calibration model training module configured to generate training samples based on the target image and the spatial positions of the feature points in the calibration board in the virtual scene, so as to train an original camera calibration model based on the training samples; the device further comprises: a conversion relationship determining module configured to determine the first coordinate conversion relationship according to the radial distortion of the lens of the virtual camera and a distortion radius; the conversion relationship determining module comprises: a conversion relationship determining sub-module configured to calculate the horizontal coordinate and the vertical coordinate of the distortion point according to the following formula: ， ， wherein and are the horizontal and vertical coordinates of the distortion point, and are the horizontal and vertical coordinates of the imaging point in the camera coordinate system, , , ... is the radial distortion of the virtual camera lens, and are the tangential distortion of the virtual camera lens, r is the distortion radius.

7. The apparatus of claim 6, wherein, the image position marking module comprises: an image position marking sub-module configured to mark the image positions of the feature points in the calibration board in the corrected image, according to the spatial position coordinates of the feature points in the calibration board in the virtual scene, a second coordinate conversion relationship and the first coordinate conversion relationship, to obtain a target image, wherein the second coordinate conversion relationship is a coordinate conversion relationship between a spatial coordinate system and the first coordinate system; the virtual image determining module comprises: a virtual image determining first sub-module configured to determine each virtual image formed by light rays emitted by the calibration board and sensed by the virtual camera under the condition that no lens distortion occurs and under different light source conditions, according to the target parameters, wherein the different light source conditions include a strong light source; the virtual image determining module comprises: a virtual image determining second sub-module configured to determine each virtual image formed by light rays emitted by the calibration board and sensed by the virtual camera under the condition that no lens distortion occurs and when the calibration board is in different states, according to the target parameters, wherein the different states of the calibration board include one of a calibration board wrinkle, a damaged calibration board and a stained calibration board; the virtual scene is further provided with a car model, and the virtual camera is a fisheye camera arranged on the car model.

8. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-5.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-5.

Images