Image conversion method and device, electronic equipment, storage medium and vehicle

Abstract

The invention relates to an image conversion method and device, electronic equipment, a storage medium and a vehicle, and the method comprises the steps: obtaining a camera visual angle corresponding to each vehicle-mounted camera on the vehicle and an original image collected by each vehicle-mounted camera; according to the camera visual angle corresponding to each vehicle-mounted camera, a conversion matrix corresponding to each camera visual angle is determined, and the conversion matrix corresponding to each camera visual angle is determined according to a camera parameter matrix of the vehicle-mounted cameras of the corresponding visual angles of different vehicle types; and performing view angle conversion on the original image of the corresponding view angle based on the conversion matrix corresponding to each camera view angle to obtain a virtual image of the corresponding view angle. By adopting the above technical scheme, the original images collected by the vehicle-mounted cameras of different vehicle types can be converted to a unified visual angle, and unification of the image data collected by different vehicle types is realized, so that the influence of the difference of camera parameters of different vehicle types on the visual perception model can be reduced, and the training effect of the model is ensured.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image conversion method, device, electronic device, storage medium and vehicle. Background Art

[0002] Currently, the application of visual perception technology in the field of autonomous driving is becoming more and more widespread and active. Most of the current deep learning-based visual perception models are driven by data. Images collected by vehicles are acquired and annotated for training to obtain visual perception models for different tasks. Therefore, good data plays a vital role in improving the performance of the model.

[0003] The on-board cameras on vehicles usually have multiple viewing angles, including front view, left front view, left rear view, right front view, right rear view and rear view. Due to the differences in camera models, camera intrinsic parameters and installation positions of on-board cameras of different models, the image viewing angles collected by on-board cameras of different models will be different. Some visual perception tasks are strongly dependent on camera parameters and viewing angles. Therefore, when using images collected by different models for model training, the model training effect will be unsatisfactory. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, at least one embodiment of the present disclosure provides an image conversion method, device, electronic device, storage medium and vehicle.

[0005] In a first aspect, the present disclosure provides an image conversion method, comprising:

[0006] Obtaining the camera angle corresponding to each on-board camera on the vehicle and the original image captured by each on-board camera;

[0007] Determine a conversion matrix corresponding to each camera angle according to the camera angle corresponding to each vehicle-mounted camera, wherein the conversion matrix corresponding to each camera angle is determined according to a camera parameter matrix of vehicle-mounted cameras of corresponding angles of different vehicle models;

[0008] Based on the conversion matrix corresponding to each camera perspective, the original image of the corresponding perspective is transformed to obtain a virtual image of the corresponding perspective.

[0009] In a second aspect, the present disclosure provides an image conversion device, comprising:

[0010] A data acquisition module, used to acquire the camera angle corresponding to each on-board camera on the vehicle and the original image captured by each on-board camera;

[0011] A matrix determination module, used to determine a conversion matrix corresponding to each camera angle according to the camera angle corresponding to each vehicle-mounted camera, wherein the conversion matrix corresponding to each camera angle is determined according to a camera parameter matrix of vehicle-mounted cameras of corresponding angles of different vehicle models;

[0012] The image conversion module is used to perform perspective conversion on the original image of the corresponding perspective based on the conversion matrix corresponding to each camera perspective to obtain a virtual image of the corresponding perspective.

[0013] In a third aspect, the present disclosure provides an electronic device, including: a processor and a memory;

[0014] The processor is used to execute any of the image conversion methods provided by the embodiments of the present disclosure by calling the program or instruction stored in the memory.

[0015] In a fourth aspect, the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program or instruction, wherein the program or instruction enables a computer to execute any of the image conversion methods provided by the embodiments of the present disclosure.

[0016] In a fifth aspect, the present disclosure provides a vehicle, comprising the image conversion device as described in the second aspect or the electronic device as described in the third aspect or the computer-readable storage medium as described in the fourth aspect.

[0017] In a sixth aspect, the present disclosure provides a computer program product, which is used to execute any of the image conversion methods provided by the embodiments of the present disclosure.

[0018] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has at least the following advantages:

[0019] In the disclosed embodiment, the camera angle corresponding to each on-board camera on the vehicle and the original image captured by each on-board camera are obtained, and the conversion matrix corresponding to each camera angle is determined according to the camera angle corresponding to each on-board camera, wherein the conversion matrix corresponding to each camera angle is determined according to the camera parameter matrix of the on-board camera of the corresponding angle of view of different vehicle models, and then the original image of the corresponding angle of view is converted based on the conversion matrix corresponding to each camera angle of view to obtain a virtual image of the corresponding angle of view. The above technical solution is adopted, and the conversion matrix corresponding to the camera angle of view is determined according to the camera parameter matrix of the on-board camera of different vehicle models with different camera angles, so as to achieve the unification of the conversion matrix, and then the original image of the corresponding angle of view is converted using the conversion matrix of each camera angle of view, so as to achieve the conversion of the original images captured by the on-board cameras of different vehicle models to a unified angle of view for any camera angle of view, so as to achieve the unification of the image data collected by different vehicle models, thereby reducing the influence of the difference in camera parameters of different vehicle models on the visual perception model, so that the model training can make full use of multiple data and ensure the training effect of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0022] Figure 1 A flowchart of an image conversion method provided by an exemplary embodiment of the present disclosure;

[0023] Figure 2 A flowchart of an image conversion method provided by another exemplary embodiment of the present disclosure;

[0024] Figure 3 A schematic diagram of a spatial three-dimensional coordinate system constructed with the origin of the self-vehicle coordinate system of a reference vehicle as the center provided in an embodiment of the present disclosure;

[0025] Figure 4 A schematic diagram of the mapping relationship between spherical space calibration points on a camera image provided by an embodiment of the present disclosure;

[0026] Figure 5 A schematic diagram of the structure of an image conversion device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to more clearly understand the above-mentioned purposes, features and advantages of the present disclosure, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understandable that the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments, and the specific embodiments described herein are only used to explain the present disclosure, rather than to limit the present disclosure. In the absence of conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in the field belong to the scope of protection of the present disclosure.

[0028] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0029] At present, the application of visual perception technology in the field of autonomous driving is becoming more and more extensive and active. Most of the current deep learning-based visual perception models are driven by data, so good data plays a vital role in improving the performance of the model. Data collection and annotation usually require huge resources. Data in the field of autonomous driving is mainly collected and annotated by real value vehicles. The real value vehicle contains multiple sensors such as lidar and on-board cameras. On-board cameras usually have multiple perspectives, including front view, left front view, left rear view, right front view, right rear view and rear view. However, due to the inconsistency of camera models, intrinsic parameters and installation positions of different models, that is, the internal and external parameters of on-board cameras of different models are different, the data perspectives collected by on-board cameras will be different. However, in the field of autonomous driving, some visual perception tasks are strongly dependent on camera parameters and perspectives. In visual perception tasks, the model is more sensitive to the perspective of the image. When image data is directly migrated to model training of other models, the training effect of the model will be poor and the performance of the trained model will deteriorate.

[0030] In order to make the best use of the images collected by different models, it is necessary to pre-process the images collected by different models, that is, convert the perspective of the image data of different models, set the virtual perspective of the camera in advance, fix the internal and external parameters of the virtual camera, and transfer the data to the pre-set virtual perspective during model training and model vehicle deployment, so as to make the best use of different data and improve the performance of the model. However, there is no solution in the prior art to convert images collected by different models to the same virtual perspective.

[0031] In view of the above problems, the present disclosure provides an image conversion method, which mainly solves the following technical problems:

[0032] (1) Different models have inconsistent internal and external parameters. The image data collected by different models have different camera internal and external parameters and viewing angles. Therefore, it is necessary to select a set of appropriate camera internal and external parameters to fix and unify the camera viewing angle;

[0033] (2) The perspectives of image data of different car models are not unified. On the basis of fixed camera parameters, the image data of different car models need to be processed and converted to a unified virtual perspective.

[0034] This solution determines the conversion matrix of the corresponding camera perspective according to the camera parameter matrix of the vehicle-mounted camera with different camera perspectives of different vehicle models, and then converts the original image collected by the vehicle into a virtual image according to the conversion matrix corresponding to the camera perspective of the vehicle-mounted camera. This achieves better combination of parameter information of different vehicle-mounted cameras when determining the conversion matrix, and realizes conversion of image data of different vehicle models to a unified perspective, thereby reducing the impact of different vehicle models on image data, ensuring that the visual perception task model has data input in a unified format, thereby eliminating the impact of differences in camera parameters on the model, and being able to make more full use of a variety of data to ensure the training effect of the model.

[0035] The specific implementations of the image conversion method, device, electronic device, storage medium and vehicle disclosed in the present invention are explained in detail below with reference to the accompanying drawings.

[0036] Figure 1 A flow chart of an image conversion method provided for an exemplary embodiment of the present disclosure. The method can be executed by an image conversion device provided for an embodiment of the present disclosure. The image conversion device can be implemented by software and / or hardware and can be integrated in an electronic device. The electronic device can be an image processing device deployed in a vehicle, or can be other devices such as a computer.

[0037] like Figure 1 As shown, the image conversion method may include the following steps:

[0038] Step 101, obtaining the camera angle corresponding to each on-board camera on the vehicle and the original image captured by each on-board camera.

[0039] Usually, one or more vehicle-mounted cameras are installed on a vehicle, and each vehicle-mounted camera corresponds to a camera viewing angle, and the camera viewing angles include, for example, front view, left front view, left rear view, right front view, right rear view, and rear view. The vehicle-mounted camera of each camera viewing angle captures an image of the corresponding viewing angle. In the disclosed embodiment, the electronic device can obtain the camera viewing angle of each vehicle-mounted camera on the vehicle and the image captured by each vehicle-mounted camera (referred to as an original image).

[0040] Among them, the camera viewing angle of each vehicle-mounted camera can be determined according to the installation position of each vehicle-mounted camera, or the relevant information of each vehicle-mounted camera (including camera viewing angle, camera internal and external parameters, etc.) can be pre-stored in the vehicle, and the camera viewing angle of each vehicle-mounted camera can be obtained by reading the relevant information of each vehicle-mounted camera from the vehicle.

[0041] Step 102, determining a conversion matrix corresponding to each camera angle according to the camera angle corresponding to each vehicle-mounted camera, wherein the conversion matrix corresponding to each camera angle is determined according to a camera parameter matrix of a vehicle-mounted camera of a corresponding angle of view of different vehicle models.

[0042] In the disclosed embodiment, the conversion matrices corresponding to different camera perspectives are predetermined, and the conversion matrices corresponding to the camera perspectives of different vehicle models can be used to determine the conversion matrices corresponding to the camera perspectives. The original image captured by the vehicle camera can be converted to the virtual perspective corresponding to the conversion matrix through the conversion matrix to obtain the converted virtual image. Therefore, in practical applications, a matching conversion matrix can be determined for image conversion from the conversion matrices corresponding to the predetermined multiple different camera perspectives according to the camera perspective of the vehicle camera installed on the vehicle.

[0043] Step 103 , performing perspective conversion on the original image of the corresponding perspective based on the conversion matrix corresponding to each camera perspective to obtain a virtual image of the corresponding perspective.

[0044] In the disclosed embodiment, after obtaining the conversion matrix corresponding to each camera perspective, or using the conversion matrix corresponding to each camera perspective, the original image of the corresponding camera perspective is converted to obtain a virtual image of the corresponding perspective.

[0045] It should be noted that the image conversion solution disclosed in the present invention can be applied to the training scenarios of visual perception models, and can also be deployed on vehicles and applied to image recognition scenarios after model training.

[0046] For example, in a model training scenario, it is necessary to obtain original images captured by vehicle-mounted cameras of different vehicle models and different camera perspectives, and perform perspective conversion on the original images according to a predetermined transformation matrix of each camera perspective to obtain virtual images and annotate them to form a training sample set, which is then used to train the visual perception task model to ensure the training effect of the model.

[0047] Exemplarily, in an image recognition scenario, the solution of the present disclosure is deployed on a vehicle. After the on-board cameras of various camera angles on the vehicle capture the original image, the original image is converted into a virtual image according to the transformation matrix corresponding to each camera angle. The virtual image is then input into a pre-trained visual perception task model for image recognition, and the vehicle performs corresponding operations based on the image recognition results.

[0048] The image conversion method of the disclosed embodiment obtains the camera angle corresponding to each on-board camera on the vehicle and the original image captured by each on-board camera, and determines the conversion matrix corresponding to each camera angle according to the camera angle corresponding to each on-board camera, wherein the conversion matrix corresponding to each camera angle is determined according to the camera parameter matrix of the on-board camera of the corresponding angle of view of different vehicle models, and then the original image of the corresponding angle of view is converted based on the conversion matrix corresponding to each camera angle of view to obtain a virtual image of the corresponding angle of view. The above technical solution is adopted, and the conversion matrix corresponding to the camera angle of view is determined according to the camera parameter matrix of the on-board camera of different vehicle models with different camera angles, so as to achieve the unification of the conversion matrix, and then the original image of the corresponding angle of view is converted using the conversion matrix of each camera angle of view, so as to achieve the conversion of the original images captured by the on-board cameras of different vehicle models to a unified angle of view for any camera angle, so as to achieve the unification of the image data collected by different vehicle models, thereby reducing the influence of the differences in camera parameters of different vehicle models on the visual perception model, so that the model training can make full use of multiple data and ensure the training effect of the model.

[0049] In an optional embodiment of the present disclosure, Figure 2 As shown, in Figure 1 Based on the embodiment shown, in the embodiment of the present disclosure, the transformation matrix corresponding to the target camera viewing angle can be determined by the following steps:

[0050] Step 201 : obtaining a target camera parameter matrix corresponding to the vehicle-mounted cameras of the target camera viewing angles of multiple different vehicle models.

[0051] The target camera perspective refers to any camera perspective among the front view, left front view, left rear view, right front view, right rear view and rear view. That is to say, for each camera perspective, the following is used: Figure 2 The conversion matrix determination method shown is used to determine the corresponding conversion matrix.

[0052] In the disclosed embodiment, when determining the conversion matrix corresponding to the target camera viewing angle, it is necessary to obtain the camera parameter matrices corresponding to the vehicle-mounted cameras of the target camera viewing angle of multiple different vehicle models (referred to as the target camera parameter matrix). For example, assuming that the target camera viewing angle of the conversion matrix to be determined is the front view, the camera parameter matrices of the front view cameras of multiple different vehicle models are obtained as the target camera parameter matrix.

[0053] Step 202: determining a virtual camera parameter matrix according to the target camera parameter matrix.

[0054] In the disclosed embodiment, after the target camera parameter matrix corresponding to the target camera viewing angle of different vehicle models is obtained, the virtual camera parameter matrix corresponding to the target camera viewing angle can be determined according to the target camera parameter matrix.

[0055] As an example, when determining the virtual camera parameter matrix corresponding to the target camera perspective, the target camera parameter matrices of the onboard cameras of the target camera perspective of each vehicle model can be added and divided by the number of target camera parameter matrices, and the result is determined as the virtual camera parameter matrix. In other words, the virtual camera parameter matrix corresponding to the target camera perspective can be obtained by averaging the target camera parameter matrices of the onboard cameras of the target camera perspectives of all different vehicle models. It can be understood that if the target camera parameter matrix includes a camera intrinsic parameter matrix and a camera extrinsic parameter matrix, the camera intrinsic parameter matrix and the camera extrinsic parameter matrix are respectively solved according to the above-mentioned solution method to obtain the virtual camera intrinsic parameter matrix and the virtual camera extrinsic parameter matrix, and the virtual camera intrinsic parameter matrix and the virtual camera extrinsic parameter matrix are the obtained virtual camera parameter matrix.

[0056] As another example, when determining the virtual camera parameter matrix corresponding to the target camera perspective, the total number of images captured by the onboard cameras of the target camera perspectives of multiple different vehicle models can also be obtained, and then the virtual camera parameter matrix can be determined according to the total number of images captured by the onboard cameras of the target camera perspectives of each vehicle model in the multiple different vehicle models and the target camera parameter matrix. Among them, the total number of images captured by the onboard camera of the target camera perspective of a vehicle model refers to the sum of all historical images captured by the onboard camera of the target camera perspective of the vehicle, and the total number of images is used as the weight of the target camera parameter matrix, and the virtual camera parameter matrix corresponding to the target camera perspective is obtained by solving the weighted average method.

[0057] Assuming that the target camera parameter matrix includes the camera intrinsic parameter matrix and the camera extrinsic parameter matrix, accordingly, the virtual camera parameter matrix finally solved includes the virtual camera intrinsic parameter matrix and the virtual camera extrinsic parameter matrix. The virtual camera parameter matrix can be calculated by the following formula (1):

[0058]

[0059] In the above formula (1), K v represents the virtual camera intrinsic parameter matrix, K j The camera intrinsic parameter matrix corresponding to the vehicle-mounted camera of the target camera view of the j-th vehicle model, Rt v Represents the virtual camera external parameter matrix, Rt j The camera extrinsic matrix corresponding to the vehicle-mounted camera of the target camera view of the j-th vehicle model, S j represents the total number of images captured by the vehicle-mounted camera from the target camera perspective of the j-th vehicle model, and n represents the number of vehicle models.

[0060] In this embodiment, by obtaining the total number of images collected by the on-board camera of the target camera angle of view of each vehicle model, the target camera parameter matrix is ​​weighted and averaged according to the total number of images to obtain a virtual camera parameter matrix corresponding to the target camera angle of view, so that the determined virtual camera parameter matrix can be more biased towards the on-board cameras with a larger data volume, while taking into account the virtual cameras with a smaller data volume, which helps to reduce the impact of changes in image angle of view on image data.

[0061] Step 203: determining spherical space calibration points according to the virtual camera parameter matrix.

[0062] In the disclosed embodiment, after the virtual camera parameter matrix corresponding to the target camera viewing angle is determined, the spherical space calibration points can be further determined according to the virtual camera parameter matrix.

[0063] As an example, when determining the spherical space calibration points, you can first obtain the virtual camera's x-axis focal length, y-axis focal length, x-axis coordinate value and y-axis coordinate value of the virtual principal point from the virtual camera parameter matrix, and then determine the virtual camera's horizontal field of view angle according to the x-axis focal length and the x-axis coordinate value, and determine the virtual camera's vertical field of view angle according to the y-axis focal length and the y-axis coordinate value, and then determine the spherical space calibration points according to the horizontal field of view angle, vertical field of view angle, and the sphere radius and rotation angle corresponding to the target camera's viewing angle.

[0064] It can be understood that the virtual camera parameter matrix corresponds to a virtual camera. When the virtual camera parameter matrix is ​​known, the relevant parameters of the virtual camera can be directly obtained from the virtual camera parameter matrix, including the focal length in the x-axis direction, the focal length in the y-axis direction, the x-axis coordinate value and the y-axis coordinate value of the virtual principal point. Assuming that in the virtual camera parameter matrix, the virtual camera internal parameter matrix is ​​as shown in formula (2), then f x That is, the focal length in the x-axis direction, f y represents the focal length in the y-axis direction, c x Indicates the x-axis coordinate value of the virtual principal point, c y Indicates the y-axis coordinate value of the virtual principal point.

[0065]

[0066] In this embodiment, when determining the horizontal field of view angle and the vertical field of view angle, they can be calculated using the following formula (3):

[0067]

[0068] In the above formula (3), fov w Represents the horizontal field of view of the virtual camera, fov h represents the vertical field of view of the virtual camera, cx Indicates the x-axis coordinate value of the virtual principal point, f x represents the focal length in the x-axis direction, c y Indicates the y-axis coordinate value of the virtual principal point, f y Indicates the focal length along the y-axis.

[0069] Next, after obtaining the horizontal and vertical field angles of the virtual camera, the spherical space calibration points can be determined according to the horizontal and vertical field angles of the virtual camera, the radius of the sphere and the rotation angle corresponding to the target camera's viewing angle.

[0070] Among them, the sphere radius and rotation angle corresponding to the target camera perspective can be preset according to actual needs. Usually, the rotation angles (denoted as ω) of the front view, left front view, left rear view, right front view, right rear view and rear view cameras are not consistent. In this embodiment, the rotation angle corresponding to each camera perspective can be preset by the R&D personnel according to experience. For example, the rotation angles of the front view, left front view, left rear view, right front view, right rear view and rear view cameras can be preset to 90°, 140°, 220°, 40°, 320°, and 270°, respectively, so that the rotation angle corresponding to the target camera perspective can be determined. In addition, since the cameras with different camera perspectives have different focus areas, the sphere radius (denoted as r) corresponding to different camera perspectives is usually different. The sphere radius corresponding to each camera perspective can be preset by the R&D personnel according to experience, for example, the radius of the side view perspective is set smaller, and the radius of the front and rear view perspectives is set larger, so that the sphere radius corresponding to the target camera perspective can be determined.

[0071] Assume that the spatial three-dimensional coordinate system is constructed with the origin of the ego-vehicle coordinate system of the reference vehicle (which can be any vehicle model, used to provide the reference camera parameter matrix) as the center (usually the position of the grounding point of the rear axle of the vehicle is used as the origin of the ego-vehicle coordinate system) as follows Figure 3 As shown in the figure, a spatial sphere is constructed with the origin O of the ego-vehicle coordinate system as the center and the radius of the sphere corresponding to the target camera's perspective as the radius r. The four points A, B, C and D are located on the spatial sphere. For virtual cameras with different perspectives, assuming that they rotate around the z-axis, and the rotation angle with the positive direction of the x-axis (that is, the rotation angle of OG with the positive direction of the x-axis) is ω, then the ω of the front view, left front view, left rear view, right front view, right rear view and rear view cameras are all inconsistent.

[0072] like Figure 3 As shown, φ=∠AOE, we can get:

[0073]

[0074] Thus, the spatial coordinates of points A, B, C and D on the spatial sphere are:

[0075]

[0076] Therefore, points A, B, C and D on the spatial sphere are the determined spherical space calibration points.

[0077] Step 204 : determining virtual camera plane coordinate points based on the spherical space calibration points and the virtual camera parameter matrix, and determining real camera plane coordinate points based on the spherical space calibration points and the reference camera parameter matrix of the target camera viewing angle.

[0078] Among them, the reference camera parameter matrix can be a camera parameter matrix corresponding to the on-board camera of the target camera perspective installed on the reference vehicle, the reference camera parameter matrix includes a reference camera intrinsic parameter matrix and a reference camera extrinsic parameter matrix, and the reference vehicle can be a vehicle of any model.

[0079] In the disclosed embodiment, after the spherical space calibration points are determined, the virtual camera plane coordinate points can be determined based on the spherical space calibration points and the obtained virtual camera parameter matrix, and the real camera plane coordinate points can be determined based on the spherical space calibration points and the reference camera parameter matrix.

[0080] For the target camera perspective, the corresponding reference camera parameter matrix (i.e., the camera parameter matrix of the real camera) is known, and the virtual camera parameter matrix corresponding to the virtual camera has been obtained. Assuming that the determined spherical space calibration points are A, B, C, and D, and the positions of the spherical space calibration points are known, the real camera plane coordinate points and the virtual camera plane coordinate points corresponding to the spherical space calibration points can be solved according to the camera parameter matrix of the corresponding camera. The solution formula is shown in formula (4). It can be understood that the solved real camera plane coordinate points and virtual camera plane coordinate points are both pixel points.

[0081]

[0082] In the above formula (4), K represents the camera's intrinsic parameter matrix, which is a 3×3 matrix, Rt represents the camera's extrinsic parameter matrix, which is a 3×4 matrix, (x, y, z) represents the coordinates of the spherical space calibration points, and (u, v) is the coordinates of the camera plane coordinate points obtained. Therefore, in this embodiment, by substituting the virtual camera parameter matrix and the spherical space calibration points into the above formula (4), the corresponding virtual camera plane coordinate points can be obtained, and by substituting the reference camera parameter matrix and the spherical space calibration points into the above formula (4), the corresponding real camera plane coordinate points can be obtained.

[0083] Assuming that the spherical space calibration points are A, B, C and D, the mapping relationship of each spherical space calibration point on the virtual camera image and the mapping relationship on the real camera image are as follows: Figure 4As shown, the mapping point of point A on the real camera image (i.e., the real camera plane coordinate point) is P0, and the mapping point on the virtual camera image (i.e., the virtual camera plane coordinate point) is P0', the mapping point of point B on the real camera image is P1, and the mapping point on the virtual camera image is P1', the mapping point of point C on the real camera image is P2, and the mapping point on the virtual camera image is P2', the mapping point of point D on the real camera image is P3, and the mapping point on the virtual camera image is P3', then the coordinates of point A and the virtual camera parameter matrix are substituted into the mapping relationship represented by the above formula (4), and the coordinates of P0' can be determined. Similarly, the coordinates of point A and the reference camera parameter matrix are substituted into the above formula (4), and the coordinates of P0 can be determined. By analogy, the final real camera plane coordinate point is P0 (u 0 ,v 0 ), P1(u 1 ,v 1 ), P2(u 2 ,v 2 ), P3(u 3 ,v 3 ), the final virtual camera plane coordinate point is P0′(u′ 0 ,v′ 0 ), P1′(u 1 ′,v 1 ′), P2′(u′ 2 ,v′ 2 ), P3′(u′ 3 ,v′ 3 ).

[0084] Step 205 : determining a transformation matrix corresponding to the target camera viewing angle based on the virtual camera plane coordinate points and the real camera plane coordinate points.

[0085] In the disclosed embodiment, after the virtual camera plane coordinate points and the real camera plane coordinate points corresponding to the spherical space calibration points are determined, the transformation matrix corresponding to the target camera viewing angle can be obtained based on the virtual camera plane coordinate points and the real camera plane coordinate points.

[0086] Exemplarily, the transformation matrix from the real camera to the virtual camera can be obtained using OpenCV according to the corresponding points in the virtual camera plane coordinate points and the real camera plane coordinate points, that is, the transformation matrix corresponding to the target camera viewing angle is determined.

[0087] In this embodiment, according to Figure 2 The steps shown can determine the transformation matrix corresponding to each camera perspective, and can establish and store the corresponding relationship between the camera perspective and the transformation matrix, so that in actual application, the corresponding transformation matrix can be determined according to the camera perspective of the vehicle-mounted camera for image conversion.

[0088] The image conversion method of the disclosed embodiment obtains the target camera parameter matrix corresponding to the vehicle-mounted camera of the target camera perspective of multiple different vehicle models, determines the virtual camera parameter matrix according to the target camera parameter matrix, then determines the spherical space calibration points according to the virtual camera parameter matrix, determines the virtual camera plane coordinate points based on the spherical space calibration points and the virtual camera parameter matrix, and determines the real camera plane coordinate points based on the spherical space calibration points and the reference camera parameter matrix, and then determines the conversion matrix corresponding to the target camera perspective based on the virtual camera plane coordinate points and the real camera plane coordinate points. Thus, the conversion matrix corresponding to each camera perspective can be determined, which lays the foundation for using the conversion matrix to convert the perspective of the image during model training and vehicle-side visual perception task recognition. This solution unifies the perspective of the image through the virtual camera perspective conversion, which will be more conducive to the training and reasoning of the perspective perception task model, improve the model performance, and also help to improve the accuracy of vehicle-side visual perception task recognition.

[0089] In order to implement the above embodiments, the present disclosure further provides an image conversion device, which can be implemented by software and / or hardware and can be integrated in an electronic device.

[0090] Figure 5 A schematic diagram of the structure of an image conversion device provided by an embodiment of the present disclosure is shown in FIG. Figure 5 As shown, the image conversion device 30 may include: a data acquisition module 310 , a matrix determination module 320 and an image conversion module 330 .

[0091] The data acquisition module 310 is used to acquire the camera angle corresponding to each on-board camera on the vehicle and the original image captured by each on-board camera;

[0092] A matrix determination module 320, configured to determine a conversion matrix corresponding to each camera angle according to the camera angle corresponding to each vehicle-mounted camera, wherein the conversion matrix corresponding to each camera angle is determined according to a camera parameter matrix of vehicle-mounted cameras of corresponding angles of different vehicle models;

[0093] The image conversion module 330 is used to perform perspective conversion on the original image of the corresponding perspective based on the conversion matrix corresponding to each camera perspective to obtain a virtual image of the corresponding perspective.

[0094] Optionally, the image conversion device 30 further includes:

[0095] A first acquisition module is used to acquire a target camera parameter matrix corresponding to the vehicle-mounted camera of the target camera angle of view of multiple different vehicle models;

[0096] A first determination module, used to determine a virtual camera parameter matrix according to the target camera parameter matrix;

[0097] A second determination module, used to determine the spherical space calibration points according to the virtual camera parameter matrix;

[0098] A third determination module is used to determine virtual camera plane coordinate points based on the spherical space calibration points and the virtual camera parameter matrix, and to determine real camera plane coordinate points based on the spherical space calibration points and a reference camera parameter matrix of the target camera viewing angle;

[0099] The fourth determination module is used to determine the transformation matrix corresponding to the target camera viewing angle based on the virtual camera plane coordinate point and the real camera plane coordinate point.

[0100] Further optionally, the first determining module is further configured to:

[0101] Obtaining a total number of images captured by the vehicle-mounted camera of the target camera perspective of the multiple different vehicle models;

[0102] A virtual camera parameter matrix is ​​determined according to the total number of images captured by the vehicle-mounted camera of the target camera viewing angle of each vehicle model among the multiple different vehicle models and the target camera parameter matrix.

[0103] Further optionally, the target camera parameter matrix includes a camera intrinsic parameter matrix and a camera extrinsic parameter matrix, and correspondingly, the virtual camera parameter matrix includes a virtual camera intrinsic parameter matrix and a virtual camera extrinsic parameter matrix; the first determination module determines the virtual camera parameter matrix by the following formula:

[0104]

[0105] Among them, K v represents the virtual camera intrinsic parameter matrix, K j Represents the camera intrinsic parameter matrix corresponding to the vehicle-mounted camera of the target camera angle of view of the jth vehicle model, Rt v Represents the virtual camera external parameter matrix, Rt j represents the camera extrinsic matrix corresponding to the vehicle-mounted camera of the target camera angle of view of the j-th vehicle model, S j represents the total number of images captured by the vehicle-mounted camera of the target camera angle of view of the j-th vehicle model, and n represents the number of vehicle models.

[0106] Optionally, the second determining module is further used to:

[0107] Obtaining the x-axis focal length, y-axis focal length, x-axis coordinate value and y-axis coordinate value of the virtual principal point of the virtual camera from the virtual camera parameter matrix;

[0108] Determine the horizontal field of view angle of the virtual camera according to the focal length in the x-axis direction and the coordinate value in the x-axis direction, and determine the vertical field of view angle of the virtual camera according to the focal length in the y-axis direction and the coordinate value in the y-axis direction;

[0109] A spherical space calibration point is determined according to the horizontal field of view angle, the vertical field of view angle, and the sphere radius and the rotation angle corresponding to the target camera viewing angle.

[0110] Further optionally, the horizontal field of view angle and the vertical field of view angle of the virtual camera are determined by the following formula:

[0111]

[0112] Among them, fov w Represents the horizontal field of view of the virtual camera, fov h represents the vertical field of view of the virtual camera, c x Indicates the x-axis coordinate value of the virtual principal point, f x represents the focal length in the x-axis direction, c y Indicates the y-axis coordinate value of the virtual principal point, f y Represents the focal length in the y-axis direction.

[0113] The image conversion device applicable to electronic devices provided in the embodiments of the present disclosure can execute the image conversion method provided in the embodiments of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method. For the contents not described in detail in the embodiments of the present disclosure, reference can be made to the description in any method embodiment of the present disclosure.

[0114] The embodiment of the present disclosure also provides an electronic device, including a processor and a memory; the processor calls the program or instructions stored in the memory to execute the steps of each embodiment of the image conversion method as described in any of the above embodiments. To avoid repeated description, they are not repeated here.

[0115] The embodiments of the present disclosure also provide a computer-readable storage medium, which is non-transitory and stores programs or instructions. The programs or instructions enable a computer to execute the steps of each embodiment of the image conversion method as described in any of the aforementioned embodiments. To avoid repeated description, they will not be repeated here.

[0116] An embodiment of the present disclosure also provides a vehicle, comprising an image conversion device as described in any of the foregoing embodiments of the present disclosure, or an electronic device as described in any of the foregoing embodiments of the present disclosure, or a computer-readable storage medium as described in any of the foregoing embodiments of the present disclosure.

[0117] The embodiments of the present disclosure further provide a computer program product, which is used to execute the steps of each embodiment of the image conversion method as described in any of the aforementioned embodiments.

[0118] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0119] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image conversion method, characterized in that: The method comprises: Obtaining the camera angle corresponding to each on-board camera on the vehicle and the original image captured by each on-board camera; Determine a conversion matrix corresponding to each camera angle according to the camera angle corresponding to each vehicle-mounted camera, wherein the conversion matrix corresponding to each camera angle is determined according to a camera parameter matrix of vehicle-mounted cameras of corresponding angles of different vehicle models; Based on the conversion matrix corresponding to each camera perspective, the original image of the corresponding perspective is transformed to obtain a virtual image of the corresponding perspective.

2. The method according to claim 1, characterized in that The transformation matrix corresponding to the target camera's viewing angle is determined as follows: Obtaining a target camera parameter matrix corresponding to the vehicle-mounted cameras of the target camera viewing angles of multiple different vehicle models; Determine a virtual camera parameter matrix according to the target camera parameter matrix; Determine spherical space calibration points according to the virtual camera parameter matrix; Determine virtual camera plane coordinate points based on the spherical space calibration points and the virtual camera parameter matrix, and determine real camera plane coordinate points based on the spherical space calibration points and a reference camera parameter matrix of the target camera viewing angle; A transformation matrix corresponding to the target camera viewing angle is determined based on the virtual camera plane coordinate points and the real camera plane coordinate points.

3. The method according to claim 2, characterized in that Determining a virtual camera parameter matrix according to the target camera parameter matrix comprises: Obtaining a total number of images captured by the vehicle-mounted camera of the target camera perspective of the multiple different vehicle models; A virtual camera parameter matrix is ​​determined according to the total number of images captured by the vehicle-mounted camera of the target camera viewing angle of each vehicle type among the multiple different vehicle types and the target camera parameter matrix.

4. The method according to claim 3, characterized in that The target camera parameter matrix includes a camera intrinsic parameter matrix and a camera extrinsic parameter matrix. Correspondingly, the virtual camera parameter matrix includes a virtual camera intrinsic parameter matrix and a virtual camera extrinsic parameter matrix. The virtual camera parameter matrix is ​​determined by the following formula: Among them, K v represents the virtual camera intrinsic parameter matrix, K j Represents the camera intrinsic parameter matrix corresponding to the vehicle-mounted camera of the target camera angle of view of the jth vehicle model, Rt v Represents the virtual camera external parameter matrix, Rt j represents the camera extrinsic matrix corresponding to the vehicle-mounted camera of the target camera angle of view of the j-th vehicle model, S j represents the total number of images captured by the vehicle-mounted camera of the target camera angle of view of the j-th vehicle model, and n represents the number of vehicle models.

5. The method according to claim 2, characterized in that: The determining of the spherical space calibration points according to the virtual camera parameter matrix comprises: Obtaining the x-axis focal length, y-axis focal length, x-axis coordinate value and y-axis coordinate value of the virtual principal point of the virtual camera from the virtual camera parameter matrix; Determine the horizontal field of view angle of the virtual camera according to the focal length in the x-axis direction and the coordinate value in the x-axis direction, and determine the vertical field of view angle of the virtual camera according to the focal length in the y-axis direction and the coordinate value in the y-axis direction; A spherical space calibration point is determined according to the horizontal field of view angle, the vertical field of view angle, and the sphere radius and the rotation angle corresponding to the target camera viewing angle.

6. The method according to claim 5, characterized in that The horizontal field of view angle and the vertical field of view angle of the virtual camera are determined by the following formula: Among them, fov w Represents the horizontal field of view of the virtual camera, fov h represents the vertical field of view of the virtual camera, c x Indicates the x-axis coordinate value of the virtual principal point, f x represents the focal length in the x-axis direction, c y Indicates the y-axis coordinate value of the virtual principal point, f y Represents the focal length in the y-axis direction.

7. An image conversion device, characterized in that: include: A data acquisition module, used to acquire the camera angle corresponding to each on-board camera on the vehicle and the original image captured by each on-board camera; A matrix determination module, used to determine a conversion matrix corresponding to each camera angle according to the camera angle corresponding to each vehicle-mounted camera, wherein the conversion matrix corresponding to each camera angle is determined according to a camera parameter matrix of vehicle-mounted cameras of corresponding angles of different vehicle models; The image conversion module is used to perform perspective conversion on the original image of the corresponding perspective based on the conversion matrix corresponding to each camera perspective to obtain a virtual image of the corresponding perspective.

8. An electronic device, characterized in that: include: Processor and memory; The processor is used to execute the image conversion method according to any one of claims 1 to 6 by calling the program or instruction stored in the memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or an instruction, wherein the program or the instruction enables a computer to execute the image conversion method according to any one of claims 1 to 6.

10. A vehicle, characterized in that: It includes the image conversion device as claimed in claim 7, the electronic device as claimed in claim 8, or the computer-readable storage medium as claimed in claim 9.

Images