Projection parameter detection method and device, electronic equipment and vehicle

Abstract

The invention provides a projection parameter detection method and device, electronic equipment and a vehicle, relates to the technical field of sensor simulation, and can determine whether overall projection parameters are accurate or not according to geometric analysis results of a detection frame and an insertion frame in a camera image so as to further ensure the accuracy of a sensor simulation result. According to the method, the electronic equipment obtains a camera image shot by a camera of a vehicle, geometric analysis in the aspects of height or distance and the like is carried out on a detection frame corresponding to a background vehicle and an insertion frame corresponding to a virtual vehicle in the camera image, and therefore a geometric analysis result is obtained. Afterwards, the electronic device obtains the accuracy of the projection parameters according to the geometric analysis result. Wherein the geometric analysis result can reflect whether the overall projection parameters are accurate or not. And when the projection parameters are accurate, the whole projection parameters can be used in the subsequent sensor simulation process, so that the accuracy of the sensor simulation result is further ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sensor simulation, and in particular to a projection parameter detection method and device, an electronic device, and a vehicle. BACKGROUND

[0002] Sensor simulation refers to constructing highly realistic simulation data by projecting a three-dimensional (3D) foreground into each camera image of a vehicle, thereby supporting training and testing of automatic driving functions of the vehicle. In sensor simulation, road reconstruction parameters, camera internal and external parameters, and ego vehicle pose data are required as projection parameters to calculate the projection of the foreground in each camera image. The overall precision or accuracy of the projection parameters, including the road reconstruction parameters, camera internal and external parameters, and ego vehicle pose data, directly affects the accuracy of the sensor simulation result.

[0003] However, in some current technical solutions, the detection of the projection parameters is relatively single, for example, only the camera external parameters are detected, or only the ego vehicle pose is detected. Neither of the solutions can accurately obtain the overall precision or accuracy of the projection parameters, including the road reconstruction parameters, camera internal and external parameters, and ego vehicle pose data, thereby affecting the accuracy of the sensor simulation result. SUMMARY

[0004] The present application provides a projection parameter detection method and device, an electronic device, and a vehicle, which can determine whether the overall projection parameters are accurate according to the geometric analysis result of the detection frame and the insertion frame in the camera image, thereby further ensuring the accuracy of the sensor simulation result.

[0005] To achieve the above object, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, a projection parameter detection method is provided. Taking the case where the method is applied to an electronic device, the electronic device obtains a camera image captured by a camera of an ego vehicle, wherein the camera image includes a two-dimensional detection frame corresponding to a background vehicle and a two-dimensional insertion frame corresponding to a virtual vehicle, and the insertion frame is obtained by projecting the virtual vehicle onto the camera image. The electronic device performs geometric analysis on the detection frame and the insertion frame in the camera image to obtain a geometric analysis result, wherein the geometric analysis includes height or distance analysis of the detection frame and the insertion frame. The electronic device then obtains the accuracy of the projection parameters according to the geometric analysis result.

[0007] In the method, the insertion frame corresponding to the virtual vehicle can be constructed in the camera image according to the overall projection parameter, and geometric analysis can be performed on the insertion frame and the detection frame corresponding to the background vehicle in the camera image. The result of the geometric analysis can reflect whether the overall projection parameter is accurate. When the projection parameter is accurate, the overall projection parameter can be used in the subsequent sensor simulation process, so as to further ensure the accuracy of the sensor simulation result.

[0008] In addition, the method can perform geometric analysis on the insertion frame corresponding to the virtual vehicle and the detection frame corresponding to the background vehicle in the camera image without special reference objects, and the operation is relatively simple, the implementation difficulty is small, the detection process is short, the efficiency is high, and the result of whether the projection parameter is accurate can be given, so that the user can clearly understand the projection parameter detection situation.

[0009] In an implementation manner of the first aspect, the geometric analysis includes height analysis of the detection frame and the insertion frame, and when the geometric analysis result is obtained, the electronic device obtains distances between the detection frame and the insertion frame and the ego vehicle based on depth estimation for the camera image. When the distance between the detection frame and the ego vehicle is less than or equal to the distance between the insertion frame and the ego vehicle, the electronic device obtains the geometric analysis result according to a size relationship between a height of the detection frame and a first height threshold, where the first height threshold is less than a height of the insertion frame. When the distance between the detection frame and the ego vehicle is greater than the distance between the insertion frame and the ego vehicle, the electronic device obtains the geometric analysis result according to a size relationship between the height of the detection frame and a second height threshold, where the second height threshold is greater than the height of the insertion frame.

[0010] In the implementation manner, the electronic device can determine the geometric relationship between the detection frame and the insertion frame based on the height analysis of the detection frame and the insertion frame, that is, the electronic device can judge the detection frame and the insertion frame based on the rule of "near large and far small", so as to more accurately determine the accuracy of the projection parameter.

[0011] In an implementation manner of the first aspect, when the geometric analysis result is obtained according to the size relationship between the height of the detection frame and the first height threshold, when the height of the detection frame is greater than the first height threshold, the electronic device obtains the geometric analysis result that the geometric relationship between the detection frame and the insertion frame is normal. When the height of the detection frame is less than or equal to the first height threshold, the electronic device obtains the geometric analysis result that the geometric relationship between the detection frame and the insertion frame is abnormal.

[0012] In an implementation form of the first aspect, the geometric analysis further comprises a ratio analysis of the detection bounding box and the insertion bounding box; and the electronic device, when obtaining the geometric analysis result, obtains the geometric analysis result that the geometric relationship between the detection bounding box and the insertion bounding box is normal, if the height of the detection bounding box is less than or equal to a third height threshold, when the height of the detection bounding box is greater than a first height threshold, wherein the third height threshold is determined according to the second height threshold.

[0013] In an implementation form of the first aspect, the geometric analysis further comprises a ratio analysis of the detection bounding box and the insertion bounding box; and the electronic device, when obtaining the geometric analysis result, obtains the geometric analysis result that the geometric relationship between the detection bounding box and the insertion bounding box is normal, if the height of the detection bounding box is less than or equal to a third height threshold, when the height of the detection bounding box is greater than a first height threshold, wherein the third height threshold is determined according to the second height threshold.

[0014] In the above implementation form, the electronic device can judge the detection bounding box and the insertion bounding box from multiple angles based on the height analysis and the ratio analysis, so that the geometric analysis manner provided by the present application is more comprehensive, and the accuracy of the projection parameter can be more accurately obtained.

[0015] In an implementation form of the first aspect, the geometric analysis further comprises a ratio analysis of the detection bounding box and the insertion bounding box; and the electronic device, when obtaining the geometric analysis result, obtains the geometric analysis result that the geometric relationship between the detection bounding box and the insertion bounding box is normal, if the height of the detection bounding box is less than or equal to a third height threshold, when the height of the detection bounding box is greater than a first height threshold, wherein the third height threshold is determined according to the second height threshold.

[0016] In the above implementation form, the electronic device can judge the detection bounding box and the insertion bounding box from multiple angles based on the height analysis and the ratio analysis, so that the geometric analysis manner provided by the present application is more comprehensive, and the accuracy of the projection parameter can be more accurately obtained.

[0017] In an implementation form of the first aspect, the geometric analysis comprises a distance analysis of the detection bounding box and the insertion bounding box; and the electronic device, when obtaining the geometric analysis result, obtains, for the camera image, a distance between the detection bounding box and the ego vehicle based on the depth estimation. The electronic device obtains the geometric analysis result that the geometric relationship between the detection bounding box and the insertion bounding box is normal, if the distance between the detection bounding box and the ego vehicle is less than or equal to a first distance threshold. The electronic device obtains the geometric analysis result that the geometric relationship between the detection bounding box and the insertion bounding box is abnormal, if the distance between the detection bounding box and the ego vehicle is greater than the first distance threshold.

[0018] In the implementation manners, the situation of the insertion box or the detection box being abnormal in geometric relationship due to an error in vanishing point estimation on the camera image can be excluded, the detection box and the insertion box are judged from the perspective of vanishing point estimation, the geometric analysis manner provided by the application is more comprehensive, and therefore the accuracy of the projection parameter can be more accurately obtained.

[0019] In an implementation manner of the first aspect, the ego vehicle includes a plurality of cameras, each camera is configured to capture a plurality of camera images in time sequence; the detection box and the insertion box being abnormal in geometric relationship indicates that the camera image is unavailable, and the detection box and the insertion box being normal in geometric relationship indicates that the camera image is available. When the electronic device obtains the accuracy of the projection parameter according to the geometric analysis result, for one camera, if the number of unavailable camera images in the plurality of camera images corresponding to the camera is greater than or equal to a first preset threshold, the electronic device determines that a sub-projection parameter corresponding to the camera is abnormal, wherein the sub-projection parameter includes an intrinsic and extrinsic parameter of the camera, an ego vehicle pose corresponding to the camera, and a road reconstruction parameter corresponding to the camera. If the number of cameras with abnormal sub-projection parameters in the plurality of cameras is greater than or equal to a second preset threshold, the electronic device determines that the projection parameter is inaccurate, wherein the projection parameter includes the intrinsic and extrinsic parameter of each camera, the ego vehicle pose corresponding to each camera, and the road reconstruction parameter corresponding to each camera.

[0020] In the implementation manners, the electronic device can determine the accuracy of the overall projection parameter in the case that the plurality of camera images captured by the plurality of cameras are obtained, so that the detection including all the projection parameters corresponding to all the cameras is accurate.

[0021] In an implementation manner of the first aspect, the ego vehicle includes a plurality of cameras, each camera is configured to capture a plurality of camera images in time sequence; the detection box and the insertion box being abnormal in geometric relationship indicates that the camera image is unavailable, and the detection box and the insertion box being normal in geometric relationship indicates that the camera image is available. When the electronic device obtains the accuracy of the projection parameter according to the geometric analysis result, for one camera, if the number of unavailable camera images in the plurality of camera images corresponding to the camera is greater than or equal to a first preset threshold, the electronic device determines that a sub-projection parameter corresponding to the camera is abnormal. If all the cameras with abnormal sub-projection parameters belong to a preset type of cameras, the electronic device determines that the projection parameter is inaccurate, wherein the preset type includes a fisheye camera or a pinhole camera, and the projection parameter includes the intrinsic and extrinsic parameter of each camera, the ego vehicle pose corresponding to each camera, and the road reconstruction parameter corresponding to each camera.

[0022] In the implementation manners, similarly, the electronic device can also determine the accuracy of the overall projection parameter in the case that the plurality of camera images captured by the plurality of cameras are obtained, so that the detection including all the projection parameters corresponding to all the cameras is accurate.

[0023] In an implementation form of the first aspect, the camera image includes at least one detection box, the virtual vehicle includes at least one, and the camera image includes at least one insertion box; when the geometric analysis result is obtained by the electronic device, when the detection box is multiple and the insertion box is multiple, the electronic device determines, for each insertion box, whether there is any detection box that has an abnormal geometric relationship with the insertion box, and if so, determines that the insertion box is abnormal; and if the number of abnormal insertion boxes in one camera image is greater than or equal to a third preset threshold, the geometric analysis result that the detection box and the insertion box have an abnormal geometric relationship is obtained.

[0024] In the implementation form described above, when there are multiple detection boxes or multiple insertion boxes in one camera image, the electronic device can perform geometric analysis based on the multiple detection boxes or the multiple insertion boxes in this case, so as to more accurately obtain the geometric analysis result.

[0025] In a second aspect, a device for detecting projection parameters is provided, and the device includes:

[0026] An image acquisition module is configured to acquire a camera image captured by a camera of a host vehicle; the camera image includes a two-dimensional detection box corresponding to a background vehicle and a two-dimensional insertion box corresponding to a virtual vehicle, and the insertion box is obtained by projecting the virtual vehicle onto the camera image.

[0027] In an implementation form of the second aspect, the geometric analysis includes height analysis of the detection box and the insertion box; the analysis module is specifically configured to acquire, for the camera image, distances between the detection box and the insertion box and the host vehicle based on depth estimation; when the distance between the detection box and the host vehicle is less than or equal to the distance between the insertion box and the host vehicle, the geometric analysis result is obtained according to a size relationship between a height of the detection box and a first height threshold; the first height threshold is less than the height of the insertion box; and when the distance between the detection box and the host vehicle is greater than the distance between the insertion box and the host vehicle, the geometric analysis result is obtained according to a size relationship between the height of the detection box and a second height threshold; the second height threshold is greater than the height of the insertion box.

[0028] In an implementation form of the second aspect, the analysis module is specifically configured to obtain a geometric analysis result that the detection box and the insertion box have a normal geometric relationship when the height of the detection box is greater than a first height threshold; and obtain a geometric analysis result that the detection box and the insertion box have an abnormal geometric relationship when the height of the detection box is less than or equal to the first height threshold.

[0029] In an implementation form of the second aspect, the analysis module is specifically configured to: when the height of the detection box is greater than the second height threshold, obtain a geometric analysis result that the geometric relationship between the detection box and the insertion box is abnormal; and when the height of the detection box is less than or equal to the second height threshold, obtain a geometric analysis result that the geometric relationship between the detection box and the insertion box is normal.

[0030] In an implementation form of the second aspect, the geometric analysis further comprises scale analysis of the detection box and the insertion box; and the analysis module is specifically configured to: when the height of the detection box is greater than the first height threshold, if the height of the detection box is less than or equal to a third height threshold, obtain a geometric analysis result that the geometric relationship between the detection box and the insertion box is normal; and the third height threshold is determined according to the second height threshold.

[0031] In an implementation form of the second aspect, the geometric analysis further comprises scale analysis of the detection box and the insertion box; and the analysis module is specifically configured to: when the height of the detection box is less than or equal to the second height threshold, if the height of the detection box is greater than a fourth height threshold, obtain a geometric analysis result that the geometric relationship between the detection box and the insertion box is normal; and the fourth height threshold is determined according to the first height threshold.

[0032] In an implementation form of the second aspect, the geometric analysis comprises distance analysis of the detection box and the insertion box; and the analysis module is specifically configured to: for the camera image, based on the depth estimation, obtain a distance between the detection box and the ego vehicle; when the distance between the detection box and the ego vehicle is less than or equal to a first distance threshold, obtain a geometric analysis result that the geometric relationship between the detection box and the insertion box is normal; and when the distance between the detection box and the ego vehicle is greater than the first distance threshold, obtain a geometric analysis result that the geometric relationship between the detection box and the insertion box is abnormal.

[0033] In an implementation form of the second aspect, the ego vehicle comprises a plurality of cameras, each camera is configured to capture a plurality of camera images in time sequence; the geometric relationship between the detection box and the insertion box being abnormal indicates that the camera image is unusable, and the geometric relationship between the detection box and the insertion box being normal indicates that the camera image is usable; and the projection parameter detection module is specifically configured to: for one camera, if a number of unusable camera images in the plurality of camera images corresponding to the camera is greater than or equal to a first preset threshold, determine that a sub-projection parameter corresponding to the camera is abnormal; the sub-projection parameter comprises internal and external parameters of the camera, a pose of the ego vehicle corresponding to the camera, and road reconstruction parameters corresponding to the camera; and if a number of cameras with abnormal sub-projection parameters in the plurality of cameras is greater than or equal to a second preset threshold, determine that the projection parameter is inaccurate; the projection parameter comprises internal and external parameters of each camera, a pose of the ego vehicle corresponding to each camera, and road reconstruction parameters corresponding to each camera.

[0034] In an implementation form of the second aspect, the ego vehicle comprises a plurality of cameras, each camera configured to capture a plurality of camera images in time sequence; the geometric relationship between the bounding box and the insertion box being abnormal indicates that the camera image is unavailable, and the geometric relationship between the bounding box and the insertion box being normal indicates that the camera image is available; the projection parameter detection module is specifically configured to, for one camera, if the number of unavailable camera images in the plurality of camera images corresponding to the camera is greater than or equal to a first preset threshold, determine that the sub-projection parameter corresponding to the camera is abnormal; the sub-projection parameter comprises intrinsic and extrinsic parameters of the camera, the ego vehicle pose corresponding to the camera, and the road reconstruction parameter corresponding to the camera; if all the cameras with abnormal sub-projection parameters belong to a preset type of cameras, it is determined that the projection parameter is inaccurate; the preset type comprises fisheye cameras or pinhole cameras; the projection parameter comprises intrinsic and extrinsic parameters of each camera, the ego vehicle pose corresponding to each camera, and the road reconstruction parameter corresponding to each camera.

[0035] In an implementation form of the second aspect, there is at least one bounding box, at least one virtual vehicle, and at least one insertion box; the analysis module is specifically configured to, when there are a plurality of bounding boxes and a plurality of insertion boxes, for each insertion box, if there is any bounding box with an abnormal geometric relationship with the insertion box, it is determined that the insertion box is abnormal; if the number of abnormal insertion boxes in one camera image is greater than or equal to a third preset threshold, a geometric analysis result of the geometric relationship between the bounding box and the insertion box is obtained.

[0036] In a third aspect, an electronic device is provided, comprising a memory and one or more processors; the memory is coupled to the processor; wherein the memory stores computer program code, and the computer program code comprises computer instructions, which, when executed by the processor, cause the electronic device to perform the projection parameter detection method in the first aspect and any implementation form thereof.

[0037] In a fourth aspect, a vehicle is provided, comprising a memory and one or more processors; the memory is coupled to the processor; wherein the memory stores computer program code, and the computer program code comprises computer instructions, which, when executed by the processor, cause the vehicle to perform the projection parameter detection method in the first aspect and any implementation form thereof.

[0038] In a fifth aspect, a computer readable storage medium is provided, comprising computer instructions, which, when executed on an electronic device or a vehicle, cause the electronic device or the vehicle to perform the projection parameter detection method in the first aspect and any implementation form thereof.

[0039] In a sixth aspect, a computer program product is provided, which, when executed on a computer, causes the computer to perform the projection parameter detection method in the first aspect and any implementation form thereof.

[0040] The projection parameter detection apparatus provided in the second aspect, the electronic device provided in the third aspect, the vehicle provided in the fourth aspect, the computer readable storage medium provided in the fifth aspect and the computer program product provided in the sixth aspect can achieve the beneficial effects as described in the first aspect and any one of the implementation manners, which will not be described herein again. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 A perspective projection diagram provided by the embodiments of the present application is shown;

[0042] Figure 2 A structure diagram of an electronic device provided by the embodiments of the present application is shown Figure One .

[0043] Figure 3 A camera image diagram provided by the embodiments of the present application is shown Figure One .

[0044] Figure 4 A camera image diagram provided by the embodiments of the present application is shown Figure Two .

[0045] Figure 5 A camera image diagram provided by the embodiments of the present application is shown Figure Three .

[0046] Figure 6 A flow diagram of a projection parameter detection method provided by the embodiments of the present application is shown Figure One .

[0047] Figure 7 A flow diagram of a projection parameter detection method provided by the embodiments of the present application is shown Figure Two .

[0048] Figure 8 A flow diagram of a projection parameter detection method provided by the embodiments of the present application is shown Figure Three .

[0049] Figure 9 A structure diagram of a projection parameter detection apparatus provided by the embodiments of the present application is shown;

[0050] Figure 10 A structure diagram of an electronic device provided by the embodiments of the present application is shown Figure Two . DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. In the description of the present application, unless otherwise specified, " / " represents an "or" relationship between the objects before and after the " / ", for example, A / B can represent A or B; in the present application, "and / or" is only a description of the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In addition, in the description of the present application, unless otherwise specified, "multiple" means two or more than two. "At least one of the following" or similar expressions means any combination of the items, including any combination of single item or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple. In addition, in order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, "first", "second", and the like are used to distinguish the same items or similar items with basically the same function and effect. Those skilled in the art can understand that "first", "second", and the like do not limit the quantity and execution order, and "first", "second", and the like do not necessarily mean different. At the same time, in the embodiments of the present application, "exemplary" or "for example" means to serve as an example, illustration or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the use of "exemplary" or "for example" is intended to present the relevant concept in a specific manner, for understanding.

[0052] In addition, the business scenarios described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that as new business scenarios appear, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0053] In sensor simulation, road reconstruction parameters, camera internal and external parameters, ego pose and other data are needed as projection parameters, and these projection parameters are used to calculate the projection of the foreground in each camera image, as shown in Figure 1 The accuracy of the projection parameters directly affects the accuracy of the results of the sensor simulation work, so quality detection or accuracy detection of the projection parameters is a very important task.

[0054] In some technical solutions, the error value of the spatial point cloud re-projection can be calculated, and the ego vehicle pose can be optimized according to the error value. Alternatively, the quality of the camera extrinsic parameters of the vehicle-mounted camera can be detected mainly by two types of methods. One type of method is to perform static quality detection of the camera extrinsic parameters based on special targets with geometric features. This type of method can obtain relatively accurate detection results, but requires special detection targets and needs to be performed in a static or empty state of the vehicle, and the application occasions are limited. Another type of method is to perform dynamic detection of the camera extrinsic parameters based on existing features such as traffic marking line vanishing points and parallel lines, but this type of method requires that the reference objects be clearly visible and not be occluded.

[0055] In the above technical solutions, the detection of the projection parameters is relatively single, for example, only the camera extrinsic parameters are detected, or only the ego vehicle pose is detected. However, the overall accuracy or accuracy of the projection parameters including the road reconstruction parameters, the camera intrinsic and extrinsic parameters, and the ego vehicle pose directly affects the accuracy of the sensor simulation results. None of the above detection methods of the projection parameters can accurately obtain the overall accuracy or accuracy of the projection parameters including the road reconstruction parameters, the camera intrinsic and extrinsic parameters, and the ego vehicle pose, thereby affecting the accuracy of the sensor simulation results.

[0056] In addition, the above method for optimizing the ego vehicle pose has a large implementation difficulty, a long time consumption, and cannot provide a detection result, so the user cannot explicitly know whether the ego vehicle pose is accurate. In addition, the above detection methods of the camera extrinsic parameters all require special reference objects, and the operation is relatively complex, so the detection process has a long time consumption and low efficiency.

[0057] Based on the above, the embodiments of the present application provide a detection method of projection parameters. The method can obtain a camera image captured by a camera of an ego vehicle, wherein the camera image includes a two-dimensional detection box corresponding to a background vehicle and a two-dimensional insertion box corresponding to a virtual vehicle, and the insertion box is obtained by projecting the virtual vehicle onto the camera image. Then, the detection box and the insertion box in the camera image are subjected to geometric analysis to obtain a geometric analysis result, wherein the geometric analysis includes height or distance analysis of the detection box and the insertion box. Finally, the accuracy of the projection parameters is obtained according to the geometric analysis result.

[0058] In the above method, the insertion box corresponding to the virtual vehicle can be constructed in the camera image according to the overall projection parameters, and the insertion box and the detection box corresponding to the background vehicle in the camera image are subjected to geometric analysis. The result of the geometric analysis can reflect whether the overall projection parameters are accurate. When the projection parameters are accurate, the overall projection parameters can be used in the subsequent sensor simulation process, thereby further ensuring the accuracy of the sensor simulation results.

[0059] In addition, the method performs geometric analysis on the insertion frame corresponding to the virtual vehicle and the detection frame corresponding to the background vehicle in the camera image, without the need for special reference objects and without requirements for the driving state of the vehicle, and is simple to operate, easy to implement, short in detection process, high in efficiency, and capable of providing a result of whether the projection parameters are accurate, so that the user can clearly understand the detection situation of the projection parameters.

[0060] The projection parameter detection method can be applied to an electronic device for detecting projection parameters. As shown in the figure, the electronic device can include a virtual frame projection module 201, a background vehicle detection module 202, and a quality inspection judgment module 203. Figure 2

[0061] The virtual frame projection module 201 is mainly used to construct a virtual vehicle in a world coordinate system in which the ego vehicle is located, obtain a 3D frame of the virtual vehicle in the world coordinate system, and project the 3D frame onto a camera image of the ego vehicle by using projection parameters. After projection, the 3D frame corresponds to a 2D insertion frame in the camera image.

[0062] It can be understood that the "ego vehicle" refers to a vehicle in which the electronic device is located, or refers to a source vehicle of the camera image, wherein at least one camera is arranged on the vehicle, and each camera can obtain a camera image from the respective position, direction, etc.

[0063] The ego vehicle corresponds to an ego vehicle pose in the world coordinate system. Since the camera image is captured by the camera of the ego vehicle, objects or background vehicles in the camera image also correspond to spatial coordinates in the world coordinate system in which the ego vehicle is located, and correspond to two-dimensional coordinates in the camera image. In addition, after the virtual frame projection module 201 constructs the virtual vehicle, the virtual vehicle also corresponds to spatial coordinates in the world coordinate system, and correspondingly, the 3D frame corresponding to the virtual vehicle also corresponds to spatial coordinates in the world coordinate system, and the 2D frame corresponds to two-dimensional coordinates in the camera image.

[0064] The background vehicle detection module 202 is mainly used to detect the background vehicle in the camera image and obtain a 2D detection frame corresponding to the background vehicle.

[0065] The quality inspection judgment module 203 is mainly used to perform geometric analysis (or quality inspection judgment) on the insertion frame and the detection frame in the camera image, so as to determine whether the projection parameters are accurate according to the result of the geometric analysis.

[0066] The geometric analysis includes height or distance analysis of the detection frame and the insertion frame.

[0067] In some possible implementation manners, the height analysis can be performed based on the rule that "the nearer is larger and the farther is smaller". For example, the detection frame or the insertion frame in the camera image that is closer to the ego vehicle should meet the rule of being larger in size (e.g., being higher). ​

[0068] For example, referring to (a) and (b) of FIG. 13, when performing height analysis, the quality inspection judgment module 203 can obtain the distances between the detection frame a1 and the ego vehicle and between the insertion frame b1 and the ego vehicle based on the depth estimation for the camera image, where the distance between the detection frame a1 and the ego vehicle is L1, and the distance between the insertion frame b1 and the ego vehicle is M1. Figure 3 When L1 is less than or equal to M1, it indicates that the detection frame a1 is closer to the ego vehicle, and then according to the rule of "closer larger and farther smaller", as shown in (a) of FIG. 13, the height of the detection frame a1 should be higher, or the height should be greater than a certain threshold (such as a first height threshold), where the first height threshold can be determined according to the height of the insertion frame b1, for example, less than the height of the insertion frame b1. If it is detected that the height of the detection frame a1 is greater than the first height threshold, it indicates that the detection frame a1 and the insertion frame b1 meet the rule of "closer larger and farther smaller", and the geometric relationship between the detection frame a1 and the insertion frame b1 is normal, and if it is detected that the height of the detection frame a1 is less than or equal to the first height threshold, it indicates that the detection frame a1 and the insertion frame b1 do not meet the rule of "closer larger and farther smaller", and the geometric relationship between the detection frame a1 and the insertion frame b1 is abnormal.

[0069] Figure 3 When L1 is greater than M1, it indicates that the detection frame a1 is farther from the ego vehicle, and then according to the rule of "closer larger and farther smaller", as shown in (b) of FIG. 13, the height of the detection frame a1 should be lower, or the height should be less than or equal to a certain threshold (such as a second height threshold), where the second height threshold can be determined according to the height of the insertion frame b1, for example, greater than the height of the insertion frame b1. If it is detected that the height of the detection frame a1 is greater than the second height threshold, it indicates that the detection frame a1 and the insertion frame b1 do not meet the rule of "closer larger and farther smaller", and the geometric relationship between the detection frame a1 and the insertion frame b1 is abnormal, and if it is detected that the height of the detection frame a1 is less than or equal to the second height threshold, it indicates that the detection frame a1 and the insertion frame b1 meet the rule of "closer larger and farther smaller", and the geometric relationship between the detection frame a1 and the insertion frame b1 is normal.

[0070] In other possible implementations, the height analysis can also be performed based on the rule of "closer larger and farther smaller" and the rule of proportion. For example, the detection frame or the insertion frame closer to the ego vehicle on the camera image should meet the rule of larger size (such as higher), and the detection frame and the insertion frame should also meet the requirement of the proportion relationship. Figure 3

[0071]

[0072] ​​​For example, when the height of the detection frame a1 is greater than the first height threshold, if the height of the detection frame a1 is less than or equal to a third height threshold, it indicates that the detection frame a1 and the insertion frame b1 meet the requirement of the proportional relationship, and the proportional relationship between the detection frame a1 and the insertion frame b1 is normal. The third height threshold is determined according to the second height threshold, for example, can be the second height threshold multiplied by a proportional value. When the detection frame a1 and the insertion frame b1 meet the "near large and far small" rule, if the proportional relationship between the detection frame a1 and the insertion frame b1 is normal, it can be further determined that the geometric relationship between the detection frame a1 and the insertion frame b1 is normal.

[0073] When the height of the detection frame a1 is greater than the first height threshold, if the height of the detection frame a1 is greater than the third height threshold, it indicates that the detection frame a1 and the insertion frame b1 do not meet the requirement of the proportional relationship, and the proportional relationship between the detection frame a1 and the insertion frame b1 is abnormal. When the detection frame a1 and the insertion frame b1 meet the "near large and far small" rule, if the proportional relationship between the detection frame a1 and the insertion frame b1 is abnormal, it can be further determined that the geometric relationship between the detection frame a1 and the insertion frame b1 is abnormal.

[0074] For example, when the height of the detection frame a1 is less than or equal to the second height threshold, if the height of the detection frame a1 is greater than a fourth height threshold, it indicates that the detection frame a1 and the insertion frame b1 meet the requirement of the proportional relationship, and the proportional relationship between the detection frame a1 and the insertion frame b1 is normal. The fourth height threshold is determined according to the first height threshold, for example, can be the first height threshold multiplied by a proportional value. When the detection frame a1 and the insertion frame b1 meet the "near large and far small" rule, if the proportional relationship between the detection frame a1 and the insertion frame b1 is normal, it can be further determined that the geometric relationship between the detection frame a1 and the insertion frame b1 is normal.

[0075] When the height of the detection frame a1 is less than or equal to the second height threshold, if the height of the detection frame a1 is less than or equal to the fourth height threshold, it indicates that the detection frame a1 and the insertion frame b1 do not meet the requirement of the proportional relationship, and the proportional relationship between the detection frame a1 and the insertion frame b1 is abnormal. When the detection frame a1 and the insertion frame b1 meet the "near large and far small" rule, if the proportional relationship between the detection frame a1 and the insertion frame b1 is abnormal, it can be further determined that the geometric relationship between the detection frame a1 and the insertion frame b1 is abnormal.

[0076] In other possible implementations, height analysis may also include distance analysis. For example, the quality inspection judgment module 203 determines the position of the vanishing point (such as the horizon) in the camera image by detecting or estimating its depth. Based on the position of the vanishing point, it detects background vehicles in the camera image, determines the detection boxes corresponding to the background vehicles, and projects the 3D insertion boxes corresponding to the virtual vehicles into the camera image according to projection parameters to obtain the insertion boxes corresponding to the virtual vehicles. If the vanishing point position is accurate, the determined positions of the detection boxes, insertion boxes, etc., will also be relatively accurate; if the vanishing point position is inaccurate, the determined positions of the detection boxes, insertion boxes, etc., will also be inaccurate.

[0077] For example, see Figure 4 As shown in (a) and (b) in the figure, when performing distance analysis, the quality inspection judgment module 203 can obtain the distances between the detection box a1 and the insertion box b1 and the vehicle respectively based on depth estimation for the camera image, wherein the distance between the detection box a1 and the vehicle is L1 and the distance between the insertion box b1 and the vehicle is M1.

[0078] If the vanishing point estimation on the camera image is abnormal, causing the vanishing point to appear as if it is "floating upwards" on the camera image, such as... Figure 4 As shown in (a), or, this causes the vanishing point to be closer to the vehicle in the camera image, such as... Figure 4 As shown in (b), this situation can easily lead to abnormal position determination of the detection box a1 and the insertion box b1, such as the detection box a1 and the insertion box b1 being far apart. A preset threshold can be used to determine the position of the detection box a1 and the insertion box b1. If L1 is less than or equal to the first distance threshold, it indicates that the distance between the detection box a1 and the vehicle meets the distance requirements, the vanishing point estimation in the camera image is correct, or the vanishing point estimation has no effect on the detection box a1, thus confirming that the geometric relationship between the detection box and the insertion box is normal. Alternatively, if M1 is less than or equal to the second distance threshold, it indicates that the distance between the insertion box b1 and the vehicle meets the distance requirements, the vanishing point estimation in the camera image is correct, or the vanishing point estimation has no effect on the insertion box b1, thus confirming that the geometric relationship between the detection box and the insertion box is normal.

[0079] If L1 is greater than the first distance threshold, it indicates that the distance between the detection box a1 and the vehicle does not meet the distance requirements, the vanishing point estimation in the camera image is incorrect, or the vanishing point estimation affects the detection box a1. Therefore, it can be determined that the geometric relationship between the detection box and the insertion box is abnormal. Alternatively, if M1 is greater than the second distance threshold, it indicates that the distance between the insertion box b1 and the vehicle does not meet the distance requirements, the vanishing point estimation in the camera image is incorrect, or the vanishing point estimation affects the insertion box b1. Therefore, it can be determined that the geometric relationship between the detection box and the insertion box is normal.

[0080] The above implementation provides multiple ways of geometric analysis, so that the detection frame and the insertion frame can be judged from multiple aspects or angles, making the geometric analysis provided by the present application more comprehensive, and thus the accuracy of the projection parameters can be more accurately obtained.

[0081] In some possible implementation ways, no matter which way is used to determine that the geometric relationship between the detection frame and the insertion frame is abnormal, it can be considered that the projection parameters are inaccurate. Specifically, due to the inaccuracy of the projection parameters, the position of the insertion frame of the virtual vehicle projected onto the camera image is also not accurate enough, so the distance between the insertion frame and the ego vehicle determined according to the depth estimation is not accurate enough, and further, the results of the geometric analysis such as the height analysis and the distance analysis of the insertion frame and the detection frame are affected, so that the geometric relationship between the detection frame and the insertion frame is abnormal.

[0082] Therefore, no matter which way is used to determine that the geometric relationship between the detection frame and the insertion frame is abnormal, it can be considered that the projection parameters are inaccurate. Further, other projection parameters can be used in the subsequent sensor simulation process, or the projection parameters can be modified. No matter which way is used to determine that the geometric relationship between the detection frame and the insertion frame is normal, it can be considered that the projection parameters are accurate. Further, the projection parameters can be used in the subsequent sensor simulation process.

[0083] The above implementation can be understood as how the quality inspection judgment module 203 determines whether the projection parameters are accurate when the camera image includes one detection frame and one insertion frame.

[0084] In some possible implementation ways, the background vehicle detection module 202 can detect more than one background vehicle from the camera image, and correspondingly, the camera image can include detection frames corresponding to more than one background vehicle. In addition, the virtual frame projection module 201 can also construct more than one virtual vehicle in the world coordinate system, and correspondingly, the camera image can also include insertion frames corresponding to more than one virtual vehicle.

[0085] In this case, the insertion frame and the detection frame can be paired for geometric analysis respectively, so as to obtain multiple geometric analysis results, and finally determine whether the projection parameters are accurate in combination with the multiple geometric analysis results.

[0086] For example, referring to FIG. 8, Figure 5 As shown in FIG. 8, the camera image includes detection frames a1, a2 and a3, and also includes insertion frames b1, b2 and b3. The quality inspection judgment module 203 can obtain distances L1, L2 and L3 between the detection frames a1, a2 and a3 and the ego vehicle respectively based on depth estimation, and also obtain distances M1, M2 and M3 between the insertion frames b1, b2 and b3 and the ego vehicle respectively.

[0087] Wherein, when performing the geometric analysis, the quality inspection judgment module 203 can compare each detection frame with each insertion frame respectively for each insertion frame, and if there is any detection frame with abnormal geometric relationship with the insertion frame, it is determined that the insertion frame is abnormal. For example, the detection frames a1, a2 and a3 in the insertion frame b1 are respectively analyzed for geometric relationship, and if the geometric relationship between the detection frame a1 and the insertion frame b1 is abnormal, it is determined that the insertion frame b1 is abnormal. Figure 5

[0088] Further, if the number of abnormal insertion frames in a camera image is greater than or equal to a third preset threshold, it means that the number of abnormal insertion frames is too large, so it can be further determined that the geometric relationship between the detection frame and the insertion frame in the camera image is abnormal, that is, the current geometric analysis result is that the geometric relationship between the detection frame and the insertion frame is abnormal, and it is further determined that the projection parameter is inaccurate.

[0089] And if the number of abnormal insertion frames is less than the third preset threshold, it means that the number of abnormal insertion frames is very small, so it can be further determined that the geometric relationship between the detection frame and the insertion frame in the camera image is normal, that is, the current geometric analysis result is that the geometric relationship between the detection frame and the insertion frame is normal, and it is further determined that the projection parameter is accurate.

[0090] The above implementation manner can be used for the case where there are multiple detection frames and / or multiple insertion frames in a camera image, and the geometric analysis result of the detection frame and the insertion frame can be further determined in combination with the abnormal situation of the multiple detection frames and / or multiple insertion frames, fully considering the influence of each detection frame and / or insertion frame on the geometric analysis, and thus the projection parameter can be more accurately determined.

[0091] In some possible implementation manners, the above self-driving vehicle can include multiple cameras, for example, 11 cameras are configured at different positions of some vehicles, which can include 4 fisheye cameras and 7 pinhole cameras, etc. Each camera can capture multiple camera images in time sequence. That is, the self-driving vehicle camera obtains time sequence frame images, which include at least one camera image in time sequence.

[0092] ​When performing sensor simulation, projection needs to be performed for each camera image taken by each camera. Since the position, angle and the like of each camera are different, the projection parameters used when projecting the camera image of each camera are different. In order to ensure that the detection including all projection parameters corresponding to all cameras is accurate, in some possible implementation manners, the electronic device can further determine whether the projection parameter (or referred to as a sub-projection parameter) corresponding to each camera is normal based on the content in the above implementation manner, and further determine the accuracy of the projection parameter of the whole ego vehicle based on the condition of the projection parameter corresponding to each camera.

[0093] The projection parameter (or referred to as a sub-projection parameter) corresponding to the camera includes the intrinsic and extrinsic parameters of the camera to which the camera image belongs, the ego vehicle pose corresponding to the camera and the road reconstruction parameter corresponding to the camera. The projection parameter of the whole ego vehicle includes the intrinsic and extrinsic parameters of each camera, the ego vehicle pose corresponding to each camera and the road reconstruction parameter corresponding to each camera.

[0094] The virtual frame projection module 201 in the electronic device can project the 3D frame corresponding to the virtual vehicle into the plurality of camera images corresponding to the plurality of cameras respectively, and the background vehicle detection module 202 can detect the background vehicle in the plurality of camera images respectively and obtain the detection frame in each camera image. The quality inspection judgment module 203 determines whether the sub-projection parameter corresponding to each camera is normal, and further determines the accuracy of the projection parameter of the whole ego vehicle based on the condition of the projection parameter corresponding to each camera.

[0095] For example, for a camera, if the number of unusable camera images in the plurality of camera images corresponding to the camera is greater than or equal to a first preset threshold, it means that there are too many unusable camera images, which may have a great impact on the process of sensor simulation, and the quality inspection judgment module 203 determines that the sub-projection parameter corresponding to the camera is abnormal; if the number of unusable camera images in the plurality of camera images corresponding to the camera is less than the first preset threshold, it means that there are few unusable camera images, which has little impact on the process of sensor simulation, and the quality inspection judgment module 203 determines that the sub-projection parameter corresponding to the camera is normal. The availability of the camera image can be represented by the geometric relationship between the detection frame and the inserted frame, wherein the abnormal geometric relationship between the detection frame and the inserted frame indicates that the camera image is unusable, and the normal geometric relationship between the detection frame and the inserted frame indicates that the camera image is usable.

[0096] If the number of cameras with abnormal sub-projection parameters is greater than or equal to the second preset threshold, it indicates that there are too many cameras with abnormal sub-projection parameters, which may greatly affect the process of sensor simulation, and the quality inspection determination module 203 determines that the projection parameters are inaccurate; if the number of cameras with abnormal sub-projection parameters is less than the second preset threshold, it indicates that there are few cameras with abnormal sub-projection parameters, which has little effect on the process of sensor simulation, and the quality inspection determination module 203 determines that the projection parameters are accurate.

[0097] In some possible implementation manners, according to actual business requirements, the projection parameters determined by processing the camera images captured by specific cameras may be more meaningful, where the specific cameras can be pinhole cameras or fisheye cameras. In this case, after the quality inspection determination module 203 determines the sub-projection parameters corresponding to each camera, if all the cameras with abnormal sub-projection parameters belong to a preset type of cameras, such as fisheye cameras or pinhole cameras, the quality inspection determination module 203 determines that the projection parameters are inaccurate. If none of the cameras with abnormal sub-projection parameters belongs to the preset type of cameras, the quality inspection determination module 203 determines that the projection parameters are accurate.

[0098] For example, the preset type is a fisheye camera. If all the cameras with abnormal sub-projection parameters are fisheye cameras, and all the cameras with normal sub-projection parameters are pinhole cameras, the influence of the sub-projection parameters corresponding to the fisheye cameras on the overall projection parameter determination is small. Therefore, even if the sub-projection parameters corresponding to the fisheye cameras are abnormal, it is considered that the overall projection parameters are accurate and available.

[0099] In the above several implementation manners, for the case of obtaining multiple camera images captured by multiple cameras, the electronic device can determine the accuracy of the projection parameters from different aspects in multiple ways, fully consider the influence of different situations on the projection parameters, and then more accurately determine whether the projection parameters are accurate.

[0100] In some possible implementation manners, when the background vehicle detection module 202 detects the background vehicles, misidentification may occur, such as identifying trees in the camera images as background vehicles. In order to exclude the influence of these misidentification situations on the geometric analysis, the quality inspection determination module 203 can also perform post-processing on the detection boxes on the camera images that have been determined, that is, the quality inspection determination module 203 removes the misidentified detection boxes, thereby excluding interference and ensuring the accuracy of the geometric analysis process.

[0101] It can be understood that the conditions of data comparison in the foregoing implementation manners, such as less than or equal to, greater than or equal to, are only examples, and in some other possible implementation manners, the conditions of less than or equal to, greater than or equal to can also be set as simply less than, greater than, and the conditions corresponding thereto, such as greater than (less than or equal to corresponding), less than (greater than or equal to corresponding), can be set as greater than or equal to, less than or equal to. The embodiments of the present application do not make specific limitations in this regard.

[0102] For example, in the foregoing implementation manner, the quality inspection judgment module 203 can also determine that the detection frame a1 is relatively close to the ego vehicle when L1 is less than M1. If the height of the detection frame a1 is greater than or equal to the first height threshold, it can also be indicated that the detection frame a1 and the insertion frame b1 meet the rule of "near large and far small", otherwise, if the height of the detection frame a1 is less than the first height threshold, it can also be indicated that the detection frame a1 and the insertion frame b1 do not meet the rule of "near large and far small".

[0103] The foregoing method provided by the embodiments of the present application can construct the insertion frame corresponding to the virtual vehicle in the camera image according to the overall projection parameter, and perform geometric analysis on the insertion frame and the detection frame corresponding to the background vehicle in the camera image. The result of the geometric analysis can reflect whether the overall projection parameter is accurate. When the projection parameter is accurate, the overall projection parameter can be used in the subsequent sensor simulation process, thereby further ensuring the accuracy of the sensor simulation result.

[0104] In addition, the foregoing method performs geometric analysis on the insertion frame corresponding to the virtual vehicle and the detection frame corresponding to the background vehicle in the camera image by constructing the insertion frame, without the need for special reference objects, and the operation is relatively simple, the implementation difficulty is small, the detection process is short, the efficiency is high, and the result of whether the projection parameter is accurate can be given, so that the user can clearly understand the projection parameter detection situation.

[0105] The foregoing projection parameter detection method is applied to an electronic device as an example, and the projection parameter detection method is described as follows. Figure 6 As shown in FIG. 8, the method can include the following steps S601-S603.

[0106] S601, the electronic device acquires a camera image photographed by a camera of an ego vehicle.

[0107] The camera image includes a two-dimensional detection frame corresponding to a background vehicle and a two-dimensional insertion frame corresponding to a virtual vehicle, and the insertion frame is obtained by projecting the virtual vehicle onto the camera image.

[0108] In some embodiments, the background vehicle in the foregoing camera image can be recognized by the electronic device from the camera image based on a recognition algorithm. In addition, the electronic device continues to determine the detection frame corresponding to the background vehicle on the camera image.

[0109] The virtual vehicle can be constructed by the electronic device in a world coordinate system. In addition, the electronic device continues to determine a 3D box corresponding to the virtual vehicle, and projects the 3D box onto the camera image according to the projection parameter, and further obtains an insertion box corresponding to the virtual vehicle.

[0110] In some other embodiments, the camera image has included the foreground that can be the virtual vehicle before being acquired by the electronic device, or other devices such as cameras have projected the foreground that can be the virtual vehicle into the camera image in other ways; no matter which way, the electronic device can directly obtain the insertion box corresponding to the foreground that can be the virtual vehicle after acquiring the camera image.

[0111] S602, the electronic device performs geometric analysis on the detection box and the insertion box in the camera image, and obtains a geometric analysis result.

[0112] The geometric analysis includes height or distance analysis of the detection box and the insertion box.

[0113] For example, when the geometric analysis includes height analysis of the detection box and the insertion box, referring to FIG. 6B, the electronic device can obtain the distances between the detection box and the insertion box and the ego vehicle based on the depth estimation for the camera image. In addition, when the distance between the detection box and the ego vehicle is less than or equal to the distance between the insertion box and the ego vehicle, the electronic device obtains the geometric analysis result according to the size relationship between the height of the detection box and the first height threshold. When the distance between the detection box and the ego vehicle is greater than the distance between the insertion box and the ego vehicle, the electronic device obtains the geometric analysis result according to the size relationship between the height of the detection box and the second height threshold. That is, the electronic device can judge the detection box and the insertion box based on the rule of "near large and far small", and thus obtain the geometric analysis result. Figure 7 The electronic device can obtain the distance between the detection box and the ego vehicle based on the depth estimation for the camera image. In addition, when the distance between the detection box and the ego vehicle is less than or equal to the distance between the insertion box and the ego vehicle, the electronic device obtains the geometric analysis result according to the size relationship between the height of the detection box and the first height threshold. When the distance between the detection box and the ego vehicle is greater than the distance between the insertion box and the ego vehicle, the electronic device obtains the geometric analysis result according to the size relationship between the height of the detection box and the second height threshold. That is, the electronic device can judge the detection box and the insertion box based on the rule of "near large and far small", and thus obtain the geometric analysis result.

[0114] Further, for the case that the distance between the detection box and the ego vehicle is less than or equal to the distance between the insertion box and the ego vehicle, when the height of the detection box is greater than the first height threshold, the geometric analysis result obtained by the electronic device is that the geometric relationship between the detection box and the insertion box is normal. When the height of the detection box is less than or equal to the first height threshold, the geometric analysis result obtained by the electronic device is that the geometric relationship between the detection box and the insertion box is abnormal.

[0115] In addition, for the case that the distance between the detection box and the ego vehicle is greater than the distance between the insertion box and the ego vehicle, when the height of the detection box is greater than the second height threshold, the geometric analysis result obtained by the electronic device is that the geometric relationship between the detection box and the insertion box is abnormal. When the height of the detection box is less than or equal to the second height threshold, the geometric analysis result obtained by the electronic device is that the geometric relationship between the detection box and the insertion box is normal.

[0116] The first height threshold and the second height threshold are respectively determined according to the height of the insertion frame, for example, the first height threshold is min height, and the second height threshold is max height, wherein min height can represent the height of the insertion frame multiplied by a coefficient less than 1, such as 0.7, 0.8, 0.9, etc., and the specific coefficient value can be set according to actual needs; max height can be the height of the insertion frame multiplied by a coefficient greater than 1, such as 1.1, 1.2, 1.3, etc., and the specific coefficient value can be set according to actual needs.

[0117] In this way, the electronic device can determine the geometric relationship between the detection frame and the insertion frame based on the height analysis of the detection frame and the insertion frame, thereby more accurately determining the accuracy of the projection parameter.

[0118] Further, when the geometric analysis includes the height analysis of the detection frame and the insertion frame, the electronic device can further include a ratio analysis based on the determination that the height of the detection frame is greater than the first height threshold. Figure 7 Further, when the geometric analysis includes the height analysis of the detection frame and the insertion frame, the electronic device can further include a ratio analysis based on the determination that the height of the detection frame is greater than the first height threshold.

[0119] Figure 7 Further, when the geometric analysis includes the height analysis of the detection frame and the insertion frame, the electronic device can further include a ratio analysis based on the determination that the height of the detection frame is greater than the first height threshold.

[0120] The third height threshold is determined according to the second height threshold, and the fourth height threshold is determined according to the first height threshold, for example, the third height threshold is max height multiplied by a coefficient F1, and the fourth height threshold is min height multiplied by a coefficient F2, and the specific coefficient value can be set according to actual needs.

[0121] In this way, the electronic device can judge the detection frame and the insertion frame from multiple angles based on the height analysis and the ratio analysis, so that the geometric analysis method provided by the present application is more comprehensive, and the accuracy of the projection parameter can be more accurately determined.

[0122] Further, when the geometric analysis includes the distance analysis of the detection frame and the insertion frame, referring to FIG. 6, the electronic device can further include a ratio analysis based on the determination that the distance between the detection frame and the insertion frame is greater than the first distance threshold.​Figure 8 As shown, the electronic device can obtain the distance between the detection frame and the ego vehicle based on the depth estimation for the camera image. When the distance between the detection frame and the ego vehicle is less than or equal to the first distance threshold, the geometric analysis result obtained by the electronic device is that the geometric relationship between the detection frame and the insertion frame is normal. When the distance between the detection frame and the ego vehicle is greater than the first distance threshold, the geometric analysis result obtained by the electronic device is that the geometric relationship between the detection frame and the insertion frame is abnormal.

[0123] This analysis manner can exclude the case that the geometric relationship between the insertion frame or the detection frame is abnormal due to the vanishing point estimation error on the camera image. The detection frame and the insertion frame are judged from the perspective of vanishing point estimation, so that the geometric analysis manner provided by the present application is more comprehensive, and the accuracy of the projection parameter can be more accurately obtained.

[0124] In some embodiments, there is at least one detection frame and at least one insertion frame in the camera image. When there are multiple detection frames and multiple insertion frames, for each insertion frame, if there is any detection frame with abnormal geometric relationship with the insertion frame, the electronic device determines that the insertion frame is abnormal. Further, if the number of abnormal insertion frames in one camera image is greater than or equal to a third preset threshold, the geometric analysis result obtained by the electronic device is that the geometric relationship between the detection frame and the insertion frame is abnormal. If the number of abnormal insertion frames is less than the third preset threshold, the geometric analysis result obtained by the electronic device is that the geometric relationship between the detection frame and the insertion frame is normal.

[0125] The third preset threshold can be set according to actual needs or according to the total number of insertion frames in the camera image. For example, if there are four insertion frames in the camera image, the third preset threshold can be 2. Once the number of abnormal insertion frames is greater than or equal to 2, the electronic device can determine that the geometric relationship between the detection frame and the insertion frame is abnormal. That is, once the number of abnormal insertion frames exceeds 50% of the total number of insertion frames, the electronic device can determine that the camera image is unusable.

[0126] In the above manner, when there are multiple detection frames or multiple insertion frames in one camera image, the electronic device can perform geometric analysis based on multiple detection frames or multiple insertion frames in this case, so as to more accurately obtain the geometric analysis result.

[0127] It can be understood that when there are multiple detection boxes or multiple insertion boxes in a camera image, the electronic device can respectively perform geometric analysis on each detection box and each insertion box. In the process of geometric analysis, the height of the insertion box, for example, the first height threshold and the second height threshold, is determined according to the height of the currently compared insertion box. For example, when geometric analysis is performed on detection box a1 and insertion box b1, the first height threshold, the second height threshold, and the like are determined according to the height of insertion box b1; when geometric analysis is performed on detection box a1 and insertion box b2, the first height threshold, the second height threshold, and the like are determined according to the height of insertion box b2.

[0128] S603, the electronic device obtains the accuracy of the projection parameter according to the geometric analysis result.

[0129] If the geometric analysis result is that the geometric relationship between the detection box and the insertion box is abnormal, the electronic device can determine that the projection parameter is inaccurate. If the geometric analysis result is that the geometric relationship between the detection box and the insertion box is normal, the electronic device can determine that the projection parameter is accurate.

[0130] In some embodiments, the ego vehicle can include multiple cameras, each camera is used to shoot a plurality of camera images in time sequence, and the abnormal geometric relationship between the detection box and the insertion box indicates that the camera image is unusable, and the normal geometric relationship between the detection box and the insertion box indicates that the camera image is usable.

[0131] In this case, the electronic device can perform the above geometric analysis for each camera image to determine whether each camera image is usable, and further determine the accuracy of the sub-projection parameter corresponding to each camera, and determine the accuracy of the overall projection parameter based on the accuracy of the sub-projection corresponding to each camera.

[0132] For example, for a camera, if the number of unusable camera images in the plurality of camera images corresponding to the camera is greater than or equal to a first preset threshold, the electronic device determines that the sub-projection parameter corresponding to the camera is abnormal, wherein the sub-projection parameter includes the internal and external parameters of the camera, the ego vehicle pose corresponding to the camera, and the road reconstruction parameter corresponding to the camera. And the number of unusable camera images is less than the first preset threshold, the electronic device determines that the sub-projection parameter corresponding to the camera is normal.

[0133] And if the number of cameras with abnormal sub-projection parameters in the plurality of cameras is greater than or equal to a second preset threshold, the electronic device determines that the projection parameter is inaccurate, wherein the projection parameter includes the internal and external parameters of each camera, the ego vehicle pose corresponding to each camera, and the road reconstruction parameter corresponding to each camera. And the number of cameras with abnormal sub-projection parameters is less than the second preset threshold, the electronic device determines that the projection parameter is accurate.

[0134] Furthermore, if all camera images are available, the electronic equipment can also determine that the projection parameters are accurate.

[0135] The aforementioned first preset threshold can be determined based on actual needs, or it can be determined based on the total number of camera images corresponding to a camera. For example, if a camera corresponds to 100 camera images, the first preset threshold can be 20. Once the number of unusable camera images is greater than or equal to 20, the electronic device can determine that the sub-projection parameters corresponding to this camera are abnormal. In other words, once the number of unusable camera images exceeds 20% of the total number of camera images, the electronic device can determine that the sub-projection parameters corresponding to this camera are abnormal.

[0136] The aforementioned second preset threshold can be determined based on actual needs, or it can be determined based on the total number of cameras corresponding to the vehicle. For example, if the vehicle includes 11 cameras, the second preset threshold can be 4. Once the number of cameras with abnormal sub-projection parameters is greater than or equal to 4, the electronic device can determine that the overall projection parameters are inaccurate.

[0137] In the above method, the electronic device can determine the accuracy of the overall projection parameters when multiple camera images are acquired, thereby ensuring that the detection of all projection parameters corresponding to all cameras is accurate.

[0138] For another example, for a single camera, if the number of unusable camera images among multiple camera images corresponding to that camera is greater than or equal to a first preset threshold, then the sub-projection parameters corresponding to the camera are determined to be abnormal. Conversely, if the number of unusable camera images is less than the first preset threshold, then the electronic device determines that the sub-projection parameters corresponding to the camera are normal.

[0139] Furthermore, if all cameras exhibiting abnormal sub-projection parameters belong to a preset type, the electronic device determines that the projection parameters are inaccurate. The preset types include fisheye cameras and pinhole cameras. If none of the cameras exhibiting abnormal sub-projection parameters belong to a preset type, the electronic device determines that the projection parameters are accurate.

[0140] Furthermore, if all camera images are available, the electronic equipment can also determine that the projection parameters are accurate.

[0141] The camera types mentioned above can be set according to actual conditions or business needs. For example, it can be a pinhole camera or a fisheye camera.

[0142] In the above manner, the electronic device can determine the accuracy of the overall projection parameters in the case that multiple camera images captured by multiple cameras are obtained, so as to ensure that the detection of all projection parameters corresponding to all cameras is accurate. The projection parameter detection can be performed on both pinhole cameras and fisheye cameras, and the applicability is strong. When the preset type of camera is a fisheye camera, the influence of the fisheye camera distortion can also be considered.

[0143] In addition, as can be known from the above, the electronic device can determine the accuracy of the projection parameters in various manners from different aspects, fully consider the influence of different situations on the projection parameters, and then more accurately determine whether the projection parameters are accurate.

[0144] Other contents of the above steps S601-S603 can refer to the contents that can be achieved by the modules of the electronic device in the foregoing embodiments, which will not be described here.

[0145] In addition, the result of the geometric analysis in the above method can reflect whether the overall projection parameters are accurate. When the projection parameters are accurate, the overall projection parameters can be used in the subsequent sensor simulation process, so as to further ensure the accuracy of the sensor simulation result. In addition, the above method performs geometric analysis on the insertion frame corresponding to the virtual vehicle and the detection frame corresponding to the background vehicle in the camera image without special reference objects, and the operation is relatively simple, the implementation difficulty is small, the detection process is short and efficient, and the result of whether the projection parameters are accurate can be given to the user to clearly understand the projection parameter detection situation.

[0146] In addition, the method in the embodiments of the present application has good robustness, and the projection parameter detection can be implemented even if there are background vehicles on the road, which makes up for the poor performance of the traditional method in the case that there are few lane lines and street lamps on the road.

[0147] In some schemes, multiple embodiments of the present application can be combined, and the combined scheme can be implemented. Optionally, some operations in the flow of each method embodiment are combined, and / or the order of some operations is changed. In addition, the execution order between the steps of each flow is only exemplary, and does not constitute a limitation on the execution order between the steps. The execution order between the steps can also be other execution orders. It is not intended to indicate that the execution order is the only execution order in which these operations can be performed. Those skilled in the art can think of various ways to reorder the operations described in the embodiments of the present application. In addition, it should be pointed out that the process details of some embodiments of the present application are also applicable in a similar manner to other embodiments, or different embodiments can be combined for use.

[0148] In addition, some steps in the method embodiments can be replaced by other possible steps. Alternatively, some steps in the method embodiments can be optional and can be deleted in some use scenarios. Alternatively, other possible steps can be added in the method embodiments.

[0149] In addition, each method embodiment can be implemented independently or in combination.

[0150] It can be understood that, in order to realize the above functions, the foregoing electronic device comprises hardware and / or software modules corresponding to each function. The algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is realized in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application in combination with the embodiments, but such implementation should not be considered beyond the scope of the present application.

[0151] The present embodiment can divide the functional modules of the electronic device according to the above method examples, for example, each functional module can be divided according to each function, or two or more functions can be integrated in one processing module. The integrated module can be realized in the form of hardware. It should be noted that the division of modules in the present embodiment is illustrative, and is only a logical functional division. When actually implemented, there can be another division manner.

[0152] The present embodiment provides a projection parameter detection device, as shown in Figure 9 The projection parameter detection device can include an image acquisition module 901, an analysis module 902, and a projection parameter detection module 903.

[0153] The image acquisition module 901 is configured to acquire a camera image captured by a camera of a host vehicle. The camera image includes a two-dimensional detection frame corresponding to a background vehicle and a two-dimensional insertion frame corresponding to a virtual vehicle. The insertion frame is obtained by projecting the virtual vehicle onto the camera image. For example, the image acquisition module 901 can also be configured to perform the related content of the foregoing step S601.

[0154] The analysis module 902 is configured to perform geometric analysis on the detection frame and the insertion frame in the camera image to obtain a geometric analysis result. The geometric analysis includes height or distance analysis of the detection frame and the insertion frame. For example, the analysis module 902 can also be configured to perform the related content of the foregoing step S602.

[0155] The projection parameter detection module 903 is configured to obtain the accuracy of the projection parameter according to the geometric analysis result. For example, the projection parameter detection module 903 can also be configured to perform the related content of the foregoing step S603.

[0156] The embodiments of the present application further provide an electronic device, as shown in the accompanying drawings, which can include one or more processors 1001, a memory 1002, and a communication interface 1003. Figure 10 The memory 1002, the communication interface 1003, and the processor 1001 are coupled together, for example, through a bus 1004.

[0157] The memory 1002, the communication interface 1003, and the processor 1001 are coupled together, for example, through a bus 1004.

[0158] The communication interface 1003 is configured to perform data transmission with other devices. The memory 1002 stores computer program code. The computer program code includes computer instructions, which, when executed by the processor 1001, causes the electronic device to perform the projection parameter detection method in the embodiments of the present application.

[0159] The processor 1001 can be a processor or a controller, for example, a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The processor can implement or execute the various exemplary logical blocks, modules, and circuits described in connection with the present disclosure. The processor can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, and the like.

[0160] The bus 1004 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 1004 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 10 Only one thick line is used in the figure to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0161] The embodiments of the present application also provide a vehicle, which can include, for example, the processor, the memory, the communication interface and the bus shown in the above Figure 10 The memory stores computer program codes. The computer program codes include computer instructions, which, when executed by the processor, cause the vehicle to perform the projection parameter detection method in the embodiments of the present application.

[0162] The embodiments of the present application also provide a computer readable storage medium, which includes computer instructions, when the computer instructions are run on an electronic device or a vehicle, the electronic device or the vehicle performs the related method steps in the above method embodiments.

[0163] The embodiments of the present application also provide a computer program product, when the computer program product is run on a computer, the computer executes the related method steps in the above method embodiments.

[0164] The projection parameter detection device, the electronic device, the vehicle, the computer readable storage medium or the computer program product provided by the present application are all used to execute the corresponding method provided above, so the beneficial effects that can be achieved are referred to the beneficial effects of the corresponding method provided above, which will not be repeated here.

[0165] Through the above description of the implementation, those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of each functional module is taken as an example, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0166] In several embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented by other means. For example, the device embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and actual implementation can be in another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0167] The units described as separate components can or can not be physically separate, and the components shown as units can be one physical unit or multiple physical units, that is, can be located in one place, or can be distributed to multiple different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0168] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0169] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product in essence or in the form of a part or all of the technical solutions or the technical solutions. The software product is stored in a storage medium and includes a plurality of instructions for causing an apparatus (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various storage medium capable of storing program codes.

[0170] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any change or replacement within the technical scope disclosed in the present application should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of detecting a projection parameter, characterized by, The method comprises: obtaining a camera image captured by a camera of a vehicle; the camera image comprises a two-dimensional detection box corresponding to a background vehicle and a two-dimensional insertion box corresponding to a virtual vehicle, the insertion box being obtained by projecting the virtual vehicle onto the camera image; performing geometric analysis on the detection box and the insertion box in the camera image to obtain a geometric analysis result; the geometric analysis comprises height or distance analysis of the detection box and the insertion box; obtaining the accuracy of the projection parameter according to the geometric analysis result.

2. The method of claim 1, wherein, The geometric analysis comprises height analysis of the detection box and the insertion box; the geometric analysis on the detection box and the insertion box in the camera image to obtain a geometric analysis result comprises: obtaining the distance between the detection box and the insertion box and the vehicle based on depth estimation for the camera image; when the distance between the detection box and the vehicle is less than or equal to the distance between the insertion box and the vehicle, obtaining a geometric analysis result according to the size relationship between the height of the detection box and a first height threshold; the first height threshold is less than the height of the insertion box; when the distance between the detection box and the vehicle is greater than the distance between the insertion box and the vehicle, obtaining a geometric analysis result according to the size relationship between the height of the detection box and a second height threshold; the second height threshold is greater than the height of the insertion box.

3. The method of claim 2, wherein, The geometric analysis result obtained according to the size relationship between the height of the detection box and the first height threshold comprises: when the height of the detection box is greater than the first height threshold, obtaining a geometric analysis result that the geometric relationship between the detection box and the insertion box is normal; when the height of the detection box is less than or equal to the first height threshold, obtaining a geometric analysis result that the geometric relationship between the detection box and the insertion box is abnormal.

4. The method according to claim 2 or 3, characterized in that, The geometric analysis result obtained according to the size relationship between the height of the detection box and the second height threshold comprises: when the height of the detection box is greater than the second height threshold, obtaining a geometric analysis result that the geometric relationship between the detection box and the insertion box is abnormal; when the height of the detection box is less than or equal to the second height threshold, obtaining a geometric analysis result that the geometric relationship between the detection box and the insertion box is normal.

5. The method of claim 3, wherein, The geometric analysis further comprises scale analysis of the detection box and the insertion box; the geometric analysis result that the geometric relationship between the detection box and the insertion box is normal when the height of the detection box is greater than the first height threshold comprises: when the height of the detection box is greater than the first height threshold, if the height of the detection box is less than or equal to a third height threshold, a geometric analysis result that the geometric relationship between the detection box and the insertion box is normal is obtained; the third height threshold is determined according to the second height threshold.

6. The method of claim 4, wherein, The geometric analysis further comprises scale analysis of the detection box and the insertion box; the geometric analysis result that the geometric relationship between the detection box and the insertion box is normal when the height of the detection box is less than or equal to the second height threshold comprises: When the height of the detection frame is less than or equal to the second height threshold, if the height of the detection frame is greater than a fourth height threshold, a geometric analysis result that the detection frame and the insertion frame have a normal geometric relationship is obtained; the fourth height threshold is determined according to the first height threshold.

7. The method of claim 1, wherein, The geometric analysis includes distance analysis of the detection frame and the insertion frame; the geometric analysis of the detection frame and the insertion frame in the camera image obtains a geometric analysis result, including: For the camera image, a distance between the detection frame and the ego vehicle is obtained based on depth estimation; When the distance between the detection frame and the ego vehicle is less than or equal to a first distance threshold, a geometric analysis result that the detection frame and the insertion frame have a normal geometric relationship is obtained; When the distance between the detection frame and the ego vehicle is greater than the first distance threshold, a geometric analysis result that the detection frame and the insertion frame have an abnormal geometric relationship is obtained.

8. The method according to any one of claims 3 to 7, characterized in that, The ego vehicle includes a plurality of cameras, each of which is used to capture a plurality of camera images in time sequence; the detection frame and the insertion frame having an abnormal geometric relationship indicates that the camera image is unusable, and the detection frame and the insertion frame having a normal geometric relationship indicates that the camera image is usable; The accuracy of the projection parameter is obtained according to the geometric analysis result, including: For one camera, if the number of unusable camera images in the plurality of camera images corresponding to the camera is greater than or equal to a first preset threshold, it is determined that a sub-projection parameter corresponding to the camera is abnormal; the sub-projection parameter includes internal and external parameters of the camera, an ego vehicle pose corresponding to the camera, and a road reconstruction parameter corresponding to the camera; If the number of cameras with the abnormal sub-projection parameter in the plurality of cameras is greater than or equal to a second preset threshold, it is determined that the projection parameter is inaccurate; the projection parameter includes internal and external parameters of each camera, the ego vehicle pose corresponding to each camera, and the road reconstruction parameter corresponding to each camera.

9. The method according to any one of claims 3-7, characterized in that, The ego vehicle includes a plurality of cameras, each of which is used to capture a plurality of camera images in time sequence; the detection frame and the insertion frame having an abnormal geometric relationship indicates that the camera image is unusable, and the detection frame and the insertion frame having a normal geometric relationship indicates that the camera image is usable; The accuracy of the projection parameter is obtained according to the geometric analysis result, including: For one camera, if the number of unusable camera images in the plurality of camera images corresponding to the camera is greater than or equal to a first preset threshold, it is determined that a sub-projection parameter corresponding to the camera is abnormal; the sub-projection parameter includes internal and external parameters of the camera, an ego vehicle pose corresponding to the camera, and a road reconstruction parameter corresponding to the camera; If all the cameras with the abnormal sub-projection parameter belong to a preset type of camera, it is determined that the projection parameter is inaccurate; the preset type includes a fisheye camera or a pinhole camera; the projection parameter includes internal and external parameters of each camera, the ego vehicle pose corresponding to each camera, and the road reconstruction parameter corresponding to each camera.

10. The method according to any one of claims 2-9, characterized in that, The detection boxes are at least one, the virtual vehicle is at least one, and the insertion boxes are at least one; the detection boxes and the insertion boxes in the camera image are subjected to geometric analysis to obtain a geometric analysis result, including: When the detection boxes are multiple and the insertion boxes are multiple, For each insertion box, if there is any detection box with an abnormal geometric relationship with the insertion box, the insertion box is determined to be abnormal; If the number of abnormal insertion boxes in one camera image is greater than or equal to a third preset threshold, the geometric analysis result of the detection boxes and the insertion boxes with an abnormal geometric relationship is obtained.

11. A projection parameter detection apparatus, characterized by comprising: Including: An image acquisition module is configured to acquire a camera image captured by a camera of a vehicle; the camera image includes a two-dimensional detection box corresponding to a background vehicle and a two-dimensional insertion box corresponding to a virtual vehicle, and the insertion box is obtained by projecting the virtual vehicle onto the camera image; An analysis module is configured to perform geometric analysis on the detection boxes and the insertion boxes in the camera image to obtain a geometric analysis result; the geometric analysis includes height or distance analysis of the detection boxes and the insertion boxes; A projection parameter detection module is configured to obtain the accuracy of the projection parameter based on the geometric analysis result.

12. An electronic device, comprising: An electronic device including a memory and one or more processors; the memory is coupled to the processor; wherein the memory stores computer program code, the computer program code includes computer instructions, when the computer instructions are executed by the processor, the electronic device executes the projection parameter detection method as claimed in any one of claims 1-10.

13. A vehicle characterized by comprising: A vehicle including a memory and one or more processors; the memory is coupled to the processor; wherein the memory stores computer program code, the computer program code includes computer instructions, when the computer instructions are executed by the processor, the vehicle executes the projection parameter detection method as claimed in any one of claims 1-10.

14. A computer-readable storage medium, characterized in that, Computer instructions, when the computer instructions run on an electronic device or a vehicle, make the electronic device or the vehicle execute the projection parameter detection method as claimed in any one of claims 1-10.

15. A computer program product, characterised in that, When the computer program product runs on a computer, the computer executes the projection parameter detection method as claimed in any one of claims 1-10.

Images