Orientation angle determination and model training method and device, equipment, medium and vehicle

Abstract

The invention relates to an orientation angle determination and model training method and device, equipment, a medium and a vehicle, and the orientation angle determination method comprises the steps: obtaining a to-be-detected image collected by a vehicle-mounted camera and internal and external parameters of the vehicle-mounted camera; determining a conversion relation between a vehicle-mounted camera coordinate system and a vehicle coordinate system by using internal and external parameters of the camera; inputting the to-be-detected image and the internal and external parameters of the vehicle-mounted camera into the BEV model, and outputting a target three-dimensional feature map of a target object in the to-be-detected image by the BEV model; inputting the target three-dimensional feature map into an angle regression model, and outputting a to-be-converted orientation angle of the target object in a vehicle-mounted camera coordinate system by the angle regression model; according to the method, the to-be-converted orientation angle in the vehicle-mounted camera coordinate system is converted according to the conversion relation between the vehicle-mounted camera coordinate system and the vehicle coordinate system, the actual orientation angle of the target object in the vehicle coordinate system is obtained, and therefore the accuracy of the actual orientation angle obtained on the basis of the BEV model is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of information processing technology, and in particular to a method, device, equipment, medium and vehicle for determining a heading angle and training a model. Background Art

[0002] For intelligent driving vehicles, accurate and real-time target object detection is one of the important functions that enables intelligent driving vehicles to accurately perceive the complex environment around them. However, how to accurately judge and identify the three-dimensional information such as the position, size, and orientation angle of target objects in the complex environment around the vehicle is the main difficulty in target detection.

[0003] An existing method for determining the orientation angle of target objects (such as other vehicles) around a vehicle is to use an on-board camera to obtain images around the vehicle, input the images into a bird's-eye-view (BEV) perception model, and use the BEV perception model to recognize the camera images to obtain the orientation angle of the target objects (such as target bicycles, target motorcycles, target vehicles, target pedestrians, etc.) relative to the vehicle.

[0004] However, in the process of training the BEV perception model, the training data used are basically images collected by vehicles. However, when vehicles collect images, they are mostly on the road, resulting in most target objects traveling in the same direction as the vehicle (i.e., the target object's orientation angle is about 0°) or in the opposite direction of the vehicle (i.e., the target object's orientation angle is about 180°). As a result, the trained BEV perception model can only give accurate orientation angles for target objects with orientation angles close to 0° and 180°, while it is difficult to give accurate orientation angles for target objects at other angles. Therefore, how to obtain a more accurate orientation angle based on the use of the BEV perception model has become an urgent problem to be solved. Summary of the invention

[0005] In order to solve the above technical problems, the present disclosure provides a method, device, equipment, medium and vehicle for determining a heading angle and training a model.

[0006] A first aspect of an embodiment of the present disclosure provides a method for determining a heading angle, the method comprising:

[0007] Obtain the image to be detected collected by the vehicle-mounted camera and the internal and external parameters of the vehicle-mounted camera;

[0008] The conversion relationship between the on-board camera coordinate system and the vehicle coordinate system is determined using the internal and external parameters of the on-board camera;

[0009] The image to be detected and the internal and external parameters of the vehicle-mounted camera are input into the BEV model, and the BEV model outputs a target three-dimensional feature map of the target object in the image to be detected;

[0010] The target 3D feature map is input into the angle regression model, and the angle regression model outputs the orientation angle to be converted of the target object in the vehicle-mounted camera coordinate system;

[0011] The conversion relationship between the on-board camera coordinate system and the vehicle coordinate system is used to convert the orientation angle to be converted in the on-board camera coordinate system to obtain the actual orientation angle of the target object in the vehicle coordinate system.

[0012] A second aspect of the embodiments of the present disclosure provides an angular regression model training method, the method comprising:

[0013] Obtaining a training sample set and a labeling result corresponding to the training sample set, wherein the labeling result includes the orientation angle of the object in the training sample in the camera coordinate system;

[0014] The training sample set is input into a preset result prediction network, and the preset result prediction network identifies the orientation angle of the training sample set to obtain a predicted orientation angle;

[0015] Compare the preset orientation angle with the annotation result to obtain the loss value of the preset result prediction network;

[0016] The loss value is input into the preset loss function to adjust the parameters of the preset result prediction network to obtain the angular regression model.

[0017] A third aspect of an embodiment of the present disclosure provides a device for determining a heading angle, the device comprising:

[0018] An image acquisition module is used to acquire the image to be detected collected by the vehicle-mounted camera and the internal and external parameters of the vehicle-mounted camera;

[0019] A relation conversion module, used to determine the conversion relationship between the on-board camera coordinate system and the vehicle coordinate system using the internal and external parameters of the on-board camera;

[0020] The first acquisition module is used to input the image to be detected and the internal and external parameters of the vehicle-mounted camera into the BEV model, and the BEV model outputs a target three-dimensional feature map of the target object in the image to be detected;

[0021] The second acquisition module is used to input the target three-dimensional feature map into the angle regression model, and the angle regression model outputs the orientation angle to be converted of the target object in the vehicle-mounted camera coordinate system;

[0022] The third acquisition module is used to transform the orientation angle to be transformed in the vehicle camera coordinate system by using the conversion relationship between the vehicle camera coordinate system and the vehicle coordinate system, so as to obtain the actual orientation angle of the target object in the vehicle coordinate system.

[0023] A fourth aspect of an embodiment of the present disclosure provides an angular regression model training device, the device comprising:

[0024] A sample set acquisition module is used to acquire a training sample set and a labeling result corresponding to the training sample set, wherein the labeling result includes the orientation angle of the object in the training sample in the camera coordinate system;

[0025] A prediction result acquisition module is used to input the training sample set into a preset result prediction network, and the preset result prediction network identifies the orientation angle of the training sample set to obtain a predicted orientation angle;

[0026] A loss value determination module is used to compare the preset orientation angle with the annotation result to obtain a preset loss value of the result prediction network;

[0027] The parameter adjustment module is used to input the loss value into the preset loss function to adjust the parameters of the preset result prediction network to obtain the angular regression model.

[0028] A fifth aspect of the embodiments of the present disclosure provides an electronic device, the device comprising:

[0029] Memory;

[0030] Processor; and

[0031] A computer program, wherein the computer program is stored in a memory and is configured to be executed by a processor to implement the orientation angle determination method provided in the first aspect or the angle regression model training method provided in the second aspect.

[0032] The sixth aspect of the embodiments of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for determining the orientation angle provided in the first aspect or the method for training the angle regression model provided in the second aspect is implemented.

[0033] A seventh aspect of an embodiment of the present disclosure provides a vehicle, comprising the electronic device of the fifth aspect described above.

[0034] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0035] The method, device, equipment, medium and vehicle for determining the orientation angle and training the model provided by the embodiments of the present disclosure can obtain the image to be detected collected by the vehicle-mounted camera and the internal and external parameters of the vehicle-mounted camera, and use the internal and external parameters of the camera to determine the conversion relationship between the vehicle-mounted camera coordinate system and the vehicle coordinate system. At the same time, the image to be detected and the internal and external parameters of the vehicle-mounted camera are input into the BEV model, and the BEV model outputs the target three-dimensional feature map of the target object in the image to be detected, and the target three-dimensional feature map is input into the angle regression model, and the angle regression model outputs the orientation angle to be converted of the target object in the vehicle-mounted camera coordinate system, and uses the conversion relationship between the vehicle-mounted camera coordinate system and the vehicle coordinate system to perform the conversion of the vehicle-mounted camera. The orientation angle to be converted in the coordinate system is converted to obtain the actual orientation angle of the target object in the vehicle coordinate system. Therefore, the orientation angle to be converted of the target object in the vehicle camera coordinate system can be determined through the angular regression model, and then the actual orientation angle of the target object in the vehicle coordinate system is obtained according to the orientation angle to be converted in the vehicle camera coordinate system and the conversion relationship between the vehicle camera coordinate system and the vehicle coordinate system. The angular regression model is combined with the BEV model, so that angles other than 0° or 180° are considered when obtaining the actual orientation angle of the target object in the vehicle coordinate system, thereby improving the accuracy of the actual orientation angle obtained based on the BEV model. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0038] Figure 1 is a flow chart of a method for determining a heading angle provided by an embodiment of the present disclosure;

[0039] Figure 2 is a flow chart of an angular regression model training method provided by an embodiment of the present disclosure;

[0040] Figure 3 is a structural schematic diagram of a device for determining a direction angle provided by an embodiment of the present disclosure;

[0041] Figure 4 It is a structural schematic diagram of an angular regression model training device provided by an embodiment of the present disclosure;

[0042] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0044] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0045] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0046] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0047] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0048] Generally, the existing method for determining the orientation angle of target objects (such as other vehicles) around a vehicle is to use an on-board camera to obtain images around the vehicle, input the images into a bird's-eye-view (BEV) perception model, and use the BEV perception model to recognize the camera images, thereby obtaining the orientation angle of the target object (such as a target bicycle, a target motorcycle, a target vehicle, and a target pedestrian, etc.) relative to the vehicle. However, in the process of training the BEV perception model, the training data used are basically images collected by the vehicle, but the vehicle is mostly on the road when collecting images, resulting in most target objects traveling in the same direction as the vehicle (i.e., the orientation angle of the target object is about 0°) or traveling in the opposite direction of the vehicle (i.e., the orientation angle of the target object is about 180°). Therefore, the trained BEV perception model can only give accurate orientation angles for target objects with orientation angles close to 0° and 180°, while it is difficult to give accurate orientation angles for target objects at other angles. Therefore, how to obtain a more accurate orientation angle based on the use of the BEV perception model has become an urgent problem to be solved. To address this problem, an embodiment of the present disclosure provides a method for determining a heading angle, which is described below in conjunction with a specific embodiment.

[0049] Figure 1 It is a flow chart of a method for determining an orientation angle provided in an embodiment of the present disclosure. The method can be executed by a device for determining an orientation angle. The device for determining an orientation angle can be implemented in software and / or hardware. The device for determining an orientation angle can be configured in an electronic device, such as a server or a terminal or a server cluster, wherein the terminal can specifically include a computer or a tablet computer, a vehicle-mounted terminal, or any device that can be used to process the method for determining an orientation angle.

[0050] like Figure 1 As shown, the method for determining the orientation angle provided by the embodiment of the present disclosure includes the following steps.

[0051] S110, obtaining an image to be detected captured by a vehicle-mounted camera and internal and external parameters of the vehicle-mounted camera.

[0052] In the disclosed embodiment, the electronic device can obtain the image to be detected captured by the vehicle-mounted camera and the internal and external parameters of the vehicle-mounted camera when it is necessary to determine the orientation angle of the target object around the vehicle; in addition, the internal and external parameters of the vehicle-mounted camera can also be stored in advance and directly read when needed.

[0053] In the embodiment of the present disclosure, the number of vehicle-mounted cameras may be one or more, which is not limited herein.

[0054] In the embodiments of the present disclosure, the internal and external parameters of the camera refer to the internal and external parameters of the camera, wherein the internal parameters of the camera are parameters related to the camera's own characteristics, such as the camera's aperture, focal length, pixel size, etc., and the external parameters of the camera are parameters that reflect the relative position relationship between the camera and the vehicle, such as parameters in the vehicle coordinate system or in the world coordinate system, or the camera's position, posture, orientation, etc.

[0055] In some embodiments of the present disclosure, when there are multiple vehicle-mounted cameras, the multiple vehicle-mounted cameras are all set on the vehicle and face different directions. Each vehicle-mounted camera can capture images, and each vehicle-mounted camera has its own internal and external parameters.

[0056] Among them, different directions include the left front, front, right front, left rear, rear, and right rear of the vehicle. The directions can also be set according to the camera angle and actual conditions. Multiple cameras are set at various directions of the vehicle so that the multiple images captured by the multiple cameras can cover the directions around the vehicle.

[0057] In some embodiments of the present disclosure, when an electronic device receives an instruction to perform orientation angle regression on a target object around a vehicle, the electronic device parses the instruction, which includes the corresponding image to be detected and the internal and external parameters of the corresponding vehicle-mounted camera to be detected. The electronic device then directly obtains the image to be detected captured by the vehicle-mounted camera and the internal and external parameters of the vehicle-mounted camera.

[0058] In some other embodiments of the present disclosure, when the electronic device needs to perform orientation angle regression on a target object around a vehicle, the electronic device controls at least one vehicle-mounted camera installed in the vehicle to capture images around the vehicle. After capturing the images around the vehicle, the at least one vehicle-mounted camera uploads the images to the electronic device, so that the electronic device obtains at least one image to be detected captured by the at least one vehicle-mounted camera, and obtains internal and external parameters of the at least one vehicle-mounted camera from the memory according to the identification of the vehicle-mounted camera.

[0059] Furthermore, for each of the at least one vehicle-mounted camera, a corresponding relationship between the internal and external parts of the vehicle-mounted camera and the images to be detected captured by the vehicle-mounted camera is established.

[0060] S120: Determine the conversion relationship between the onboard camera coordinate system and the vehicle coordinate system using the internal and external parameters of the camera.

[0061] In the disclosed embodiment, after acquiring the internal and external parameters of the vehicle-mounted camera, the electronic device uses the internal and external parameters of the camera to determine the conversion relationship between the vehicle-mounted camera coordinate system and the vehicle coordinate system.

[0062] In some embodiments of the present disclosure, when the number of the vehicle-mounted camera is one, the conversion relationship between the vehicle-mounted camera coordinate system and the vehicle coordinate system is directly determined according to the internal and external parameters of the vehicle-mounted camera.

[0063] In some other embodiments of the present disclosure, when there are multiple vehicle-mounted cameras, the conversion relationship between the vehicle-mounted camera coordinate system and the vehicle coordinate system of each vehicle-mounted camera is determined according to the internal and external parameters of each vehicle-mounted camera, and according to the correspondence between the internal and external parameters of the vehicle-mounted camera and the image to be detected captured by the vehicle-mounted camera, a correspondence between the conversion relationship and the image to be detected is established.

[0064] In the disclosed embodiment, the specific implementation method of determining the conversion relationship between the vehicle-mounted camera coordinate system and the vehicle coordinate system based on the internal and external parameters of the vehicle-mounted camera is similar to the existing implementation method of determining the conversion relationship between the camera coordinate system and the vehicle coordinate system, and will not be repeated here.

[0065] S130, inputting the image to be detected and the internal and external parameters of the vehicle-mounted camera into the BEV model, and the BEV model outputs a target three-dimensional feature map of the target object in the image to be detected.

[0066] In the disclosed embodiment, after obtaining the internal and external parameters of the image to be detected and the vehicle-mounted camera, the electronic device inputs the internal and external parameters of the image to be detected and the vehicle-mounted camera into the BEV model, and the feature extraction module of the BEV model outputs a two-dimensional feature map corresponding to the image to be detected, and the perspective transformation module in the BEV model converts the two-dimensional feature map from a two-dimensional coordinate system to a three-dimensional coordinate system according to the internal and external parameters of the vehicle-mounted camera to obtain a three-dimensional feature map corresponding to the image to be detected, and inputs the three-dimensional feature map into the BEV model, and the BEV model identifies the target object in the three-dimensional feature map to obtain the position information of the target object, and segments the image area corresponding to the target object from the three-dimensional feature map according to the position information to obtain the target three-dimensional feature map corresponding to the target object.

[0067] The position information of the target object may be the position of the target object in the three-dimensional feature map, such as pixel coordinates.

[0068] In the embodiment of the present disclosure, the target object may be an object around the vehicle that creates an obstacle to the driving of the vehicle, such as a building, other vehicles around the vehicle, trees, etc.

[0069] S140, inputting the target three-dimensional feature map into the angle regression model, and the angle regression model outputs the orientation angle to be converted of the target object in the vehicle-mounted camera coordinate system.

[0070] In the disclosed embodiment, after obtaining the target three-dimensional feature map, the electronic device inputs the target three-dimensional feature map into the angular regression model, and the angular regression model recognizes the target three-dimensional feature map and outputs the orientation angle to be converted of the target object in the vehicle-mounted camera coordinate system.

[0071] Among them, the angle regression model is a pre-trained machine learning model used to identify the three-dimensional feature map and obtain the orientation angle of the target object in the three-dimensional feature map in the camera coordinate system.

[0072] S150, using the conversion relationship between the onboard camera coordinate system and the vehicle coordinate system, convert the orientation angle to be converted in the onboard camera coordinate system to obtain the actual orientation angle of the target object in the vehicle coordinate system.

[0073] In some embodiments of the present disclosure, when the number of vehicle-mounted cameras is one, the orientation angle to be converted in the vehicle-mounted camera coordinate system is directly converted according to the conversion relationship between the vehicle-mounted camera coordinate system and the vehicle coordinate system to obtain the actual orientation angle of the target object in the vehicle coordinate system.

[0074] In other embodiments of the present disclosure, when there are multiple vehicle-mounted cameras, the corresponding relationship between the conversion relationship between the vehicle-mounted camera coordinate system and the vehicle coordinate system and the image to be detected is obtained. For each image to be detected, the conversion relationship between the vehicle-mounted camera coordinate system and the vehicle coordinate system is used to convert the orientation angle to be converted in the vehicle-mounted camera coordinate system to obtain the actual orientation angle of the target object in the vehicle coordinate system.

[0075] It should be noted that in the embodiment of the present disclosure, there is no limitation on the execution order of step S120 and steps S130 - S140 , and step S120 and steps S130 - S140 may be executed simultaneously, or step S120 may be executed after steps S130 - S140 .

[0076] In the disclosed embodiment, the image to be detected captured by the vehicle-mounted camera and the internal and external parameters of the vehicle-mounted camera can be obtained, and the conversion relationship between the vehicle-mounted camera coordinate system and the vehicle coordinate system can be determined by using the internal and external parameters of the camera. At the same time, the image to be detected and the internal and external parameters of the vehicle-mounted camera are input into the BEV model, and the BEV model outputs a target three-dimensional feature map of the target object in the image to be detected. The target three-dimensional feature map is input into the angular regression model, and the angular regression model outputs the orientation angle to be converted of the target object in the vehicle-mounted camera coordinate system. The conversion relationship between the vehicle-mounted camera coordinate system and the vehicle coordinate system is used to perform the transformation of the orientation angle to be converted in the vehicle-mounted camera coordinate system. The conversion is performed to obtain the actual orientation angle of the target object in the vehicle coordinate system. Therefore, the orientation angle to be converted of the target object in the vehicle camera coordinate system can be determined through the angular regression model, and then the actual orientation angle of the target object in the vehicle coordinate system is obtained according to the orientation angle to be converted in the vehicle camera coordinate system and the conversion relationship between the vehicle camera coordinate system and the vehicle coordinate system. The angular regression model is combined with the BEV model to take into account angles other than 0° or 180° when obtaining the actual orientation angle of the target object in the vehicle coordinate system, thereby improving the accuracy of the actual orientation angle obtained based on the BEV model.

[0077] Figure 2 It is a flowchart of an angular regression model training method provided by an embodiment of the present disclosure. The method can be executed by an angular regression model training device. The angular regression model training device can be implemented in software and / or hardware. The angular regression model training device can be configured in an electronic device, such as a server or a terminal or a server cluster, wherein the terminal can specifically include a computer or a tablet computer, a vehicle-mounted terminal, or any device that can be used to process the angular regression model training method.

[0078] like Figure 2 As shown, the angular regression model training method provided by the embodiment of the present disclosure includes the following steps.

[0079] S210: Obtain a training sample set and a labeling result corresponding to the training sample set, where the labeling result includes the orientation angle of the object in the training sample in the camera coordinate system.

[0080] In an embodiment of the present disclosure, when the electronic device receives an instruction to train a preset result prediction model, it obtains a training sample set and a labeling result corresponding to the training sample set.

[0081] In the embodiment of the present disclosure, obtaining a training sample set may specifically include: obtaining multiple images captured by different vehicle-mounted cameras, inputting each of the multiple images into a BEV model respectively, and having the BEV model output a three-dimensional feature map of the object in each image, and determining a set of three-dimensional feature maps of the object in each image as a training sample set.

[0082] In the embodiments of the present disclosure, different vehicle-mounted cameras may be vehicle-mounted cameras of the same specifications and models arranged at different positions on the vehicle, or vehicle-mounted cameras of different specifications and models arranged at the same position on the vehicle, or cameras of different specifications and models arranged at different positions on any device other than the vehicle.

[0083] The multiple images may be images captured by multiple different vehicle-mounted cameras at the same time, or may be a sequence of images from the same vehicle-mounted camera within a time period, or may be a sequence of images from multiple different vehicle-mounted cameras within a time period.

[0084] In the disclosed embodiment, the specific implementation method of the BEV model outputting the three-dimensional feature map of the object in each image is similar to the implementation method of the existing BEV model to perform feature recognition on the object in the image to obtain the three-dimensional feature map, and will not be repeated here.

[0085] In the embodiment of the present disclosure, the labeling result may be manually labeled. When there are multiple objects in a training sample, the orientation angles of the multiple objects in the camera coordinate system are labeled respectively.

[0086] S220, inputting the training sample set into a preset result prediction network, and using the preset result prediction network to identify the orientation angle of the training sample set to obtain a predicted orientation angle.

[0087] In the embodiment of the present disclosure, the preset result preset network model can be the same network structure as the detector in the BEV model, or it can be any network structure in the existing machine learning model that identifies objects and obtains recognition results.

[0088] Specifically, after obtaining the training sample set, the electronic device inputs the training sample set into a preset result prediction network, and the preset result prediction network identifies the orientation angle of the object in each training sample in the training sample set and outputs the predicted orientation angle of the object in each training sample.

[0089] S230: Compare the preset orientation angle with the marked result to obtain a preset loss value of the result prediction network.

[0090] In an embodiment of the present disclosure, after obtaining the predicted orientation angle of the object in each training sample, the electronic device compares the preset orientation angle with the orientation angle of the object in the training sample in the annotation result in the camera coordinate system, obtains the difference between the preset orientation angle and the orientation angle of the object in the annotation result in the camera coordinate system, and determines the difference as the loss value of the preset result prediction network.

[0091] S240, inputting the loss value into a preset loss function to adjust the parameters of the preset result prediction network to obtain an angular regression model.

[0092] In the embodiment of the present disclosure, the preset loss function is a pre-set function for adjusting parameters during the training process of the preset result prediction network.

[0093] Specifically, the loss value is input into a preset loss function to obtain adjustment parameters corresponding to a preset result prediction network, and the parameters of the preset result prediction network are adjusted according to the adjustment parameters to obtain an angular regression model.

[0094] In the disclosed embodiments, a training sample set and annotation results corresponding to the training sample set can be obtained, the annotation results including the orientation angles of objects in the training samples in the camera coordinate system, the training sample set is input into a preset result prediction network, the preset result prediction network performs orientation angle recognition on the training sample set to obtain a predicted orientation angle, the preset orientation angle is compared with the annotation results to obtain a loss value of the preset result prediction network; the loss value is input into a preset loss function to adjust the parameters of the preset result prediction network to obtain an angle regression model, thereby enabling the obtained angle regression model to accurately obtain the orientation angles of objects in the three-dimensional feature map in the camera coordinate system, thereby improving the accuracy of the obtained orientation angles of the objects in the camera coordinate system.

[0095] Figure 3 : is a schematic diagram of the structure of a device for determining a heading angle provided in an embodiment of the present disclosure. The device for determining a heading angle in an embodiment of the present disclosure may be arranged in an electronic device, and the electronic device may be a server or a terminal or a server cluster, wherein the terminal may specifically include a computer or a tablet computer, a vehicle-mounted terminal, or any device that can be used to process the method for determining a heading angle, etc., and is not limited here.

[0096] like Figure 3 As shown, the orientation angle determination device 300 may include an image acquisition module 310 , a relationship conversion module 320 , a first acquisition module 330 , a second acquisition module 340 and a third acquisition module 350 .

[0097] The image acquisition module 310 can be used to acquire the image to be detected captured by the vehicle-mounted camera and the internal and external parameters of the vehicle-mounted camera.

[0098] The relationship conversion module 320 can be used to determine the conversion relationship between the vehicle-mounted camera coordinate system and the vehicle coordinate system using the internal and external parameters of the vehicle-mounted camera.

[0099] The first acquisition module 330 can be used to input the image to be detected and the internal and external parameters of the vehicle-mounted camera into the BEV model, and the BEV model outputs a target three-dimensional feature map of the target object in the image to be detected.

[0100] The second acquisition module 340 can be used to input the target three-dimensional feature map into the angle regression model, and the angle regression model outputs the orientation angle to be converted of the target object in the vehicle-mounted camera coordinate system.

[0101] The third acquisition module 350 can be used to transform the orientation angle to be transformed in the vehicle camera coordinate system by using the conversion relationship between the vehicle camera coordinate system and the vehicle coordinate system, and obtain the actual orientation angle of the target object in the vehicle coordinate system.

[0102] In the disclosed embodiment, the image to be detected captured by the vehicle-mounted camera and the internal and external parameters of the vehicle-mounted camera can be obtained, and the conversion relationship between the vehicle-mounted camera coordinate system and the vehicle coordinate system can be determined by using the internal and external parameters of the camera. At the same time, the image to be detected and the internal and external parameters of the vehicle-mounted camera are input into the BEV model, and the BEV model outputs a target three-dimensional feature map of the target object in the image to be detected. The target three-dimensional feature map is input into the angular regression model, and the angular regression model outputs the orientation angle to be converted of the target object in the vehicle-mounted camera coordinate system. The conversion relationship between the vehicle-mounted camera coordinate system and the vehicle coordinate system is used to perform the transformation of the orientation angle to be converted in the vehicle-mounted camera coordinate system. The conversion is performed to obtain the actual orientation angle of the target object in the vehicle coordinate system. Therefore, the orientation angle to be converted of the target object in the vehicle camera coordinate system can be determined through the angular regression model, and then the actual orientation angle of the target object in the vehicle coordinate system is obtained according to the orientation angle to be converted in the vehicle camera coordinate system and the conversion relationship between the vehicle camera coordinate system and the vehicle coordinate system. The angular regression model is combined with the BEV model to take into account angles other than 0° or 180° when obtaining the actual orientation angle of the target object in the vehicle coordinate system, thereby improving the accuracy of the actual orientation angle obtained based on the BEV model.

[0103] In some embodiments of the present disclosure, there are multiple vehicle-mounted cameras, which are respectively arranged on the vehicle and facing different directions, including the left front, front, right front, left rear, rear, and right rear of the vehicle.

[0104] In some embodiments of the present disclosure, the orientation angle determining device 300 may further include a relationship establishing module.

[0105] The relationship establishment module can be used to establish a corresponding relationship between the internal and external parameters of the vehicle camera and the image to be detected captured by the vehicle camera after obtaining the image to be detected captured by the vehicle camera and the internal and external parameters of the vehicle camera for each of the multiple vehicle cameras.

[0106] It should be noted that Figure 3 The illustrated orientation angle determination device 300 can execute each step in the above-mentioned orientation angle determination method embodiment, and realize each process and effect in the above-mentioned orientation angle determination method embodiment, which will not be described in detail here.

[0107] Figure 4 It is a schematic diagram of the structure of an angular regression model training device provided in an embodiment of the present disclosure. The angular regression model training device in the embodiment of the present disclosure can be set in an electronic device, and the electronic device can be a server or a terminal or a server cluster, wherein the terminal can specifically include a computer or a tablet computer, a vehicle-mounted terminal, or any device that can be used to process the angular regression model training method, etc., which is not limited here.

[0108] like Figure 4 As shown, the angular regression model training device 400 may include a sample set acquisition module 410, a prediction result acquisition module 420, a loss value determination module 430 and a parameter adjustment module 440.

[0109] The sample set acquisition module 410 may be used to acquire a training sample set and a labeling result corresponding to the training sample set, where the labeling result includes the orientation angle of the object in the training sample in the camera coordinate system.

[0110] The prediction result acquisition module 420 may be used to input the training sample set into a preset result prediction network, and the preset result prediction network may perform orientation angle recognition on the training sample set to obtain a predicted orientation angle.

[0111] The loss value determination module 430 can be used to compare the preset orientation angle with the annotation result to obtain a preset loss value of the result prediction network.

[0112] The parameter adjustment module 440 can be used to input the loss value into a preset loss function to adjust the parameters of the preset result prediction network to obtain an angular regression model.

[0113] In the disclosed embodiments, a training sample set and annotation results corresponding to the training sample set can be obtained, the annotation results including the orientation angles of objects in the training samples in the camera coordinate system, the training sample set is input into a preset result prediction network, the preset result prediction network identifies the orientation angles of the training sample set to obtain predicted orientation angles, the preset orientation angles are compared with the annotation results to obtain a loss value of the preset result prediction network; the loss value is input into a preset loss function to adjust the parameters of the preset result prediction network to obtain an angle regression model, thereby enabling the obtained angle regression model to accurately obtain the orientation angles of objects in the three-dimensional feature map in the camera coordinate system, thereby improving the accuracy of the obtained orientation angles of the objects in the camera coordinate system.

[0114] In some embodiments of the present disclosure, the sample set acquisition module 410 can be specifically used to acquire multiple images captured by different vehicle-mounted cameras, input each of the multiple images into a BEV model respectively, and have the BEV model output a three-dimensional feature map of the object in each image, and determine the set of three-dimensional feature maps of the object in each image as a training sample set.

[0115] It should be noted that Figure 4 The angular regression model training device 400 shown can execute each step in the above-mentioned angular regression model training method embodiment, and realize each process and effect in the above-mentioned angular regression model training method embodiment, which will not be described in detail here.

[0116] Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown.

[0117] In the disclosed embodiment, Figure 5 The electronic device shown may be a server or a terminal or a server cluster, wherein the terminal may specifically include a computer or a tablet computer, or a vehicle-mounted terminal, which is not limited here.

[0118] like Figure 5 As shown, the electronic device may include a processor 510 and a memory 520 storing computer program instructions.

[0119] Specifically, the processor 510 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0120] The memory 520 may include a large capacity memory for information or instructions. By way of example and not limitation, the memory 520 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 520 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 520 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 520 is a non-volatile solid-state memory. In a particular embodiment, the memory 520 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (Electrically Erasable Programmable ROM, EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0121] The processor 510 reads and executes the computer program instructions stored in the memory 520 to perform the steps of the orientation angle determination method or the angle regression model training method provided in the embodiments of the present disclosure.

[0122] In one example, the electronic device may further include a transceiver 530 and a bus 540. Figure 5As shown, the processor 510, the memory 520 and the transceiver 530 are connected via a bus 540 and communicate with each other.

[0123] The bus 540 includes hardware, software, or both. For example, but not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a Memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 540 may include one or more buses.

[0124] The embodiments of the present disclosure also provide a computer-readable storage medium, which may store a computer program. When the computer program is executed by a processor, the processor implements the orientation angle determination method or angle regression model training method provided by the embodiments of the present disclosure.

[0125] The above-mentioned storage medium may, for example, include a memory 520 of computer program instructions, and the above-mentioned instructions may be executed by the processor 510 of the electronic device to complete the orientation angle determination method or angle regression model training method provided in the embodiment of the present disclosure. Optionally, the storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a ROM, a random access memory (Random Access Memory, RAM), a compact disc read-only memory (Compact Disc ROM, CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0126] The embodiments of the present disclosure also provide a vehicle, which includes electronic equipment and can implement the various processes and effects in the above embodiments of the present disclosure, which will not be elaborated here.

[0127] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising" is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0128] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for determining a heading angle, characterized in that: The method comprises: Acquire the image to be detected captured by the vehicle-mounted camera and the internal and external parameters of the vehicle-mounted camera; Determine the conversion relationship between the on-board camera coordinate system and the vehicle coordinate system using the internal and external parameters of the on-board camera; Inputting the image to be detected and the internal and external parameters of the vehicle-mounted camera into the BEV model, and the BEV model outputs a target three-dimensional feature map of the target object in the image to be detected; Inputting the target three-dimensional feature map into an angle regression model, and having the angle regression model output the orientation angle to be converted of the target object in the vehicle-mounted camera coordinate system; The conversion relationship between the on-board camera coordinate system and the vehicle coordinate system is used to convert the orientation angle to be converted in the on-board camera coordinate system to obtain the actual orientation angle of the target object in the vehicle coordinate system.

2. The method according to claim 1, characterized in that There are multiple vehicle-mounted cameras, which are respectively arranged on the vehicle and facing different directions, including the left front, front, right front, left rear, rear, and right rear of the vehicle.

3. The method according to claim 2, characterized in that After acquiring the image to be detected captured by the vehicle-mounted camera and the internal and external parameters of the vehicle-mounted camera, the method further includes: For each of the multiple vehicle-mounted cameras, a corresponding relationship between the interior and exterior of the vehicle-mounted camera and the image to be detected captured by the vehicle-mounted camera is established.

4. A method for training an angular regression model, characterized in that: The method further comprises: Acquire a training sample set and a labeling result corresponding to the training sample set, wherein the labeling result includes the orientation angle of the object in the training sample in the camera coordinate system; Inputting the training sample set into a preset result prediction network, and using the preset result prediction network to identify the orientation angle of the training sample set to obtain a predicted orientation angle; Comparing the preset orientation angle with the annotation result to obtain the preset loss value of the result prediction network; The loss value is input into a preset loss function to adjust the parameters of the preset result prediction network to obtain the angle regression model.

5. The method according to claim 4, characterized in that The step of obtaining a training sample set includes: Acquire multiple images captured by different vehicle-mounted cameras, input each of the multiple images into a BEV model respectively, and have the BEV model output a three-dimensional feature map of the object in each image, and determine a set of the three-dimensional feature maps of the object in each image as the training sample set.

6. A device for determining a direction angle, characterized in that: include: An image acquisition module, used to acquire the image to be detected captured by the vehicle-mounted camera and the internal and external parameters of the vehicle-mounted camera; A relationship conversion module, used to determine the conversion relationship between the on-board camera coordinate system and the vehicle coordinate system using the internal and external parameters of the on-board camera; A first acquisition module, used for inputting the image to be detected and the internal and external parameters of the vehicle-mounted camera into a BEV model, and the BEV model outputs a target three-dimensional feature map of the target object in the image to be detected; A second acquisition module is used to input the target three-dimensional feature map into an angle regression model, and the angle regression model outputs the orientation angle to be converted of the target object in the vehicle-mounted camera coordinate system; The third acquisition module is used to transform the orientation angle to be transformed in the vehicle-mounted camera coordinate system by using the transformation relationship between the vehicle-mounted camera coordinate system and the vehicle coordinate system, so as to obtain the actual orientation angle of the target object in the vehicle coordinate system.

7. An angular regression model training device, characterized in that: include: A sample set acquisition module, used to acquire a training sample set and a labeling result corresponding to the training sample set, wherein the labeling result includes the orientation angle of the object in the training sample in the camera coordinate system; A prediction result acquisition module, used for inputting the training sample set into a preset result prediction network, and having the preset result prediction network perform orientation angle recognition on the training sample set to obtain a predicted orientation angle; A loss value determination module, used to compare the preset orientation angle with the annotation result to obtain the loss value of the preset result prediction network; A parameter adjustment module is used to input the loss value into a preset loss function to adjust the parameters of the preset result prediction network to obtain the angular regression model.

8. An electronic device, characterized in that: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 3 or the method according to any one of claims 4 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 or the method according to any one of claims 4 to 5 is implemented.

10. A vehicle, characterized in that: Comprising the electronic device as claimed in claim 8.

Images