Vehicle orientation angle calculation method and system, electronic equipment and medium

Abstract

The invention provides a vehicle orientation angle calculation method and system, electronic equipment and a medium. The method comprises the following steps: acquiring image data of a target vehicle according to a look-around camera of a vehicle; acquiring position information of the target vehicle by utilizing a trained target detection model according to the image data; the position information comprises an attribute category and a rotating frame, and the rotating frame comprises center point coordinates, length and width and a rotating angle; projection coordinate information is obtained through calculation according to the center point coordinates, the length, the width and the rotation angle in the rotation frame; obtaining a target coordinate according to a second preset reference, set lookup table information corresponding to the image data and the projection coordinate information; and obtaining the orientation angle of the target vehicle according to the target coordinate, the projection coordinate information, a third preset reference and the attribute category. According to the invention, the calculation power and data cost brought by a 3D target detection model are reduced by adopting a 2D rotating target detection method.

Description

Technical Field

[0001] This application belongs to the technical field of intelligent driving, and particularly relates to a method, system, electronic device and medium for calculating the vehicle orientation angle. Background Art

[0002] With the rapid development of artificial intelligence technology, the intelligent development of automobiles is also accelerating. With the maturity of L1-level assisted driving, L2-level advanced assisted driving has also flourished. In particular, advanced assisted driving based on pure vision perception has become increasingly popular in the industry.

[0003] In automatic parking perception, strategies for vehicle planning and control are often based on the surround view BEV. When judging the orientation angle between the target vehicle and the host vehicle, the current mainstream methods all require 3D information of the target vehicle. The 3D target information is generally obtained through a 3D target detection model. The current mainstream 3D target detection models all require a large amount of 3D point cloud data, and the 3D point cloud data is collected based on lidar. This requires a large amount of manpower for data collection, and has relatively high requirements for data cleaning and annotation, as well as high hardware requirements. It is easily restricted by computing power resources. Moreover, there are four fish-eye cameras in the front, rear, left and right in the development of the surround view system. This method that requires high computing power needs to be executed four times simultaneously, which is not conducive to transplantation to a board with low computing power and is not conducive to cost control in the development of the surround view system. Summary of the Invention

[0004] This application provides a method, system, electronic device and medium for calculating the vehicle orientation angle, which is used to solve the problems of large computing power and data pressure of the target detection algorithm and high development cost when judging the orientation angle between the target vehicle and the host vehicle in the prior art.

[0005] First aspect, the present application provides a method for calculating the vehicle orientation angle. The method includes: obtaining image data of a target vehicle according to the surround cameras of the host vehicle; obtaining the position information of the target vehicle by using the trained target detection model according to the image data; the position information includes an attribute category and a rotated bounding box, and the rotated bounding box includes center point coordinates, length and width, and a rotation angle; calculating and obtaining projection coordinate information according to the center point coordinates, length and width, and rotation angle in the rotated bounding box; the projection coordinate information is the target rotated bounding box coordinate information of the target vehicle; obtaining target coordinates according to a second preset reference, the set lookup table information corresponding to the image data, and the projection coordinate information; the second preset reference is a preset projection reference of the target vehicle; the target coordinates include a first target coordinate and a second target coordinate, which are used to represent the coordinate information of the target rotated bounding box of the target vehicle in the surround bird's-eye view; obtaining the orientation angle of the target vehicle according to the target coordinates, the projection coordinate information, a third preset reference, and the attribute category; the third preset reference is a preset orientation reference of the target vehicle.

[0006] In an implementation manner of the first aspect, the trained target detection model is obtained by training based on a first preset reference; the first preset reference is a preset calibration reference for the position information of the target vehicle, and the training method of the trained target detection model includes: obtaining a surround fisheye image of the target vehicle; performing annotation according to the surround fisheye image of the target vehicle and the first preset reference to obtain the annotated surround fisheye image data as a training data set; training a first neural network model according to the training data set to obtain the trained target detection model.

[0007] In an implementation manner of the first aspect, the first preset reference includes: if the included angle between the head direction of the target vehicle and the reference normal plane is an acute angle, then the attribute category of the target vehicle is rear-facing; if the included angle between the head direction of the target vehicle and the reference normal plane is an obtuse angle, then the attribute category of the target vehicle is forward-facing; if the included angle between the head direction of the target vehicle and the reference normal plane is a right angle, and the direction of the head of the target vehicle in the image data of the target vehicle is to the left, then the attribute category of the target vehicle is left-facing; if the included angle between the head direction of the target vehicle and the reference normal plane is a right angle, and the direction of the head of the target vehicle in the image data of the target vehicle is to the right, then the attribute category of the target vehicle is right-facing.

[0008] In an implementation of the first aspect, obtaining the target coordinates based on the second preset reference, the set lookup table information corresponding to the image data, and the projection coordinate information includes: obtaining the coordinates to be projected based on the projection coordinate information; the coordinates to be projected include the coordinate with the largest ordinate and the coordinate with the second largest ordinate in the projection coordinate information; obtaining the corresponding projection coordinate information based on the coordinates to be projected, the set lookup table information corresponding to the image data, and the second preset reference, and using it as the target coordinates.

[0009] In an implementation of the first aspect, the second preset reference includes: if the coordinate with the largest ordinate is not found in the set lookup table information corresponding to the image data to obtain the corresponding projection coordinate, it means that the first target coordinate is not found, and the calculation of the orientation angle of the target vehicle ends; if the coordinate with the largest ordinate is found in the set lookup table information corresponding to the image data to obtain the corresponding projection coordinate, and the coordinate with the second largest ordinate is not found in the set lookup table information corresponding to the image data to obtain the corresponding projection coordinate, then assign the midpoint coordinate of the line connecting the coordinate with the largest ordinate and the coordinate with the second largest ordinate to the coordinate with the second largest ordinate, and re-obtain the new second target coordinate based on the re-assigned coordinate with the second largest ordinate; perform iterative processing until the new coordinate with the second largest ordinate is found in the set lookup table information corresponding to the image data to obtain the corresponding projection coordinate, and obtain the first target coordinate and the second target coordinate.

[0010] In an implementation of the first aspect, obtaining the orientation angle of the target vehicle based on the target coordinates, the projection coordinate information, the third preset reference, and the attribute category includes: obtaining the orientation of the target vehicle based on the target coordinates, the projection coordinate information, the third preset reference, and the attribute category; obtaining the orientation angle of the target vehicle relative to the host vehicle based on the orientation of the target vehicle and the orientation of the host vehicle.

[0011] In an implementation of the first aspect, the third preset reference includes: when the attribute category is vehicle head forward, the corresponding directed vertical line information is calculated based on the undirected straight line information and the directed projection straight line, and the directed vertical line information is used as the orientation of the target vehicle; the directed vertical line information is opposite to the direction of the directed projection straight line; when the attribute category is vehicle head rearward, the corresponding directed vertical line information is calculated based on the undirected straight line information and the directed projection straight line, and the directed vertical line information is used as the orientation of the target vehicle; the directed vertical line information is in the same direction as the directed projection straight line; when the attribute category is vehicle head left, the point with the largest horizontal coordinate in the projection coordinate information and the second largest vertical coordinate is directed toward the direction of the coordinate point with the smallest horizontal coordinate. Map onto the straight line information, obtain directed straight line information, and use the direction of the directed straight line information as the orientation of the target vehicle; when the attribute category is right-facing, map the coordinate point with a smaller horizontal coordinate in the maximum vertical coordinate and the second largest vertical coordinate in the projection coordinate information toward the coordinate point with a larger horizontal coordinate onto the straight line information, obtain directed straight line information, and use the direction of the directed straight line information as the orientation of the target vehicle; wherein, obtain the corresponding undirected straight line information according to the target coordinates; obtain the corresponding reference normal plane according to the image data of the target vehicle, and determine the directed projection straight line of the reference normal plane according to the camera to which the image data of the target vehicle belongs; the directed projection straight line is a directed straight line perpendicular to the ground from the reference normal plane.

[0012] In a second aspect, the present application provides a vehicle heading angle calculation system, the system comprising: an image acquisition module, configured to acquire image data of a target vehicle; a position acquisition module, configured to acquire position information of the target vehicle using a trained target detection model according to the image data; the position information comprises an attribute category and a rotation box, the rotation box comprising center point coordinates, length, width and rotation angle; a coordinate conversion module, configured to acquire projection coordinate information according to the center point coordinates, length, width and rotation angle in the rotation box; the projection coordinate information is target rotation box coordinate information of the target vehicle; a projection processing module, configured to acquire target coordinates according to a second preset reference, set lookup table information corresponding to the image data and the projection coordinate information; the second preset reference is a preset projection reference of the target vehicle; the target coordinates include a first target coordinate and a second target coordinate, which are used to characterize the coordinate information of the target rotation box coordinate information of the target vehicle under a bird's-eye view; a heading angle calculation module, configured to acquire the heading angle of the target vehicle according to the target coordinates, the projection coordinate information, a third preset reference and the attribute category; the third preset reference is a preset heading reference of the target vehicle.

[0013] In a third aspect, the present application provides an electronic device, which includes: a memory configured to store a computer program; a processor configured to run the computer program to implement the vehicle orientation angle calculation method as described above.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by an electronic device, it implements the vehicle orientation angle calculation method as described above.

[0015] As described above, a vehicle orientation angle calculation method, system, electronic device, and medium according to the present application have the following

[0016] Beneficial effects:

[0017] In the present application, image data of a target vehicle is obtained according to the surround-view camera of the host vehicle; position information of the target vehicle is obtained according to the image data by using a trained target detection model; the position information includes an attribute category and a rotated bounding box, and the rotated bounding box includes center point coordinates, length and width, and a rotation angle; projection coordinate information is calculated and obtained according to the center point coordinates, length and width, and rotation angle in the rotated bounding box; the projection coordinate information is the target rotated bounding box coordinate information of the target vehicle; target coordinates are obtained according to a second preset reference, the set lookup table information corresponding to the image data, and the projection coordinate information; the second preset reference is a preset projection reference of the target vehicle; the target coordinates include a first target coordinate and a second target coordinate, which are used to represent the coordinate information of the target rotated bounding box coordinate information of the target vehicle in the surround-view bird's-eye view; the orientation angle of the target vehicle is obtained according to the target coordinates, the projection coordinate information, a third preset reference, and the attribute category; the third preset reference is a preset orientation reference of the target vehicle. In the present application, the target detection model is trained by a first preset reference, and set lookup table information of the surround-view BEV and the original image pixels is generated. Based on the set lookup table information, point coordinate projection is performed, and the result of 2D rotated target detection is projected onto the surround-view BEV through the lookup table, and then the orientation angle of the target vehicle in the host vehicle coordinate system is calculated. In this way, the computing power and data cost brought by the 3D target detection model can be reduced by using the 2D rotated target detection method.

[0018] In the present application, the surround-view BEV is also used to unify the coordinate systems of the host vehicle and the target vehicle, which can reduce the error and time consumption caused by coordinate conversion between multiple fish-eye cameras, reduce the complexity of the 3D target detection orientation angle post-processing algorithm, and is convenient for transplantation to low-computing-power mobile terminal devices. Description of the Drawings

[0019] Figure 1 It shows a schematic diagram of an application scenario of the vehicle orientation angle calculation method described in an embodiment of the present application.

[0020] Figure 2 It shows a schematic flowchart of the vehicle orientation angle calculation method described in the embodiments of the present application.

[0021] Figure 3 It shows a schematic flowchart of obtaining the target look-up table described in the embodiments of the present application.

[0022] Figure 4 It shows a schematic diagram of the reference normal plane of the front view camera of the vehicle described in the embodiments of the present application.

[0023] Figure 5 It shows a schematic flowchart of the vehicle orientation angle calculation method described in the embodiments of the present application.

[0024] Figure 6 It shows a schematic diagram of the target rotation box of the target vehicle described in the embodiments of the present application.

[0025] Figure 7 It shows a schematic diagram of the projection of the projection coordinate information described in the embodiments of the present application.

[0026] Figure 8 It shows a schematic structural diagram of the vehicle orientation angle calculation system described in the embodiments of the present application.

[0027] Figure 9 It shows a schematic structural diagram of the electronic device described in the embodiments of the present application.

[0028] Description of component labels

[0029] 1 Vehicle orientation angle calculation device

[0030] 11 Processor

[0031] 12 Memory

[0032] 13 Input device

[0033] 14 Output device

[0034] 3 Vehicle orientation angle calculation system

[0035] 31 Image acquisition module

[0036] 32 Position acquisition module

[0037] 33 Coordinate conversion module

[0038] 34 Projection processing module

[0039] 35 Orientation angle calculation module

[0040] 4 Electronic device

[0041] 41 Processing unit

[0042] 42 Memory

[0043] 421 RAM

[0044] 422 Cache Memory

[0045] 423 Storage System

[0046] 424 Utilities

[0047] 4241 Program Modules

[0048] 43 Bus

[0049] 44 I / O Interface

[0050] 45 Network Adapter

[0051] Steps S1 to S5 Detailed Implementation Manner

[0052] The following uses specific specific examples to illustrate the implementation manners of the present application. Those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0053] It should be noted that the drawings provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner. Therefore, only the components related to the present application are shown in the drawings, rather than being drawn according to the number, shape, and size of the components during actual implementation. The types, quantities, and proportions of the components during actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0054] The following embodiments of the present application provide a vehicle orientation angle calculation method, system, electronic device, and medium, which solve the problems in the prior art that when determining the orientation angle between a target vehicle and the own vehicle, the computing power and data pressure of the target detection algorithm are relatively large, the algorithm runs slowly, and the development cost is relatively high.

[0055] The vehicle orientation angle calculation method provided by the embodiments of the present application can run in a vehicle orientation angle calculation device. Taking Figure 1 as an example, Figure 1 is the hardware structure block diagram of the vehicle orientation angle calculation device for running the vehicle orientation angle calculation method. The vehicle orientation angle calculation device 1 includes but is not limited to a processor 11 and a memory 12. The processor 11 is connected to the memory 12 through a bus.

[0056] The memory 12 is the non-transitory computer-readable storage medium provided by this application. Among them, the memory stores instructions executable by at least one processor, so that the at least one processor 11 executes the vehicle orientation angle calculation method provided by this application. The non-transitory computer-readable storage medium of this application stores computer instructions, and these computer instructions are used to cause a computer to execute the vehicle orientation angle calculation method provided by this application.

[0057] The memory 12 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store image data required for vehicle orientation angle calculation and data created according to the use of the electronic device determined by the vehicle orientation angle, etc. In addition, the memory 12 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory 12 may optionally include a memory remotely arranged relative to the processor 11, and these remote memories can be connected to the vehicle orientation angle calculation device 1 for determining the vehicle orientation angle through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0058] The vehicle orientation angle calculation device 1 further includes an input device 13 and an output device 14. The input device 13 can receive input image data, such as original fish-eye images, and these original fish-eye images can be stored in the memory 12, so that the processor 11 calculates the orientation angle based on these original fish-eye images and outputs the calculated orientation angle through the output device 14.

[0059] Among them, the input device 13 may include but is not limited to a fish-eye camera and a camera, etc. The output device 14 may include but is not limited to a liquid crystal display (LCD), a light-emitting diode (LED) display, a plasma display, and a touch screen. The embodiments of this application do not make limitations.

[0060] In the embodiments of this application, the above components of the vehicle orientation angle calculation device 1 and Figure 1 other components not shown in the figure may also be connected to each other, for example, through a bus. It should be understood that Figure 1 the structural block diagram of the computing device shown is only for illustrative purposes and is not a limitation on the scope of this application. Those skilled in the art can add or replace other components as needed.

[0061] The vehicle orientation angle calculation device 1 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The vehicle orientation angle calculation device 1 can also be a mobile or stationary server. The embodiments of the present application do not make any limitations.

[0062] Next, the technical solutions in the embodiments of the present application will be described in detail with reference to the accompanying drawings in the embodiments of the present application.

[0063] The following embodiments of the present application provide a vehicle orientation angle calculation method, which can be implemented, for example, by Figure 1 the shown processor 11. Figure 2 It is shown as a schematic flow chart of the vehicle orientation angle calculation method described in the embodiments of the present application. As Figure 2 shown, the present embodiment provides a vehicle orientation angle calculation method, and the vehicle orientation angle calculation method includes the following steps S1 to S5.

[0064] Step S1: Obtain the image data of the target vehicle according to the surround view camera of the host vehicle.

[0065] Specifically, the surround view camera of the host vehicle is a camera arranged in the front, rear, left and / or right of the host vehicle, and the camera numbers CamID are 0, 1, 2, 3 (front, rear, left, right) respectively. The image data is the image information collected by the camera of the host vehicle. The surround view camera of the host vehicle can collect several pieces of image data. The present application only calculates the orientation angle for the image data containing the target vehicle, and does not calculate the orientation angle for the image data not containing the target vehicle. When the surround view camera of the host vehicle collects several pieces of image data including the target vehicle, the present application performs orientation angle calculation processing on each piece of image data respectively to calculate the orientation angle between the target vehicle in the current image and the host vehicle.

[0066] Step S2: Obtain the position information of the target vehicle according to the image data by using the trained target detection model; the position information includes the attribute category and the rotated bounding box, and the rotated bounding box includes the center point coordinates, length and width, and rotation angle. Among them, the rotated bounding box is the minimum circumscribed rectangle of the target vehicle.

[0067] Specifically, the present application uses the trained target detection model to obtain the position information of the target vehicle according to the image data, and the trained target detection model is a model for obtaining the position information such as the attribute category and the rotated bounding box of the target vehicle.

[0068] Step S3: Calculate and obtain projection coordinate information based on the center point coordinates, length and width, and rotation angle in the rotation box; the projection coordinate information is the target rotation box coordinate information of the target vehicle.

[0069] Specifically, the center point coordinates, length and width, and rotation angle in the rotation box cannot directly represent the position and direction of the target vehicle. In this application, calculations are performed based on the center point coordinates, length and width, and rotation angle in the rotation box to obtain projection coordinate information, which is the four vertex coordinates of the rotation box, that is, the four vertex coordinates of the minimum circumscribed rectangle of the target vehicle. In this way, the position and direction of the target vehicle can be better represented, facilitating the calculation of the orientation angle of the target vehicle later.

[0070] Step S4: Obtain target coordinates based on the second preset reference, the set lookup table information corresponding to the image data, and the projection coordinate information; the second preset reference is the preset projection reference of the target vehicle; the target coordinates include a first target coordinate and a second target coordinate, which are used to characterize the coordinate information of the target rotation box coordinate information of the target vehicle in the panoramic bird's-eye view.

[0071] Step S5: Obtain the orientation angle of the target vehicle based on the target coordinates, the projection coordinate information, the third preset reference, and the attribute category; the third preset reference is the preset orientation reference of the target vehicle.

[0072] It should be noted that the above labels S1 to S5 are only used to identify different steps and are not used to limit the execution order between these steps.

[0073] For the image data of the target vehicle including several images obtained by the surround-view camera of the host vehicle, this application performs orientation angle calculation processing on each image respectively to obtain the orientation angle between the target vehicle detected in the current image and the host vehicle.

[0074] The set lookup table information and the trained target detection model for each camera are obtained in advance. When processing the current image, this application further calculates the orientation angle between the target vehicle in the current image and the host vehicle through the set lookup table information of the camera to which the current image belongs and the trained target detection model.

[0075] In one implementation, the method for obtaining the setting lookup table information of each camera includes: the application first needs to obtain the internal and external parameters of the fisheye cameras with CamID numbers 0, 1, 2, 3 (front view, rear view, left view, right view) in the vehicle surround view system. There are two methods for obtaining the internal parameters. One is to read the parameters that come with the camera factory settings, and the other is to calibrate the internal parameters of the fisheye camera using the Zhang Zhengyou calibration method or the polynomial fitting method. The method for obtaining the external parameters is mainly to obtain the pixel coordinate points in the original image corresponding to the reference points around the vehicle, and establish the translation matrix T and rotation matrix R from the image coordinate system to the camera coordinate system, and then to the world coordinate system.

[0076] Based on the camera internal parameters under a single view, the original fisheye image of the fisheye camera is distorted and corrected to generate a single mapping relationship matrix map1 (i.e., pixel coordinate lookup table) from the original fisheye image to the correction image. Then, based on the camera external parameters T and R, a single correction image is generated to a single bird's-eye view. Similarly, the bird's-eye view under the other three camera perspectives is generated. Then, the four single-view bird's-eye views are fused to generate the surround view BEV (Bird's-Eye View). Finally, for each camera perspective, map1, T, and R are fused to generate the original fisheye image under the four perspectives directly to the target lookup table Map of the bird's-eye view, i.e., the target lookup table Map corresponding to each of the four cameras. Among them, the surround view BEV (Bird's-Eye View, BEV) is a technology used to perceive and understand the surrounding environment in the field of autonomous driving and computer vision.

[0077] In this implementation, the present application adopts the calibration method of inverse polynomial fitting parameters to calibrate the internal parameters of the fisheye cameras of the vehicle with CamID numbers 0, 1, 2, and 3 (front view, rear view, left view, and right view), and obtain the internal parameters of the four fisheye cameras. Based on the internal parameters of each fisheye camera, the original fisheye image of the current fisheye camera is distorted and corrected, and the mapping relationship matrix map1 (i.e., pixel coordinate lookup table) from the original fisheye image to the correction image is obtained; then a calibration checkerboard is laid around the vehicle. For example, the present application adopts a checkerboard pattern calibration object laid 10m around the vehicle, that is, the vehicle surround view BEV field of view is a square area of ​​20mx20m, which is used to establish a one-to-one correspondence between the corner points on the image and the 3D points in the real world coordinate system, and then the pnp algorithm is used to obtain the rotation matrix R and translation matrix T from the image coordinate system to the camera coordinate system and then to the world coordinate system. Based on R and T, the correction image can be projected to a bird's-eye view from a bird's-eye view.

[0078] Figure 3 The flowchart of obtaining the target lookup table described in the embodiment of the present application is shown. Figure 3As shown in the figure, in order to reduce the amount of calculation, reduce the computing power requirements, and facilitate transplantation to low-computing-power terminals for operation, the lookup tables map1, R, and T are fused to obtain a lookup table Map directly from the original image to the bird's-eye view image. In order to unify the position of the target vehicle in the ego-vehicle coordinate system, it is necessary to rotate the Map obtained from cameras in different orientations. When CamID = 0, the Map does not need to be rotated; when CamID = 1, the Map is rotated clockwise by 180°; when CamID = 2, the Map is rotated counterclockwise by 90°; when CamID = 3, the Map is rotated clockwise by 90°.

[0079] This application uses surround-view BEV to unify the coordinate systems of the ego-vehicle and the target vehicle, which can reduce the errors and time consumption caused by coordinate conversion between multiple fish-eye cameras, and further reduce the complexity of the 3D target detection orientation angle post-processing algorithm.

[0080] In an embodiment of the present application, the trained target detection model is obtained by training based on a first preset benchmark; the first preset benchmark is a calibration benchmark for the position information of the preset target vehicle, and the training method of the trained target detection model includes the following steps S211 to S213.

[0081] Step S211: Obtain the surround-view fish-eye image of the target vehicle.

[0082] Step S212: Perform annotation according to the surround-view fish-eye image of the target vehicle and the first preset benchmark to obtain the annotated surround-view fish-eye image data as the training data set.

[0083] Step S213: Train the first neural network model according to the training data set to obtain the trained target detection model.

[0084] Specifically, this application collects vehicle surround-view fish-eye image data, uses the vehicle head direction annotation rule as the first preset benchmark to perform manual data annotation on the vehicle attributes in the image. There are 4 categories of the attributes cls of the annotated target vehicle, and cls are respectively: carHead (front of the vehicle head), carTail (rear of the vehicle head), carLeft (left of the vehicle head), carRight (right of the vehicle head). Based on the annotated data, through the trained target detection model (training a 2D rotation target detection CNN model), the attribute classification of the target vehicle and the position localization coordinates in the image are obtained. The localization coordinates are mainly the center point coordinates, length and width, and rotation angle of the rotated bounding box. Among them, the trained target detection model is obtained by training a 2D rotation target detection CNN model. In this way, the computing power and data costs brought by the 3D target detection model can be reduced by using the 2D rotation target detection method.

[0085] Among them, carHead (front of the vehicle), carTail (rear of the vehicle), carLeft (front of the vehicle to the left), and carRight (front of the vehicle to the right) respectively refer to the orientation of the target vehicle relative to the current camera (the camera to which the surround fisheye image belongs). For example, the current state of the target vehicle is that the front of the target vehicle faces the camera, and the attribute category cls of the target vehicle is represented as carHead; the current state of the target vehicle is that the rear of the target vehicle faces the camera, and the attribute category cls of the target vehicle is represented as carTail; the current state of the target vehicle is that the front of the target vehicle faces right in the camera, and the attribute category cls of the target vehicle is represented as carRight; the current state of the target vehicle is that the front of the target vehicle faces left in the camera, and the attribute category cls of the target vehicle is represented as carLeft.

[0086] The target detection model used in this application is obtained by training a 2D rotation target detection CNN model. Compared with the 3D target detection model in traditional technology, the target detection model used in this application brings lower computing power and data costs.

[0087] In an embodiment of the present application, the first preset reference includes the following steps S221 to S224.

[0088] Step S221: If the angle between the head direction of the target vehicle and the reference normal plane is an acute angle, the attribute category of the target vehicle is head-on rearward. The reference normal plane is defined as a plane passing through the optical axis of the camera to which the image data belongs and perpendicular to the ground. The head direction of the target vehicle is obtained based on the image data of the target vehicle.

[0089] Step S222: If the angle between the head direction of the target vehicle and the reference normal plane is an obtuse angle, the attribute category of the target vehicle is head-on.

[0090] Step S223: If the angle between the front direction of the target vehicle and the reference normal plane is a right angle, and the front direction of the target vehicle in the image data of the target vehicle is left, then the attribute category of the target vehicle is left-facing.

[0091] Step S224: If the angle between the front direction of the target vehicle and the reference normal plane is a right angle, and the front direction of the target vehicle in the image data of the target vehicle is right, then the attribute category of the target vehicle is right front direction.

[0092] Specifically, based on the panoramic fisheye camera, this application collects the panoramic fisheye image dataset required for training the CNN model, performs manual annotation under the first preset benchmark, and then trains the CNN model based on the annotated data to obtain the trained target detection model. The specific description of the first preset benchmark is as follows:

[0093] First, define the reference normal plane β that passes through the optical axis of the camera and is perpendicular to the ground. As Figure 4 shown, the black car model is the ego vehicle, the camera is a front view camera, the dashed line is the optical axis of the camera, and the optical axis is on the plane β( Figure 4 shown as a schematic diagram of the reference normal plane of the front view camera of the ego vehicle described in the embodiment of this application).

[0094] When the included angle formed by the head direction of the target vehicle and β is an acute angle, the category is carTail.

[0095] When the included angle formed by the head direction of the target vehicle and β is an obtuse angle, the category is carHead.

[0096] When the included angle formed by the head direction of the target vehicle and β is a right angle, there are two cases. In the current image, if the direction of the target vehicle is to the left, the category is carLeft; if the direction is to the right, the category is carRight.

[0097] It should be noted that in the judgment process of the first preset benchmark, the judgment process of the included angle between the head direction of the target vehicle and the reference normal plane is a parallel process without a sequential order, that is, steps S221 - S224 are a parallel process.

[0098] In an embodiment of this application, obtaining the target coordinates according to the second preset benchmark, the set lookup table information corresponding to the image data, and the projection coordinate information includes the following steps S41 to S42.

[0099] Step S41: Obtain the coordinate information to be projected according to the projection coordinate information; the coordinate information to be projected includes the coordinate with the largest ordinate and the coordinate with the second largest ordinate in the projection coordinate information.

[0100] Step S42: Obtain the corresponding projection coordinate information according to the coordinate information to be projected, the set lookup table information corresponding to the image data, and the second preset benchmark, and use it as the target coordinates.

[0101] In an embodiment of this application, the second preset benchmark includes the following steps S4211 to S4213.

[0102] Step S4211: If the coordinate with the largest ordinate is not found in the corresponding projection coordinate in the set lookup table information corresponding to the image data, it means that the first target coordinate has not been queried, and the calculation of the orientation angle of the target vehicle ends.

[0103] Step S4212: If the coordinate with the largest ordinate is found in the corresponding projection coordinate in the set lookup table information corresponding to the image data, and the coordinate with the second largest ordinate is not found in the corresponding projection coordinate in the set lookup table information corresponding to the image data, then assign the midpoint coordinate of the line connecting the coordinate with the largest ordinate and the coordinate with the second largest ordinate to the coordinate with the second largest ordinate, and re-obtain the new second target coordinate according to the re-assigned coordinate with the second largest ordinate.

[0104] Step S4213: Iteratively process until the new coordinate with the second largest ordinate is found in the corresponding projection coordinate in the set lookup table information corresponding to the image data, and obtain the first target coordinate and the second target coordinate.

[0105] Specifically, based on the trained object detection model, the present application can obtain the approximate minimum circumscribed rectangle bounding box positioning (i.e., the rotated box) and the corresponding attribute category (i.e., the attribute category) of the target vehicle in the current image. In the original image (the current image) from the current perspective, then project the point p1 with the largest ordinate value and the second largest point p2 among the coordinate points of the minimum circumscribed rectangle box onto the surround view BEV through the target lookup table Map to obtain the corresponding projection coordinate information p1' and p2', which are used as the target coordinates. Among them, projecting the point p1 with the largest ordinate value and the second largest point p2 among the coordinate points of the minimum circumscribed rectangle box onto the surround view BEV through the target lookup table Map to obtain the corresponding projection coordinate information p1' and p2', that is, finding the corresponding projection coordinate information p1' and p2' for the point p1 with the largest ordinate value and the second largest point p2 among the coordinate points of the minimum circumscribed rectangle box through the target lookup table Map, that is, the projection coordinate information p1' and p2' corresponding to p1 and p2 on the original image on the surround view BEV (as Figures 6 - 7 shown).

[0106] The present application sorts the coordinates in the projection coordinate information (for example, sorting from largest to smallest or from smallest to largest is acceptable), finds the point coordinate p1 with the largest ordinate and the point coordinate p2 with the second largest ordinate from the sorted projection coordinate information as the coordinate information to be projected; and finds the corresponding target lookup table Map according to the number of the camera to which the current image belongs, and finds the projection coordinate information p1' and p2' corresponding to the coordinate information to be projected from the corresponding target lookup table Map according to the second preset criterion, which are used as the target coordinates.

[0107] The setting rule of the second preset reference is as follows: If p1' corresponding to p1 cannot be found in the target lookup table Map, that is, p1' = (0, 0), it means that p1 is not found, and at this time, the calculation of the orientation angle is exited; if p1' ≠ (0, 0) and p2' ≠ (0, 0) at the same time, then p1' and p2' are directly connected; if p1' ≠ (0, 0) and p2' = (0, 0) at the same time, then the midpoint coordinates of the line connecting p1 and p2 are assigned to point p2, and the new p2' is obtained by looking up the table for point p2 again. Repeat the above steps until p2' is not the zero coordinate point. At this time, p1' and p2' are used as the target coordinates.

[0108] In an embodiment of the present application, obtaining the orientation angle of the target vehicle according to the target coordinates, the projection coordinate information, the third preset reference, and the attribute category includes the following steps S51 to S52.

[0109] Step S51: Obtain the orientation of the target vehicle according to the target coordinates, the projection coordinate information, the third preset reference, and the attribute category.

[0110] Step S52: Obtain the orientation angle of the target vehicle relative to the host vehicle according to the orientation of the target vehicle and the orientation of the host vehicle.

[0111] In an embodiment of the present application, the third preset reference includes the following steps S5111 to S5114.

[0112] Step S5111: When the attribute category is forward-facing of the vehicle head, calculate the corresponding directed perpendicular line information according to the undirected line information and the directed projection line, and use the directed perpendicular line information as the orientation of the target vehicle; the direction of the directed perpendicular line information is opposite to the direction of the directed projection line.

[0113] Step S5112: When the attribute category is rear-facing of the vehicle head, calculate the corresponding directed perpendicular line information according to the undirected line information and the directed projection line, and use the directed perpendicular line information as the orientation of the target vehicle; the direction of the directed perpendicular line information is the same as the direction of the directed projection line.

[0114] Step S5113: When the attribute category is left-facing of the vehicle head, map the point with the larger abscissa among the coordinates with the largest ordinate and the second-largest ordinate in the projection coordinate information to the line information in the direction of the coordinate point with the smaller abscissa to obtain the directed line information, and use the direction of the directed line information as the orientation of the target vehicle.

[0115] Step S5114: When the attribute category is right-facing of the vehicle head, map the coordinate point with the smaller abscissa among the coordinate with the largest ordinate and the coordinate with the second-largest ordinate in the projection coordinate information in the direction of the coordinate point with the larger abscissa onto the straight line information to obtain the directed straight line information, and use the direction of the directed straight line information as the orientation of the target vehicle.

[0116] Among them, the corresponding undirected straight line information is obtained according to the target coordinate; the corresponding reference normal plane is obtained according to the image data of the target vehicle, and the directed projection straight line of the reference normal plane is determined according to the camera to which the image data of the target vehicle belongs; the directed projection straight line is the directed straight line perpendicular to the ground of the reference normal plane.

[0117] Specifically, in this application, the first target coordinate p1’ and the second target coordinate p2’ are connected, denoted as L2, and it is judged whether L2 or the perpendicular line of L2 is the straight line where the orientation of the target vehicle is located based on the attribute category of the target vehicle. Among them, p1 and p2 are respectively the two points with the largest ordinate values among the coordinate points of the minimum circumscribed rectangle of the target vehicle, and they are located at the wheels or the bottom of the front edge of the target vehicle. Therefore, the connection line L2 between the corresponding projection point coordinates of p1 and p2 is in the same direction as or perpendicular to the target vehicle, that is, as Figure 7 shown, for the target vehicle car1 on the left side of the vehicle, L2 is parallel to its direction; for the target vehicle car2 in the front of the vehicle, the direction perpendicular to L2 is the same as its direction. Finally, calculate the included angle between the straight line where the orientation of the target vehicle is located and the straight line where the advancing direction of the host vehicle is located, which is the orientation angle λ;

[0118] It should be noted that in the judgment process of the third preset reference, the process of judging the orientation of the target vehicle according to the attribute category is a parallel process without a sequential order, that is, steps S5111 - S5114 are a parallel process.

[0119] For the image data of the target vehicle including several images obtained by the surround-view camera of the host vehicle, in this application, the orientation angle calculation process is performed on each image respectively to obtain the orientation angle between the target vehicle detected in the current image and the host vehicle. The following takes the calculation of the vehicle orientation angle of a single image as an example for illustration.

[0120] Figure 5 It is shown as the flowchart of the vehicle orientation angle calculation method described in the embodiment of this application. As Figure 5As shown, the present application inputs the acquired image into the trained object detection model to obtain the attribute category and rotation bounding box of the target vehicle, and obtains the orientation of the target vehicle according to the second preset criterion, the third preset criterion, the setting lookup table information of the camera to which the current image belongs, and the attribute category and rotation bounding box of the target vehicle, and further calculates the orientation angle of the target vehicle according to the orientation of the target vehicle and the orientation of the host vehicle.

[0121] Specifically, after completing the dataset annotation using the first preset criterion, the present application selects a CNN model for model training. The present application selects the YOLOv8n-obb model to train the dataset, and finally, based on the trained weights, inference can obtain the rotation bounding box and the status attribute category cls of the target vehicle. The position is mainly the center point (x, y), length and width (w, h), and rotation angle (θ) of the rotation bounding box, and the status attribute category is mainly carHead, carTail, carLeft, and carRight.

[0122] To better obtain the specific representation of the direction of the target vehicle, the (x, y, w, h, θ) output by the CNN model is converted into the four vertex coordinates of the minimum bounding rectangle of the target vehicle, that is, the four vertex coordinates [p1, p2, p3, p4] of the target rotation bounding box.

[0123] In one implementation, the present application obtains the position information of the target vehicle by using the trained object detection model according to the image data; the position information includes the attribute category cls and the rotation bounding box, the rotation bounding box includes the center point coordinates (x, y), length and width (w, h), and rotation angle (θ), and the status attribute category is mainly carHead, carTail, carLeft, and carRight; to better obtain the specific representation of the direction of the target vehicle, the center point coordinates (x, y), length and width (w, h), and rotation angle (θ) in the rotation bounding box are calculated to obtain the projection coordinate information [p1(x1, y1), p2(x2, y2), p3(x3, y3), p4(x4, y4)].

[0124] The calculation formula is as follows:

[0125]

[0126] x1 = x + x 01 + x 02 ,y1 = y + y 01 + y 02 (3)

[0127] x2 = x + x 01 - x 02 ,y2 = y + y 01 - y 02 (4)

[0128] x3=xx 01 -x 02 , y3=yy 01 -y 02 (5)

[0129] x4=xx 01 +x 02 , y4=yy 01 +y 02 (6)

[0130] Among them, x 01 Indicates the projection distance of the width w of the rotation box on the horizontal axis, y 01 Indicates the projection distance of the width w of the rotation box on the ordinate axis; x 02 Indicates the projection distance of the height h of the rotating frame on the horizontal axis; y 02 Indicates the projection distance of the height h of the rotating box on the vertical axis. The horizontal and vertical axes here refer to the pixel coordinate system.

[0131] Figure 6 Shown is a schematic diagram of a target rotation frame of a target vehicle according to an embodiment of the present application. Figure 7 The diagram shows a projection of the projection coordinate information described in the embodiment of the present application. Figures 6 - 7 As shown, for the four points of the target rotation box [p1(x1,y1), p2(x2,y2), p3(x3,y3), p4(x4,y4)], sort them according to the value of the ordinate (either large to small or small to large), find the point with the largest ordinate value as p1, and the point with the second largest ordinate value as p2. According to the CamID number of the fisheye camera, use the corresponding target lookup table Map and based on the second preset reference, project the two points p1 and p2 to the bird's-eye view of the surround view BEV to obtain p1' and p2'. The example of point projection detected by CNN is as follows Figure 7 As shown, the black vehicle is the ego vehicle, the yellow vehicle is the target vehicle, the left side is the target vehicle position information output by CNN in the original image, and the right side is a schematic diagram of the calculated point projection and orientation angle. The second preset benchmark specifically includes:

[0132] If p1 does not find the corresponding projection coordinate p1' in the setting lookup table information corresponding to the image data, that is, p1'=(0,0), it means that the projection point coordinate p1' of p1 is not found, and the orientation angle calculation is exited at this time.

[0133] If p1 can find the corresponding projection coordinate p1' in the set lookup table information corresponding to the image data, that is, p1'≠(0,0), and at the same time, p2 can find the corresponding projection coordinate p2' in the set lookup table information corresponding to the image data, that is, p2'≠(0,0), then directly connect p1' and p2'; if p2 cannot find the corresponding projection coordinate p2' in the set lookup table information corresponding to the image data, that is, p2'=(0,0), then take the midpoint coordinate of the line connecting p1 and p2 and assign it to point p2, re-lookup point p2 to get the new p2', repeat the above steps until p2' is not the zero coordinate, and then connect p1' and p2', denoted as L2.

[0134] Based on the camera ID number to which the image data belongs, confirm that the directed line of the vertical ground projection of the reference normal plane β (each camera has a β) is L1, as Figure 7 shown, OF, OB, OL, and OR correspond to L1 with different camera ID numbers. Currently, the undirected line L2 is known. In order to confirm the direction of the target vehicle, it is necessary to judge the direction of L2 in combination with the attribute category information of the target vehicle. The judgment criterion is the third preset criterion, as follows:

[0135] When cls = carHead, then calculate the perpendicular line of the straight line L2, and the direction of the perpendicular line is the opposite direction of L1, that is, the current directed perpendicular line represents the direction of the target vehicle.

[0136] When cls = carTail, then calculate the perpendicular line of the straight line L2, and the direction of the perpendicular line is the same as the direction of L1, that is, the current directed perpendicular line represents the direction of the target vehicle.

[0137] When cls = carLeft, then among points p1 and p2, map the point with the larger abscissa to L2 in the direction of the point with the smaller abscissa, that is, the current direction of L2 represents the direction of the target vehicle.

[0138] When cls = carRight, then among points p1 and p2, map the point with the smaller abscissa to L2 in the direction of the point with the larger abscissa, that is, the current direction of L2 represents the direction of the target vehicle.

[0139] In Figures 6 - 7 car1 represents the cases of cls = carLeft and carRight. The gray directed dotted line is L2, which is the orientation of the target vehicle; car2 represents the cases of cls = carHead\carTail. The black dotted line is L2, and the gray directed dotted line is the perpendicular line of L2, which is the orientation of the target vehicle; for the above cases, directly calculate the angle between the gray directed dotted line and the straight line where the forward direction of the black self-vehicle is located, and the final orientation angle λ between the target vehicle and the self-vehicle can be obtained.

[0140] The steps in the above embodiment are the calculation process when only the left camera detects the target vehicle. If the four cameras detect more than one target vehicle respectively, the above steps are executed for the detected target vehicles respectively to confirm the orientation angle between the vehicle and all the surrounding target vehicles for subsequent calculations.

[0141] The protection scope of the vehicle heading angle calculation method described in the embodiment of the present application is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.

[0142] An embodiment of the present application also provides a vehicle heading angle calculation system, which can implement the vehicle heading angle calculation method described in the present application. However, the implementation device of the vehicle heading angle calculation method described in the present application includes but is not limited to the structure of the vehicle heading angle calculation system listed in the present embodiment. All structural deformations and replacements of the prior art made according to the principles of the present application are included in the protection scope of the present application.

[0143] like Figure 8 As shown, this embodiment provides a vehicle heading angle calculation system, and the system 3 includes: an image acquisition module 31, a position acquisition module 32, a coordinate conversion module 33, a projection processing module 34 and a heading angle calculation module 35.

[0144] The image acquisition module 31 is configured to acquire image data of the target vehicle.

[0145] The position acquisition module 32 is configured to acquire the position information of the target vehicle according to the image data using the trained target detection model; the position information includes attribute categories and a rotation box, and the rotation box includes center point coordinates, length, width and rotation angle.

[0146] The coordinate conversion module 33 is configured to calculate and obtain projection coordinate information according to the center point coordinates, length, width and rotation angle in the rotation frame; the projection coordinate information is the target rotation frame coordinate information of the target vehicle.

[0147] The projection processing module 34 is configured to obtain the target coordinates according to a second preset reference, the set lookup table information corresponding to the image data and the projection coordinate information; the second preset reference is a preset projection reference of the target vehicle; the target coordinates include a first target coordinate and a second target coordinate, which are used to characterize the coordinate information of the target rotation box coordinate information of the target vehicle under the bird's-eye view perspective.

[0148] The orientation angle calculation module 35 is configured to obtain the orientation angle of the target vehicle according to the target coordinates, the projection coordinate information, a third preset reference, and the attribute category; the third preset reference is a preset orientation reference of the target vehicle.

[0149] It should be noted that the functions or operations of the image acquisition module 31, the position acquisition module 32, the coordinate conversion module 33, the projection processing module 34, and the orientation angle calculation module 35 described in the embodiments of the present disclosure correspond one by one to the steps in the vehicle orientation angle calculation method described above, so details are not repeated here.

[0150] In several embodiments provided in the present application, it should be understood that the disclosed system, apparatus, or method can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of modules / units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some interfaces, and the indirect coupling or communication connection of devices or modules or units can be in electrical, mechanical, or other forms.

[0151] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they can be located in one place, or distributed to multiple network units. Some or all of the modules / units can be selected according to actual needs to achieve the purpose of the embodiments of the present application. For example, in each embodiment of the present application, the various functional modules / units can be integrated in a processing module, or each module / unit can exist physically alone, or two or more modules / units can be integrated in one module / unit.

[0152] Those of ordinary skill in the art should also further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0153] The embodiments of the present application also provide an electronic device, which includes: a memory and a processor; wherein,

[0154] The memory is used to store a computer program;

[0155] The memory includes various media that can store program codes, such as ROM, RAM, magnetic disks, USB flash drives, memory cards, or optical discs.

[0156] The processor is used to execute the computer program stored in the memory, so that the electronic device executes the vehicle orientation angle calculation method as described above.

[0157] Preferably, the processor may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP for short), an application specific integrated circuit (ASIC for short), a field programmable gate array (FPGA for short), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0158] As Figure 9 As shown, the electronic device 4 of the present application is presented in the form of a general computing device, and its specific implementation products are diverse, including intelligent mobile devices, such as mobile phones, laptop computers, tablets, intelligent wearable devices, in-vehicle units, etc. The components of the electronic device may include, but are not limited to: one or more processors or processing units 41, a memory 42, and a bus 43 connecting different system components (including the memory 42 and the processing unit 41).

[0159] The bus 43 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the multiple bus structures. For example, these architectures include, but are not limited to, an industry standard architecture (ISA) bus, a microchannel architecture (MAC) bus, an enhanced ISA bus, a video electronics standards association (VESA) local bus, and a peripheral component interconnect (PCI) bus.

[0160] The electronic device 4 typically includes a variety of computer system-readable media. These media can be any available media that can be accessed by the control terminal, including volatile and non-volatile media, removable and non-removable media.

[0161] The memory 42 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 421 and / or cache memory 422. The control terminal may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the memory 42 may be used for reading and writing on non-removable, non-volatile magnetic media ( Figure 9 not shown, commonly referred to as a "hard disk drive"). Although Figure 9 not shown in, a disk drive for reading and writing on removable non-volatile disks (such as "floppy disks") and an optical disk drive for reading and writing on removable non-volatile optical disks (such as CD-ROM, DVD-ROM or other optical media) may be provided. In these cases, each drive may be connected to the bus 43 through one or more data media interfaces. The memory 42 may include at least one program product having a set (such as at least one) of program modules configured to perform the functions of the embodiments of the present disclosure.

[0162] A program / utility 424 having a set (at least one) of program modules 4241 may be stored, for example, in the memory 42. Such program modules 4241 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment. The program modules 4241 generally perform the functions and / or methods in the embodiments described in the present disclosure.

[0163] The electronic device 4 may also communicate with one or more external devices (such as a keyboard, a pointing device, a display, etc.), and may also communicate with one or more devices that enable a user to interact with the control terminal, and / or communicate with any device that enables the control terminal to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication may be carried out through an input / output (I / O) interface 44. Also, the control terminal may communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 45. As Figure 9 shown, the network adapter 45 communicates with other modules of the control terminal through the bus 43. It should be understood that although not shown in the figure, other hardware and / or software modules may be used in conjunction with the control terminal, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0164] The embodiments of the present application also provide a computer-readable storage medium. Those of ordinary skill in the art can understand that all or part of the steps in the methods of the above embodiments can be completed by instructing a processor through a program. The program can be stored in a computer-readable storage medium. The storage medium is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center integrating one or more available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a digital video disc (DVD)), or a semiconductor medium (such as a solid-state disk (SSD)), etc.

[0165] The embodiments of the present application can also provide a computer program product, which includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in the embodiments of the present application are generated. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, or data center to another website, computer, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.).

[0166] When the computer program product is executed by a computer, the computer executes the method described in the foregoing method embodiments. The computer program product can be a software installation package. In the case where the foregoing method needs to be used, the computer program product can be downloaded and executed on the computer.

[0167] In summary, a method, a system, an electronic device, and a medium for calculating a vehicle orientation angle according to the present application have the following

[0168] Advantageous effects:

[0169] This application obtains the image data of the target vehicle according to the surround cameras of the host vehicle; obtains the position information of the target vehicle by using the trained target detection model according to the image data; the position information includes an attribute category and a rotated bounding box, and the rotated bounding box includes center point coordinates, length and width, and a rotation angle; calculates and obtains the projection coordinate information according to the center point coordinates, length and width, and rotation angle in the rotated bounding box; the projection coordinate information is the target rotated bounding box coordinate information of the target vehicle; obtains the target coordinates according to a second preset reference, the set look-up table information corresponding to the image data, and the projection coordinate information; the second preset reference is the projection reference of the preset target vehicle; the target coordinates include a first target coordinate and a second target coordinate, which are used to represent the coordinate information of the target rotated bounding box of the target vehicle in the surround bird's-eye view; obtains the orientation angle of the target vehicle according to the target coordinates, the projection coordinate information, a third preset reference, and the attribute category; the third preset reference is the orientation reference of the preset target vehicle. This application trains the target detection model through a first preset reference, generates the set look-up table information of the surround BEV and the original image pixels, projects the point coordinates based on the set look-up table information, projects the result of the 2D rotated target detection onto the surround BEV through the look-up table, and then calculates the orientation angle of the target vehicle in the host vehicle coordinate system. In this way, the computing power and data cost brought by the 3D target detection model can be reduced by using the 2D rotated target detection method.

[0170] This application also unifies the coordinate systems of the host vehicle and the target vehicle by using the surround BEV, which can reduce the error and time consumption caused by the coordinate conversion between multiple fisheye cameras, reduce the complexity of the 3D target detection orientation angle post-processing algorithm, and is also convenient to be transplanted to low-computing-power mobile terminal devices.

[0171] The descriptions of the processes or structures corresponding to the above respective drawings have their own emphases. For parts not detailed in a certain process or structure, reference can be made to the relevant descriptions of other processes or structures.

[0172] The above embodiments are only illustrative of the principles and effects of this application, and are not used to limit this application. Any person familiar with this technology can modify or change the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or changes made by those with ordinary knowledge in the technical field without departing from the spirit and technical ideas disclosed in this application should still be covered by the claims of this application.

Claims

1. A method for calculating a vehicle heading angle, characterized in that: The method comprises: Acquire image data of the target vehicle based on the surround view camera of the vehicle; Acquire the position information of the target vehicle using the trained target detection model according to the image data; the position information includes attribute categories and a rotation box, and the rotation box includes center point coordinates, length, width and rotation angle; The projection coordinate information is obtained by calculating the center point coordinates, length, width and rotation angle in the rotation frame; the projection coordinate information is the target rotation frame coordinate information of the target vehicle; The target coordinates are acquired according to a second preset reference, the set lookup table information corresponding to the image data and the projection coordinate information; the second preset reference is a preset projection reference of the target vehicle; the target coordinates include a first target coordinate and a second target coordinate, which are used to characterize the coordinate information of the target rotation frame coordinate information of the target vehicle under the bird's-eye view perspective; The orientation angle of the target vehicle is acquired according to the target coordinates, the projection coordinate information, a third preset reference and the attribute category; the third preset reference is a preset orientation reference of the target vehicle.

2. The vehicle heading angle calculation method according to claim 1, characterized in that: The trained target detection model is obtained based on a first preset benchmark training; The first preset reference is a preset reference for calibrating the position information of the target vehicle, and the trained target detection model is a training method comprising: Acquire a surround fisheye image of the target vehicle; Annotating the surround fisheye image of the target vehicle and the first preset reference to obtain annotated surround fisheye image data as a training data set; The first neural network model is trained according to the training data set to obtain a trained target detection model.

3. The vehicle heading angle calculation method according to claim 2, characterized in that: The first preset benchmark includes: If the angle between the front direction of the target vehicle and the reference normal plane is an acute angle, the attribute category of the target vehicle is rearward facing; If the angle between the head direction of the target vehicle and the reference normal plane is an obtuse angle, the attribute category of the target vehicle is head-on; If the angle between the front direction of the target vehicle and the reference normal plane is a right angle, and the front direction of the target vehicle in the image data of the target vehicle is left, then the attribute category of the target vehicle is left-facing; If the angle between the front direction of the target vehicle and the reference normal plane is a right angle, and the front direction of the target vehicle in the image data of the target vehicle is right, then the attribute category of the target vehicle is right-facing.

4. The vehicle heading angle calculation method according to claim 1, characterized in that: Acquiring the target coordinates according to the second preset reference, the set lookup table information corresponding to the image data, and the projection coordinate information includes: Acquire the coordinate information to be projected according to the projection coordinate information; the coordinate information to be projected includes the coordinate with the largest ordinate and the coordinate with the second largest ordinate in the projection coordinate information; Corresponding projection coordinate information is acquired according to the coordinate information to be projected, the setting lookup table information corresponding to the image data and the second preset reference as the target coordinates.

5. The vehicle heading angle calculation method according to claim 1 or 4, characterized in that: The second preset reference includes: If the coordinate with the largest ordinate has not found the corresponding projection coordinate in the set lookup table information corresponding to the image data, it means that the first target coordinate has not been found, and the calculation of the orientation angle of the target vehicle ends; If the coordinate with the largest vertical coordinate has a corresponding projection coordinate found in the setting lookup table information corresponding to the image data, and the coordinate with the second largest vertical coordinate has not a corresponding projection coordinate found in the setting lookup table information corresponding to the image data, assign the midpoint coordinate of the line connecting the coordinate with the largest vertical coordinate and the coordinate with the second largest vertical coordinate to the coordinate with the second largest vertical coordinate, and obtain a new second target coordinate based on the assigned coordinate with the second largest vertical coordinate; Iterate the process until the new coordinate with the second largest ordinate finds the corresponding projection coordinate in the setting lookup table information corresponding to the image data, and obtains the first target coordinate and the second target coordinate.

6. The vehicle heading angle calculation method according to claim 1, characterized in that: Acquiring the orientation angle of the target vehicle according to the target coordinates, the projection coordinate information, the third preset reference and the attribute category includes: Acquire the orientation of the target vehicle according to the target coordinates, the projection coordinate information, the third preset reference and the attribute category; The orientation angle of the target vehicle relative to the own vehicle is obtained according to the orientation of the target vehicle and the orientation of the own vehicle.

7. The vehicle heading angle calculation method according to claim 1 or 6, characterized in that: The third preset reference includes: When the attribute category is vehicle head forward, the corresponding directed vertical line information is calculated according to the undirected straight line information and the directed projection straight line, and the directed vertical line information is used as the direction of the target vehicle; the directed vertical line information is opposite to the direction of the directed projection straight line; When the attribute category is vehicle head-rearward, the corresponding directed vertical line information is calculated according to the undirected straight line information and the directed projection straight line, and the directed vertical line information is used as the direction of the target vehicle; the directed vertical line information has the same direction as the directed projection straight line; When the attribute category is left-facing, the point with the largest horizontal coordinate in the maximum vertical coordinate and the second largest vertical coordinate in the projection coordinate information is mapped to the straight line information in the direction of the coordinate point with the smallest horizontal coordinate, to obtain directed straight line information, and the direction of the directed straight line information is used as the direction of the target vehicle; When the attribute category is that the vehicle is facing right, the coordinate point with a smaller horizontal coordinate in the maximum vertical coordinate and the second largest vertical coordinate in the projection coordinate information is mapped to the straight line information in the direction of the coordinate point with a larger horizontal coordinate, and directed straight line information is obtained, and the direction of the directed straight line information is used as the direction of the target vehicle; Among them, the corresponding undirected straight line information is obtained according to the target coordinates; the corresponding reference normal plane is obtained according to the image data of the target vehicle, and the directed projection straight line of the reference normal plane is determined according to the camera to which the image data of the target vehicle belongs; the directed projection straight line is a directed straight line of the reference normal plane perpendicular to the ground.

8. A vehicle heading angle calculation system, characterized in that: The system comprises: An image acquisition module is configured to acquire image data of a target vehicle; A position acquisition module is configured to acquire the position information of the target vehicle using the trained target detection model according to the image data; the position information includes an attribute category and a rotation box, and the rotation box includes a center point coordinate, a length, a width, and a rotation angle; A coordinate conversion module is configured to calculate and obtain projection coordinate information according to the center point coordinates, length, width and rotation angle in the rotation frame; the projection coordinate information is the target rotation frame coordinate information of the target vehicle; The projection processing module is configured to obtain the target coordinates according to a second preset reference, the set lookup table information corresponding to the image data and the projection coordinate information; the second preset reference is a preset projection reference of the target vehicle; the target coordinates include a first target coordinate and a second target coordinate, which are used to represent the coordinate information of the target rotation frame coordinate information of the target vehicle under the bird's-eye view perspective; The heading angle calculation module is configured to obtain the heading angle of the target vehicle according to the target coordinates, the projection coordinate information, a third preset reference and the attribute category; the third preset reference is a preset heading reference of the target vehicle.

9. An electronic device, characterized in that: The electronic device comprises: a memory configured to store a computer program; A processor is configured to run the computer program to implement the vehicle heading angle calculation method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by an electronic device, the vehicle heading angle calculation method according to any one of claims 1 to 7 is implemented.

Images