A method, device, electronic device, and storage medium for identifying vehicle orientation.

By identifying key parts and relative positions in the target vehicle image, the orientation of the obstacle vehicle can be determined, solving the problem of difficulty in accurately perceiving vehicle orientation in existing technologies and improving the stability of obstacle avoidance and the accuracy of driving trajectory prediction.

CN118692053BActive Publication Date: 2026-11-17CHINA AUTOMOTIVE INNOVATION CORP
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410795019.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-19
Publication Date
2026-11-17
Estimated Expiration
2044-06-19

AI Technical Summary

Technical Problem

Existing technology makes it difficult to accurately detect the orientation of vehicles facing obstacles, which affects the stable driving of the vehicle itself.

Method used

By acquiring images of the target vehicle from an image acquisition device, the target vehicle and its key parts are identified, key points are selected based on their relative positions, and the vehicle's orientation is determined using the coordinate system of the image acquisition device.

Benefits of technology

It enables accurate identification of the orientation of obstacle vehicles, improving the stability of obstacle avoidance and the accuracy of trajectory prediction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118692053B_ABST
    Figure CN118692053B_ABST
Patent Text Reader

Abstract

This application discloses a method, apparatus, electronic device, and storage medium for vehicle orientation recognition. The vehicle orientation recognition method can be applied to the field of intelligent driving technology. The method includes: acquiring a target vehicle image captured by an image acquisition device; identifying the target vehicle in the target vehicle image to obtain a first target region, and identifying key parts of the target vehicle in the target vehicle image to obtain a second target region; selecting target key points of the target vehicle based on the relative positional relationship between the first and second target regions; and determining the orientation of the target vehicle relative to the image acquisition device based on the position of the target key points in the coordinate system corresponding to the image acquisition device. This application achieves two-dimensional vehicle orientation recognition by identifying the target vehicle and its key parts in the image, selecting target key points based on their relative positional relationship, and determining the orientation of the target vehicle based on the position of the target key points.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to a method, device, electronic device and storage medium for recognizing vehicle orientation. Background Technology

[0002] With the rapid development of technology, intelligent driving technology has gradually become a hot research direction in automotive engineering. Obstacle visual perception technology is a key technology in the field of intelligent driving, used to perceive information about obstacles such as vehicles, pedestrians, and electric vehicles encountered during the driving process, including information such as the vehicle's position, orientation, speed, and acceleration. Among these, vehicle visual perception technology occupies an important position in current assisted driving or intelligent driving systems.

[0003] Vehicle vision perception technology is a technique that predicts three-dimensional space from two-dimensional space, including perceiving the position, orientation, length, width, height, and speed of obstacle vehicles. However, because vision is a sensing technology that maps three-dimensional space into two-dimensional images, it lacks one-dimensional depth information compared to sensors such as LiDAR and millimeter-wave radar. For example, LiDAR's point cloud information can directly obtain the orientation information of obstacle vehicles, while obtaining the orientation information of obstacle vehicles from images is more difficult. Nevertheless, recognizing the orientation of obstacle vehicles is particularly important for the prediction, planning, and control modules of a vehicle.

[0004] Currently, it is difficult to accurately perceive the orientation of obstacle vehicles using either camera models or neural networks. If the orientation of obstacle vehicles changes during driving, it will directly affect the stable driving of the vehicle. Summary of the Invention

[0005] To address the problems of existing technologies, this application provides a method, apparatus, electronic device, and storage medium for identifying vehicle orientation. The technical solution is as follows:

[0006] On the one hand, a method for recognizing vehicle orientation is provided, the method comprising:

[0007] Acquire images of the target vehicle captured by the image acquisition device;

[0008] Identify the target vehicle in the target vehicle image to obtain a first target region, and identify the key parts of the target vehicle in the target vehicle image to obtain a second target region;

[0009] Based on the relative positional relationship between the first target area and the second target area, the target key points of the target vehicle are selected;

[0010] Based on the position of the target key points in the coordinate system corresponding to the image acquisition device, the orientation of the target vehicle relative to the image acquisition device is determined.

[0011] On the other hand, a vehicle orientation recognition device is provided, the device comprising:

[0012] The image acquisition module is used to acquire images of the target vehicle captured by the image acquisition device;

[0013] The image recognition module is used to identify the target vehicle in the target vehicle image to obtain a first target region, and to identify key parts of the target vehicle in the target vehicle image to obtain a second target region;

[0014] The target selection module is used to select the target key points of the target vehicle based on the relative positional relationship between the first target area and the second target area;

[0015] An orientation determination module is used to determine the orientation of the target vehicle relative to the image acquisition device based on the position of the target key points in the coordinate system corresponding to the image acquisition device.

[0016] In one exemplary embodiment, the target selection module includes:

[0017] The proportion determination module is used to determine the proportion of the second target area in the first target area;

[0018] The first key point module is used to determine the two endpoints of the key part that are close to the image acquisition device as the first target key point and the second target key point when the proportion is greater than the first preset threshold.

[0019] In one exemplary implementation, the orientation determining module includes:

[0020] An azimuth angle determination module is used to determine whether the azimuth angle of the first target key point and the azimuth angle of the second target key point are complementary in the coordinate system corresponding to the image acquisition device.

[0021] The first included angle module is used to determine the included angle between the side of the target vehicle and the optical axis of the image acquisition device based on the elevation angle of the first target key point and the elevation angle of the second target key point when the judgment result is negative, so as to obtain the target included angle.

[0022] The first orientation module is used to determine the orientation of the target vehicle relative to the image acquisition device based on the target angle and the relative positional relationship between the key part and the target vehicle.

[0023] In one exemplary embodiment, the apparatus further includes a second orientation module for determining the orientation of a target vehicle when the azimuth angle of the first target key point is complementary to the azimuth angle of the second target key point, the second orientation module comprising:

[0024] The second angle module is used to determine that the angle between the side of the target vehicle and the optical axis of the image acquisition device is zero when the judgment result is yes.

[0025] The third orientation module is used to determine the orientation of the target vehicle relative to the image acquisition device based on the relative positional relationship between the key parts and the target vehicle.

[0026] In one exemplary embodiment, the target selection module includes:

[0027] The difference determination module is used to determine the difference between the coordinates of the first target region and the coordinates of the second target region;

[0028] The target side determination module is used to determine the two sides of the target vehicle in the target vehicle image as the first target side and the second target side when the difference is greater than a second preset threshold.

[0029] The second key point module is used to determine the two endpoints of the first target side and the two endpoints of the second target side as the target key points; the first target side and the second target side have overlapping endpoints.

[0030] In one exemplary implementation, the orientation determining module includes:

[0031] The orientation determination module is used to determine the orientation of the target vehicle relative to the image acquisition device in the coordinate system corresponding to the image acquisition device, based on the azimuth angle of the first target key point and the azimuth angle of the second target key point;

[0032] The third angle module is used to determine the angle between the side of the target vehicle and the optical axis of the image acquisition device based on the elevation angle of the target key point, the length and width of the target vehicle, and the orientation of the target vehicle relative to the image acquisition device, so as to obtain the target angle.

[0033] The fourth orientation module is used to determine the orientation of the target vehicle relative to the image acquisition device based on the target angle and the relative positional relationship between the key part and the target vehicle.

[0034] On the other hand, an electronic device is provided, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the vehicle orientation recognition method of any of the above aspects.

[0035] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or the at least one program being loaded and executed by a processor to implement a vehicle orientation recognition method as described above.

[0036] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the vehicle orientation recognition method described above.

[0037] This application embodiment acquires a target vehicle image captured by an image acquisition device; identifies the target vehicle in the target vehicle image to obtain a first target region, and identifies key parts of the target vehicle in the target vehicle image to obtain a second target region; selects target key points of the target vehicle based on the relative positional relationship between the first and second target regions; and determines the orientation of the target vehicle relative to the image acquisition device based on the position of the target key points in the coordinate system corresponding to the image acquisition device. This application achieves two-dimensional vehicle orientation recognition by identifying the target vehicle and its key parts in the image, selecting target key points based on their relative positional relationship, and determining the orientation of the target vehicle based on the position of the target key points. It can accurately calculate the orientation of the target vehicle during driving, and when applied to vehicle visual perception technology, it can predict the driving trajectory of the target vehicle, improving the stability of obstacle avoidance. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart illustrating a vehicle orientation recognition method provided in an embodiment of this application;

[0040] Figure 2 This is a schematic diagram illustrating the relative positional relationship between the first target region and the second target region, as provided in the embodiments of this application.

[0041] Figure 3 This is a schematic diagram illustrating the relative positional relationship between the first target region and the second target region, as provided in the embodiments of this application.

[0042] Figure 4 This is a schematic diagram illustrating the relative positional relationship between the first target region and the second target region in the third type of embodiment provided in this application;

[0043] Figure 5 This is a schematic diagram illustrating the relative positional relationship between the first target region and the second target region in the fourth type of embodiment provided in this application;

[0044] Figure 6 This is a schematic diagram illustrating the relative positional relationship between the first target region and the second target region in the fifth embodiment of this application.

[0045] Figure 7 This is a schematic diagram illustrating the relative positional relationship between key parts in a target vehicle image and the target vehicle in the first type of target vehicle image provided in this application embodiment;

[0046] Figure 8 This is a schematic diagram illustrating the relative positional relationship between key parts and the target vehicle in a second type of target vehicle image provided in this application embodiment;

[0047] Figure 9 This is a schematic diagram illustrating the relative positional relationship between key parts and the target vehicle in a third type of target vehicle image provided in this application embodiment;

[0048] Figure 10 This is a schematic diagram illustrating the relative positional relationship between key parts and the target vehicle in the fourth type of target vehicle image provided in this application embodiment;

[0049] Figure 11 This is a schematic diagram illustrating the relative positional relationship between key parts and the target vehicle in the fifth type of target vehicle image provided in this application embodiment;

[0050] Figure 12 This is a schematic diagram illustrating a method for defining the orientation of a target vehicle, as provided in an embodiment of this application.

[0051] Figure 13 This is a structural block diagram of the first type of target vehicle provided in the embodiments of this application;

[0052] Figure 14 This is a schematic diagram of the first type of target vehicle orientation provided in the embodiments of this application;

[0053] Figure 15 This is a schematic diagram of the second type of target vehicle orientation provided in the embodiments of this application;

[0054] Figure 16 This is a schematic diagram of the third type of target vehicle orientation provided in the embodiments of this application;

[0055] Figure 17 This is a schematic diagram of the fourth target vehicle orientation provided in the embodiments of this application;

[0056] Figure 18 This is a schematic diagram of the fifth target vehicle orientation provided in the embodiments of this application;

[0057] Figure 19 This is a structural block diagram of the second type of target vehicle provided in the embodiments of this application;

[0058] Figure 20 This is a schematic diagram of the sixth target vehicle orientation provided in the embodiments of this application;

[0059] Figure 21 This is a schematic diagram of the seventh target vehicle orientation provided in the embodiments of this application;

[0060] Figure 22 This is a structural block diagram of the third type of target vehicle provided in the embodiments of this application;

[0061] Figure 23 This is a schematic diagram of the eighth target vehicle orientation provided in the embodiments of this application;

[0062] Figure 24 This is a schematic diagram of the ninth target vehicle orientation provided in the embodiments of this application;

[0063] Figure 25 This is a structural block diagram of the fourth type of target vehicle provided in the embodiments of this application;

[0064] Figure 26 This is a structural block diagram of the fifth type of target vehicle provided in the embodiments of this application;

[0065] Figure 27 This is a flowchart illustrating another vehicle orientation recognition method provided in an embodiment of this application;

[0066] Figure 28 This is a structural block diagram of a vehicle orientation recognition device provided in an embodiment of this application;

[0067] Figure 29 This is a hardware structure block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0068] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0069] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0070] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0071] Please see Figure 1 The diagram illustrates a flowchart of a vehicle orientation recognition method provided in an embodiment of this application. It should be noted that while this specification provides method steps as shown in the embodiments or flowcharts, more or fewer steps may be included based on conventional or non-inventive methods. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or product execution, the methods can be executed sequentially according to the embodiments or accompanying drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown... Figure 1 As shown, the method may include:

[0072] S101, acquire the target vehicle image captured by the image acquisition device.

[0073] The image acquisition device is installed on the vehicle and is used to acquire images of obstacles within the vehicle. Specifically, the image acquisition device is mainly used to acquire images of target vehicles that act as obstacles.

[0074] The target vehicle image includes target vehicles that act as obstacles for the vehicle itself. In practice, the target vehicle image may include multiple target vehicles in different poses.

[0075] S103, identify the target vehicle in the target vehicle image to obtain the first target region, and identify the key parts of the target vehicle in the target vehicle image to obtain the second target region.

[0076] The target vehicle is the vehicle that acts as an obstacle in the driving environment.

[0077] The first target region is the area covered by the entirety of a single target vehicle in the target vehicle image.

[0078] The relative positional relationship between the key parts of the target vehicle and the target vehicle as a whole is known. In practice, any one of the iconic parts of the target vehicle, such as the front or rear, can be selected as the key part.

[0079] The second target region is the area covered by key parts of the target vehicle in the target vehicle image. Specifically, the second target region and the first target region indicate the same target vehicle.

[0080] In practice, the target vehicle image is processed based on a neural network model. Specifically, the target vehicle as a whole and its key parts are identified and marked with rectangular frames to obtain the first target area and the second target area of ​​the target vehicle. The first target area is the area covered by the outer rectangular frame of the target vehicle as a whole, and the second target area is the area covered by the outer rectangular frame of the key parts of the target vehicle.

[0081] S105, based on the relative positional relationship between the first target area and the second target area, select the target key points of the target vehicle.

[0082] For example, if the key part is the front or rear of the vehicle, the relative positional relationship between the first target area and the second target area can be divided into five cases, as shown in the reference. Figures 2-6 This is a schematic diagram illustrating the relative positional relationships between five first target regions and second target regions provided in the embodiments of this application; wherein, the solid-line frame corresponds to the first target region, and the dashed-line frame corresponds to the second target region.

[0083] Specifically, we will use the front of the vehicle as a key part for explanation. Figure 2 ,correspond Figure 7 The relative positional relationship between the front of the vehicle and the target vehicle in the image shown indicates that the orientation of the target vehicle, acting as an obstacle, is parallel or nearly parallel to the orientation of the vehicle itself; for Figure 3 ,correspond Figure 8 The relative positional relationship between the front of the vehicle and the target vehicle in the image shown is... Figure 4 ,correspond Figure 9 The image shows the relative positional relationship between the front of the vehicle and the target vehicle. The orientation of the target vehicle, acting as an obstacle, intersects with, but is not perpendicular to, the orientation of the vehicle itself. Figure 5 ,correspond Figure 10 The relative positional relationship between the front of the vehicle and the target vehicle in the image shown is... Figure 6 ,correspond Figure 11The image shows the relative positional relationship between the front of the vehicle and the target vehicle. The target vehicle, acting as an obstacle, is oriented perpendicular to the orientation of the vehicle itself.

[0084] The relative positional relationship between the target key points and the target vehicle is known, and the target key points are visible in the target vehicle image.

[0085] In one exemplary embodiment, step S105 may include the following steps:

[0086] Determine the proportion of the second target area within the first target area;

[0087] When the proportion is greater than the first preset threshold, the two endpoints of the key part closest to the image acquisition device are identified as the first target key point and the second target key point.

[0088] The first target region is the area covered by the entirety of a single target vehicle in the target vehicle image.

[0089] The second target region is the area covered by key parts of the target vehicle in the target vehicle image. Specifically, the second target region and the first target region indicate the same target vehicle.

[0090] The relative positional relationship between the key components and the target vehicle as a whole is known. In practice, any one of the iconic parts of the target vehicle, such as the front or rear, can be selected as the key component.

[0091] In practice, the target vehicle image is processed based on a neural network model. Specifically, the target vehicle as a whole and its key parts are identified and marked with rectangular frames to obtain the first target area and the second target area of ​​the target vehicle. The first target area is the area covered by the outer rectangular frame of the target vehicle as a whole, and the second target area is the area covered by the outer rectangular frame of the key parts of the target vehicle.

[0092] For example, the key part is the front or rear of the vehicle. In the target vehicle image, if the proportion of the second target region in the first target region is greater than a first preset threshold, then the first target region and the second target region are considered to overlap. Figure 13 The diagram shows a structural block diagram of a first type of target vehicle provided in this application embodiment; wherein the first preset threshold can be set to 0.8. Further, the target vehicle is mapped to the coordinate system O corresponding to the image acquisition device. c -X c Y c Z c ,like Figure 12 The diagram shown is a schematic diagram of a method for defining the orientation of a target vehicle according to an embodiment of this application, wherein O cLet Z be the optical center of the image acquisition device, and let Z be the direction along the optical axis of the image acquisition device. c Positive direction of the axis, X c Axis, Y c The axis is defined according to the right-hand rule. In O c -X c Z c In the middle, for reference Figures 14-18 The diagram shows five target vehicle orientations provided in the embodiments of this application. The key parts are marked with line segment ab on the side closer to the image acquisition device. Point a in the diagram is the first target key point, and point b in the diagram is the second target key point.

[0093] As can be seen from the above technical solutions of the embodiments of this application, when the first target area and the second target area are almost overlapping, the embodiments of this application select two endpoints of the key parts near the image acquisition device as target key points to calculate the orientation of the target vehicle relative to the image acquisition device, and can accurately calculate the orientation of the target vehicle during driving.

[0094] In one exemplary embodiment, step S105 may further include the following steps:

[0095] Determine the difference between the coordinates of the first target region and the coordinates of the second target region;

[0096] When the difference is greater than the second preset threshold, the two sides of the target vehicle in the target vehicle image are determined as the first target side and the second target side;

[0097] The two endpoints of the first target side and the two endpoints of the second target side are identified as target key points.

[0098] The first target region is the area covered by the entire target vehicle in the image. Specifically, the coordinates of the first target region are the coordinates of the first target region in the image coordinate system.

[0099] The second target region is the area covered by key parts of the target vehicle in the target vehicle image. Specifically, the second target region and the first target region indicate the same target vehicle. Specifically, the coordinates of the second target region are the coordinates of the second target region in the image coordinate system.

[0100] In practice, the target vehicle image is processed based on a neural network model. Specifically, the target vehicle as a whole and its key parts are identified and marked with rectangular frames to obtain the first target area and the second target area of ​​the target vehicle. The first target area is the area covered by the outer rectangular frame of the target vehicle as a whole, and the second target area is the area covered by the outer rectangular frame of the key parts of the target vehicle.

[0101] Among them, the first target side and the second target side are the two visible sides of the four sides of the target vehicle in the target vehicle image, and the first target side and the second target side have overlapping endpoints.

[0102] The relative positional relationship between the target key points and the target vehicle is known, and the target key points are visible in the target vehicle image. In specific implementation, there are 3 target key points.

[0103] For example, the key part is the front or rear of the vehicle. In the target vehicle image, if the difference between the coordinates of the first target area and the coordinates of the second target area is greater than a second preset threshold, then the orientation of the target vehicle, which acts as an obstacle, intersects with the orientation of the vehicle itself. In specific implementations, this can be divided into... Figures 8-11 The four scenarios are shown.

[0104] Specifically, in the target vehicle image, when the difference between the x-coordinate of the first target region and the x-coordinate of the second target region is greater than a second preset threshold, a reference is made. Figure 19 The diagram shown is a structural block diagram of the second type of target vehicle provided in this application embodiment; here, the second preset threshold can be set to 30 pixels. Further, the target vehicle is mapped to the coordinate system O corresponding to the image acquisition device. c -X c Y c Z c ,like Figure 12 The diagram shown is a schematic diagram of a method for defining the orientation of a target vehicle according to an embodiment of this application, wherein O c Let Z be the optical center of the image acquisition device, and let Z be the direction along the optical axis of the image acquisition device. c Positive direction of the axis, X c Axis, Y c The axis is defined according to the right-hand rule. In O c -X c Z c In the middle, for reference Figures 20-21 The diagram shows two target vehicle orientations provided in the embodiments of this application. Line segment ac represents the first target side, line segment bc represents the second target side, and points a, b, and c represent key target points.

[0105] Specifically, in the target vehicle image, when the difference between the x-coordinate of the second target region and the x-coordinate of the first target region is greater than a second preset threshold, a reference is made. Figure 22 The diagram shows a structural block diagram of the third type of target vehicle provided in this application embodiment; here, the second preset threshold can be set to 30 pixels. Further, the target vehicle is mapped to the coordinate system O corresponding to the image acquisition device. c -X c Y c Z c In O c -Xc Z c In the middle, for reference Figures 23-24 The diagram shows two target vehicle orientations provided in the embodiments of this application. Line segment ac represents the first target side, line segment bc represents the second target side, and points a, b, and c represent key target points.

[0106] Specifically, if the difference between the x-coordinate of the first target region and the x-coordinate of the second target region is greater than a second preset threshold, and if the width of the first target region is greater than a certain multiple of the width of the second target region, and the x-coordinate of the second target region is less than the x-coordinate of the first target region, then refer to... Figure 25 The diagram shows a structural block diagram of the fourth type of target vehicle provided in this application embodiment; the second preset threshold can be set to 0.8, and the multiple can be set to 4. In this case, the pixel width of the second target area can be set to transform... Figure 19 The situation shown is used to determine the target key points and calculate the vehicle orientation. In specific implementation, the pixel width of the second target area can be set to a value greater than or equal to 1.

[0107] Specifically, if the difference between the x-coordinate of the second target region and the x-coordinate of the first target region is greater than a second preset threshold, and if the width of the first target region is greater than a certain multiple of the width of the second target region, and the x-coordinate of the second target region is greater than the x-coordinate of the first target region, then refer to... Figure 26 The diagram shows a structural block diagram of the fourth type of target vehicle provided in this application embodiment; the second preset threshold can be set to 0.8, and the multiple can be set to 4. In this case, the pixel width of the second target area can be set to transform... Figure 22 The situation shown is used to determine the target key points and calculate the vehicle orientation. In specific implementation, the pixel width of the second target area can be set to a value greater than or equal to 1.

[0108] As can be seen from the above technical solutions of the embodiments of this application, when the difference between the coordinates of the first target area and the coordinates of the second target area is greater than a second preset threshold, the endpoints on both sides of the target vehicle in the target vehicle image are selected as target key points to calculate the orientation of the target vehicle relative to the image acquisition device, thereby accurately calculating the orientation of the target vehicle during driving.

[0109] S107, Based on the position of the target key points in the coordinate system corresponding to the image acquisition device, determine the orientation of the target vehicle relative to the image acquisition device.

[0110] Specifically, such as Figure 12 The diagram shown is a schematic of a method for defining the orientation of a target vehicle according to an embodiment of this application, which maps the target vehicle to the coordinate system O corresponding to the image acquisition device.c -X c Y c Z c O c Let Z be the optical center of the image acquisition device, and let Z be the direction along the optical axis of the image acquisition device. c Positive direction of the axis, X c Axis, Y c According to the right-hand rule, the orientation of the target vehicle relative to the image acquisition device is defined as the yaw angle, which is defined as the angle between the target vehicle's side profile and the Z-axis. c The angle between the axes; the direction of the yaw angle is defined according to the rotation direction of the right-hand rule, from Y... c Looking at the axis from the positive direction to the negative direction, counterclockwise rotation is positive, and clockwise rotation is negative. Specifically, this can be determined based on the origin O of the target key point. c To the target key point and Z c The included angle along the positive axis, i.e., the elevation angle of the target key point, is used to calculate the yaw angle. The elevation angle of the target key point can be calculated based on the intrinsic parameters of the image acquisition device. Specifically, the definition of the intrinsic parameter matrix K of the image acquisition device is given in formula (1):

[0111]

[0112] Among them, f x Let f be the projection of the focal length f of the image acquisition device in the x-direction. y Let (u0, v0) be the projection of the focal length f in the y direction, and let (u0, v0) be the coordinates of the optical axis projection in the image pixel coordinate system.

[0113] As can be seen from the above technical solutions of the embodiments of this application, the embodiments of this application acquire images of the target vehicle, identify the target vehicle and key parts of the target vehicle in the image, select the target key points of the target vehicle based on the relative positional relationship between the two, and determine the orientation of the target vehicle based on the position of the target key points, thereby realizing the vehicle orientation recognition in two-dimensional space. It can accurately calculate the orientation of the target vehicle during driving. When applied to vehicle visual perception technology, it can predict the driving trajectory of the target vehicle and improve the stability of vehicle obstacle avoidance.

[0114] In one exemplary implementation, such as Figures 14-17 The diagram shown illustrates four target vehicle orientations provided in this application embodiment. Step S107 may include the following steps:

[0115] In the coordinate system corresponding to the image acquisition device, determine whether the azimuth angles of the first target key point and the second target key point are complementary;

[0116] If the judgment result is negative, the angle between the side of the target vehicle and the optical axis of the image acquisition device is determined based on the elevation angle of the first target key point and the elevation angle of the second target key point, thus obtaining the target angle.

[0117] Based on the target angle and the relative positional relationship between the key parts and the target vehicle, the orientation of the target vehicle relative to the image acquisition device is determined.

[0118] The coordinate system corresponding to the image acquisition device is: Figure 12 The coordinate system O shown c -X c Y c Z c Specifically, O c Let Z be the optical center of the image acquisition device, and let Z be the direction along the optical axis of the image acquisition device. c Positive direction of the axis, X c Axis, Y c According to the right-hand rule, the orientation of the target vehicle relative to the image acquisition device is defined as the target angle yaw, and the yaw angle is defined as the angle between the side profile of the target vehicle and the Z-axis. c The angle between the axes; the direction of the yaw angle is defined according to the rotation direction of the right-hand rule, from Y... c Looking from the positive direction of the axis towards the negative direction, counterclockwise rotation is positive, and clockwise rotation is negative.

[0119] Specifically, point a is the first target key point, point b in the diagram is the second target key point, and point c is the midpoint of line segment ab. The case where the azimuth angles of the first and second target key points are complementary is excluded, meaning point c is excluded from the Z-plane. c O c Y c The above situation.

[0120] Specifically, θ a Let θ be the angle of elevation of point a. b Let θ be the elevation angle of point b, and θ be the side profile of the target vehicle relative to Z. c The included angle of the axle is yaw = θ if the critical part is the rear of the vehicle; and yaw = θ + π if the critical part is the front of the vehicle. θ can be calculated using formula (2):

[0121]

[0122] Specifically, in Figure 14 In the case shown, exist Figure 15 In the case shown, exist Figure 16 In the case shown, exist Figure 17 In the case shown,

[0123] As can be seen from the above technical solutions of the embodiments of this application, the embodiments of this application map the target vehicle to the coordinate system corresponding to the image acquisition device, calculate the orientation of the target vehicle based on the elevation angle of the target key points and the relative positional relationship between the key parts and the target vehicle, realize the vehicle orientation recognition in two-dimensional space, can accurately calculate the orientation of the target vehicle during driving, and when applied to vehicle visual perception technology, can predict the driving trajectory of the target vehicle and improve the stability of vehicle obstacle avoidance.

[0124] In one exemplary implementation, reference Figure 18 The diagram shown is a fifth type of target vehicle orientation provided in this application embodiment. After the step of determining whether the azimuth angle of the first target key point and the azimuth angle of the second target key point are complementary, the following steps may also be included:

[0125] When the judgment result is yes, the angle between the side of the target vehicle and the optical axis of the image acquisition device is determined to be zero;

[0126] Based on the relative positional relationship between key components and the target vehicle, the orientation of the target vehicle relative to the image acquisition equipment is determined.

[0127] Specifically, the coordinate system corresponding to the image acquisition device is: Figure 12 The coordinate system O shown c -X c Y c Z c Specifically, O c Let Z be the optical center of the image acquisition device, and let Z be the direction along the optical axis of the image acquisition device. c Positive direction of the axis, X c Axis, Y c According to the right-hand rule, the orientation of the target vehicle relative to the image acquisition device is defined as the target angle yaw, and the yaw angle is defined as the angle between the side profile of the target vehicle and the Z-axis. c The angle between the axes; the direction of the yaw angle is defined according to the rotation direction of the right-hand rule, from Y... c Looking from the positive direction of the axis towards the negative direction, counterclockwise rotation is positive, and clockwise rotation is negative.

[0128] Specifically, point a is the first key point of the objective, point b in the diagram is the second key point of the objective, point c is the midpoint of line segment ab, and θ a Let θ be the angle of elevation of point a. b Let θ be the elevation angle of point b, and θ be the side profile of the target vehicle relative to Z. c The included angle of the axes; when the azimuth angles of the first target key point and the second target key point are complementary, point c lies on plane Z. c O c Y cAt the top, θ = 0. If the critical part is the rear of the car, then the yaw angle = θ; if the critical part is the front of the car, then the yaw angle = θ + π.

[0129] As can be seen from the above technical solutions of the embodiments of this application, when the orientation of the target vehicle is parallel to the orientation of the vehicle, the orientation of the target vehicle is directly determined, realizing the vehicle orientation recognition in two-dimensional space, improving the response speed of identifying the orientation of the target vehicle, and when applied to vehicle visual perception technology, it can predict the driving trajectory of the target vehicle and improve the stability of the vehicle's obstacle avoidance.

[0130] In one exemplary implementation, reference Figures 20-21 Figures 23 and 24 illustrate four target vehicle orientations provided in the embodiments of this application. Step S107 may further include the following steps:

[0131] In the coordinate system corresponding to the image acquisition device, the orientation of the target vehicle relative to the image acquisition device is determined based on the azimuth angle of the first target key point and the azimuth angle of the second target key point.

[0132] Based on the elevation angle of the target key points, as well as the length and width of the target vehicle, and combined with the orientation of the target vehicle relative to the image acquisition device, the angle between the side of the target vehicle and the optical axis of the image acquisition device is determined, thus obtaining the target angle.

[0133] Based on the target angle and the relative positional relationship between the key parts and the target vehicle, the orientation of the target vehicle relative to the image acquisition device is determined.

[0134] The coordinate system corresponding to the image acquisition device is: Figure 12 The coordinate system O shown c -X c Y c Z c Specifically, O c Let Z be the optical center of the image acquisition device, and let Z be the direction along the optical axis of the image acquisition device. c Positive direction of the axis, X c Axis, Y c According to the right-hand rule, the orientation of the target vehicle relative to the image acquisition device is defined as the target angle yaw, and the yaw angle is defined as the angle between the side profile of the target vehicle and the Z-axis. c The angle between the axes; the direction of the yaw angle is defined according to the rotation direction of the right-hand rule, from Y... c Looking from the positive direction of the axis towards the negative direction, counterclockwise rotation is positive, and clockwise rotation is negative.

[0135] The length and width of the target vehicle can be calculated using model inference or estimated using camera projection.

[0136] Specifically, points a, b, and c are the target key points. Let be the angle of elevation of point a. Let be the angle of elevation of point b. Let C be the elevation angle of point c, W be the width of the target vehicle, and L be the length of the target vehicle.

[0137] like Figure 20 As shown, θ represents the target vehicle's side profile and Z-axis. c The included angle of the axis, If the critical part is the rear of the car, then the yaw angle = θ; if the critical part is the front of the car, then the yaw angle = θ + π. θ can be calculated using formulas (3) to (7):

[0138]

[0139] Asin(θ+θ b )=Bcos(θ+θ a (4)

[0140]

[0141]

[0142] like Figure 21 As shown, θ represents the intersection of the short side and the Z-axis of the target vehicle. c The included angle of the axis, If the critical part is the rear of the car, then If the critical part is the front of the car, then θ can be calculated using formulas (8) to (12):

[0143]

[0144] Asin(θ+θ b )=Bcos(θ+θ a (9)

[0145]

[0146] like Figure 23 As shown, θ represents the target vehicle's side profile and Z-axis. c The included angle of the axis, If the critical part is the rear of the car, then the yaw angle = θ; if the critical part is the front of the car, then the yaw angle = θ + π. θ can be calculated using formulas (13) to (17):

[0147]

[0148] Asin(θ+θ b )=Bcos(θ+θ a (14)

[0149]

[0150] like Figure 24 As shown, θ represents the intersection of the short side and the Z-axis of the target vehicle. c The included angle of the axle, if the critical part is the rear of the vehicle, then If the critical part is the front of the car, then θ can be calculated using formulas (18) to (22):

[0151]

[0152] Asin(θ+θ b )=Bcos(θ+θ a (19)

[0153]

[0154] As can be seen from the above technical solutions of the embodiments of this application, the embodiments of this application map the target vehicle to the coordinate system corresponding to the image acquisition device, calculate the orientation of the target vehicle based on the elevation angle of the target key points, the length of the target vehicle, the width of the target vehicle, the orientation of the target vehicle relative to the image acquisition device, and the relative positional relationship between the key parts and the target vehicle, and realize vehicle orientation recognition in two-dimensional space. It can accurately calculate the orientation of the target vehicle during driving. When applied to vehicle visual perception technology, it can predict the driving trajectory of the target vehicle and improve the stability of vehicle obstacle avoidance.

[0155] To facilitate a full understanding of the solution presented in this application, the process of identifying the orientation of the target vehicle is summarized below. Please refer to [link / reference needed]. Figure 27 The diagram shown is a flowchart illustrating another vehicle orientation recognition method provided in this application embodiment, which may specifically include the following steps:

[0156] (1) Acquire visual images captured by the image acquisition device, the visual images including the target vehicle;

[0157] (2) The visual image is used as the input of the neural network model to obtain the full frame and the sub-frame. The full frame is used to mark the whole vehicle of the target vehicle, and its coverage area is the first target area. The sub-frame is used to mark the key parts of the target vehicle, and its coverage area is the second target area.

[0158] (3) Determine the relative positional relationship between the full frame and the child frames, denoted as Rect type;

[0159] (4) Determine the sub-frame category based on the key parts marked by the sub-frame, such as the front frame, the rear frame, etc.

[0160] (5) Estimate the length and width of the target vehicle based on the camera parameters;

[0161] (6) Calculate the orientation of the target vehicle based on the Rect type, subframe category, and the length and width of the target vehicle.

[0162] Corresponding to the vehicle orientation recognition methods provided in the above embodiments, this application also provides a vehicle orientation recognition device. Since the vehicle orientation recognition device provided in this application corresponds to the vehicle orientation recognition methods provided in the above embodiments, the implementation methods of the aforementioned vehicle orientation recognition methods are also applicable to the vehicle orientation recognition device provided in this embodiment, and will not be described in detail in this embodiment.

[0163] Please see Figure 28 The diagram shows a structural schematic of a vehicle orientation recognition device provided in an embodiment of this application. This device has the function of implementing the vehicle orientation recognition method described in the above-described method embodiments. This function can be implemented in hardware or by hardware executing corresponding software. Figure 28 As shown, the device may include:

[0164] Image acquisition module 2810 is used to acquire the target vehicle image captured by the image acquisition device;

[0165] The image recognition module 2820 is used to identify the target vehicle in the target vehicle image to obtain a first target region, and to identify the key parts of the target vehicle in the target vehicle image to obtain a second target region;

[0166] The target selection module 2830 is used to select target key points of the target vehicle based on the relative positional relationship between the first target area and the second target area;

[0167] Orientation determination module 2840 is used to determine the orientation of the target vehicle relative to the image acquisition device based on the position of the target key points in the coordinate system corresponding to the image acquisition device.

[0168] In one exemplary implementation, the target selection module includes:

[0169] The proportion determination module is used to determine the proportion of the second target area within the first target area;

[0170] The first key point module is used to identify the two endpoints of the key part closest to the image acquisition device as the first target key point and the second target key point when the proportion is greater than the first preset threshold.

[0171] In one exemplary implementation, the orientation determining module includes:

[0172] The azimuth angle determination module is used to determine whether the azimuth angles of the first target key point and the second target key point are complementary in the coordinate system corresponding to the image acquisition device.

[0173] The first included angle module is used to determine the included angle between the side of the target vehicle and the optical axis of the image acquisition device based on the elevation angle of the first target key point and the elevation angle of the second target key point when the judgment result is negative, so as to obtain the target included angle.

[0174] The first orientation module is used to determine the orientation of the target vehicle relative to the image acquisition device based on the target angle and the relative positional relationship between the key parts and the target vehicle.

[0175] In one exemplary embodiment, the apparatus further includes a second orientation module for determining the orientation of a target vehicle when the azimuth angle of the first target key point is complementary to the azimuth angle of the second target key point. The second orientation module includes:

[0176] The second angle module is used to determine that the angle between the side of the target vehicle and the optical axis of the image acquisition device is zero when the judgment result is yes.

[0177] The third orientation module is used to determine the orientation of the target vehicle relative to the image acquisition device based on the relative positional relationship between the key parts and the target vehicle.

[0178] In one exemplary implementation, the target selection module includes:

[0179] The difference determination module is used to determine the difference between the coordinates of the first target area and the coordinates of the second target area;

[0180] The target side determination module is used to determine the two sides of the target vehicle in the target vehicle image as the first target side and the second target side when the difference is greater than the second preset threshold.

[0181] The second key point module is used to determine the two endpoints of the first target side and the two endpoints of the second target side as target key points; the first target side and the second target side have overlapping endpoints.

[0182] In one exemplary implementation, the orientation determining module includes:

[0183] The orientation determination module is used to determine the orientation of the target vehicle relative to the image acquisition device in the coordinate system corresponding to the image acquisition device, based on the azimuth angles of the first target key point and the second target key point.

[0184] The third angle module is used to determine the angle between the side of the target vehicle and the optical axis of the image acquisition device based on the elevation angle of the target key point, the length and width of the target vehicle, and the orientation of the target vehicle relative to the image acquisition device, so as to obtain the target angle.

[0185] The fourth orientation module is used to determine the orientation of the target vehicle relative to the image acquisition device based on the target angle and the relative positional relationship between the key parts and the target vehicle.

[0186] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0187] This application provides an electronic device including a processor and a memory. The memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement any of the vehicle orientation recognition methods provided in the above method embodiments.

[0188] Memory is used to store software programs and modules. The processor executes these stored software programs and modules to perform various functional applications and data processing. Memory can primarily consist of a program storage area and a data storage area. The program storage area stores the operating system, application programs required for functionality, etc.; the data storage area stores data created based on device usage, etc. Furthermore, memory can include high-speed random access memory (RAM) and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory can also include a memory controller to provide the processor with access to the memory.

[0189] The method embodiments provided in this application can be executed in a computer terminal, server or similar computing device, that is, the above-mentioned electronic device may include a computer terminal, server or similar computing device. Figure 29 This is a hardware structure block diagram of a computer device for running a vehicle orientation recognition method provided in an embodiment of the present invention, such as... Figure 29 As shown, the internal structure of this computer device may include, but is not limited to, a processor, a network interface, and a memory. The processor, network interface, and memory within the computer device can be connected via a bus or other means, as illustrated in the embodiments of this specification. Figure 29 Taking the example of a connection between China and Israel via a bus.

[0190] The processor (or CPU, Central Processing Unit) is the computing and control core of the computer device. The network interface may optionally include a standard wired interface or a wireless interface (such as Wi-Fi, mobile communication interface, etc.). The memory is the storage device in the computer device used to store programs and data. It is understood that the memory here can be a high-speed RAM storage device, or a non-volatile memory device, such as at least one disk storage device; optionally, it can also be at least one storage device located remotely from the aforementioned processor. The memory provides storage space, which stores the operating system of the electronic device, including but not limited to: Windows (an operating system), Linux (an operating system), Android (a mobile operating system), iOS (a mobile operating system), etc., which are not limited in this invention; and the storage space also stores one or more instructions suitable for being loaded and executed by the processor, which can be one or more computer programs (including program code). In the embodiments of this specification, the processor loads and executes one or more instructions stored in the memory to implement the vehicle orientation recognition method provided in the above method embodiments.

[0191] Embodiments of this application also provide a computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a vehicle orientation recognition method. The at least one instruction or the at least one program is loaded and executed by the processor to implement any of the vehicle orientation recognition methods provided in the above-described method embodiments.

[0192] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0193] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0194] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0195] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0196] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for recognizing vehicle orientation, characterized in that, The method includes: Acquire images of the target vehicle captured by the image acquisition device; The target vehicle in the target vehicle image is identified to obtain a first target area covered by an outer frame used to mark the target vehicle as a whole, and key parts of the target vehicle in the target vehicle image are identified to obtain a second target area covered by an outer frame used to mark the key parts; the key parts are the iconic parts of the target vehicle. Based on the relative positional relationship between the first target area and the second target area, target key points of the target vehicle are selected; the target key points are the two endpoints of the key part closer to the image acquisition device, or the endpoints of the target vehicle on both sides visible in the target vehicle image; Based on the position of the target key points in the coordinate system corresponding to the image acquisition device, the orientation of the target vehicle relative to the image acquisition device is determined.

2. The vehicle orientation recognition method according to claim 1, characterized in that, The step of selecting target key points of the target vehicle based on the relative positional relationship between the first target region and the second target region includes: Determine the proportion of the second target region within the first target region; When the proportion is greater than the first preset threshold, the two endpoints of the key part that are close to the image acquisition device are determined as the first target key point and the second target key point.

3. The vehicle orientation recognition method according to claim 2, characterized in that, Determining the orientation of the target vehicle relative to the image acquisition device based on the position of the target key points in the coordinate system corresponding to the image acquisition device includes: In the coordinate system corresponding to the image acquisition device, determine whether the azimuth angle of the first target key point and the azimuth angle of the second target key point are complementary; If the result of the judgment is negative, the angle between the side of the target vehicle and the optical axis of the image acquisition device is determined based on the elevation angle of the first target key point and the elevation angle of the second target key point, thus obtaining the target angle; Based on the target angle and the relative positional relationship between the key parts and the target vehicle, the orientation of the target vehicle relative to the image acquisition device is determined.

4. The vehicle orientation recognition method according to claim 3, characterized in that, The method further includes: When the result of the judgment is yes, the angle between the side of the target vehicle and the optical axis of the image acquisition device is determined to be zero; Based on the relative positional relationship between the key components and the target vehicle, the orientation of the target vehicle relative to the image acquisition device is determined.

5. The vehicle orientation recognition method according to claim 1, characterized in that, The step of selecting target key points of the target vehicle based on the relative positional relationship between the first target region and the second target region includes: Determine the difference between the coordinates of the first target region and the coordinates of the second target region; When the difference is greater than a second preset threshold, the two sides of the target vehicle in the target vehicle image are determined as the first target side and the second target side. The two endpoints of the first target side and the two endpoints of the second target side are defined as the target key points; the first target side and the second target side have overlapping endpoints.

6. The vehicle orientation recognition method according to claim 5, characterized in that, Determining the orientation of the target vehicle relative to the image acquisition device based on the position of the target key points in the coordinate system corresponding to the image acquisition device includes: In the coordinate system corresponding to the image acquisition device, the orientation of the target vehicle relative to the image acquisition device is determined based on the azimuth angle of the first target key point and the azimuth angle of the second target key point; Based on the elevation angle of the target key point, the length and width of the target vehicle, and the orientation of the target vehicle relative to the image acquisition device, the angle between the side of the target vehicle and the optical axis of the image acquisition device is determined, thus obtaining the target angle. Based on the target angle and the relative positional relationship between the key parts and the target vehicle, the orientation of the target vehicle relative to the image acquisition device is determined.

7. A vehicle orientation recognition device, characterized in that, The device includes: The image acquisition module is used to acquire images of the target vehicle captured by the image acquisition device; An image recognition module is used to identify a target vehicle in the target vehicle image, obtain a first target area covered by an outer frame used to mark the entire target vehicle, and identify key parts of the target vehicle in the target vehicle image, obtain a second target area covered by an outer frame used to mark the key parts; the key parts are the iconic parts of the target vehicle. The target selection module is used to select target key points of the target vehicle based on the relative positional relationship between the first target area and the second target area; the target key points are the two endpoints of the key part closer to the image acquisition device, or the endpoints of the target vehicle on both sides visible in the target vehicle image; An orientation determination module is used to determine the orientation of the target vehicle relative to the image acquisition device based on the position of the target key points in the coordinate system corresponding to the image acquisition device.

8. An electronic device, characterized in that, The system includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the vehicle orientation recognition method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing at least one instruction or at least one program, said at least one instruction or said at least one program being loaded and executed by a processor to implement the vehicle orientation recognition method as described in any one of claims 1 to 6.

10. A computer program, characterized in that, When the computer program is executed by the processor, it implements the vehicle orientation recognition method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Vehicle pose recognition method and device, computer equipment and storage medium

    CN116597012A

  • Vehicle orientation estimation method and device based on multi-task network, equipment and medium

    CN117274678A