Visual sensor blind spot monitoring method, device, equipment, medium and program product
Patent Information
- Application Number
- CN202411960027.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-27
- Publication Date
- 2025-04-04
Smart Images
Figure CN119749532A_ABST
Abstract
Description
Background Art
[0002] With the development of autonomous driving, more and more vehicles are automatically controlling their operations through autonomous driving systems. At present, the main means used by autonomous driving systems to ensure driving safety is to use visual sensors installed in the vehicle, such as radar, lidar, cameras, etc., to detect and identify the driving environment, including surrounding obstacles, traffic lights, etc.
[0003] However, due to many considerations such as the sensor's field of view, composition, installation and use, it is generally difficult to cover all spaces within a certain radius around the main vehicle. At the same time, due to mutual occlusion between obstacles, some obstacles are located in the blind spot of the visual sensor's field of view. The visual sensor cannot perceive the existence of the obstacle, affecting the safety of autonomous driving.
[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention
[0005] The purpose of the present disclosure is to provide a visual sensor blind spot monitoring method, device, equipment, medium and program product, which can simplify the calculation complexity to at least a certain extent, reduce the difficulty of monitoring blind spots, improve accuracy, reduce the data processing volume of hardware, and realize efficient and accurate blind spot monitoring.
[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.
[0007] According to one aspect of the present disclosure, a method for monitoring blind spots of a visual sensor is provided, the method comprising:
[0008] Perform obstacle target detection based on visual sensors to determine at least one obstacle around the main vehicle and multiple vertices of each obstacle;
[0009] Projecting multiple vertices into the coordinate system of the visual sensor to obtain sensor coordinates of multiple vertices of each obstacle;
[0010] For any obstacle, the sensor coordinates of each vertex are projected onto a two-dimensional surface centered on the vehicle to obtain a polygon; a polygon is composed of the two-dimensional coordinates of multiple vertices on the two-dimensional surface;
[0011] Determine the field of view rectangle of the vision sensor;
[0012] Based on the polygon and the field of view rectangle, determine whether any obstacle is in the sensor's blind spot.
[0013] In some embodiments, the sensor coordinates of each vertex are projected onto a two-dimensional surface centered on the main vehicle to obtain a polygon, including:
[0014] The sensor coordinates of each vertex are projected onto a two-dimensional sphere or a two-dimensional cylinder centered on the vehicle to obtain a polygon.
[0015] In some embodiments, determining whether any obstacle is in the blind spot according to the polygon and the field of view rectangle includes:
[0016] Determine a first overlapping area of the polygon and the field of view rectangle;
[0017] determining a first ratio of the first overlapping area to the polygon;
[0018] If the first ratio is greater than a first preset threshold, it is determined that any obstacle is not in the blind spot;
[0019] If the first ratio is less than or equal to the first preset threshold, it is determined that any obstacle is in the blind spot.
[0020] In some embodiments, the method further comprises:
[0021] Acquire a first target obstacle that is not in the blind spot;
[0022] Get the polygon corresponding to each first target obstacle;
[0023] For any two first target obstacles among the plurality of first target obstacles, determining a second overlapping area of the polygons of the any two first target obstacles;
[0024] Determine a second ratio of the second overlapping area to the polygonal area corresponding to the long-distance obstacle; the long-distance obstacle is the first target obstacle far from the main vehicle among the two first target obstacles for determining the second overlapping area;
[0025] According to the second ratio, it is determined whether the long-distance obstacle of the first target obstacle is blocked.
[0026] In some embodiments, the polygon corresponding to the obstacle is at least: a minimum circumscribed rectangle composed of multiple vertices and aligned with the coordinate axes of the two-dimensional surface, or a minimum convex polygon.
[0027] In some embodiments, the method further comprises:
[0028] Acquire a first target obstacle that is not in the blind spot;
[0029] Projecting multiple vertices corresponding to the first target obstacle into a local coordinate system centered on the main vehicle to obtain coordinates of the local coordinate system;
[0030] Obtain the minimum bounding rectangle corresponding to the local coordinate system coordinates of each first target obstacle; the minimum bounding rectangle is formed by projecting the local coordinate system coordinates of multiple vertices on a two-dimensional surface and aligning with the coordinate axis;
[0031] For any two first target obstacles, determining whether corresponding minimum circumscribed rectangles overlap;
[0032] If they do not overlap, it is determined that any two first target obstacles are not blocked;
[0033] If there is overlap, the overlapping second target obstacle is obtained.
[0034] In some embodiments, after acquiring the overlapping second target obstacle, the method further includes:
[0035] Obtain the minimum convex polygon corresponding to each second target obstacle;
[0036] Determine the fourth overlapping area of every two polygons in the minimum convex polygon corresponding to the second target obstacle;
[0037] Determine a third ratio of the fourth overlapping area to the polygonal area corresponding to the long-distance obstacle; the long-distance obstacle is the second target obstacle farthest from the main vehicle among the two second target obstacles for determining the second overlapping area;
[0038] It is determined whether a long-distance obstacle in the second target obstacle is blocked according to the third ratio.
[0039] According to another aspect of the present disclosure, there is also provided a visual sensor blind spot monitoring device, comprising:
[0040] A detection unit, used to detect obstacles based on a visual sensor, and determine the world coordinate system coordinates of at least one obstacle around the main vehicle and multiple vertices of each obstacle;
[0041] A first determining unit, used to project the world coordinate system coordinates into the coordinate system of the visual sensor to obtain the sensor coordinates of the multiple vertices of each obstacle;
[0042] A projection unit is used to project the sensor coordinates of each vertex of any obstacle onto a two-dimensional surface centered on the main vehicle to obtain a polygon; the polygon is composed of the two-dimensional coordinates of multiple vertices on the two-dimensional surface;
[0043] A second determining unit, used to determine a field of view rectangle of the visual sensor;
[0044] The third determination unit is used to determine, for any obstacle, whether any obstacle is in the blind area of the sensor according to the polygon and the field of view rectangle.
[0045] In some embodiments, the visual sensor blind spot monitoring device further includes: an occlusion monitoring unit;
[0046] An occlusion monitoring unit, used for acquiring a first target obstacle that is not in the blind spot;
[0047] Get the polygon corresponding to each first target obstacle;
[0048] For any two first target obstacles among the plurality of first target obstacles, determining a second overlapping area of the polygons of the any two first target obstacles;
[0049] Determine a second ratio of the second overlapping area to the polygonal area corresponding to the long-distance obstacle; the long-distance obstacle is the first target obstacle far from the main vehicle among the two first target obstacles for determining the second overlapping area;
[0050] According to the second ratio, it is determined whether the long-distance obstacle of the first target obstacle is blocked.
[0051] In some embodiments, the occlusion monitoring unit is further used to:
[0052] Acquire a first target obstacle that is not in the blind spot;
[0053] Projecting multiple vertices corresponding to the first target obstacle into a local coordinate system centered on the main vehicle to obtain coordinates of the local coordinate system;
[0054] Obtain the minimum bounding rectangle corresponding to the local coordinate system coordinates of each first target obstacle; the minimum bounding rectangle is formed by projecting the local coordinate system coordinates of multiple vertices on a two-dimensional surface and aligning with the coordinate axis;
[0055] For any two first target obstacles, determining whether corresponding minimum circumscribed rectangles overlap;
[0056] If they do not overlap, it is determined that any two first target obstacles are not blocked;
[0057] If there is overlap, the overlapping second target obstacle is obtained.
[0058] In some embodiments, the occlusion monitoring unit is further used to:
[0059] Obtain the minimum convex polygon corresponding to each second target obstacle;
[0060] Determine the fourth overlapping area of every two polygons in the minimum convex polygon corresponding to the second target obstacle;
[0061] Determine a third ratio of the fourth overlapping area to the polygonal area corresponding to the long-distance obstacle; the long-distance obstacle is the second target obstacle farthest from the main vehicle among the two second target obstacles for determining the second overlapping area;
[0062] It is determined whether a long-distance obstacle in the second target obstacle is blocked according to the third ratio.
[0063] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any one of the above methods by executing the executable instructions.
[0064] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any one of the above methods is implemented.
[0065] According to another aspect of the present disclosure, a computer program product is provided, including: a computer program or instructions, and when the computer program or instructions are executed by a processor, any one of the above methods is implemented.
[0066] The embodiments of the present disclosure provide a method, device, equipment, medium and program product for monitoring blind spots of visual sensors. The method includes: performing obstacle target detection based on visual sensors, determining at least one obstacle around the main vehicle and multiple vertices of each obstacle, projecting the multiple vertices into the coordinate system of the visual sensor, obtaining the sensor coordinates of the multiple vertices of each obstacle, and for any obstacle, projecting the sensor coordinates of each vertex onto a two-dimensional surface centered on the main vehicle to obtain a polygon, wherein the polygon is composed of the two-dimensional coordinates of multiple vertices on the two-dimensional surface; determining the field of view rectangle of the visual sensor, and determining whether any obstacle is in the blind spot of the sensor based on the polygon and the field of view rectangle. The 3D object is directly projected from the world coordinate system to a perspective view centered on the sensor, and then projected onto a two-dimensional plane. The blind spot of the sensor under this perspective is a simple 2D graphic. The complex 3D operation is converted into a 2D operation, which can easily determine whether the obstacle is in the visible area, reduce the difficulty of monitoring, improve the monitoring accuracy, simplify the calculation complexity, and reduce the data processing amount of the hardware.
[0067] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0068] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.
[0069] Figure 1 A flow chart of a visual sensor blind spot monitoring method according to an embodiment of the present disclosure is shown;
[0070] Figure 2 A flowchart of determining whether an obstacle is in a blind spot according to an embodiment of the present disclosure is shown;
[0071] Figure 3 One of the schematic diagrams showing a road traffic situation in an embodiment of the present disclosure;
[0072] Figure 4 A schematic diagram showing blind spot detection of a laser radar in an embodiment of the present disclosure is shown;
[0073] Figure 5 One of the flow charts for identifying mutual occlusion of obstacles in an embodiment of the present disclosure is shown;
[0074] Figure 6 A second flowchart of identifying mutual occlusion of obstacles in an embodiment of the present disclosure is shown;
[0075] Figure 7 A third flowchart of identifying mutual occlusion of obstacles in an embodiment of the present disclosure is shown;
[0076] Figure 8 A second schematic diagram showing a road traffic situation in an embodiment of the present disclosure;
[0077] Fig. 9 One of the schematic diagrams showing an obstacle projection occlusion relationship in an embodiment of the present disclosure;
[0078] Fig.10 A third schematic diagram showing a road traffic situation in an embodiment of the present disclosure;
[0079] Fig.11 A second schematic diagram showing an obstacle projection occlusion relationship in an embodiment of the present disclosure;
[0080] Fig.12 A flow chart for determining mutual occlusion of obstacles in an embodiment of the present disclosure is shown;
[0081] Fig.13 A fourth flowchart of identifying mutual occlusion of obstacles in an embodiment of the present disclosure is shown;
[0082] Fig.14 A schematic diagram of a blind spot monitoring device for a visual sensor in an embodiment of the present disclosure is shown;
[0083] Fig.15 A schematic structural diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0084] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0085] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.
[0086] For ease of understanding, before introducing the embodiments of the present disclosure, several terms involved in the embodiments of the present disclosure are first explained as follows:
[0087] FOV: Field of view, also known as field of view, is the range of the "observable world" that can be seen through vision or device at any given moment.
[0088] Due to considerations such as composition, installation and use, it is generally difficult to cover all spaces within a certain radius around the main vehicle. At the same time, due to mutual occlusion between obstacles, some obstacles are located in the blind spot of the visual sensor's field of view. The visual sensor cannot perceive the existence of the obstacle, affecting the safety of autonomous driving.
[0089] In the related art, the accuracy of monitoring radar blind spots is relatively rough. For example, the world coordinate system coordinates, the distance between the obstacle and the main vehicle, and the obstacle height information are used to perform blind spot monitoring. This method is only suitable for estimating the approximate blind spot range of the top radar. It is not applicable to blind spot radars with various installation methods, or multiple obstacles blocking each other, and other scenarios that require calculation of the degree of object blocking, etc., which require higher blind spot detection accuracy.
[0090] The disclosed embodiment provides a visual sensor blind spot monitoring method, device, equipment, medium and program product, the method comprising: performing obstacle target detection based on a visual sensor, determining at least one obstacle around the main vehicle and multiple vertices of each obstacle, projecting the multiple vertices into the coordinate system of the visual sensor, obtaining the sensor coordinates of the multiple vertices of each obstacle, and for any obstacle, projecting the sensor coordinates of each vertex onto a two-dimensional surface centered on the main vehicle to obtain a polygon; the polygon is composed of the two-dimensional coordinates of multiple vertices on the two-dimensional surface; determining the visual sensor's field of view rectangle; and determining whether any obstacle is in the sensor blind spot based on the polygon and the field of view rectangle. Projecting a 3D object from the world coordinate system to a perspective view centered on the sensor, and projecting it to a two-dimensional plane, the blind spot of the sensor under this perspective is a simple 2D graphic, converting complex 3D operations into 2D operations, and easily determining whether the obstacle is in the visible area, reducing the difficulty of monitoring, and improving the monitoring accuracy, simplifying the calculation complexity, and reducing the data processing amount of the hardware.
[0091] Given the position parameters determined during the installation of the known radar, and while meeting the real-time requirements of the autonomous driving vehicle, it is possible to accurately estimate whether the obstacle is in the sensor's field of view and to draw conclusions on whether the obstacle is in the sensor's blind spot.
[0092] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.
[0093] Figure 1 A flow chart of a blind spot monitoring method of a visual sensor in an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the visual sensor blind spot monitoring method provided in the embodiment of the present disclosure includes the following steps:
[0094] S102: Perform obstacle target detection based on the visual sensor to determine at least one obstacle around the main vehicle and multiple vertices of each obstacle.
[0095] The visual sensor may include: radar, laser radar, camera, etc., which are used for detection on the autonomous driving vehicle and assist the autonomous driving sensor. The plurality of vertices may be 8 vertices.
[0096] The method of determining multiple vertices of each obstacle may include: taking any obstacle as an example, obtaining a bounding box of the obstacle through a radar point cloud and a deep learning model, and after determining the bounding box corresponding to the obstacle, obtaining multiple vertices of the obstacle according to the geometric size of the obstacle.
[0097] In the autonomous driving scenario, obstacles are usually represented by boxes, or polygons with heights (i.e. bosses). That is to say, all objects can be roughly represented by a box to represent their geometric size, and then multiple vertices corresponding to the object can be obtained.
[0098] In a possible embodiment, before obstacle target detection is performed, a conversion relationship between parameters of a visual sensor and a map is obtained.
[0099] S104: Projecting the multiple vertices into the coordinate system of the visual sensor to obtain sensor coordinates of the multiple vertices of each obstacle.
[0100] In a possible embodiment, for any visual sensor, the coordinate transformation matrix can be used to project multiple vertices of each obstacle into the coordinate system of any visual sensor to obtain the sensor coordinates P of the multiple vertices of each obstacle. 1,i , i represents the number of vertices, and the sensor coordinates are three-dimensional coordinates.
[0101] S106: For any obstacle, project the sensor coordinates of each vertex onto a two-dimensional surface centered on the main vehicle to obtain a polygon, where the polygon is composed of two-dimensional coordinates of multiple vertices on the two-dimensional surface.
[0102] In a possible embodiment, for any obstacle, the sensor coordinates of each vertex are projected onto a two-dimensional sphere or a two-dimensional cylinder centered on the main vehicle to obtain a polygon.
[0103] Embodiment 1: Project the sensor coordinates of each vertex onto a two-dimensional spherical surface centered on the vehicle.
[0104] The sensor coordinates of each vertex of any obstacle can be projected onto a two-dimensional sphere with a radius of 1 centered on the main vehicle to obtain the two-dimensional spherical coordinates. A polygon is formed by the two-dimensional spherical coordinates of multiple vertices on the two-dimensional sphere.
[0105] Among them, the abscissa of the two-dimensional spherical coordinates is: The ordinate of the two-dimensional spherical coordinates is:
[0106] in, The horizontal coordinate representing the sensor coordinates; The ordinate represents the sensor coordinates; The vertical coordinate representing the sensor coordinates.
[0107] Embodiment 2: The sensor coordinates of each vertex are projected onto a two-dimensional cylinder centered on the vehicle.
[0108] The sensor coordinates of each vertex of any obstacle can be projected onto a two-dimensional cylinder with a radius of 1 centered on the main vehicle to obtain the two-dimensional cylindrical coordinates. A polygon is formed by the two-dimensional cylindrical coordinates of multiple vertices on the two-dimensional cylindrical surface.
[0109] Among them, the abscissa of the two-dimensional cylindrical coordinates is: The ordinate of the two-dimensional cylindrical coordinates is:
[0110] It should be noted that the disclosed embodiment does not limit the specific value of the radius used when projecting onto a two-dimensional surface. The radius of 1 is used in the above embodiment in order to remove the influence of depth (i.e., the distance from the main vehicle) and perform normalization processing to make the blind spot monitoring accuracy higher and the results more accurate.
[0111] Among them, polygons can include multiple types, and the specific shapes of polygons are not limited in the embodiments of the present disclosure.
[0112] The polygon can at least be: a minimum circumscribed rectangle composed of multiple vertices and aligned with the coordinate axes of the two-dimensional surface, or a minimum convex polygon.
[0113] The minimum convex polygon is a regular convex polygon composed of multiple vertices, and the minimum convex polygon does not include a minimum circumscribed rectangle aligned with the coordinate axes of the two-dimensional surface.
[0114] The polygon may also include a concave polygon. Compared with a concave polygon, using a convex polygon to calculate the overlapping area is simpler, has lower computational complexity, and can meet accuracy requirements.
[0115] S108: Determine the field of view rectangle of the visual sensor.
[0116] In a possible embodiment, the field of view rectangle may be defined by the transverse FOV angle and the longitudinal FOV angle of the visual sensor.
[0117] S110: Determine whether any obstacle is in the blind area of the sensor according to the polygon and the field of view rectangle.
[0118] In one possible embodiment, Figure 2 A flowchart of determining whether an obstacle is in a blind spot in an embodiment of the present disclosure is shown. Figure 2 As shown, the following steps are included:
[0119] S202: Determine a first overlapping area between the polygon and the field of view rectangle.
[0120] S204: Determine a first ratio of the first overlapping area to the polygon.
[0121] S206: Determine whether the first ratio is greater than a first preset threshold, if so, execute S208; if not, execute S210.
[0122] S208: Determine that any obstacle is not in the blind spot.
[0123] S210: Determine whether any obstacle is in the blind spot.
[0124] Through the above method, the 3D object is directly projected from the world coordinate system to a perspective view centered on the sensor, and then projected to a two-dimensional plane. The blind spot of the sensor under this perspective is a simple 2D graphic. By converting complex 3D operations into 2D operations, it is easy to determine whether the obstacle is in the visible area, thereby reducing the difficulty of monitoring, improving the monitoring accuracy, simplifying the calculation complexity, and reducing the data processing amount of the hardware.
[0125] Figure 3 A schematic diagram of a road traffic situation in an embodiment of the present disclosure is shown, such as Figure 3 As shown in the figure, the vehicle in the center is the main vehicle, and the surrounding boxes are other obstacle vehicles. The perspective in the figure is a bird's-eye view. Figure 3 It can be seen that the main vehicle is facing the lower right direction, trying to bypass the vehicle in front, and there is also a vehicle in front of the main vehicle. They are represented by obstacle 1 and obstacle 2 respectively, which is similar in the following embodiments.
[0126] Take the visual sensor as a laser radar, and project the sensor coordinates of multiple vertices onto a two-dimensional sphere as an example. Figure 4 A schematic diagram of blind spot detection of a laser radar in an embodiment of the present disclosure is shown. Figure 4 Take the detection angle of the left laser radar of the main vehicle as an example. Figure 4 There are two boxes in the figure. One is the visible area of the left laser radar, that is, the field of view rectangle of the left laser radar, and the other box is the vehicle projection in front of the main vehicle under the perspective view of the left laser radar, that is, the polygon formed by the vehicles in front of the main vehicle, which correspond to obstacles 1 and 2 respectively.
[0127] It can be seen that the first overlapping area between the field of view rectangle of the left laser radar and the polygon formed by the vehicle in front of the main vehicle can be used to determine whether the vehicle in front of the main vehicle is in the laser radar blind area. It should be noted that the horizontal coordinate direction of the image is opposite to the left and right.
[0128] In the process of identifying obstacles, in addition to determining whether the obstacle is in the blind spot of the sensor based on the field of view rectangle, there may be other situations. For example, the identified obstacle is confirmed not to be in the blind spot of the sensor. Although the first overlapping area with the field of view rectangle is greater than the first preset threshold, it is blocked by other obstacles and is in the occlusion blind spot.
[0129] In a possible implementation, after determining that an obstacle is in the blind area of the sensor according to the field of view rectangle in the above manner, further determination is made on the obstacle that is not in the blind area of the sensor.
[0130] Figure 5 A flowchart of identifying mutual occlusion of obstacles in an embodiment of the present disclosure is shown. Figure 5 As shown, the following steps are included:
[0131] S502: Acquire a first target obstacle that is not in the blind spot.
[0132] S504: Obtain the polygon corresponding to each first target obstacle.
[0133] S506: Determine, for any two first target obstacles among the multiple first target obstacles, a second overlapping area of the polygons of any two first target obstacles.
[0134] S508: Determine a second ratio of the second overlapping area to the polygonal area corresponding to the long-distance obstacle, where the long-distance obstacle is the first target obstacle that is farther from the host vehicle than the two first target obstacles for determining the second overlapping area.
[0135] S510: Determine whether the long-distance obstacle of the first target obstacle is in the blind spot according to the second ratio.
[0136] In a possible embodiment, if the number of detected obstacles is less than a preset number, the polygon is a minimum convex polygon, and if the number of detected obstacles is greater than or equal to the preset number, the polygon is a minimum circumscribed rectangle.
[0137] When there are few obstacles, the computing power requirement can meet the high-precision calculation method, and the minimum convex polygon of each obstacle can be directly used for calculation. If there are many obstacles, the minimum enclosing rectangle can be used for judgment first.
[0138] In a possible embodiment, taking projecting each vertex onto a two-dimensional cylinder centered on the main vehicle as an example, after obtaining the second ratio, if the second ratio is greater than a second preset threshold, it is determined that the long-distance obstacle is in the blind spot; if the second ratio is less than or equal to the second preset threshold, it is determined that the long-distance obstacle is not in the blind spot.
[0139] In the above manner, different types of polygons are used to calculate the second overlapping area, and when determining whether the final long-distance obstacle is in the blind spot, the minimum circumscribed rectangle is used to calculate whether it overlaps, which reduces the computational complexity and consumes less computing power. The minimum convex polygon is used to determine the second overlapping area with higher accuracy, and the blind spot monitoring result is more accurate. Based on this, the following implementation is shown in the embodiment of the present disclosure, which monitors whether obstacles block each other in a joint manner.
[0140] Figure 6 Another flow chart for identifying mutual occlusion of obstacles in an embodiment of the present disclosure is shown. Figure 6 As shown, the following steps are included:
[0141] S602: Acquire a first target obstacle that is not in the blind spot.
[0142] S604: Projecting multiple vertices corresponding to the first target obstacle into a local coordinate system centered on the main vehicle to obtain coordinates of the local coordinate system.
[0143] The local coordinate system coordinates are P 2,i express.
[0144] S606: Obtain a minimum bounding rectangle corresponding to the local coordinate system coordinates of each first target obstacle, where the minimum bounding rectangle is formed by projecting the local coordinate system coordinates of multiple vertices onto a two-dimensional surface and aligning with the coordinate axis.
[0145] S608: For any two first target obstacles among the first target obstacles, determine whether corresponding minimum circumscribed rectangles overlap.
[0146] S610: If there is no overlap, determine that any two first target obstacles are not blocked.
[0147] S612: If there is overlap, obtain the overlapping second target obstacle.
[0148] In the disclosed embodiment, whether there is an obstruction is determined by the above method. If there is no overlap, it means there is no obstruction. Considering the actual application of autonomous driving, the actual number of obstacles that obstruct each other is not large, and most obstacles will not overlap.
[0149] Based on this, for situations where there is overlap, Figure 7 FIG. 4 shows another flow chart of identifying mutual occlusion of obstacles in an embodiment of the present disclosure. Figure 7 As shown, the following steps are included:
[0150] S702: Obtain the minimum convex polygon corresponding to each second target obstacle.
[0151] S704: Determine a fourth overlapping area between every two polygons in the minimum convex polygon corresponding to the second target obstacle.
[0152] S706: Determine a third ratio of the fourth overlapping area to the polygonal area corresponding to the long-distance obstacle, where the long-distance obstacle is the second target obstacle far from the host vehicle among the two second target obstacles for determining the second overlapping area.
[0153] S708: Determine whether a long-distance obstacle in the second target obstacle is in the blind spot according to the third ratio.
[0154] First, the minimum circumscribed rectangle is used for judgment. If there is overlap, the minimum convex polygon is used to judge whether they block each other. This can minimize the number of obstacles that need to be calculated using the minimum convex polygon, improve the accuracy of judging whether obstacles block each other, and improve the accuracy of blind spot monitoring.
[0155] Further, Figure 8 A schematic diagram of a road traffic situation in an embodiment of the present disclosure is shown, such as Figure 8 As shown in the figure, the vehicle in the center is the main vehicle, and the surrounding boxes are other obstacle vehicles. The perspective in the figure is a bird's-eye view. Figure 8 It can be seen that obstacles 3 and 4.
[0156] Fig. 9 A schematic diagram showing an obstacle projection occlusion relationship in an embodiment of the present disclosure is shown. Fig. 9 As shown in the figure, on the cylinder centered on the main vehicle, the minimum circumscribed rectangle of the projections of the eight vertices of obstacles 3 and 4, the black box is obstacle 4.
[0157] Furthermore, as the road traffic conditions change, Fig.10 A third schematic diagram showing a road traffic situation in an embodiment of the present disclosure is shown. Fig.10 As shown in the figure, the vehicle in the center is the main vehicle, and the surrounding boxes are other obstacle vehicles. The perspective in the figure is a bird's-eye view. Fig.10 It can be seen that obstacles 5 and 6.
[0158] Fig.11 A schematic diagram showing an occlusion relationship between obstacle projections in an embodiment of the present disclosure is shown. Fig.11 As shown in FIG. 1 , the minimum circumscribed rectangle of the projections of the eight vertices of obstacles 5 and 6 on the cylinder centered on the main vehicle.
[0159] After S208, Fig.12 A flow chart of determining mutual occlusion of obstacles in an embodiment of the present disclosure is shown. Fig.12 As shown, the following steps are included:
[0160] S1202: Acquire a first target obstacle that is not in the blind spot.
[0161] S1204: Projecting multiple vertices corresponding to the first target obstacle into a local coordinate system centered on the main vehicle to obtain coordinates of the local coordinate system.
[0162] S1206: Obtain a minimum bounding rectangle corresponding to the local coordinate system coordinates of each first target obstacle, where the minimum bounding rectangle is formed by projecting the local coordinate system coordinates of multiple vertices onto a two-dimensional surface and aligning them with the coordinate axes.
[0163] S1208: For any two first target obstacles among the first target obstacles, determine whether the corresponding minimum circumscribed rectangles overlap, if so, execute S1212; if not, execute S1210.
[0164] S1210: Determine whether any two first target obstacles are not blocked.
[0165] S1212: Determine whether all obstacles have been identified. If so, execute S1214; if not, return to execute S1208.
[0166] S1214: Acquire the overlapping second target obstacle.
[0167] S1216: Obtain the minimum convex polygon corresponding to each second target obstacle.
[0168] S1218: Determine the fourth overlapping area of every two polygons in the minimum convex polygon corresponding to the second target obstacle.
[0169] S1220: Determine a third ratio of the fourth overlapping area to the polygonal area corresponding to the long-distance obstacle; the long-distance obstacle is the second target obstacle far from the host vehicle among the two second target obstacles for determining the second overlapping area.
[0170] S1222: Determine whether a long-distance obstacle in the second target obstacle is in the blind spot according to the third ratio.
[0171] In another possible implementation, whether obstacles block each other may be determined in the following manner: Fig.13 FIG. 4 shows another flow chart of identifying mutual occlusion of obstacles in an embodiment of the present disclosure. Fig.13 As shown, the following steps are included:
[0172] S1302: Acquire a first target obstacle that is not in the blind spot.
[0173] S1304: For any first target obstacle, obtain a target distance between any first target obstacle and the host vehicle.
[0174] S1306: Acquire other first target obstacles whose distance from the host vehicle is less than the target distance.
[0175] S1308: Determine a fifth overlapping area between polygons corresponding to other first target obstacles and a polygon corresponding to any one of the first target obstacles.
[0176] S1310: Determine a fourth ratio of the fifth overlapping area to a polygon corresponding to any one of the first target obstacles.
[0177] S1312: If the fourth ratio is greater than the third preset threshold, any one of the first target obstacles is blocked.
[0178] S1314: If the fourth ratio is less than or equal to the third preset threshold, any first target obstacle is not blocked.
[0179] In the above manner, during the judgment process, it is possible to avoid the situation where the overlap between obstacle A and obstacle B does not exceed the threshold, and obstacle A and obstacle C do not exceed the threshold. However, there is a situation where obstacles B and obstacle C jointly block obstacle A, and the overlapping areas of obstacles B and obstacle C with obstacle A exceed the set threshold, making the result of judging mutual blocking of obstacles more accurate and more reliable.
[0180] It should be noted that the acquisition, storage, use, and processing of data in the technical solution disclosed in this disclosure are in compliance with the relevant provisions of national laws and regulations, and various types of data such as personal identity data, operation data, behavioral data, etc. related to individuals, customers, and groups obtained in the embodiments of this disclosure have all been authorized.
[0181] Based on the same inventive concept, the present disclosure also provides a visual sensor blind spot monitoring device, such as the following embodiment. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.
[0182] Fig.14 A schematic diagram of a blind spot monitoring device for a visual sensor in an embodiment of the present disclosure is shown. Fig.14As shown, the device 1400 includes: a detection unit 1401, which is used to perform obstacle target detection based on a visual sensor, and determine the world coordinate system coordinates of at least one obstacle around the main vehicle and multiple vertices of each obstacle; a first determination unit 1402, which is used to project the world coordinate system coordinates into the coordinate system of the visual sensor to obtain the sensor coordinates of multiple vertices of each obstacle; a projection unit 1403, which is used to project the sensor coordinates of each vertex of any obstacle onto a two-dimensional surface centered on the main vehicle to obtain a polygon; the polygon is composed of two-dimensional coordinates of multiple vertices on the two-dimensional surface; a second determination unit 1404, which is used to determine the field of view rectangle of the visual sensor; a third determination unit 1405, which is used to determine whether any obstacle is in the sensor blind spot according to the polygon and the field of view rectangle.
[0183] It should be noted that the examples and application scenarios implemented by the modules in the above-mentioned device embodiment are the same as those of the corresponding steps in the method embodiment, but are not limited to the contents disclosed in the above-mentioned method embodiment. It should be noted that the above-mentioned modules as part of the device can be executed in a computer system such as a set of computer executable instructions.
[0184] Those skilled in the art will appreciate that various aspects of the present disclosure may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation that combines hardware and software aspects, which may be collectively referred to herein as a "circuit," "module," or "system."
[0185] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present disclosure, the electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above methods by executing the executable instructions. Since the principle of solving the problem in the electronic device embodiment is similar to that in the above method embodiment, the implementation of the electronic device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.
[0186] Refer to the following Fig.15 15 to describe the electronic device 1500 according to this embodiment of the present disclosure. Fig.15 The electronic device 1500 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0187] like Fig.15As shown, the electronic device 1500 is in the form of a general computing device. The components of the electronic device 1500 may include but are not limited to: at least one processing unit 1510, at least one storage unit 1520, and a bus 1530 connecting different system components (including the storage unit 1520 and the processing unit 1510).
[0188] The storage unit stores program codes, which can be executed by the processing unit 1510, so that the processing unit 1510 executes the steps of various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 1510 can execute the steps of the above method embodiment.
[0189] The storage unit 1520 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 15201 and / or a cache storage unit 15202 , and may further include a read-only storage unit (ROM) 15203 .
[0190] The storage unit 1520 may also include a program / utility 15204 having a set (at least one) of program modules 15205, such program modules 15205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0191] Bus 1530 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0192] The electronic device 1500 may also communicate with one or more external devices 1540 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1500, and / or communicate with any device that enables the electronic device 1500 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1550. Furthermore, the electronic device 1500 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1560. As shown, the network adapter 1560 communicates with other modules of the electronic device 1500 via a bus 1530. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0193] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.
[0194] Based on the same inventive concept, a computer-readable storage medium is also provided in the embodiment of the present disclosure, on which a computer program is stored, and when the computer program is executed by a processor, any of the above methods is implemented. Since the principle of solving the problem in the embodiment of the computer-readable storage medium is similar to that in the above method embodiment, the implementation of the embodiment of the computer-readable storage medium can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.
[0195] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0196] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0197] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.
[0198] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).
[0199] Based on the same inventive concept, a computer program product is also provided in the embodiments of the present disclosure, including a computer program product, including: a computer program or an instruction, and when the computer program or the instruction is executed by a processor, the method of any one of the above method embodiments is implemented. Since the principle of solving the problem in the computer program product embodiment is similar to that in the above method embodiment, the implementation of the computer program product embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.
[0200] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.
[0201] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.
[0202] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.
[0203] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.
Claims
1. A visual sensor blind spot monitoring method, characterized in that: The method comprises: Perform obstacle target detection based on a visual sensor to determine at least one obstacle around the main vehicle and multiple vertices of each obstacle; Projecting the multiple vertices into a coordinate system of a visual sensor to obtain sensor coordinates of the multiple vertices of each obstacle; For any obstacle, the sensor coordinates of each vertex are projected onto a two-dimensional surface centered on the main vehicle to obtain a polygon; the polygon is composed of the two-dimensional coordinates of the multiple vertices on the two-dimensional surface; Determining a field of view rectangle of the visual sensor; Determine whether any obstacle is in a blind spot of the sensor according to the polygon and the field of view rectangle.
2. The method according to claim 1, characterized in that The sensor coordinates of each vertex are projected onto a two-dimensional surface centered on the main vehicle to obtain a polygon, including: The sensor coordinates of each vertex are projected onto a two-dimensional sphere or a two-dimensional cylinder centered on the main vehicle to obtain the polygon.
3. The method according to claim 1, characterized in that The determining, based on the polygon and the field of view rectangle, whether any obstacle is in a blind spot includes: Determine a first overlapping area between the polygon and the field of view rectangle; determining a first ratio of the first overlapping area to the polygon; If the first ratio is greater than a first preset threshold, it is determined that any one of the obstacles is not in the blind spot; If the first ratio is less than or equal to the first preset threshold, it is determined that any one obstacle is in the blind spot.
4. The method according to claim 1, characterized in that: The method further comprises: Acquire a first target obstacle that is not in the blind spot; Obtaining a polygon corresponding to each of the first target obstacles; For any two of the first target obstacles among the plurality of the first target obstacles, determining a second overlapping area of the polygons of the any two first target obstacles; Determine a second ratio of the second overlapping area to the polygonal area corresponding to the long-distance obstacle; the long-distance obstacle is the first target obstacle far from the main vehicle among the two first target obstacles for determining the second overlapping area; According to the second ratio, it is determined whether the long-distance obstacle of the first target obstacle is blocked.
5. The method according to any one of claims 1 to 4, characterized in that: The polygon corresponding to the obstacle is at least: a minimum circumscribed rectangle composed of multiple vertices and aligned with the coordinate axes of the two-dimensional surface, or a minimum convex polygon.
6. The method according to claim 1, characterized in that The method further comprises: Acquire a first target obstacle that is not in the blind spot; Projecting multiple vertices corresponding to the first target obstacle into a local coordinate system centered on the main vehicle to obtain coordinates of the local coordinate system; Obtaining a minimum bounding rectangle corresponding to the local coordinate system coordinates of each of the first target obstacles; the minimum bounding rectangle is formed by projecting the local coordinate system coordinates of multiple vertices on a two-dimensional surface and aligning with the coordinate axis; For any two of the first target obstacles, determining whether corresponding minimum circumscribed rectangles overlap; If they do not overlap, it is determined that the two first target obstacles are not blocked; If there is overlap, the overlapping second target obstacle is obtained.
7. The method according to claim 6, characterized in that After acquiring the overlapping second target obstacle, the method further includes: Obtaining the minimum convex polygon corresponding to each of the second target obstacles; Determine a fourth overlapping area of every two of the minimum convex polygons corresponding to the second target obstacle; Determine a third ratio of the fourth overlapping area to the polygonal area corresponding to the long-distance obstacle; the long-distance obstacle is the second target obstacle far from the main vehicle among the two second target obstacles for determining the second overlapping area; According to the third ratio, it is determined whether the long-distance obstacle in the second target obstacle is blocked.
8. A visual sensor blind spot monitoring device, characterized in that: include: A detection unit, used to detect obstacles based on a visual sensor, and determine the world coordinate system coordinates of at least one obstacle around the main vehicle and multiple vertices of each obstacle; A first determining unit, used for projecting the coordinates of the world coordinate system into the coordinate system of the visual sensor to obtain the sensor coordinates of the multiple vertices of each obstacle; A projection unit, for projecting the sensor coordinates of each vertex of any obstacle onto a two-dimensional surface centered on the main vehicle to obtain a polygon; the polygon is composed of the two-dimensional coordinates of the multiple vertices on the two-dimensional surface; A second determining unit, used to determine the field of view rectangle of the visual sensor; The third determining unit is used to determine, for any obstacle, whether the any obstacle is in a blind spot of the sensor according to the polygon and the field of view rectangle.
9. An electronic device, characterized in that: include: processor; and a memory for storing executable instructions for the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
11. A computer program product comprising: A computer program or instruction, characterized in that when the computer program or instruction is executed by a processor, the method described in any one of claims 1 to 7 is implemented.