Driveable region fusion method and device based on vehicle-mounted multi-camera system

Abstract

The embodiment of the application provides a drivable area fusion method and device based on a vehicle-mounted multi-camera system, and belongs to the technical field of intelligent driving visual perception. The method comprises the following steps: acquiring boundary point coordinates of a drivable area of each peripheral camera in a vehicle-mounted multi-camera system in a self-vehicle coordinate system and position attributes of each peripheral camera; calculating angle information of all boundary points of each peripheral camera according to the boundary point coordinates of the drivable area; determining an overlapping area between adjacent peripheral cameras according to the angle information; adjusting the boundary of the drivable area in the peripheral camera according to the overlapping area and the position attributes of each peripheral camera according to a set fusion strategy, obtaining an adjusted boundary; and determining the drivable area around the self-vehicle according to the adjusted boundary. The method disclosed by the application can realize the fusion output of the drivable area through the peripheral camera surrounding the self-vehicle, and meets the requirements of fast calculation speed and high detection precision.

Description

Technical Field

[0001] This application relates to the field of intelligent driving visual perception technology, specifically to a drivable area fusion method based on an in-vehicle multi-camera system, a drivable area fusion device based on an in-vehicle multi-camera system, a machine-readable storage medium, and an intelligent driving processor. Background Technology

[0002] With the development of intelligent driving technology, the requirements for the accuracy of environmental perception are becoming increasingly stringent. Environmental perception is a prerequisite for safe driving in intelligent vehicles and a crucial component of the entire system, directly determining the level of intelligence in the decision-making and planning of autonomous vehicles. Environmental perception technology processes perception information output from various onboard sensors to acquire the intelligent vehicle's own state information, the state information of surrounding traffic participants, and traffic environment information, providing necessary data support for the intelligent vehicle's path planning and driving behavior decisions.

[0003] However, the current mainstream environmental perception fusion has the following problems: (1) The current mainstream environmental perception fusion mostly relies on the fusion of multiple sensor systems, which involves the alignment and calibration of cross-class sensors, resulting in slow calculation speed and low detection accuracy. (2) The current mainstream environmental perception fusion mostly uses surround-view fisheye cameras, which are only suitable for short-distance environmental perception such as parking environment, and are not suitable for high-speed driving environment. Summary of the Invention

[0004] The purpose of this application is to provide a drivable area fusion method and apparatus based on a vehicle-mounted multi-camera system. It does not rely on a multi-sensor system and can achieve fusion output of the drivable area using only a panoramic camera that surrounds the vehicle. It is applicable to high-speed driving environments and meets the requirements of fast calculation speed and high detection accuracy.

[0005] To achieve the above objectives, the first aspect of this application provides a drivable area fusion method based on an in-vehicle multi-camera system, the method comprising:

[0006] Obtain the coordinates of the boundary point of the drivable area of ​​each panoramic camera in the vehicle coordinate system and the position attributes of each panoramic camera.

[0007] Based on the coordinates of the boundary points of the drivable area of ​​each panoramic camera in the vehicle-mounted multi-camera system in the vehicle coordinate system, calculate the angle information of all boundary points of each panoramic camera in the vehicle coordinate system.

[0008] Based on the angle information of all boundary points of each panoramic camera in the vehicle coordinate system, the overlapping area between adjacent panoramic cameras is determined;

[0009] Based on the overlapping area between adjacent panoramic cameras and the position attributes of each panoramic camera, the boundary of the drivable area within each panoramic camera is adjusted according to the set fusion strategy to obtain the adjusted boundary.

[0010] The drivable area around the vehicle is determined based on the adjusted boundary.

[0011] In this embodiment of the application, before obtaining the coordinates of the drivable area boundary point of each panoramic camera in the vehicle-mounted multi-camera system in the vehicle coordinate system, the method further includes:

[0012] Images of the vehicle's surroundings at the same time are acquired by multiple panoramic cameras in an onboard multi-camera system;

[0013] Semantic segmentation results are obtained by using model inference on images of the vehicle's surroundings at the same time.

[0014] The semantic segmentation results are analyzed by connected component analysis to obtain the boundary points of the drivable region in the image coordinate system.

[0015] The boundary points of the drivable area in the image coordinate system are converted into the coordinates of the boundary points of the drivable area in the vehicle coordinate system.

[0016] In this embodiment of the application, the step of calculating the angle information of all boundary points of each panoramic camera in the vehicle coordinate system based on the coordinates of the boundary points of the drivable area of ​​each panoramic camera in the vehicle coordinate system includes:

[0017] Based on the coordinates of the drivable area boundary points of each panoramic camera in the vehicle coordinate system in the vehicle multi-camera system, the angle information of all boundary points of each panoramic camera in the vehicle coordinate system is calculated by using the arctangent function.

[0018] In this embodiment of the application, determining the overlap region between adjacent panoramic cameras based on the angle information of all boundary points of each panoramic camera in the vehicle coordinate system includes:

[0019] The field of view of each panoramic camera is determined based on the angle information of all boundary points of each panoramic camera in the vehicle coordinate system.

[0020] Based on the field of view of each panoramic camera, the overlapping area between adjacent panoramic cameras is determined.

[0021] In this embodiment of the application, the step of adjusting the boundary of the drivable area within each panoramic camera according to a set fusion strategy based on the overlapping area between adjacent panoramic cameras and the position attributes of each panoramic camera to obtain the adjusted boundary includes:

[0022] When there is a front-view camera or a rear-view camera among two adjacent panoramic cameras, the field of view of the side-view camera is adjusted based on the overlapping area of ​​the two adjacent panoramic cameras and the field of view of the front-view camera or the rear-view camera as a reference to filter out duplicate points and obtain the boundary point of the drivable area after the adjustment of the field of view of the side-view camera.

[0023] In this embodiment of the application, the step of adjusting the boundary of the drivable area within each panoramic camera according to a set fusion strategy based on the overlapping area between adjacent panoramic cameras and the position attributes of each panoramic camera to obtain the adjusted boundary includes:

[0024] When both adjacent panoramic cameras are side-view cameras, based on the overlapping area of ​​the two adjacent panoramic cameras, the field of view boundary of the drivable area of ​​the two side-view cameras is adjusted according to half of the overlapping area to filter out duplicate points, and the boundary point of the drivable area after the adjustment of the field of view of the side-view cameras is obtained.

[0025] In this embodiment of the application, determining the drivable area around the vehicle based on the adjusted boundary includes:

[0026] The adjusted field of view is determined based on the adjusted boundary.

[0027] Based on the adjusted field of view, boundary points outside the field of view of the corresponding panoramic camera's drivable area are filtered out to obtain the filtered boundary points.

[0028] The filtered boundary points are then fused to obtain the boundary points of the drivable area surrounding the vehicle in the vehicle coordinate system.

[0029] Linear interpolation is used to smooth the boundary points of the drivable area around the vehicle in the vehicle coordinate system, thereby obtaining the smoothed boundary points of the drivable area around the vehicle in the vehicle coordinate system.

[0030] A second aspect of this application provides a drivable area fusion device based on a vehicle-mounted multi-camera system, the device comprising:

[0031] The acquisition module is used to acquire the coordinates of the boundary point of the drivable area of ​​each panoramic camera in the vehicle coordinate system and the position attributes of each panoramic camera.

[0032] The calculation module is used to calculate the angle information of all boundary points of each panoramic camera in the vehicle coordinate system based on the coordinates of the boundary points of the drivable area of ​​each panoramic camera in the vehicle coordinate system.

[0033] The first determining module is used to determine the overlapping area between adjacent panoramic cameras based on the angle information of all boundary points of each panoramic camera in the vehicle coordinate system.

[0034] The adjustment module is used to adjust the boundary of the drivable area within each panoramic camera according to a set fusion strategy based on the overlapping area between the adjacent panoramic cameras and the position attributes of each panoramic camera, so as to obtain the adjusted boundary.

[0035] The second determining module is used to determine the drivable area around the vehicle based on the adjusted boundary.

[0036] A third aspect of this application provides an intelligent driving processor configured to execute the aforementioned drivable area fusion method based on an onboard multi-camera system.

[0037] A fourth aspect of this application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the aforementioned drivable area fusion method based on an in-vehicle multi-camera system.

[0038] Compared with the prior art, the above-mentioned technical solution of this application has the following beneficial effects:

[0039] (1) This application calculates the angles of all boundary points of each panoramic camera in the vehicle coordinate system based on the coordinates of the boundary points of the drivable area of ​​each panoramic camera in the vehicle coordinate system. Then, it determines the overlapping area between adjacent panoramic cameras based on the angles of all boundary points of each panoramic camera in the vehicle coordinate system. Finally, it adjusts the boundary of the drivable area according to the set fusion strategy based on the overlapping area between adjacent panoramic cameras and the position attributes of each panoramic camera, thereby achieving accurate perception of the drivable area information around the vehicle. This application does not rely on the fusion of multiple sensor systems, thus avoiding the alignment and calibration of cross-class sensors, and has the advantages of fast calculation speed and high detection accuracy.

[0040] (2) The method described in this application can achieve accurate perception of the drivable area information around the vehicle by using a panoramic camera around the vehicle, which meets the requirements of fast calculation speed and high detection accuracy. It is applicable to high-speed driving environment and provides a better driving assistance experience.

[0041] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0042] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:

[0043] Figure 1 The schematic diagram illustrates a flow chart of a drivable area fusion method based on an in-vehicle multi-camera system according to an embodiment of this application;

[0044] Figure 2 A schematic diagram of a vehicle coordinate system according to an embodiment of this application is shown;

[0045] Figure 3 An illustrative view of the overlapping drivable area according to an embodiment of this application is shown.

[0046] Figure 4 This illustration schematically shows a fusion diagram of the front-view camera and the left front-view camera according to an embodiment of this application;

[0047] Figure 5 This illustration schematically shows a fusion diagram of the left front-view camera and the left rear-view camera according to an embodiment of this application;

[0048] Figure 6 This illustration schematically shows a left rear-view camera and a rear-view camera fusion diagram according to an embodiment of this application;

[0049] Figure 7 This illustration schematically shows an overall process diagram of a drivable area fusion method based on an in-vehicle multi-camera system according to an embodiment of this application;

[0050] Figure 8 The diagram illustrates a structural block diagram of a drivable area fusion device based on an in-vehicle multi-camera system according to an embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0052] It should be noted that if the embodiments of this application involve directional indications (such as up, down, left, right, front, back, etc.), these directional indications are only used to explain the relative positional relationships and movement of the components in a specific posture. If the specific posture changes, the directional indications will also change accordingly. Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" can explicitly or implicitly include at least one of those features. The technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0053] The drivable area refers to the area around an autonomous vehicle that it can drive in, that is, the area that can be planned and controlled. The algorithm for determining the drivable area is one of the important perception algorithms in the field of autonomous driving, playing a crucial "safety net" role in assisted driving or autonomous driving. Panoramic cameras typically have a field of view of 60°–120° and are mainly used to perceive scenes at a greater distance, suitable for driving scenarios. This application uses six panoramic cameras to perceive the environment around the vehicle while driving. After detecting the drivable area using semantic segmentation and connected component analysis algorithms, a multi-camera drivable area fusion strategy is formulated to achieve accurate perception of the drivable area information around the vehicle.

[0054] The drivable area fusion method based on an in-vehicle multi-camera system provided in this application can be applied to the driving environment of an in-vehicle multi-camera system. The in-vehicle multi-camera system consists of six panoramic cameras: front-view, rear-view, left front-view, left rear-view, right front-view, and right rear-view.

[0055] Figure 1 The illustration schematically shows a flowchart of a drivable area fusion method based on an in-vehicle multi-camera system according to an embodiment of this application. For example... Figure 1 As shown in one embodiment of this application, a drivable area fusion method based on an in-vehicle multi-camera system is provided. This embodiment mainly illustrates the application of this method to an in-vehicle multi-camera system, including the following steps:

[0056] Step 110: Obtain the coordinates of the drivable area boundary point of each panoramic camera in the vehicle coordinate system and the position attributes of each panoramic camera.

[0057] In this embodiment, the coordinates of the drivable area boundary points in the vehicle coordinate system of the panoramic camera can be obtained through semantic segmentation results. For example, multiple panoramic images of the vehicle's surroundings at the same time can be acquired using multiple panoramic cameras, and then semantic segmentation processing can be performed on these multiple panoramic images to obtain the coordinates of the drivable area boundary points in the vehicle coordinate system of the panoramic camera.

[0058] In this embodiment, the position attribute of the panoramic camera refers to its installation location. For example, a panoramic camera installed at the front of the vehicle (front end) is a front-view camera. A panoramic camera installed at the rear of the vehicle (rear end) is a rear-view camera. In this embodiment, there are two panoramic cameras installed on the right side of the vehicle: a right front-view camera and a right rear-view camera. There are also two panoramic cameras installed on the left side of the vehicle: a left front-view camera and a left rear-view camera. Figure 2 A schematic diagram of a vehicle coordinate system according to an embodiment of this application is shown. Figure 2 As shown, the vehicle-mounted multi-camera system in this embodiment consists of six panoramic cameras: a front-view camera, a rear-view camera, a left front-view camera, a left rear-view camera, a right front-view camera, and a right rear-view camera. This embodiment acquires six panoramic images of the vehicle's surroundings at the same time using these six cameras. Semantic segmentation processing is then performed on each of these six panoramic images to obtain the coordinates of the drivable area boundary points in the vehicle's coordinate system.

[0059] Step 120: Based on the coordinates of the boundary points of the drivable area of ​​each panoramic camera in the vehicle-mounted multi-camera system in the vehicle coordinate system, calculate the angle information of all boundary points of each panoramic camera in the vehicle coordinate system.

[0060] Figure 2 A schematic diagram of a vehicle coordinate system according to an embodiment of this application is shown. In this embodiment, the angle information refers to the angle α between the boundary point P(x,y) and the positive x-axis in the vehicle coordinate system, such as... Figure 2 As shown. The angle α between the boundary point and the positive x-axis is calculated using the arctangent function based on the x and y values ​​of the boundary point coordinates in the vehicle coordinate system of each panoramic camera. To consider the signs of x and y, the original boundary point is projected onto the range (0, 360). Since the coordinates of the drivable area boundary points are arranged sequentially from the image's horizontal coordinate, the angles of each boundary point are also arranged in order. The angle is calculated by taking the first and last five points, removing the maximum and minimum values, and then averaging the remaining three points as the edge point of the drivable area boundary within the camera's field of view.

[0061] Step 130: Determine the overlapping area between adjacent panoramic cameras based on the angle information of all boundary points of each panoramic camera in the vehicle coordinate system.

[0062] In this embodiment, the viewing angle range of each panoramic camera is determined by the angle information of all boundary points of each panoramic camera in the vehicle coordinate system obtained in the previous step, thereby determining the overlapping area of ​​two adjacent panoramic cameras in the vehicle coordinate system. Figure 3 An illustrative view of the overlapping drivable area according to an embodiment of this application is shown. Figure 3 As shown, the panoramic camera inputs six images, resulting in six overlapping areas for different scenarios: front-view camera and left front-view camera, left front-view camera and left rear-view camera, left rear-view camera and rear-view camera, front-view camera and right front-view camera, right front-view camera and right rear-view camera, and right rear-view camera and rear-view camera. For example, the angle of each boundary point within the drivable area of ​​a single camera is first calculated. Then, the edge of the drivable area boundary point of a single camera is determined based on the angle of each boundary point. Finally, the camera overlapping area is determined using the edge of the drivable area boundary point of a single camera.

[0063] Step 140: Based on the overlapping area between the adjacent panoramic cameras and the position attributes of each panoramic camera, adjust the boundary of the drivable area within each panoramic camera according to the set fusion strategy to obtain the adjusted boundary.

[0064] In this embodiment, when there is a front-view camera or a rear-view camera among two adjacent panoramic cameras, since the field of view of the front-view camera or the rear-view camera is larger than that of the side-view camera, after calculating the overlapping area, the field of view of the side-view camera is adjusted based on the field of view of the front-view camera or the rear-view camera to filter out duplicate points.

[0065] Figure 4 A schematic diagram illustrating the fusion of the front-view camera and the left front-view camera according to an embodiment of this application is shown. Figure 4 As shown in the figure, the solid line represents the field of view of the forward-looking camera, and the dashed line represents the field of view of the left forward-looking camera. The field of view of the forward-looking camera includes the left and right boundaries. The right boundary is located in the first quadrant, and the left boundary is located in the second quadrant. The area where the field of view of the forward-looking camera overlaps with that of the left forward-looking camera is called the overlapping area. After calculating the angle of the overlapping area, the original left front boundary of the left forward-looking camera's field of view is adjusted to the left boundary position of the forward-looking camera, using the field of view of the forward-looking camera as a reference, and duplicate points within the overlapping area are removed.

[0066] In this embodiment, the side-view cameras include a left front-view camera, a left rear-view camera, a right front-view camera, and a right rear-view camera. When two adjacent panoramic cameras are both side-view cameras, after calculating the overlapping area, the edges of each camera are readjusted by half the size of the overlapping area. For example, when one of the two adjacent panoramic cameras is a left front-view camera and the other is a left rear-view camera, after calculating the overlapping area, the field of view boundaries of the drivable areas of the left front-view camera and the left rear-view camera are adjusted by half the overlapping area to filter out duplicate points.

[0067] Figure 5 A schematic diagram illustrating the fusion of the left front-view camera and the left rear-view camera according to an embodiment of this application is shown. Figure 5 As shown in the figure, the solid line represents the field of view of the left front-view camera, and the dashed line represents the field of view of the left rear-view camera. After calculating the angle of the overlapping area, half of the angle of the overlapping area is subtracted from the lower boundary of the left front-view camera, and half of the angle of the overlapping area is added to the upper boundary of the left rear-view camera. Duplicate points within the overlapping area are then removed.

[0068] Figure 6 A schematic diagram illustrating the fusion of the left rear-view camera and the rear-view camera according to an embodiment of this application is shown. Figure 6 As shown in the figure, the solid line represents the field of view of the rear-view camera, and the dashed line represents the field of view of the left rear-view camera. After calculating the angle of the overlapping area, the original left rear boundary of the field of view of the left rear-view camera is adjusted to the left boundary position of the rear-view camera, based on the field of view of the rear-view camera, and duplicate points in the overlapping area are removed.

[0069] Step 150: Determine the drivable area around the vehicle based on the adjusted boundary.

[0070] In this embodiment, the adjusted field of view is determined based on the adjusted boundary; boundary points outside the field of view of the corresponding panoramic camera's drivable area are filtered out based on the adjusted field of view to obtain filtered boundary points; the filtered boundary points are fused to obtain the boundary points of the drivable area surrounding the vehicle in the vehicle coordinate system; and the boundary points of the drivable area surrounding the vehicle in the vehicle coordinate system are smoothed using linear interpolation to obtain the smoothed boundary points of the drivable area surrounding the vehicle in the vehicle coordinate system.

[0071] Figure 1 This is a flowchart illustrating a drivable area fusion method based on a vehicle-mounted multi-camera system in one embodiment. It should be understood that, although... Figure 1The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0072] In one embodiment, Figure 7 The illustration schematically shows an overall flow diagram of a drivable area fusion method based on an in-vehicle multi-camera system according to an embodiment of this application. For example... Figure 7 As shown in one embodiment of this application, a drivable area fusion method based on an in-vehicle multi-camera system is provided, mainly including: calculating the coordinates of the drivable area boundary points in the vehicle coordinate system of the panoramic camera using semantic segmentation result data; calculating the angle of each boundary point of the drivable area using the arctangent function based on the coordinates of the boundary points; traversing the cameras and calculating the overlapping area of ​​the drivable areas of two adjacent panoramic cameras according to the angle of the boundary points; for cameras with overlapping areas, readjusting the boundary of the drivable area according to the fusion strategy; traversing the original points and filtering out points other than the boundary points based on the adjusted boundary; and performing linear interpolation on the fused points.

[0073] In one embodiment, such as Figure 8 As shown, a drivable area fusion device based on a vehicle-mounted multi-camera system is provided, including an acquisition module 210, a calculation module 220, a first determination module 230, an adjustment module 240, and a second determination module 250, wherein:

[0074] The acquisition module 210 is used to acquire the coordinates of the boundary point of the drivable area of ​​each panoramic camera in the vehicle coordinate system and the position attributes of each panoramic camera.

[0075] The calculation module 220 is used to calculate the angle information of all boundary points of each panoramic camera in the vehicle coordinate system based on the coordinates of the boundary points of the drivable area of ​​each panoramic camera in the vehicle coordinate system.

[0076] The first determining module 230 is used to determine the overlapping area between adjacent panoramic cameras based on the angle information of all boundary points of each panoramic camera in the vehicle coordinate system.

[0077] The adjustment module 240 is used to adjust the boundary of the drivable area within each panoramic camera according to a set fusion strategy based on the overlapping area between the adjacent panoramic cameras and the position attributes of each panoramic camera, so as to obtain the adjusted boundary.

[0078] The second determining module 250 is used to determine the drivable area around the vehicle based on the adjusted boundary.

[0079] The drivable area fusion device based on the vehicle-mounted multi-camera system includes a processor and a memory. The aforementioned acquisition module 210, calculation module 220, first determination module 230, adjustment module 240, and second determination module 250 are all stored in the memory as program units. The processor executes the aforementioned program modules stored in the memory to implement the corresponding functions.

[0080] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and the drivable area fusion method based on an onboard multi-camera system can be implemented by adjusting kernel parameters.

[0081] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0082] This application provides a storage medium storing a program that, when executed by a processor, implements the above-described drivable area fusion method based on an in-vehicle multi-camera system.

[0083] In one embodiment, the drivable area fusion device based on an in-vehicle multi-camera system provided in this application can be implemented as a computer program, which can run on an in-vehicle computer device. The memory of the computer device can store the various program modules that make up the drivable area fusion device based on the in-vehicle multi-camera system, for example... Figure 8 The diagram shows an acquisition module 210, a calculation module 220, a first determination module 230, an adjustment module 240, and a second determination module 250. The computer program, comprised of these modules, causes the processor to execute the steps in the drivable area fusion method based on a vehicle-mounted multi-camera system described in the various embodiments of this application. The computer device can, as shown in the diagram... Figure 8The acquisition module 210 in the drivable area fusion device based on the vehicle-mounted multi-camera system shown executes step 110. The computer device can execute step 120 via the calculation module 220. The computer device can execute step 130 via the first determination module 230. The computer device can execute step 140 via the adjustment module 240. The computer device can execute step 150 via the second determination module 250.

[0084] This application provides a device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps:

[0085] Step 110: Obtain the coordinates of the drivable area boundary point of each panoramic camera in the vehicle coordinate system and the position attributes of each panoramic camera.

[0086] Step 120: Based on the coordinates of the boundary points of the drivable area of ​​each panoramic camera in the vehicle-mounted multi-camera system in the vehicle coordinate system, calculate the angle information of all boundary points of each panoramic camera in the vehicle coordinate system.

[0087] Step 130: Determine the overlapping area between adjacent panoramic cameras based on the angle information of all boundary points of each panoramic camera in the vehicle coordinate system.

[0088] Step 140: Based on the overlapping area between the adjacent panoramic cameras and the position attributes of each panoramic camera, adjust the boundary of the drivable area within each panoramic camera according to the set fusion strategy to obtain the adjusted boundary.

[0089] Step 150: Determine the drivable area around the vehicle based on the adjusted boundary.

[0090] In one embodiment, before obtaining the coordinates of the drivable area boundary points of each panoramic camera in the vehicle coordinate system in the vehicle multi-camera system, the method further includes:

[0091] Images of the vehicle's surroundings at the same time are acquired by multiple panoramic cameras in an onboard multi-camera system;

[0092] Semantic segmentation results are obtained by using model inference on images of the vehicle's surroundings at the same time.

[0093] The semantic segmentation results are analyzed by connected component analysis to obtain the boundary points of the drivable region in the image coordinate system.

[0094] The boundary points of the drivable area in the image coordinate system are converted into the coordinates of the boundary points of the drivable area in the vehicle coordinate system.

[0095] In one embodiment, calculating the angle information of all boundary points of each panoramic camera in the vehicle coordinate system based on the coordinates of the drivable area boundary points of each panoramic camera in the vehicle coordinate system includes:

[0096] Based on the coordinates of the drivable area boundary points of each panoramic camera in the vehicle coordinate system in the vehicle multi-camera system, the angle information of all boundary points of each panoramic camera in the vehicle coordinate system is calculated by using the arctangent function.

[0097] In one embodiment, determining the overlap region between adjacent panoramic cameras based on the angle information of all boundary points of each panoramic camera in the vehicle coordinate system includes:

[0098] The field of view of each panoramic camera is determined based on the angle information of all boundary points of each panoramic camera in the vehicle coordinate system.

[0099] Based on the field of view of each panoramic camera, the overlapping area between adjacent panoramic cameras is determined.

[0100] In one embodiment, adjusting the boundary of the drivable area within each panoramic camera according to a set fusion strategy based on the overlapping area between adjacent panoramic cameras and the position attributes of each panoramic camera to obtain the adjusted boundary includes:

[0101] When there is a front-view camera or a rear-view camera among two adjacent panoramic cameras, the field of view of the side-view camera is adjusted based on the overlapping area of ​​the two adjacent panoramic cameras and the field of view of the front-view camera or the rear-view camera as a reference to filter out duplicate points and obtain the boundary point of the drivable area after the adjustment of the field of view of the side-view camera.

[0102] In one embodiment, adjusting the boundary of the drivable area within each panoramic camera according to a set fusion strategy based on the overlapping area between adjacent panoramic cameras and the position attributes of each panoramic camera to obtain the adjusted boundary includes:

[0103] When both adjacent panoramic cameras are side-view cameras, based on the overlapping area of ​​the two adjacent panoramic cameras, the field of view boundary of the drivable area of ​​the two side-view cameras is adjusted according to half of the overlapping area to filter out duplicate points, and the boundary point of the drivable area after the adjustment of the field of view of the side-view cameras is obtained.

[0104] In one embodiment, determining the drivable area around the vehicle based on the adjusted boundary includes:

[0105] The adjusted field of view is determined based on the adjusted boundary.

[0106] Based on the adjusted field of view, boundary points outside the field of view of the corresponding panoramic camera's drivable area are filtered out to obtain the filtered boundary points.

[0107] The filtered boundary points are then fused to obtain the boundary points of the drivable area surrounding the vehicle in the vehicle coordinate system.

[0108] Linear interpolation is used to smooth the boundary points of the drivable area around the vehicle in the vehicle coordinate system, thereby obtaining the smoothed boundary points of the drivable area around the vehicle in the vehicle coordinate system.

[0109] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0111] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0113] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0114] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application.

Claims

1. A drivable area fusion method based on a vehicle-mounted multi-camera system, characterized in that, The method includes: Obtain the coordinates of the drivable area boundary point of each panoramic camera in the vehicle coordinate system and the position attributes of each panoramic camera; Based on the coordinates of the boundary points of the drivable area of ​​each panoramic camera in the vehicle-mounted multi-camera system in the vehicle coordinate system, calculate the angle information of all boundary points of each panoramic camera in the vehicle coordinate system. The angle information refers to the angle between the boundary point and the positive x-axis in the vehicle coordinate system. Based on the angle information of all boundary points of each panoramic camera in the vehicle coordinate system, the overlapping area between adjacent panoramic cameras is determined; Based on the overlapping area between adjacent panoramic cameras and the position attributes of each panoramic camera, the boundary of the drivable area within each panoramic camera is adjusted according to the set fusion strategy to obtain the adjusted boundary. The drivable area around the vehicle is determined based on the adjusted boundary, including: The adjusted field of view is determined based on the adjusted boundary. Based on the adjusted field of view, boundary points outside the field of view of the corresponding panoramic camera's drivable area are filtered out to obtain the filtered boundary points. The filtered boundary points are then fused to obtain the boundary points of the drivable area surrounding the vehicle in the vehicle coordinate system. Linear interpolation is used to smooth the boundary points of the drivable area around the vehicle in the vehicle coordinate system, thereby obtaining the smoothed boundary points of the drivable area around the vehicle in the vehicle coordinate system.

2. The drivable area fusion method based on a vehicle-mounted multi-camera system according to claim 1, characterized in that, The step of adjusting the boundary of the drivable area within each panoramic camera according to a set fusion strategy based on the overlapping area between adjacent panoramic cameras and the position attributes of each panoramic camera, to obtain the adjusted boundary, includes: When there is a front-view camera or a rear-view camera among two adjacent panoramic cameras, the field of view of the side-view camera is adjusted based on the overlapping area of ​​the two adjacent panoramic cameras and the field of view of the front-view camera or the rear-view camera as a reference to filter out duplicate points and obtain the boundary point of the drivable area after the adjustment of the field of view of the side-view camera.

3. The drivable area fusion method based on a vehicle-mounted multi-camera system according to claim 1, characterized in that, The step of adjusting the boundary of the drivable area within each panoramic camera according to a set fusion strategy based on the overlapping area between adjacent panoramic cameras and the position attributes of each panoramic camera, to obtain the adjusted boundary, includes: When both adjacent panoramic cameras are side-view cameras, based on the overlapping area of ​​the two adjacent panoramic cameras, the field of view boundary of the drivable area of ​​the two side-view cameras is adjusted according to half of the overlapping area to filter out duplicate points, and the boundary point of the drivable area after the adjustment of the field of view of the side-view cameras is obtained.

4. The drivable area fusion method based on a vehicle-mounted multi-camera system according to claim 1, characterized in that, Before acquiring the coordinates of the drivable area boundary points of each panoramic camera in the vehicle coordinate system in the vehicle multi-camera system, the method further includes: Images of the vehicle's surroundings at the same time are acquired by multiple panoramic cameras in an onboard multi-camera system; Semantic segmentation results are obtained by using model inference on images of the vehicle's surroundings at the same time. The semantic segmentation results are analyzed by connected component analysis to obtain the boundary points of the drivable region in the image coordinate system. The boundary points of the drivable area in the image coordinate system are converted into the coordinates of the boundary points of the drivable area in the vehicle coordinate system.

5. The drivable area fusion method based on a vehicle-mounted multi-camera system according to claim 1, characterized in that, The step of calculating the angle information of all boundary points of each panoramic camera in the vehicle coordinate system based on the coordinates of the drivable area boundary points of each panoramic camera in the vehicle coordinate system includes: Based on the coordinates of the drivable area boundary points of each panoramic camera in the vehicle coordinate system in the vehicle multi-camera system, the angle information of all boundary points of each panoramic camera in the vehicle coordinate system is calculated by using the arctangent function.

6. The drivable area fusion method based on a vehicle-mounted multi-camera system according to claim 1, characterized in that, The step of determining the overlap area between adjacent panoramic cameras based on the angle information of all boundary points of each panoramic camera in the vehicle coordinate system includes: The field of view of each panoramic camera is determined based on the angle information of all boundary points of each panoramic camera in the vehicle coordinate system. Based on the field of view of each panoramic camera, the overlapping area between adjacent panoramic cameras is determined.

7. A drivable area fusion device based on a vehicle-mounted multi-camera system, characterized in that, The device includes: The acquisition module is used to acquire the coordinates of the boundary point of the drivable area of ​​each panoramic camera in the vehicle coordinate system and the position attributes of each panoramic camera. The calculation module is used to calculate the angle information of all boundary points of each panoramic camera in the vehicle coordinate system based on the coordinates of the boundary points of the drivable area of ​​each panoramic camera in the vehicle coordinate system. The angle information refers to the angle between the boundary point and the positive x-axis in the vehicle coordinate system. The first determining module is used to determine the overlapping area between adjacent panoramic cameras based on the angle information of all boundary points of each panoramic camera in the vehicle coordinate system. The adjustment module is used to adjust the boundary of the drivable area within each panoramic camera according to a set fusion strategy based on the overlapping area between the adjacent panoramic cameras and the position attributes of each panoramic camera, so as to obtain the adjusted boundary. The second determining module is used to determine the drivable area around the vehicle based on the adjusted boundary, including: The adjusted field of view is determined based on the adjusted boundary. Based on the adjusted field of view, boundary points outside the field of view of the corresponding panoramic camera's drivable area are filtered out to obtain the filtered boundary points. The filtered boundary points are then fused to obtain the boundary points of the drivable area surrounding the vehicle in the vehicle coordinate system. Linear interpolation is used to smooth the boundary points of the drivable area around the vehicle in the vehicle coordinate system, thereby obtaining the smoothed boundary points of the drivable area around the vehicle in the vehicle coordinate system.

8. A machine-readable storage medium storing instructions thereon, characterized in that, When executed by a processor, this instruction causes the processor to be configured to perform the drivable area fusion method based on a vehicle-mounted multi-camera system according to any one of claims 1 to 6.

Images