Driving area detection method and device, storage medium and electronic equipment

Abstract

The invention discloses a detection method and device for a drivable area, a storage medium and electronic equipment. The method comprises the steps that a multi-view environment image collected by a vehicle is determined; processing the multi-view environment image to obtain a sight line occupancy state and an occupancy point distance corresponding to the preset sight line; according to the sight line occupancy state and the occupancy point distance, determining position information of an occupancy point in a world coordinate system; and determining a drivable area of the vehicle according to the position information of the occupied point. According to the method, the occupancy state of each preset sight line and the image distance of the occupancy point closest to the original point in the grid points on the sight line in the occupancy state are determined in the aerial view through feature transformation, and the drivable area of the vehicle is determined based on the occupancy state, so that the model calculation amount can be reduced, redundant information is greatly reduced, and the vehicle driving efficiency is improved. The accuracy of the occupancy detection result is improved, and the driving area can be determined quickly and accurately.

Description

Technical Field

[0001] The present disclosure relates to the field of intelligent driving technology, and in particular to a method, device, storage medium and electronic device for detecting a drivable area. Background Art

[0002] With the rapid development of intelligent driving technology, people have higher and higher requirements for the safety and stability of intelligent driving vehicles. The accuracy of detecting the drivable area is crucial to ensure the safe and stable driving of intelligent driving vehicles. Most of the existing methods for detecting drivable areas perform semantic segmentation on a bird's-eye view, assigning a semantic category to each spatial coordinate position as the basis for whether it is drivable. It has the advantages of intuitive output form and simple model structure, but this solution has high requirements on the capacity of the network model, many redundant features, and low accuracy. Summary of the invention

[0003] In order to solve the above technical problems, the present disclosure provides a method, device, storage medium and electronic device for detecting a drivable area, so as to achieve the effect of quickly and accurately determining the drivable area.

[0004] An embodiment of the first aspect of the present disclosure provides a method for detecting a drivable area, comprising:

[0005] Determine the multi-view environment image collected by the vehicle;

[0006] Processing the multi-view environment image to obtain a sight line occupancy state and an occupancy point distance corresponding to a preset sight line;

[0007] Determining the position information of the occupied point in the world coordinate system according to the line of sight occupation state and the distance of the occupied point;

[0008] The drivable area of ​​the vehicle is determined according to the position information of the occupied point.

[0009] An embodiment of the second aspect of the present disclosure provides a method for training an image processing model, comprising:

[0010] Determine a sample data set including a plurality of sample data, wherein the sample data includes a multi-view sample image, a true value of a sight line occupancy state of a preset sight line in a bird's-eye view corresponding to the multi-view sample image, and a true value of an occupied point distance;

[0011] Using an initial image processing model to process the multi-view sample images included in the sample data, to obtain a line of sight occupancy state prediction value and an occupancy point distance prediction value corresponding to the preset line of sight;

[0012] Determine a loss function according to the true value of the line of sight occupancy state and the predicted value of the line of sight occupancy state, and the true value of the occupied point distance and the predicted value of the occupied point distance;

[0013] The initial image processing model is trained based on the loss function to obtain a trained image processing model.

[0014] An embodiment of a third aspect of the present disclosure provides a device for detecting a drivable area, including:

[0015] A first determination module is used to determine a multi-view environment image collected by a vehicle;

[0016] A first processing module is used to process the multi-view environment image to obtain a sight line occupancy state and an occupancy point distance corresponding to a preset sight line;

[0017] A second determination module is used to determine the position information of the occupied point in the world coordinate system according to the line of sight occupation state and the occupied point distance;

[0018] The third determination module is used to determine the drivable area of ​​the vehicle according to the location information of the occupied point.

[0019] An embodiment of a fourth aspect of the present disclosure provides a training device for an image processing model, comprising:

[0020] A fourth determination module is used to determine a sample data set including a plurality of sample data, wherein the sample data includes a multi-view sample image, a true value of a sight line occupancy state of a preset sight line in a bird's-eye view corresponding to the multi-view sample image, and a true value of an occupied point distance;

[0021] A second processing module is used to process the multi-view sample images included in the sample data using an initial image processing model to obtain a line of sight occupancy state prediction value and an occupied point distance prediction value corresponding to the preset line of sight;

[0022] a fifth determination module, configured to determine a loss function according to the true value of the sight line occupancy state and the predicted value of the sight line occupancy state, and the true value of the occupied point distance and the predicted value of the occupied point distance;

[0023] A training module is used to train the initial image processing model based on the loss function to obtain a trained image processing model.

[0024] An embodiment of the fifth aspect of the present disclosure provides a computer-readable storage medium, which stores a computer program, and the computer program is used to execute the drivable area detection method proposed in the embodiment of the first aspect of the present disclosure or the image processing model training method proposed in the embodiment of the second aspect of the present disclosure.

[0025] An embodiment of a sixth aspect of the present disclosure provides an electronic device, the electronic device comprising:

[0026] processor;

[0027] a memory for storing instructions executable by the processor;

[0028] The processor is used to read the instructions from the memory and execute the instructions to implement the drivable area detection method proposed in the embodiment of the first aspect of the present disclosure or the image processing model training method proposed in the embodiment of the second aspect of the present disclosure.

[0029] An embodiment of the seventh aspect of the present disclosure provides a computer program product. When an instruction processor in the computer program product is executed, it executes the drivable area detection method proposed in the embodiment of the first aspect of the present disclosure or the image processing model training method proposed in the embodiment of the second aspect of the present disclosure.

[0030] In the disclosed embodiment, the multi-view environment images collected during the vehicle driving process are processed by using a pre-trained image processing model to determine the sight occupation state and the distance of the occupied point corresponding to the preset sight in the bird's-eye view; according to the sight occupation state and the distance of the occupied point, the position information of the occupied point in the world coordinate system is determined, and then the drivable area of ​​the vehicle is determined based on the position information of the occupied point. By transforming the features of the multi-view environment image, the occupancy state of each preset sight in the bird's-eye view and the distance of the occupied grid point (i.e., the occupied point) closest to the origin in the grid points on the sight in the occupied state are determined, and the drivable area of ​​the vehicle is determined based on this. On the one hand, it is not necessary to determine the occupancy of the grid points in the occlusion area of ​​each sight whose distance exceeds the occupied point, and on the other hand, it is not necessary to determine the occupancy of the remaining grid points except each sight, so that the model calculation amount can be reduced, the redundant information can be greatly reduced, and because the occupancy of the grid points in the occupancy area whose distance exceeds the occupied point is not considered, the introduction of the prediction data of the invisible area is avoided, so that the accuracy of the occupancy detection result can be improved, and the drivable area can be determined quickly and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a schematic diagram of the principle of a method for detecting a drivable area provided by an exemplary embodiment of the present disclosure;

[0032] Figure 2 is a schematic diagram of a network architecture of an image processing system provided by an exemplary embodiment of the present disclosure;

[0033] Figure 3 is a schematic diagram of the structure of a terminal device provided by an exemplary embodiment of the present disclosure;

[0034] Figure 4 is a flowchart of a method for detecting a drivable area provided by an exemplary embodiment of the present disclosure;

[0035] Figure 5 It is a schematic diagram of determining the sight line occupation state and the distance of the occupied point in the method for detecting the drivable area provided by an exemplary embodiment of the present disclosure;

[0036] Figure 6 is a flow chart of a method for detecting a drivable area provided by another exemplary embodiment of the present disclosure;

[0037] Figure 7 is a flowchart of a method for detecting a drivable area provided by yet another exemplary embodiment of the present disclosure;

[0038] Figure 8 is a flowchart of a method for detecting a drivable area provided by another exemplary embodiment of the present disclosure;

[0039] Fig. 9 is a flowchart of a method for detecting a drivable area provided by yet another exemplary embodiment of the present disclosure;

[0040] Fig.10 is a flowchart of a method for detecting a drivable area provided by another exemplary embodiment of the present disclosure;

[0041] Fig.11 is a flowchart of a training method for an image processing model provided by an exemplary embodiment of the present disclosure;

[0042] Fig.12 is a flowchart of a training method for an image processing model provided by another exemplary embodiment of the present disclosure;

[0043] Fig.13 is a schematic diagram of the structure of a detection device for a drivable area provided by an exemplary embodiment of the present disclosure;

[0044] Fig.14 is a schematic diagram of the composition structure of a detection device for a drivable area provided by another exemplary embodiment of the present disclosure;

[0045] Fig.15 It is a schematic diagram of the composition structure of a training device for an image processing model provided by an exemplary embodiment of the present disclosure;

[0046] Fig.16 is a schematic diagram of the composition structure of a training device for an image processing model provided by another exemplary embodiment of the present disclosure;

[0047] Fig.17 It is a schematic diagram of the composition structure of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.

[0049] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0050] Application Overview

[0051] The intelligent driving system is an assisted driving system developed based on automatic control technology. It makes driving decisions autonomously by sensing and analyzing the environmental information around the intelligent driving vehicle (hereinafter referred to as the vehicle), thereby realizing automatic or semi-automatic driving of the vehicle.

[0052] In the intelligent driving scenario, occupancy prediction refers to predicting whether a space or area is occupied based on the surrounding environment information collected by the vehicle. The task corresponding to occupancy prediction is called an occupancy prediction task, also known as an occupancy task. The occupancy task is performed to determine the occupancy status of each space around the vehicle, and based on this, the drivable area of ​​the vehicle is determined for trajectory planning.

[0053] In the related art, when the intelligent driving system perceives the surrounding environment of the vehicle, it is necessary not only to identify the target objects in the surrounding environment, but also to identify the specific category of each target object. Compared with the perception task of perceiving the target object in the related art, the occupation task does not focus on the specific category of the occupied object. Therefore, the occupation task only needs to detect all occupied objects related to the intelligent driving scenario in the surrounding environment, without further identifying the category of the occupied objects.

[0054] In the related art, the occupancy detection scheme based on the bird's-eye view, which is commonly used for occupancy tasks, mainly adopts the scheme of segmentation task: semantic segmentation is performed on the bird's-eye view features, and a semantic category is assigned to each spatial coordinate position as the basis for judging whether it is passable. Although the occupancy detection scheme based on the segmentation task has an intuitive output form and a relatively simple structure, it has the following defects:

[0055] 1) Dense output and much irrelevant information: Since the segmentation task is a dense classification task, the continuous coordinates in the space are discretized into a grid network. It is necessary to perform semantic recognition on each grid coordinate position in the grid network to predict its category. This places a very large capacity requirement on the deep learning model, with too many redundant features. It also brings huge computational overhead, which is not conducive to deployment in real-time environments.

[0056] 2) Poor accuracy: For semantic segmentation classification tasks, the prediction accuracy error is equal to half of the actual physical distance represented by each grid pixel. As the prediction accuracy requirements increase, the number of output pixels of the grid network increases at a square rate, which is too fast for deep learning models. In order to take into account the scale of the model, the prediction accuracy often has a large error.

[0057] 3) Feature redundancy: In bird's-eye view occupancy detection, especially in pure vision-based bird's-eye view perception, most of the sensors used to collect data are installed on the vehicle, so the observation area is strictly limited to the position of the vehicle. In non-occluded areas, it has a good observation effect and can ensure the accuracy of the prediction results, but in occluded areas, it can only rely on historical experience to fit the labels, resulting in too many uncertain labels introduced in model training, and it is impossible to guarantee the model's true perception ability in occluded areas.

[0058] In view of the above problems existing in the related art, the present disclosure provides a method for detecting a drivable area in an embodiment. The method converts the image features collected by the vehicle camera into the feature space of the bird's-eye view to perform occupancy prediction, thereby determining the drivable area. Figure 1 , in the bird's-eye view feature space, with the origin of the vehicle coordinate system as the endpoint, such as Figure 1 The origin O shown in the figure emits multiple rays in all directions as sight lines, such as Figure 1 The dashed rays in each direction are shown. Determine whether there is an occupied point on each line of sight based on the bird's-eye view features, that is, determine whether there is an obstacle in that direction. If it is determined that there is an occupied point on a certain line of sight, the image distance between the endpoint and the occupied point can be determined based on the bird's-eye view features, and the relative distance between the vehicle and the obstacle can be determined based on this. The position of the obstacle is determined based on the direction and distance of the obstacle relative to the vehicle, and the area corresponding to the position is impassable. If it is determined that there is no occupied point on a certain line of sight, it means that there is no obstacle in that direction and the area corresponding to that direction is passable.

[0059] In the disclosed embodiment, the features of the multi-perspective environmental image are transformed into a bird's-eye view to determine the occupancy status of each preset line of sight and the distance to the occupied point closest to the origin among the grid points on the line of sight in the occupied state, and the drivable area of ​​the vehicle is determined based on this. On the one hand, there is no need to determine the occupancy status of the grid points in the occlusion area on each line of sight whose distance exceeds the occupied point, and on the other hand, there is no need to determine the occupancy status of the remaining grid points except for each line of sight. Therefore, the amount of model calculation can be reduced, redundant information can be greatly reduced, and because the occupancy status of the grid points in the occlusion area whose distance exceeds the occupied point is not considered, the introduction of predicted data of the invisible area can be avoided, thereby improving the accuracy of the occupancy detection result and realizing fast and accurate determination of the drivable area.

[0060] Exemplary Systems

[0061] Figure 2 is a schematic diagram of a network architecture of an image processing system provided by an exemplary embodiment of the present disclosure. Figure 2 As shown, the image processing system 10 includes a terminal device 100 (the terminal device 100 is a plurality of Figure 2 , terminal device 100-1 and terminal device 100-2), network 200 and server device 300 are exemplarily shown. The terminal device 100 is connected to the server device 300 through the network 200. The network 200 can be a wide area network or a local area network, or a combination of the two, and a wired or wireless link is used to achieve data transmission. The terminal device 100 and the server device 300 can be directly or indirectly connected through wired or wireless communication, which is not limited in the embodiments of the present disclosure.

[0062] Exemplarily, the terminal device 100 can be implemented as an electronic device for detecting a drivable area provided on a smart driving vehicle. The terminal device 100 may include a vehicle-mounted camera for collecting environmental images around the vehicle. In the disclosed embodiment, there may be multiple vehicle-mounted cameras, each of which may have a different viewing angle. During the driving of the vehicle, multiple vehicle-mounted cameras collect driving environment images from different viewing angles to obtain multi-view (driving) environment images. The terminal device 100 may transmit the collected multi-view environment images as multi-view sample images to the server device 300.

[0063] Exemplarily, the server device 300 can be implemented as an electronic device for training an image processing model. The server device 300 determines the training data, and the training data may include multi-view sample images and their marking information. The server device 300 can receive multi-view sample images from the terminal device 100, and the marking information of each view sample image can be obtained by manual marking. The marking information of a sample image includes the true value of the line of sight occupancy state and the true value of the occupied point distance of the preset line of sight in the bird's-eye view corresponding to the sample image. Then, the server device 300 trains the initial image processing model according to the training data including the multi-view sample images and their marking information to obtain a trained image processing model. Finally, the trained image processing model is sent to the terminal device 100, so that the terminal device 100 uses the trained image processing model to process the multi-view environment images collected in real time during driving, determine the location information of the occupied point, and then determine the drivable area of ​​the vehicle.

[0064] In some embodiments, the network architecture of the image processing system may only include the terminal device 100. After the terminal device 100 acquires the multi-view environment image, it may perform model training on the local side to obtain a trained image processing model. During vehicle driving, the terminal device 100 uses the image processing model trained on the local side to process the multi-view environment image acquired in real time during driving, determine the location information of the occupied point, and then determine the drivable area of ​​the vehicle.

[0065] Figure 3 FIG. 1 is a schematic diagram of the structure of a terminal device provided by an exemplary embodiment of the present disclosure. Figure 3 As shown, the terminal device 100 may include a vehicle-mounted camera 101 , a drivable area detection device 102 , and a decision control device 103 .

[0066] There may be multiple onboard cameras 101, which are used to collect driving environment images from different perspectives to obtain multi-perspective environment images. Figure 3 As shown, the number of vehicle-mounted cameras is N, and the i-th vehicle-mounted camera is recorded as 201-i, which is used to collect the i-th perspective environment image, i=1, 2, ..., N, N is a positive integer. The i-th vehicle-mounted camera transmits the collected i-th perspective environment image to the detection device 202 of the drivable area.

[0067] The detection device 202 of the drivable area obtains environmental images of N perspectives transmitted by N vehicle-mounted cameras, and processes the environmental images of each perspective using the image processing model sent by the server device 300 to obtain the line of sight occupancy status and the occupied point distance of the preset line of sight in the bird's-eye view; then, according to the line of sight occupancy status and the occupied point distance, the position information of the occupied point in the world coordinate system is determined; then, according to the position information of the occupied point, the drivable area of ​​the vehicle is determined; finally, the drivable area is transmitted to the decision control device 103.

[0068] The decision control device 103 makes a decision based on the detected drivable area, determines the trajectory planning and executes the corresponding control instructions to control the vehicle to travel according to the planned trajectory within the drivable area.

[0069] The terminal device provided by the embodiment of the present disclosure processes the multi-view environment image collected during the vehicle driving process by using a pre-trained image processing model, determines the sight line occupancy state and the distance of the occupied point of the preset sight line in the bird's-eye view; determines the position information of the occupied point in the world coordinate system according to the sight line occupancy state and the distance of the occupied point, and then determines the drivable area of ​​the vehicle based on the position information of the occupied point. The terminal device transforms the features of the multi-view environment image into the feature space of the bird's-eye view through the pre-trained image processing model, determines the occupancy state of each preset sight line in the bird's-eye view and the distance of the occupied point closest to the origin among the grid points on the sight line in the occupied state, and determines the drivable area of ​​the vehicle based on this, on the one hand, it is not necessary to determine the occupancy of the grid points in the occupancy area of ​​each sight line whose distance exceeds the occupancy point, and on the other hand, it is not necessary to determine the occupancy of the remaining grid points except each sight line, so that the model calculation amount can be reduced, the redundant information can be greatly reduced, and because the occupancy of the grid points in the occupancy area whose distance exceeds the occupancy point is not considered, the introduction of the prediction data of the invisible area is avoided, so that the accuracy of the occupancy detection result can be improved, and the drivable area can be determined quickly and accurately.

[0070] Exemplary Methods

[0071] Figure 4 is a flowchart of a method for detecting a drivable area provided by an exemplary embodiment of the present disclosure. The present disclosure embodiment can be applied in Figure 3 On the detection device of the drivable area shown, Figure 4 As shown, the method for detecting the drivable area may include the following steps:

[0072] Step S401, determining a multi-view environment image collected by a vehicle.

[0073] The embodiments of the present disclosure may be executed by a drivable area detection device provided in an intelligent driving vehicle (referred to as vehicle), and specifically may be executed by a drivable area detection apparatus in the drivable area detection device.

[0074] The detection device of the drivable area can obtain multi-view environmental images from the vehicle-mounted camera. Specifically, during the driving of the vehicle, multiple vehicle-mounted cameras on the vehicle collect environmental images of the vehicle's surroundings in real time, and transmit the collected images to the detection device of the drivable area. Since different vehicle-mounted cameras have different postures, their field of view ranges are different. Each vehicle-mounted camera captures the vehicle's surrounding environment from different angles to obtain a multi-view environmental image.

[0075] For example, Figure 3 As shown in the figure, the number of onboard cameras is N, and the i-th perspective environment image collected by the i-th onboard camera can be expressed as v iTherefore, the multi-view environment image obtained by the detection device of the drivable area can be expressed as {v1, v2, …, v i , …, v N}, i = 1, 2, ..., N, N is a positive integer. For example, a vehicle is provided with 8 on-board cameras facing different angles around the vehicle to collect environmental images from different perspectives.

[0076] Step S402: Process the multi-view environment image to obtain the sight line occupancy state and the occupancy point distance corresponding to the preset sight line.

[0077] Exemplarily, the image processing model can be Figure 2 The server device in the image processing system is trained and deployed on the detection device of the terminal device. The detailed process of the server device training the image processing model can be found in the following Fig.11 Description of the illustrated embodiment.

[0078] The detection device of the drivable area acquires the multi-view environment image {v1, v2, ..., v i , …, v N The input image processing model performs a series of processing such as feature extraction, transformation, fusion, etc. to obtain an output result. The output result includes a first output result and a second output result, wherein the first output result may be a sight line occupation state corresponding to a preset sight line, and the second output result may be an occupied point distance.

[0079] Exemplarily, the preset sight line can be a plurality of rays emitted in a counterclockwise direction starting from the 0° sight line (positive direction of the horizontal axis of the vehicle coordinate system). The sight line occupation state refers to the state in which the grid points within the preset length range on the sight line are occupied. When a grid point within the preset length range on a certain sight line is occupied (that is, there is an obstacle at the position corresponding to the grid point), the occupation state of the sight line is determined to be in the occupied state; when all grid points within the preset length range on a certain sight line are not occupied, the occupation state of the sight line is determined to be in the unoccupied state.

[0080] Among them, the value of the preset length range can be set according to the driving focus range (such as 100m (meters), 200m, etc.), for example, it can be set to the image length corresponding to 100m. When driving, pay attention to the occlusion of obstacles within the range of 100m. Even if there are obstacles outside the range of 100m, the obstacles do not affect the occlusion of the line of sight, so the preset length range is set. The line of sight occupancy status only considers whether the grid points within the preset length range on the line of sight are occupied, and has nothing to do with whether the grid points outside the preset length range are occupied.

[0081] In some embodiments, two M-dimensional vectors can be used to represent the line of sight occupation status and the occupied point distance, respectively. Among them, M is a natural number, representing the vector dimension, which is consistent with the number of lines of sight, and each element in the vector is used to determine the location information of the occupied point. Each element in the vector of the line of sight occupation status indicates whether each line of sight is occupied. The vector can be composed of 1 and 0, wherein 1 indicates that the current line of sight is in an occupied state, and there are occupied grid points (i.e., occupied points) among the grid points within a preset length range on the line of sight, that is, there are obstacles that affect the passage in the direction of the line of sight; 0 indicates that the current line of sight is in a non-occupied state, and all grid points within a preset length range on the line of sight are not occupied, that is, the road in the direction of the line of sight is passable. Each element in the vector of the occupied point distance represents the image distance from the origin to the nearest occupied point on each line of sight. When there is no occupied point within the preset length range on the line of sight, the corresponding occupied point distance is the default value null.

[0082] See also Figure 5 As shown in (a) in FIG. 5 , taking 12 sight lines as an example, the angles are 0°, 30°, 60°, 90°, 120°, 150°, 180°, 210°, 240°, 270°, 300° and 330° respectively. The sight line occupancy state of each preset sight line can be expressed as (1 1 10 0 0 1 1 1 0 1 0) T The distance of each preset line of sight to occupied points can be expressed as (20 18 29null null null10 21 16null 13null) T .like Figure 5 As shown in (a), the image distance from the origin O to the nearest occupied point (i.e., the first occupied point) on the 0° line of sight is 20, and the occupied point distance corresponding to the 0° line of sight is 20. For another example, there is no occupied point on the 90° line of sight, and its corresponding occupied point distance is null.

[0083] In some other embodiments, the line of sight occupation state and the distance to the occupied point may be represented by an M*2-dimensional vector, and the M*2-dimensional vector may be a fusion of the above two M-dimensional vectors. Figure 5 As shown in (a) in the figure, the M*2-dimensional vector corresponding to the line of sight occupancy state and the distance to the occupied point can be expressed as The principle is similar to that of the two M-dimensional vectors mentioned above and will not be repeated here.

[0084] In some other embodiments, the line of sight occupation status and the distance to the occupied point can be represented by an M-dimensional vector, where the vector dimension M is consistent with the number of lines of sight, and the elements of the vector are used to determine the position information of the occupied point. The elements of the vector represent whether each line of sight is occupied, and the image distance from the origin to the nearest occupied point on each occupied line of sight. Figure 5As shown in (a) in the figure, the M-dimensional vector corresponding to the line of sight occupancy state and the distance to the occupied point can be expressed as (20 18 29 -1-1 -1 10 21 16-1 13 -1) T . The 0° line of sight is occupied, and the image distance from the origin O to the nearest occupied point (i.e., the first occupied point) on the 0° line of sight is 20, then the element corresponding to the 0° line of sight is 20. For another example, there is no occupied point on the 90° line of sight, and it is not occupied, and its corresponding element is -1.

[0085] For example, the number of sight lines can be set according to actual conditions, and the angles between the sight lines can be equal, such as Figure 5 The interval between each pair of sight lines shown in (a) is 30°. The angles between sight lines may also be unequal. For example, the vehicle pays more attention to the forward and lateral environmental information during driving, so a smaller number of sight lines may be set to the rear of the vehicle, and a larger number of sight lines may be set to the front and both sides of the vehicle. For example, more sight lines may be set within the field of view of 15° to 165°, such as 10° between each pair of sight lines, and fewer sight lines may be set within the field of view of 0° to 15° and 165° to 360°, such as 30° between each pair of sight lines. The more sight lines there are, the more accurate the determined drivable area is, but at the same time the amount of calculation will also increase. In actual implementation, the number and angle of preset sight lines may be set in combination with the specific environment, and are not limited in the embodiments of the present disclosure.

[0086] In the disclosed embodiment, only a small number of sight lines (such as Figure 5 The occupancy status of the 12 sight lines (shown in (a) in the figure) is determined by the occupancy status of the grid points on the sight lines other than the sight lines, which can reduce the amount of model calculation and greatly reduce redundant information. When determining the distance of the occupied points, it is only necessary to determine the distance of the occupied point closest to the origin on each sight line, without determining the distance of the remaining grid points on each sight line beyond the nearest occupied point (such as Figure 5 The occupancy of the occluded grid points exceeding the second occupied point as shown in (a) can further focus on the area of ​​interest and will not introduce predicted data of the invisible area, which can improve the accuracy of the occupancy detection results.

[0087] Step S403, determining the position information of the occupied point in the world coordinate system according to the line of sight occupation state and the distance of the occupied point.

[0088] After determining the sight lines of occupied points in the preset sight lines in the bird's-eye view and the distances of the occupied points from the origin, combined with the mapping relationship between the bird's-eye view coordinate system and the world coordinate system, the position information of each occupied point in the world coordinate system is determined according to the sight line occupancy status and the distance of the occupied points.

[0089] Step S404: determining the drivable area of ​​the vehicle according to the location information of the occupied point.

[0090] According to the world coordinates of multiple occupied points, adjacent occupied points are connected to obtain a driving boundary, and the area where unoccupied points are located within the driving boundary is the drivable area of ​​the vehicle.

[0091] The method for detecting a drivable area provided by the embodiment of the present disclosure first determines the multi-view environment image collected by the vehicle through the detection device of the drivable area; then processes the multi-view environment image to obtain the line of sight occupation state and the distance of the occupied point corresponding to the preset line of sight; then determines the position information of the occupied point in the world coordinate system according to the line of sight occupation state and the distance of the occupied point; finally determines the drivable area of ​​the vehicle according to the position information of the occupied point. The method transforms the features of the multi-view environment image into the feature space of the bird's-eye view, determines the occupation state of each preset line of sight and the distance of the occupied point closest to the origin in the grid point on the line of sight in the occupied state in the bird's-eye view, and determines the drivable area of ​​the vehicle based on this. On the one hand, it is not necessary to determine the occupation of the grid points in the occlusion area of ​​each line of sight whose distance exceeds the occupied point, and on the other hand, it is not necessary to determine the occupation of the remaining grid points except each line of sight, so that the amount of model calculation can be reduced, redundant information can be greatly reduced, and because the occupation of the grid points in the occlusion area whose distance exceeds the occupied point is not considered, the introduction of the prediction data of the invisible area is avoided, so that the accuracy of the occupancy detection result can be improved, and the drivable area can be determined quickly and accurately.

[0092] In some embodiments, based on the above embodiments, the above step S402 "processing the multi-view environment image to obtain the sight line occupancy state and the occupancy point distance corresponding to the preset sight line" can be performed by Figure 6 The steps S4021 to S4023 shown are implemented as follows:

[0093] Step S4021, using a pre-trained image processing model to process each environment image in the multi-view environment image, to obtain mapping features of each environment image in the bird's-eye view.

[0094] According to the above Figure 4 After determining multiple environmental images of different perspectives in step S401 of the illustrated embodiment, feature extraction is performed on each environmental image using an image processing model, and the extracted image features are transformed into a bird's-eye view feature space to obtain mapping features of each environmental image.

[0095] In one implementation, the mapping features of each environment image in the bird's-eye view can be obtained by Figure 7 The steps S40211 and S40212 shown are implemented as follows:

[0096] Step S40211, using the first feature extraction sub-model in the pre-trained image processing model to perform feature extraction processing on each environmental image in the multi-view environmental image, to obtain image features of each environmental image.

[0097] The image processing model may include a first feature extraction sub-model and a mapping sub-model. i , …, v N} in the i-th perspective environment image (i.e., v i ) as an example, the i-th perspective environment image is input into the first feature extraction sub-model for feature extraction to obtain the features of the i-th perspective environment image.

[0098] Step S40212, using the mapping sub-model in the image processing model to perform mapping processing on the image features of each environmental image respectively, to obtain the mapping features of each environmental image in the bird's-eye view.

[0099] The features of the i-th perspective environment image are input into the mapping sub-model for feature transformation. Combined with the camera parameters of the i-th vehicle-mounted camera, the mapping relationship between the pixel position of the image feature and the position in the bird's-eye view feature space is calculated. The image features in the image domain are mapped to the bird's-eye view feature space to obtain the mapping features of the i-th perspective environment image. The mapping features are the mapping features of the i-th perspective environment image in the bird's-eye view.

[0100] The above processing is performed on the environment images of N perspectives to obtain mapping features of the environment images of N perspectives.

[0101] Step S4022, using the fusion sub-model in the image processing model to fuse the mapping features corresponding to each environment image, to obtain a bird's-eye view fusion feature.

[0102] In the disclosed embodiment, after obtaining the mapping features of the environment images of each viewing angle, the detection device of the drivable area performs fusion processing on the mapping features to obtain fusion features in the feature space of the bird's-eye view.

[0103] In some embodiments, the environment images of multiple perspectives may be first fused in the image domain to obtain a fused image, and then the fused image may be subjected to feature extraction and mapping processing to obtain fusion features in the feature space of the bird's-eye view. Exemplarily, when processing the environment images of each perspective to determine the fusion features of the bird's-eye view, a pre-fusion solution may be adopted: first image fusion is performed and then feature extraction and mapping processing are performed, or a mid-fusion solution may be adopted: first feature extraction and mapping processing are performed and then feature fusion is performed in the feature space of the bird's-eye view. Compared with the pre-fusion solution, the mid-fusion solution can hand over the processing of the fusion step to the image processing model and perform fusion in a network learning manner, which helps to learn features with stronger expressiveness. Features with stronger expressiveness can improve the accuracy of the detection results.

[0104] Step S4023, using the image processing model to process the fusion features of the bird's-eye view to obtain the line of sight occupancy status and occupancy point distance of the preset line of sight.

[0105] The fused features are all features in the bird's-eye view feature space. In order to reduce the amount of calculation, in the disclosed embodiment, all the underlying features can be extracted and transformed to determine the line of sight occupancy status and occupied point distance of the preset line of sight.

[0106] In one implementation, the line of sight occupancy status and the distance of the occupancy point of the preset line of sight are determined according to the fusion feature of the bird's-eye view, which can be achieved by Figure 8 The steps S40231 to S40234 shown are implemented as follows:

[0107] Step S40231, using the second feature extraction sub-model in the image processing model to perform feature extraction processing on the bird's-eye view fusion features to obtain the bird's-eye view features.

[0108] All the features in the fused bird's-eye view feature space are subjected to feature extraction processing to extract some high-level features. Since the extracted features are partial features of all the features, compared with all the features, the amount of data can be greatly reduced and the amount of subsequent calculations can be reduced.

[0109] In some embodiments, there is no need to perform the above step S40231, and the detection device of the drivable area can directly perform subsequent processing on all the features of the bird's-eye view fusion feature. Although using all features for calculation can reduce feature loss and help improve the accuracy of the detection results, the data volume of all features is large, which will increase the amount of calculation and reduce the detection efficiency. In practical applications, a suitable second feature extraction sub-model can be selected for feature extraction according to the required accuracy, which is not limited here.

[0110] Step S40232, using the third feature extraction sub-model in the image processing model to perform feature extraction processing on the bird's-eye view features to obtain the sight line features of the preset sight line.

[0111] According to the above step S40231, the features in the entire bird's-eye view feature space are determined. In the embodiment of the present disclosure, only the occupancy of the grid points on the preset line of sight and the distance of the occupied points are considered. Based on this, the third feature extraction submodel in the image processing model is used to perform feature extraction processing on the bird's-eye view features to obtain the line of sight features of the preset line of sight. The third feature extraction submodel is used to extract the line of sight features on the preset line of sight from the bird's-eye view features.

[0112] Step S40233, using the classification recognition sub-model in the image processing model to perform recognition processing on the sight line feature to obtain a recognition result corresponding to the preset sight line.

[0113] The classification recognition sub-model in the image processing model is used to recognize and process the sight features of each preset sight, determine whether the grid points on each preset sight are occupied, and obtain the recognition results corresponding to each preset sight. The recognition results corresponding to each preset sight are the recognition results of whether the grid points on each preset sight are occupied.

[0114] In some embodiments, when determining whether a grid point on a preset line of sight is occupied, it is possible to identify whether all grid points on each preset line of sight are occupied. However, this implementation may introduce prediction data of invisible areas for grid points in the occluded area, affecting the accuracy of the occupancy detection result.

[0115] In the disclosed embodiment, when determining whether a grid point on a preset line of sight is occupied, each grid point on the line of sight can be identified separately from the origin O along the line of sight away from the origin O. When a grid point is identified as occupied, there is no need to identify subsequent grid points on the line of sight. That is, the occupancy of grid points in the occlusion area beyond the occupied point is not considered, which can reduce the amount of calculation; and because the occupancy of grid points in the occlusion area beyond the occupied point is not considered, the introduction of prediction data of the invisible area is avoided, thereby improving the accuracy of the occupancy detection result and realizing the rapid and accurate determination of the drivable area.

[0116] Step S40234, using the calculation sub-model in the image processing model to process the recognition result corresponding to the preset line of sight, and determine the line of sight occupancy state and the occupied point distance of the preset line of sight.

[0117] After the sight features of each preset sight are obtained through the feature extraction and transformation of the bird's-eye view, the sight features of each sight are input into the classification recognition sub-model in the image processing model for classification recognition processing to determine the recognition results corresponding to each preset sight. According to the recognition results corresponding to each preset sight, the sight occupancy status and the occupied point distance of each preset sight are determined.

[0118] In the disclosed embodiment, only the occupancy status of a small number of sights needs to be determined, and there is no need to determine the occupancy status of grid points on sights other than the sights, which can reduce the amount of model calculation and greatly reduce redundant information. When determining the distance of occupied points, only the distance to the occupied point closest to the origin on each sight needs to be determined, and there is no need to determine the occupancy status of remaining grid points on each sight that are farther away than the nearest occupied point. This can further focus on the area of ​​interest, and will not introduce predicted data for invisible areas, which can improve the accuracy of occupancy detection results.

[0119] In some embodiments, when it is not necessary to determine the category of the occupied object, the vector of the line of sight occupancy state can be represented as a 0-1 vector. 1 indicates that the current line of sight is in an occupied state, and 0 indicates that the current line of sight is in a non-occupied state, such as Figure 5 As shown in (a) in the figure, the vector of the line of sight occupancy state can be expressed as (1 1 1 0 0 0 1 1 1 0 1 0) T , each dimension represents whether a grid point is occupied within a preset length range in a line of sight direction.

[0120] In some embodiments, when it is necessary to determine the category of occupied objects, semantic segmentation can be performed on the bird's-eye view features, and the category of occupied points can be determined in combination with semantic information, and different labels can be set for objects of different categories. At this time, the elements in the vector of line of sight occupancy status not only indicate whether each line of sight is occupied but also indicate the category of occupied objects. Exemplarily, the vector of line of sight occupancy status can be represented as a 0-label vector, for example, 1 indicates that the current line of sight is in an occupied state and the occupied object is a vehicle, 2 indicates that the current line of sight is in an occupied state and the occupied object is a pedestrian, 3 indicates that the current line of sight is in an occupied state and the occupied object is a curb, and 0 indicates that the current line of sight is in an unoccupied state, such as Figure 5 As shown in (a) in the figure, the vector of the line of sight occupancy state can be expressed as (3 2 1 0 0 0 1 3 1 0 1 0) T , each dimension represents whether there is a grid point occupied within a preset length range in a line of sight direction, and when there is an occupied grid point, the specific category of the corresponding obstacle is.

[0121] In some embodiments, based on the above embodiments, the above step S403 "determine the position information of the occupied point in the world coordinate system according to the line of sight occupation state and the distance of the occupied point" can be performed by Fig. 9The steps S4031 to S4033 shown are implemented as follows:

[0122] Step S4031, determining the image coordinates of the occupied point on the preset line of sight according to the line of sight occupation state and the occupied point distance.

[0123] Still Figure 5 Taking the bird's-eye view coordinate system shown in (a) as an example, the vector of the line of sight occupancy state can be expressed as (1 1 1 0 0 0 1 1 1 0 1 0) T , where each dimension represents whether there is an occupied grid point within a preset length range in the direction of the line of sight, that is, whether there is an occupied point. The vector of occupied point distance can be expressed as (20 18 29nullnull null 10 21 16null 13null) T , where each dimension represents the distance from the nearest occupied point in a line of sight to the coordinate origin O. According to the elements of the corresponding dimensions of the vector of the line of sight occupancy state and the vector of the occupied point distance, the image coordinates (d1, d2) of the occupied point on each preset line of sight are determined. The occupied point specifically refers to the nearest occupied point.

[0124] The image coordinates of the occupied point on the preset line of sight are a plane coordinate system established with the origin O as the origin, d1 is the horizontal axis of the coordinate system, and d2 is the vertical axis of the coordinate system. d1 = d*cosθ, d2 = d*sinθ. Among them, θ is the angle of the line of sight where the occupied point is located, and d is the distance of the occupied point, that is, the image distance between the occupied point and the origin. For example Figure 5 For example, for the 0° line of sight in (a), the line of sight angle and the distance to the occupied point on the line of sight can be expressed as (0°, 20); for another example, for the 30° line of sight, the line of sight angle and the distance to the occupied point on the line of sight can be expressed as (30°, 18); for another example, for the 120° line of sight, there is no occupied point on the line of sight, then the line of sight angle and the distance to the occupied point on the line of sight can be expressed as (120°, null).

[0125] Step S4032, determining the horizontal coordinate and the vertical coordinate of each occupied point in the world coordinate system according to the image coordinates of each occupied point.

[0126] Step S4033, determining the position information of each occupied point according to the horizontal coordinate and the vertical coordinate of each occupied point in the world coordinate system.

[0127] Based on the image coordinates (d1, d2) of each occupied point, the horizontal coordinate x and the vertical coordinate y of the image coordinates (d1, d2) of each occupied point in the world coordinate system can be determined according to the mapping relationship between the bird's-eye view coordinate system and the world coordinate system, and the position information (x, y) of each occupied point can be obtained.

[0128] In the disclosed embodiment, the angle of sight is pre-set, and the position information of each occupied point in the world coordinate system is calculated based on the angle of sight of the occupied point and the distance of the occupied point, so as to determine the obstacles in the preset area and provide data support for the subsequent determination of the drivable area.

[0129] In some embodiments, based on the above embodiments, the above step S404 "determine the drivable area of ​​the vehicle according to the location information of the occupied point" can be performed by Fig.10 Steps S4041 to S4043 shown are used to implement:

[0130] Step S4041, determining adjacent occupied points corresponding to each occupied point according to the line of sight occupation status corresponding to the preset line of sight.

[0131] According to the sight occupation status corresponding to the preset sight, it can be determined whether each sight is occupied (i.e., whether there is an occupied point on the sight). In response to a certain sight being occupied, it is further determined whether the adjacent sight of the sight is in an occupied state, that is, it is determined whether the adjacent dimension (i.e., the previous dimension or the next dimension) of each dimension that is 1 in the vector of the above sight occupation state is also 1. If the previous dimension in the adjacent dimensions is 1, it means that the previous sight of the current sight is also in an occupied state, and the occupied point on the previous sight is an adjacent occupied point of the occupied point on the current sight. If the previous dimension in the adjacent dimensions is 0, it means that the previous sight of the current sight is in a non-occupied state, and there is no occupied point on the previous sight, so there is no adjacent occupied point on the previous sight that is on the occupied point of the current sight.

[0132] Still Figure 5 Taking the occupancy state shown in (a) as an example, the vector of the line of sight occupancy state is represented as (1 1 10 0 0 1 1 1 0 1 0) T , the dimension after the first dimension (i.e., the second dimension) is 1, and the occupied point on the second line of sight is the adjacent occupied point of the occupied point on the first line of sight. For another example, the dimension before the seventh dimension (i.e., the sixth dimension) is 0, there is no occupied point on the sixth line of sight, and there is no adjacent occupied point on this line of sight to the occupied point on the seventh line of sight.

[0133] Step S4042, determining the driving boundary according to each occupied point and its corresponding adjacent occupied point.

[0134] For example, each occupied point and its corresponding adjacent occupied point can be connected according to the location information of the occupied point to obtain the driving boundary. Generally, since the front and rear directions of the vehicle are passable, the driving boundary obtained after connecting the adjacent occupied points is a discontinuous boundary. Figure 5 The driving boundary is obtained by connecting the occupied points shown in (a) in Figure 5As shown in (b), the shaded area outside the boundary is the impassable area, and the rest of the area is the drivable area.

[0135] In one implementation, when connecting adjacent occupied points, the occupied point and its adjacent occupied points may be directly connected in a straight line to obtain a travel boundary.

[0136] In another implementation, it is possible to determine whether other grid points within a certain range near each occupied point are occupied points based on the occupied points on each line of sight, and connect the occupied points determined on each line of sight with other occupied points on non-line of sight determined based on the occupied points on the line of sight to obtain the driving boundary. Figure 5 As shown in (c) in the figure, we can start from each occupied point and detect the occupancy of each grid point whose sight angle changes within ±1° and whose distance from the origin is equal to the distance from the current occupied point. Figure 5 As shown in (c) in the figure, if the grid point within the -1° range of the third occupied point's line of sight and whose distance from the origin is equal to the distance of the third occupied point is not occupied, it is an unoccupied point; if the grid point within the +1° range of the third occupied point's line of sight and whose distance from the origin is equal to the distance of the third occupied point is occupied, it is an occupied point.

[0137] based on Figure 5 The driving boundary obtained by connecting the occupied points on the line of sight shown in (c) and other occupied points on the non-line of sight determined based on the occupied points on the line of sight is as follows Figure 5 As shown in (d) in the figure. This method can determine more occupied points based on the limited occupied points determined by sight without identifying the occupancy status of each grid in the grid, and then use more occupied points to determine the driving boundary, which can improve the accuracy of the drivable area.

[0138] Step S4043, determining the drivable area of ​​the vehicle according to the driving boundary.

[0139] After the driving boundary is determined, the area within the driving boundary is the drivable area.

[0140] In the disclosed embodiment, adjacent occupied points of each occupied point are determined, and a driving boundary is determined based on the occupied point and its adjacent occupied points, so as to achieve rapid and accurate determination of a drivable area.

[0141] Based on the above embodiments, the present disclosure further provides a training method for an image processing model. Fig.11 As shown, in some embodiments, the image processing model in the above embodiment can be trained by the following steps S1101 to S1104:

[0142] Step S1101: determine a sample data set including a plurality of sample data.

[0143] The sample data includes multi-view sample images, true values ​​of line of sight occupancy states of preset lines of sight in the bird's-eye view corresponding to the multi-view sample images, and true values ​​of occupied point distances.

[0144] The sample images can be collected by multiple vehicles. The true value of the line of sight occupancy state and the true value of the occupied point distance can be manually marked information.

[0145] Step S1102, using the initial image processing model to process the multi-view sample images included in the sample data, to obtain a line of sight occupancy state prediction value and an occupied point distance prediction value corresponding to a preset line of sight.

[0146] Construct an initial image processing model. Input the multi-view sample images into the initial image processing model for a series of processing such as feature extraction, transformation, and fusion to obtain a prediction result. The prediction result includes a first prediction result and a second prediction result. The first prediction result can be a prediction value of the line of sight occupancy state corresponding to a preset line of sight, and the second prediction result can be a prediction value of the occupied point distance corresponding to the preset line of sight.

[0147] In one implementation, the multi-view sample images included in the sample data are processed using the initial image processing model to obtain the line of sight occupancy state prediction value and the occupancy point distance prediction value corresponding to the preset line of sight, which can be obtained by Fig.12 Steps S11021 to S11026 shown are used to implement:

[0148] Step S11021, using the first feature extraction sub-model in the initial image processing model to perform feature extraction processing on each sample image in the multi-view sample image, to obtain image features of each sample image.

[0149] Step S11022, using the mapping sub-model in the initial image processing model, the image features of each sample image are mapped to obtain the mapping features of each sample image in the bird's-eye view.

[0150] Step S11023, using the fusion sub-model in the initial image processing model to fuse the mapping features corresponding to each sample image, to obtain a bird's-eye view fusion feature.

[0151] Step S11024, using the second feature extraction sub-model in the initial image processing model to perform feature extraction processing on the bird's-eye view fusion features to obtain bird's-eye view features.

[0152] Step S11025, using the third feature extraction sub-model in the initial image processing model to perform feature extraction processing on the bird's-eye view features to obtain the sight line features of the preset sight line.

[0153] Step S11026, using the classification recognition sub-model in the initial image processing model to process the sight line features, to obtain a recognition result corresponding to the preset sight line.

[0154] Step S11027, using the calculation sub-model in the initial image processing model to process the recognition result corresponding to the preset line of sight, and determine the line of sight occupancy state prediction value and the occupied point distance prediction value of the preset line of sight.

[0155] The constructed initial image processing model may include a first feature extraction sub-model, a mapping sub-model, a fusion sub-model, a second feature extraction sub-model, a third feature extraction sub-model, a classification recognition sub-model and a calculation sub-model. Each sample image in the multi-view sample image is input into the initial image processing model, and image domain feature extraction is performed in sequence, the extracted image features are mapped to the bird's-eye view feature space, feature fusion is performed in the bird's-eye view feature space, feature extraction is performed on the bird's-eye view fusion features to obtain the bird's-eye view features, feature extraction is performed on the bird's-eye view features to obtain the line of sight features, the extracted line of sight features are classified and recognized, and finally the line of sight occupancy state and the occupied point distance are calculated according to the line of sight recognition results.

[0156] In the disclosed embodiment, when processing environmental images of each perspective to determine the fusion features of the bird's-eye view, a medium fusion scheme is adopted: feature extraction and mapping processing are first performed, and then feature fusion is performed in the bird's-eye view feature space. By handing over the processing of the fusion step to the image processing model and performing fusion in a network learning manner, it is helpful to learn features with stronger expressive capabilities, and features with stronger expressive capabilities can improve the accuracy of the detection results.

[0157] Step S1103, determining a loss function according to the true value of the line of sight occupancy state and the predicted value of the line of sight occupancy state, and the true value of the occupied point distance and the predicted value of the occupied point distance.

[0158] Compare the true value of the line of sight occupancy state and the predicted value of the line of sight occupancy state, as well as the true value of the occupied point distance and the predicted value of the occupied point distance, and obtain the loss value between the true value and the predicted value.

[0159] Step S1104, training the initial image processing model based on the loss function to obtain a trained image processing model.

[0160] The parameters of the initial image processing model are adjusted according to the loss value. Repeat the above steps, and when the loss value meets the training end condition, stop training to obtain a trained image processing model.

[0161] In the disclosed embodiment, by training the image processing model, it is possible to determine the occupancy status of each preset sight line in the bird's-eye view and the image distance of the occupied point closest to the origin among the grid points on the sight line in the occupied state, which can reduce the amount of model calculation, greatly reduce redundant information, and improve the accuracy of occupancy detection results.

[0162] Exemplary Devices

[0163] Fig.13 FIG. 1 is a schematic diagram of the structure of a detection device for a drivable area provided by an exemplary embodiment of the present disclosure. Fig.13 As shown, the drivable area detection device 1300 may include:

[0164] The first determination module 1301 is used to determine the multi-view environment image collected by the vehicle;

[0165] The first processing module 1302 is used to process the multi-view environment image to obtain the sight line occupation state and the occupation point distance corresponding to the preset sight line;

[0166] A second determination module 1303 is used to determine the position information of the occupied point in the world coordinate system according to the line of sight occupation state and the occupied point distance;

[0167] The third determination module 1304 is used to determine the drivable area of ​​the vehicle according to the location information of the occupied point.

[0168] See also Fig.14 In some embodiments, the first processing module 1302 may include:

[0169] The first processing unit 13021 is used to process each environment image in the multi-view environment image respectively by using a pre-trained image processing model to obtain a mapping feature of each environment image in the bird's-eye view;

[0170] A fusion unit 13022 is used to use a fusion sub-model in the image processing model to perform fusion processing on the mapping features corresponding to each of the environment images to obtain a bird's-eye view fusion feature;

[0171] The second processing unit 13023 is used to process the bird's-eye view fusion features by using the image processing model to obtain the line of sight occupancy state and occupied point distance of the preset line of sight.

[0172] In some embodiments, the first processing unit 13021 may be specifically configured to:

[0173] Using a first feature extraction sub-model in a pre-trained image processing model, respectively performing feature extraction processing on each environmental image in the multi-view environmental image to obtain image features of each environmental image;

[0174] The image features of each of the environment images are mapped using the mapping sub-model in the image processing model to obtain the mapping features of each of the environment images in the bird's-eye view.

[0175] In some embodiments, the second processing unit 13023 may be specifically configured to:

[0176] Using the second feature extraction sub-model in the image processing model to perform feature extraction processing on the bird's-eye view fusion feature to obtain a bird's-eye view feature;

[0177] Using the third feature extraction sub-model in the image processing model to perform feature extraction processing on the bird's-eye view features to obtain the sight line features of the preset sight line;

[0178] Using the classification recognition sub-model in the image processing model to perform recognition processing on the sight line feature, and obtain a recognition result corresponding to the preset sight line;

[0179] The recognition result corresponding to the preset sight line is processed by using the calculation sub-model in the image processing model to determine the sight line occupancy state and the occupied point distance of the preset sight line.

[0180] See also Fig.14 In some embodiments, the second determining module 1303 may include:

[0181] A first determining unit 13031 is used to determine the image coordinates of the occupied point on the preset line of sight according to the line of sight occupation state and the occupied point distance;

[0182] The second determining unit 13032 is used to determine the horizontal coordinate and the vertical coordinate of each of the occupied points in the world coordinate system according to the image coordinates of each of the occupied points;

[0183] The third determining unit 13033 is used to determine the position information of each of the occupied points according to the horizontal coordinate and the vertical coordinate of each of the occupied points in the world coordinate system.

[0184] See also Fig.14 In some embodiments, the third determining module 1304 may include:

[0185] The fourth determining unit 13041 is used to determine the adjacent occupied points corresponding to each of the occupied points according to the line of sight occupation state corresponding to the preset line of sight;

[0186] A fifth determining unit 13042, configured to determine a driving boundary according to each of the occupied points and its corresponding adjacent occupied points;

[0187] The sixth determining unit 13043 is used to determine the drivable area of ​​the vehicle according to the driving boundary.

[0188] Fig.15 FIG. 1 is a schematic diagram of the structure of a training device for an image processing model provided by an exemplary embodiment of the present disclosure. Fig.15 As shown, the training device 1500 of the image processing model may include:

[0189] A fourth determination module 1501 is used to determine a sample data set including a plurality of sample data, wherein the sample data includes a multi-view sample image, a true value of a sight line occupancy state of a preset sight line in a bird's-eye view corresponding to the multi-view sample image, and a true value of an occupied point distance;

[0190] The second processing module 1502 is used to process the multi-view sample images included in the sample data using the initial image processing model to obtain a line of sight occupancy state prediction value and an occupancy point distance prediction value corresponding to the preset line of sight;

[0191] A fifth determination module 1503, configured to determine a loss function according to the line of sight occupancy state true value and the line of sight occupancy state predicted value, and the occupied point distance true value and the occupied point distance predicted value;

[0192] The training module 1504 is used to train the initial image processing model based on the loss function to obtain a trained image processing model.

[0193] See also Fig.16 In some embodiments, the second processing module 1502 may include:

[0194] A first feature extraction unit 15021 is used to perform feature extraction processing on each sample image in the multi-view sample image by using the first feature extraction sub-model in the initial image processing model to obtain image features of each sample image;

[0195] A mapping processing unit 15022 is used to perform mapping processing on the image features of each of the sample images respectively by using the mapping sub-model in the initial image processing model to obtain the mapping features of each of the sample images in the bird's-eye view feature space;

[0196] A fusion processing unit 15023 is used to perform fusion processing on the mapping features corresponding to each of the sample images using the fusion sub-model in the initial image processing model to obtain a bird's-eye view fusion feature;

[0197] A second feature extraction unit 15024 is used to perform feature extraction processing on the bird's-eye view fusion feature by using the second feature extraction sub-model in the initial image processing model to obtain a bird's-eye view feature;

[0198] A third feature extraction unit 15025 is used to perform feature extraction processing on the bird's-eye view features by using the third feature extraction sub-model in the initial image processing model to obtain the sight line features of the preset sight line;

[0199] The classification and recognition unit 15026 is used to process the sight line feature by using the classification and recognition sub-model in the initial image processing model to obtain a recognition result corresponding to the preset sight line;

[0200] The seventh determination unit 15027 is used to process the recognition result corresponding to the preset line of sight by using the calculation sub-model in the initial image processing model, and determine the line of sight occupancy state prediction value and the occupied point distance prediction value of the preset line of sight.

[0201] The beneficial technical effects corresponding to the exemplary embodiments of the device disclosed herein can be found in the corresponding beneficial technical effects of the above exemplary method section, which will not be repeated here.

[0202] Exemplary Electronic Devices

[0203] Fig.17 is a schematic diagram of the composition structure of an electronic device provided by an exemplary embodiment of the present disclosure, such as Fig.17 As shown, the electronic device 1700 includes at least one processor 1701 and a memory 1702 .

[0204] The processor 1701 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1700 to perform desired functions.

[0205] The memory 1702 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on a computer-readable storage medium, and the processor 1701 may execute one or more computer program instructions to implement the detection method of the drivable area or the training method of the image processing model and / or other desired functions of the various embodiments of the present disclosure described above.

[0206] In one example, the electronic device 1700 may further include: an input device 1703 and an output device 1704 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0207] The input device 1703 may also include, for example, a keyboard, a mouse, etc.

[0208] The output device 1704 can output various information to the outside, and may include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0209] Of course, to simplify, Fig.17 Only some of the components related to the present disclosure in the electronic device 1700 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application scenarios, the electronic device 1700 may also include any other appropriate components.

[0210] Exemplary computer program products and computer-readable storage media

[0211] In addition to the above-mentioned methods and devices, embodiments of the present disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the drivable area detection method or image processing model training method of various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section.

[0212] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0213] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps in the method for detecting the drivable area or the method for training the image processing model of various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section.

[0214] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium is, for example, but not limited to, a system, device or device including electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0215] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and cannot be considered as necessary for each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, rather than limitation, and the above details do not limit the present disclosure to being implemented by adopting the above specific details.

[0216] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present disclosure claims and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A method for detecting a drivable area, comprising: Determine the multi-view environment image collected by the vehicle; Processing the multi-view environment image to obtain a sight line occupancy state and an occupancy point distance corresponding to a preset sight line; Determining the position information of the occupied point in the world coordinate system according to the line of sight occupation state and the distance of the occupied point; The drivable area of ​​the vehicle is determined according to the position information of the occupied point.

2. The method according to claim 1, wherein: The processing of the multi-view environment image to obtain a sight line occupancy state and an occupancy point distance corresponding to a preset sight line includes: Using a pre-trained image processing model to process each of the multi-view environmental images, respectively, to obtain mapping features of each of the environmental images in the bird's-eye view; Using the fusion sub-model in the image processing model to fuse the mapping features corresponding to each of the environment images, to obtain a bird's-eye view fusion feature; The image processing model is used to process the fusion features of the bird's-eye view to obtain the line of sight occupancy state and the occupied point distance of the preset line of sight.

3. The method according to claim 2, wherein: The method of using a pre-trained image processing model to process each of the multi-view environmental images to obtain mapping features of each of the environmental images in the bird's-eye view includes: Using a first feature extraction sub-model in a pre-trained image processing model, respectively performing feature extraction processing on each environmental image in the multi-view environmental image to obtain image features of each environmental image; The image features of each of the environment images are mapped using the mapping sub-model in the image processing model to obtain the mapping features of each of the environment images in the bird's-eye view.

4. The method according to claim 2, wherein: The using the image processing model to process the bird's-eye view fusion features to obtain the sight line occupancy state and the occupied point distance of the preset sight line includes: Using the second feature extraction sub-model in the image processing model to perform feature extraction processing on the bird's-eye view fusion feature to obtain a bird's-eye view feature; Using the third feature extraction sub-model in the image processing model to perform feature extraction processing on the bird's-eye view features to obtain the sight line features of the preset sight line; Using the classification recognition sub-model in the image processing model to perform recognition processing on the sight line feature, and obtain a recognition result corresponding to the preset sight line; The recognition result corresponding to the preset sight line is processed by using the calculation sub-model in the image processing model to determine the sight line occupancy state and the occupied point distance of the preset sight line.

5. The method according to any one of claims 1 to 4, wherein: The determining, according to the line of sight occupation state and the distance to the occupied point, the position information of the occupied point in the world coordinate system comprises: Determining the image coordinates of the occupied point on the preset line of sight according to the line of sight occupation state and the occupied point distance; Determine the horizontal coordinate and the vertical coordinate of each of the occupied points in the world coordinate system according to the image coordinates of each of the occupied points; The position information of each of the occupied points is determined according to the horizontal coordinate and the vertical coordinate of each of the occupied points in the world coordinate system.

6. The method according to any one of claims 1 to 4, wherein: The step of determining the drivable area of ​​the vehicle according to the position information of the occupied point includes: Determining adjacent occupied points corresponding to each of the occupied points according to the line of sight occupation status corresponding to the preset line of sight; Determining a driving boundary according to each of the occupied points and its corresponding adjacent occupied points; A drivable area of ​​the vehicle is determined according to the driving boundary.

7. A method for training an image processing model, comprising: Determine a sample data set including a plurality of sample data, wherein the sample data includes a multi-view sample image, a true value of a sight line occupancy state of a preset sight line in a bird's-eye view corresponding to the multi-view sample image, and a true value of an occupied point distance; Using an initial image processing model to process the multi-view sample images included in the sample data, to obtain a line of sight occupancy state prediction value and an occupancy point distance prediction value corresponding to the preset line of sight; Determine a loss function according to the true value of the line of sight occupancy state and the predicted value of the line of sight occupancy state, and the true value of the occupied point distance and the predicted value of the occupied point distance; The initial image processing model is trained based on the loss function to obtain a trained image processing model.

8. The method according to claim 7, wherein: The using of the initial image processing model to process the multi-view sample images included in the sample data to obtain the sight line occupancy state prediction value and the occupancy point distance prediction value corresponding to the preset sight line includes: Using the first feature extraction submodel in the initial image processing model, respectively perform feature extraction processing on each sample image in the multi-view sample images to obtain image features of each sample image; Using the mapping sub-model in the initial image processing model, mapping processing is performed on the image features of each of the sample images to obtain the mapping features of each of the sample images in the bird's-eye view; Using the fusion sub-model in the initial image processing model to fuse the mapping features corresponding to each of the sample images, to obtain a bird's-eye view fusion feature; Using the second feature extraction sub-model in the initial image processing model to perform feature extraction processing on the bird's-eye view fusion feature to obtain a bird's-eye view feature; Using the third feature extraction sub-model in the initial image processing model to perform feature extraction processing on the bird's-eye view features, to obtain the sight line features of the preset sight line; Processing the sight line feature using the classification recognition sub-model in the initial image processing model to obtain a recognition result corresponding to the preset sight line; The recognition result corresponding to the preset sight line is processed by using the calculation sub-model in the initial image processing model to determine the sight line occupancy state prediction value and the occupied point distance prediction value of the preset sight line.

9. A device for detecting a drivable area, comprising: A first determination module is used to determine a multi-view environment image collected by a vehicle; A first processing module is used to process the multi-view environment image to obtain a sight line occupancy state and an occupancy point distance corresponding to a preset sight line; A second determination module is used to determine the position information of the occupied point in the world coordinate system according to the line of sight occupation state and the occupied point distance; The third determination module is used to determine the drivable area of ​​the vehicle according to the location information of the occupied point.

10. A training device for an image processing model, comprising: A fourth determination module is used to determine a sample data set including a plurality of sample data, wherein the sample data includes a multi-view sample image, a true value of a sight line occupancy state of a preset sight line in a bird's-eye view corresponding to the multi-view sample image, and a true value of an occupied point distance; A second processing module is used to process the multi-view sample images included in the sample data using an initial image processing model to obtain a line of sight occupancy state prediction value and an occupied point distance prediction value corresponding to the preset line of sight; a fifth determination module, configured to determine a loss function according to the true value of the sight line occupancy state and the predicted value of the sight line occupancy state, and the true value of the occupied point distance and the predicted value of the occupied point distance; A training module is used to train the initial image processing model based on the loss function to obtain a trained image processing model.

11. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the method for detecting a drivable area described in any one of claims 1 to 6 or the method for training an image processing model described in any one of claims 7 to 8.

12. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the instructions from the memory and execute the instructions to implement the drivable area detection method described in any one of claims 1-6 or the image processing model training method described in any one of claims 7-8.

Images