Vehicle driving safety protection method, system and device and storage medium

Abstract

The embodiment of the invention provides a vehicle driving safety protection method, system and device and a storage medium. The method comprises the steps of obtaining target image data collected by a target camera; determining contour points of the interested object based on the target image data; determining contour point cloud data of the contour points in the vehicle body coordinate system based on the contour points and target external parameters of the target camera; and determining a safety area and / or a risk area based on the contour point cloud data.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of unmanned driving, and in particular to a vehicle driving safety protection method, system, device and storage medium. BACKGROUND

[0002] In an automatic driving scenario, the environment around the vehicle needs to be judged to ensure the safe driving of the vehicle. Taking port automatic driving as an example, currently, automatic driving trucks usually rely on perception sensors installed on the vehicle head, such as laser radar and cameras, to ensure driving safety. However, these sensors face many challenges when dealing with long and complex driving environments. For example, although the installation of laser radars on both sides can provide a certain range of perception, when the vehicle turns, the angle formed by the vehicle trailer and the vehicle head will cause a blind area on one side of the vehicle trailer. Even in a straight driving state, due to the excessive length of the vehicle trailer, the radar beam will become sparse at the vehicle tail part, and it is also difficult to provide stable safety protection. Although installing a radar on the roof can partially solve this problem, the container cargo will block the top radar, causing the blind area to expand.

[0003] In order to solve the problem of incomplete coverage and still having a blind area of the above-mentioned perception sensor, the present specification provides a vehicle driving safety protection system to realize effective monitoring and safety protection of the vehicle trailer. SUMMARY

[0004] One or more embodiments of the present specification provide a vehicle driving safety protection method, the method comprising: acquiring target image data collected by a target camera; determining contour points of an object of interest based on the target image data; determining contour point cloud data of the contour points in a vehicle body coordinate system based on the contour points and target extrinsic parameters of the target camera; and determining a safety area and / or a risk area based on the contour point cloud data.

[0005] One or more embodiments of the present specification provide a vehicle driving safety protection system, the system comprising: an image data acquisition module configured to acquire target image data collected by a target camera; a contour point determination module configured to determine contour points of an object of interest based on the target image data; a point cloud data determination module configured to determine contour point cloud data of the contour points in a vehicle body coordinate system based on the contour points and target extrinsic parameters of the target camera; and an area determination module configured to determine a safety area and / or a risk area based on the contour point cloud data.

[0006] One or more embodiments of the present specification provide a vehicle driving safety protection device, the device comprising at least one memory and at least one processor, the at least one memory being configured to store computer instructions, and the at least one processor being configured to execute the computer instructions or part of the instructions to implement the vehicle driving safety protection method.

[0007] One or more embodiments of the present specification provide a computer readable storage medium, which stores computer instructions, when a computer reads the computer instructions in the storage medium, the computer executes the vehicle driving safety protection method. BRIEF DESCRIPTION OF DRAWINGS

[0008] The present specification will be further described in the manner of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same reference numbers denote the same structures, in which:

[0009] Figure 1 is an exemplary block diagram of a vehicle driving safety protection system according to some embodiments of the present specification;

[0010] Figure 2 is an exemplary flowchart of a vehicle driving safety protection method according to some embodiments of the present specification;

[0011] Figure 3 is an exemplary schematic diagram of determining a safety area according to some embodiments of the present specification;

[0012] Figure 4 is an exemplary schematic diagram of determining a risk area according to some embodiments of the present specification;

[0013] Figure 5 a in is an exemplary schematic diagram of selecting a target camera based on a vehicle hitch posture according to some embodiments of the present specification;

[0014] Figure 5 b in is another exemplary schematic diagram of selecting a target camera based on a vehicle hitch posture according to some embodiments of the present specification;

[0015] Figure 5 c in is another exemplary schematic diagram of selecting a target camera based on a vehicle hitch posture according to some embodiments of the present specification;

[0016] Figure 6 is an exemplary schematic diagram of data of a vehicle hitch tail according to some embodiments of the present specification;

[0017] Figure 7 is an exemplary schematic diagram of a safety area and a risk area according to some embodiments of the present specification. DETAILED DESCRIPTION

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present specification, the drawings needed to be used in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some examples or embodiments of the present specification, and for those skilled in the art, the present specification can also be applied to other similar scenarios without creative labor on the basis of these drawings. Unless it is clear from the language environment or otherwise stated, the same reference numbers in the drawings represent the same structures or operations.

[0019] It should be understood that the "system", "device", "unit" and / or "module" used herein is a method for distinguishing different components, elements, parts, sections or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0020] As shown in the specification and claims, unless the context clearly indicates otherwise, the words "one", "a", "an", and / or "the" do not refer to the singular, but can also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements explicitly identified, and these steps and elements do not constitute an exclusive list, and the method or device can also include other steps or elements.

[0021] Flowcharts are used in the present specification to illustrate the operations performed by the system according to the embodiments of the present specification. It should be understood that the preceding or subsequent operations are not necessarily performed in sequence. On the contrary, each step can be processed in reverse order or simultaneously. At the same time, other operations can also be added to these processes, or one or more steps of the operation can be removed from these processes.

[0022] Safe driving of an automatic driving truck is an important part of safe driving in a port. At present, the safety protection of the automatic driving truck is often realized through sensors. However, the existing sensors, including infrared sensors, ultrasonic sensors, millimeter wave sensors, and laser radars, all have their inherent limitations, and cannot realize effective monitoring and safety protection of the truck.

[0023] Some embodiments of the present specification provide a vehicle driving safety protection system, which sets available cameras at different positions of a truck, dynamically enables / closes the corresponding available cameras as needed during vehicle driving, so that the images obtained by the available cameras can fully reflect the surrounding environment, and effectively realize safety protection.

[0024] Figure 1 is an exemplary block diagram of the vehicle driving safety protection system according to some embodiments of the present specification.

[0025] In some embodiments, as Figure 1As shown, the vehicle driving safety protection system 100 can include an image data acquisition module 110, a contour point determination module 120, a point cloud data determination module 130, and a region determination module 140.

[0026] The image data acquisition module 110 refers to a module for acquiring relevant image data. In some embodiments, the image data acquisition module 110 can be configured to acquire target image data captured by a target camera.

[0027] The contour point determination module 120 refers to a module for determining contour points. In some embodiments, the contour point determination module 120 can be configured to determine contour points of an object of interest based on target image data.

[0028] The point cloud data determination module 130 refers to a module for determining point cloud data. In some embodiments, the point cloud data determination module 130 can be configured to determine contour point cloud data of contour points in a vehicle body coordinate system based on contour points and target extrinsic parameters of a target camera.

[0029] The region determination module 140 refers to a module for determining regions. In some embodiments, the region determination module 140 can be configured to determine a safety region and / or a risk region based on contour point cloud data.

[0030] In some embodiments, the vehicle driving safety protection system 100 can further include a processor. The processor can be used to process data related to the vehicle driving safety protection system 100. One or more of the image data acquisition module 110, the contour point determination module 120, the point cloud data determination module 130, and the region determination module 140 can be integrated into the processor.

[0031] In some embodiments, the processor can be a single server or a group of servers. The group of servers can be centralized or distributed. In some embodiments, the processor can be local or remote. In some embodiments, the processor can be implemented on a cloud platform. By way of example only, the cloud platform can include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an on-premises cloud, a multi-layer cloud, or the like, or any combination thereof.

[0032] More on the image data acquisition module 110, the contour point determination module 120, the point cloud data determination module 130, and the region determination module 140 can be found in the related description of Figures 2-7 .

[0033] It should be noted that the above description of the vehicle driving safety protection system 100 and its modules is for the convenience of description, and cannot limit the scope of the present specification to the embodiments. It can be understood that, for those skilled in the art, after understanding the principles of the system, any combination of the modules or connection of the subsystems and other modules can be made without departing from the principles. In some embodiments, Figure 1 The image data acquisition module 110, the contour point determination module 120, the point cloud data determination module 130, and the region determination module 140 disclosed in the above embodiments can be different modules in a system, or one module can implement the functions of two or more modules. For example, the modules can share one storage module, or each module can have its own storage module. Variations such as these are within the scope of the present specification.

[0034] Figure 2 is an exemplary flowchart of a vehicle driving safety protection method according to some embodiments of the present specification. In some embodiments, the flowchart 200 can be executed by a processor. As shown in Figure 2 , the flowchart 200 includes the following steps.

[0035] Step 210, acquiring target image data collected by a target camera.

[0036] The target camera refers to a selected camera for acquiring target image data. The target camera can include a surround-view camera, a multi-view camera, etc.

[0037] The target image data refers to an image of a region of interest acquired by the target camera. For example, Figure 6 the image shown in the lower left corner.

[0038] The region of interest refers to a blind area of the vehicle driving. The vehicle can include but is not limited to a truck, etc. In some embodiments, the region of interest can refer to a blind area of the truck during automatic driving or assisted driving due to the length of the trailer, the linkage of the trailer and the head, etc.

[0039] The trailer of the truck refers to the part connected to the rear of the head for loading containers. The length of the trailer is relatively long, and the tail of the trailer will form a blind area when the vehicle is reversing. The trailer and the head are linked by a hinge, which allows the trailer to have a certain degree of rotational freedom relative to the head. When the vehicle turns, the angle formed by the trailer and the head will cause a blind area on one side of the trailer.

[0040] In some embodiments, the target camera can be dynamically selected from at least one available camera based on the trailer pose of the trailer by the processor.

[0041] The hitch posture refers to state data that can represent the hitch posture such as the hitch translation deflection, etc. For example, the hitch posture can include the hitch deflection direction relative to the vehicle head, the hitch travel direction, etc. Among them, the hitch deflection direction relative to the vehicle head can include the hitch swinging on the left side of the vehicle head, the hitch being located directly behind the vehicle head, and the hitch swinging on the right side of the vehicle head. The hitch travel direction can include the hitch advancing (the vehicle advancing) and the hitch retreating (the vehicle reversing).

[0042] In some embodiments, the hitch posture can be obtained by means of integrated sensors (such as rotation angle sensors and displacement sensors) or preset algorithms built in the system (such as the self-hitch detection program), etc.

[0043] The available camera refers to the camera for the candidate target camera. In some embodiments, at least one available camera can be installed at at least one preset position of the hitch.

[0044] The preset position refers to the installation position of the available camera on the hitch which is set in advance.

[0045] In some embodiments, the at least one preset position can include the hitch body left side, the hitch body right side, and the hitch tail.

[0046] In some embodiments, the processor can dynamically adjust the on-off state of the available camera based on the hitch posture of the hitch.

[0047] The on-off state refers to the on state or the off state of the available camera. Among them, the available camera in the on state is the target camera.

[0048] In some embodiments, the processor can open the corresponding available camera according to the hitch posture. For example, the processor can open the available camera on the hitch body right side as the target camera to supplement the right side blind area (as shown in FIG. a of Figure 5 ) in response to the hitch swinging on the left side of the vehicle head according to the hitch deflection direction relative to the vehicle head and the hitch travel direction; open the available camera on the hitch tail as the target camera to supplement the tail blind area (as shown in FIG. b of Figure 5 ) in response to the hitch retreating; and open the available camera on the hitch body left side as the target camera to supplement the left side blind area (as shown in FIG. c of Figure 5 ) in response to the hitch swinging on the right side of the vehicle head.

[0049] The above various cases can be combined, for example, when the vehicle is reversing and turning, the hitch is retreating and the hitch is swinging on the left side of the vehicle head, the processor can open the available cameras on the hitch tail and the hitch body right side as the target cameras to supplement the tail and right side blind areas.

[0050] In some embodiments of the present disclosure, the on / off state of the available cameras can be dynamically adjusted in real time according to the vehicle hitch posture, so as to ensure that the target image data acquired by the target camera is reliable and effective.

[0051] In some embodiments, the mounting distance between any two of the at least one available camera can be not less than a preset distance threshold, and / or the angle between the image acquisition directions of any two of the at least one available camera can be not less than a preset angle threshold.

[0052] The mounting distance refers to the interval distance between the mounting positions of the available cameras. For example, the straight-line distance between the center points of the available cameras.

[0053] The preset distance threshold refers to the minimum value of the mounting distance.

[0054] The image acquisition direction refers to the orientation direction of the available camera. For example, the image acquisition direction is represented by the center line of the camera.

[0055] The preset angle threshold refers to the minimum value of the angle between the image acquisition directions.

[0056] In some embodiments, the preset distance threshold and the preset angle threshold can be preset by a technician or a processor based on relevant parameters (such as the type of vehicle, the size of vehicle, the model of camera, etc.), so as to ensure that the shooting range of the available cameras can cover all the regions of interest, and the overlap of the regions of interest covered by each available camera is small, thereby avoiding resource waste.

[0057] In some embodiments of the present disclosure, by limiting the mounting distance between the available cameras and the angle between the image acquisition directions, it can be ensured that the image data of all the regions of interest can be acquired, while the cost of the cameras is reduced.

[0058] In some embodiments, the at least one available camera can be an array camera comprising a plurality of camera units.

[0059] The array camera refers to a camera composed of a plurality of small camera units. In some embodiments, the target camera can be a camera unit in the on state.

[0060] In some embodiments, the processor can dynamically determine the on-off state of each camera unit based on the trailer posture. For example, when the vehicle is making a left turn, the trailer is on the left side of the vehicle head, the processor can turn on the partial or all camera units on the right side of the trailer body as target cameras to supplement the right blind area. Since the deflection angle of the trailer relative to the vehicle head dynamically changes during the turning process, the processor can dynamically adjust the on-off state of the camera units on the right side of the trailer body according to the real-time deflection angle of the trailer relative to the vehicle head, so that the target image data collected by the camera units in the on state (i.e., target cameras) can cover the right blind area.

[0061] In some embodiments of the present specification, when the available camera is an array camera including multiple camera units, dynamically determining the on-off state of each camera unit based on the trailer posture can accurately regulate the available camera and change the target camera in real time, so that the target image data obtained by the target camera is more accurate.

[0062] In some embodiments, the processor can also determine a predicted target camera at at least one future time point through a prediction model based on vehicle motion data and environment data.

[0063] The vehicle motion data refers to data related to the motion of the vehicle. For example, the navigation route, the traveled route, the vehicle speed, the current position of the vehicle, etc.

[0064] The environment data refers to data related to the environment in which the vehicle is located. For example, the weather in the region where the vehicle is located, the current time point, etc. In some embodiments, the processor can obtain the environment data from a third-party platform (e.g., a weather forecast APP, a world clock, etc.).

[0065] The predicted target camera refers to the target camera to be used at the predicted future time point.

[0066] The prediction model refers to a model used to determine the predicted target camera. In some embodiments, the prediction model can be a machine learning model. For example, the prediction model can include a deep neural network (DNN) model, etc.

[0067] In some embodiments, the input of the prediction model can include vehicle motion data and environment data, and the output can be a predicted target camera at at least one future time point.

[0068] In some embodiments, the prediction model can be trained based on a large number of labeled training samples. The training samples can include sample vehicle motion data and sample environment data of a sample vehicle at a first sample time point. The label corresponding to the training sample can be the actual target camera adopted by the sample vehicle at a second sample time point, where the second sample time point is a future time point of the first sample time point, i.e., the first sample time point is before the second sample time point.

[0069] In some embodiments, the processor can input a plurality of labeled training samples into an initial prediction model, construct a loss function based on the labels and the output results of the initial prediction model, and iteratively update the initial prediction model based on the loss function. The model training is completed when the loss function meets a preset condition, and a trained prediction model is obtained. The preset condition can be that the loss function converges, the number of iterations reaches a threshold, etc.

[0070] In some embodiments of the present specification, through the prediction model, the predicted target camera at the future time point can be automatically and accurately obtained based on the vehicle motion data and the environment data, which facilitates the early start of the camera for target image data collection, and improves the work efficiency and the accuracy of data analysis.

[0071] In some embodiments of the present specification, based on the vehicle hitch posture, the target camera can be dynamically selected from the available cameras installed at the preset positions, so that the selected target camera can accurately obtain target image data meeting the actual requirements, which is beneficial to realize the safety protection of vehicle driving.

[0072] Step 220, determining the contour points of the object of interest based on the target image data.

[0073] The object of interest refers to the object that needs to be concerned in the target image data. For example, a person, a vehicle, a road, a container, a lock station, etc.

[0074] In some embodiments, the object of interest can be determined in various ways. For example, the processor can determine the object of interest based on the target image data through a semantic segmentation model.

[0075] The semantic segmentation model refers to a model for classifying image data. For example, a fully convolutional network (FCN) and the like.

[0076] In some embodiments, the semantic segmentation model can identify the object of interest in the target image data and assign the object of interest to a corresponding category, one category corresponding to one type of object of interest, to obtain a corresponding semantic segmentation result. The semantic segmentation result can be represented by a semantic segmentation map. For example, Figure 6A semantic segmentation map of the target image data shown in the top left corner, different colors represent different categories.

[0077] A contour point refers to a point constituting the contour edge of an object. In some embodiments, a contour point can refer to a point corresponding to the contour of an object of interest in a pixel coordinate system.

[0078] The pixel coordinate system can refer to a plane coordinate system established with the top left corner of the image as the origin, the right direction along the horizontal direction of the image as the positive direction of the U axis, and the downward direction along the vertical direction of the image as the positive direction of the V axis. For example, the contour point coordinates can be represented by (u, v), where u represents the horizontal coordinate in the pixel coordinate system, and v represents the vertical coordinate in the pixel coordinate system.

[0079] In some embodiments, the processor can determine the contour points of the object of interest based on the target image data by various methods. For example, the processor can determine the contour points of the object of interest based on the target image data by an image processing algorithm. For example only, the image processing algorithm can be an edge detection algorithm, and the processor can extract the edge information of the region of interest by the edge detection algorithm to obtain the contour points of the object of interest. For another example, the processor can determine the continuous boundary of the same category region based on the semantic segmentation result to obtain the contour points of the object of interest.

[0080] In step 230, contour point cloud data of the contour points in the vehicle body coordinate system is determined based on the contour points and target camera target extrinsic parameters.

[0081] The target extrinsic parameters refer to parameters that can represent the conversion relationship between the camera coordinate system and the vehicle body coordinate system. In some embodiments, the target extrinsic parameters can be represented by a rotation matrix and a translation vector, denoted as [R cv t cv ], R cv , t cv represent the rotation matrix and the translation vector, respectively.

[0082] The vehicle body coordinate system refers to the coordinate system of the vehicle itself. In some embodiments, the vehicle body coordinate system can be a dynamic coordinate system. For example, the vehicle body coordinate system can be a three-dimensional rectangular coordinate system with the center of the rear axle of the trailer as the origin O v , with the direction pointing to the front of the trailer as the positive direction of the X v axis, the direction pointing to the left side of the trailer as the positive direction of the Y v axis, and the direction pointing to the top of the trailer as the positive direction of the Z v axis. The three-dimensional rectangular coordinate system is constructed. Among them, the center of the rear axle of the trailer can refer to the midpoint of the rear axle of the trailer.

[0083] The camera coordinate system refers to the coordinate system of the camera itself. In some embodiments, the camera coordinate system can be a dynamic coordinate system. For example, the camera coordinate system can be a three-dimensional rectangular coordinate system with the camera optical center as the origin O cX-axis positive direction, right along the horizontal direction of the image plane as Y-axis positive direction, down along the vertical direction of the image plane as Z-axis positive direction, pointing outward in the direction parallel to the camera optical axis c X-axis positive direction, right along the horizontal direction of the image plane as Y-axis positive direction, down along the vertical direction of the image plane as Z-axis positive direction, pointing outward in the direction parallel to the camera optical axis c X-axis positive direction, right along the horizontal direction of the image plane as Y-axis positive direction, down along the vertical direction of the image plane as Z-axis positive direction, pointing outward in the direction parallel to the camera optical axis c X-axis positive direction, right along the horizontal direction of the image plane as Y-axis positive direction, down along the vertical direction of the image plane as Z-axis positive direction, pointing outward in the direction parallel to the camera optical axis

[0084] In some embodiments, the processor can determine the target extrinsic parameter of the target camera in multiple ways. For example, the processor can calibrate the camera coordinate system and the vehicle body coordinate system through a checkerboard calibration board to obtain the target extrinsic parameter. For another example, the processor can construct a rotation matrix based on the rotation angles of the corresponding coordinate axes in the camera coordinate system and the vehicle body coordinate system, determine the vector between the coordinate origins of the camera coordinate system and the vehicle body coordinate system as a translation vector, and then determine the rotation matrix and the translation vector from the camera coordinate system to the vehicle body coordinate system as the target extrinsic parameter.

[0085] Since the influence of the object of interest on the vehicle driving can be represented by the projection data of the object of interest on the horizontal plane, in the subsequent process, the processor can only consider the relevant data of the horizontal plane, i.e., only consider the projection data of the object of interest on the X v O v Y v plane in the vehicle body coordinate system.

[0086] The contour point cloud data refers to the point cloud data of multiple contour points in the vehicle body coordinate system, as shown in the right side of FIG. Figure 6 In some embodiments, the contour point cloud data can be represented by the position coordinates of the contour points relative to the center of the vehicle body rear axle, i.e., the projection coordinates of the contour points on the X v O v Y v plane in the vehicle body coordinate system. For example, the contour point cloud data corresponding to one contour point can be represented by (x v ,y v ), where x v represents the horizontal coordinate in the vehicle body coordinate system, and y v represents the vertical coordinate in the vehicle body coordinate system.

[0087] In some embodiments, the processor can determine the contour point cloud data of the contour points in the vehicle body coordinate system based on the contour points and the target extrinsic parameter of the target camera. For example, the processor can convert the contour points from the pixel coordinate system to the vehicle body coordinate system through the target extrinsic parameter and the camera intrinsic parameter of the target camera to obtain the contour point cloud data. For example, the contour point cloud data can be calculated by the following formula (1):

[0088] wherein (u, v) represents the coordinates of the contour points in the pixel coordinate system, K is the camera intrinsic parameter, [R cv t cv ] represents the target extrinsic parameter of the target camera, [] col:1,2,4 represents the 1st, 2nd and 4th columns of the matrix, λ is a scalar in the calculation process, and (x v ,y v ) represents the contour point cloud data in the vehicle body coordinate system.

[0089] In step 240, the safety area and / or the risk area are determined based on the contour point cloud data.

[0090] The safety area refers to an area in which the vehicle can safely travel.

[0091] In some embodiments, the processor can determine the safety area based on the contour point cloud data in various ways. For example, the processor can determine an area with no point cloud or a very sparse point cloud as the safety area.

[0092] In some embodiments, the processor can determine first point cloud data of the contour point cloud data in the polar coordinate system based on the first conversion relationship between the vehicle body coordinate system and the polar coordinate system, filter the first point cloud data to determine candidate point cloud data in the polar coordinate system, and determine the safety area based on second point cloud data of the candidate point cloud data in the Cartesian coordinate system. The relevant description can be referred to in the related description of Figure 3 .

[0093] The risk area refers to an area in which there is a risk of vehicle travel.

[0094] In some embodiments, the processor can determine the risk area based on the contour point cloud data in various ways. For example, the processor can determine an area with a relatively dense point cloud as the risk area.

[0095] In some embodiments, the processor can determine third point cloud data of the contour point cloud data in the world coordinate system based on the third conversion relationship between the vehicle body coordinate system and the world coordinate system, and determine the risk area in combination with at least one frame of historical point cloud data. The relevant description can be referred to in the related description of Figure 4 .

[0096] In some embodiments of the present specification, by acquiring target image information around the vehicle in real time, extracting contour information of the object of interest from the target image information, and then determining contour point cloud data according to the conversion relationship between the coordinate systems, the safety area and the risk area around the vehicle can be accurately determined, which is conducive to timely discovering the risk of travel, so as to make a reaction such as avoidance or deceleration in advance, and greatly improve the safety of vehicle travel.

[0097] It should be noted that the above description of the flow 200 is merely for example and illustration, and does not limit the scope of the present specification. Various modifications and changes can be made to the flow 200 by those skilled in the art under the guidance of the present specification. However, these modifications and changes are still within the scope of the present specification.

[0098] Figure 3 is an exemplary schematic diagram of determining a safety area according to some embodiments of the present specification.

[0099] In some embodiments, as shown in Figure 3 , the processor can determine, based on a first conversion relationship 303 between the vehicle body coordinate system 301 and the polar coordinate system 302, first point cloud data 305 of the contour point cloud data 304 in the polar coordinate system; and determine, based on the first point cloud data 305, the safety area 306.

[0100] For more information about the vehicle body coordinate system, the contour point cloud data, and the safety area, please refer to Figure 2 and related descriptions thereof.

[0101] In some embodiments, the polar coordinate system can refer to a coordinate system established with the center of the trailer rear axle as the polar point, and the direction (i.e., the X v axis) pointing to the front of the trailer as the polar axis direction.

[0102] The first conversion relationship refers to the coordinate change relationship between the vehicle body coordinate system and the polar coordinate system. In some embodiments, the first conversion relationship can be represented by a direct polar coordinate conversion formula.

[0103] The first point cloud data refers to the point cloud data of the contour point cloud data in the polar coordinate system. In some embodiments, the first point cloud data can be represented by the position coordinates of the contour point cloud data in the polar coordinate. For example, the first point cloud data corresponding to one contour point can be represented by (r, θ), where r represents the polar radius and θ represents the polar angle.

[0104] In some embodiments, the processor can determine the first point cloud data based on the first conversion relationship. Exemplarily, the first point cloud data can be determined by the following formula (2) and formula (3): θ = tan -1 (y v / x v ) (3)

[0105] where (r, θ) represents the first point cloud data corresponding to the contour point, r represents the polar radius, θ represents the polar angle, and (x v ,y v ) represents the contour point cloud data corresponding to the contour point.

[0106] In some embodiments, the processor can determine the safety area based on the first point cloud data in multiple ways. For example, the processor can determine the safety area of the historical first point cloud data similar to the point cloud distribution of the first point cloud data in the historical data as the safety area.

[0107] In some embodiments, as shown in FIG. 3, the processor can filter the first point cloud data 305 based on a preset angle 307 to determine candidate point cloud data 308 in a polar coordinate system 302; determine second point cloud data 311 of the candidate point cloud data 308 in a Cartesian coordinate system 309 based on a second conversion relationship 310 between the polar coordinate system 302 and the Cartesian coordinate system 309; and determine the safety area 306 based on the second point cloud data 311. Figure 3

[0108] The preset angle refers to the interval angle of the polar angle when filtering the first point cloud data. For example, 1°.

[0109] The candidate point cloud data refers to the first point cloud data after filtering.

[0110] In some embodiments, the processor can divide the polar coordinate system according to the preset angle, and each interval of the preset angle is an polar angle range. Only the first point cloud data with the smallest polar radius in each polar angle range is retained to form the candidate point cloud data. For example, the preset angle is 1°, the point with the smallest polar radius in the first point cloud data with the polar angle of 0° to 1° is retained, the point with the smallest polar radius in the first point cloud data with the polar angle of 1° to 2° is retained, and so on. The retained points form the candidate point cloud data.

[0111] In some embodiments, the processor can take the center of the rear axle of the trailer as the origin O v , take the direction pointing to the front of the trailer as the X v axis positive direction, take the direction pointing to the left side of the trailer as the Y v axis positive direction, and construct a two-dimensional rectangular coordinate system as the Cartesian coordinate system. That is, the Cartesian coordinate system is the trailer body coordinate system without the Z v axis.

[0112] The second conversion relationship refers to the conversion relationship between the polar coordinate system and the Cartesian coordinate system. In some embodiments, the second conversion relationship can be represented by a polar-rectangular coordinate conversion formula.

[0113] The second point cloud data refers to the point cloud data of the candidate point cloud data in the Cartesian coordinate system. In some embodiments, the second point cloud data can be represented by the position coordinates of the candidate point cloud data in the Cartesian coordinate system. For example, the second point cloud data corresponding to a contour point can be represented by (x v ’, y v ​') indicates that x v ' represents the x-coordinate in the Cartesian coordinate system, y v ' represents the ordinate in the Cartesian coordinate system.

[0114] In some embodiments, the processor may determine the second point cloud data based on a second transformation relationship. For example, the second point cloud data may be determined using the following formulas (4) and (5): x v ′ =r×cos(θ) (4) y v ′ =r×sin(θ) (5)

[0115] Among them, (x v ′ ,y v ′ (r,θ) represents the second point cloud data corresponding to the contour point, and (r,θ) represents the first point cloud data corresponding to the contour point.

[0116] In some embodiments, the processor can connect the points in the second point cloud data sequentially, and the area within the connecting lines is the safe area.

[0117] In some embodiments of this specification, by filtering the first point cloud data in polar coordinates, the number of contour points can be effectively reduced while accurately retaining key contour points that can characterize the location of the nearest obstacle to the vehicle. Then, the vehicle can switch back to Cartesian coordinates to determine the safe area, making the safe area more intuitive and helping the vehicle to intuitively judge the surrounding environment and respond to risks in a timely manner.

[0118] In some embodiments of this specification, the contour point cloud data is converted to polar coordinates, making the subsequent screening of the first point cloud data more convenient and intuitive, which is beneficial for determining the safe area.

[0119] Figure 4 This is an exemplary schematic diagram illustrating the determination of risk areas according to some embodiments of this specification.

[0120] In some embodiments, such as Figure 4 As shown, the processor can determine the third point cloud data 403 of the contour point cloud data 304 in the world coordinate system based on the third transformation relationship 402 between the vehicle coordinate system 301 and the world coordinate system 401; and determine the risk area 405 based on the third point cloud data 403 and at least one frame of historical point cloud data 404.

[0121] For more information on vehicle coordinate systems and contour point cloud data, please refer to [link / reference]. Figure 2 Related descriptions.

[0122] The world coordinate system refers to a coordinate system for describing the absolute position of an object. In some embodiments, the world coordinate system can be a fixed two-dimensional coordinate system. For example, the Universal Transverse Mercator Grid System (UTM coordinate system) and the like.

[0123] The third conversion relationship refers to the coordinate conversion relationship between the vehicle body coordinate system and the world coordinate system. In some embodiments, the third conversion relationship can be represented by a rotation matrix and a translation vector.

[0124] In some embodiments, the processor can determine the third conversion relationship in various ways. For example, the processor can construct a rotation matrix based on the rotation angle of the corresponding coordinate axis from the vehicle body coordinate system to the world coordinate system, determine the vector between the coordinate origins of the vehicle body coordinate system and the world coordinate system as a translation vector, and then determine the rotation matrix and the translation vector from the vehicle body coordinate system to the world coordinate system as the third conversion relationship.

[0125] The third point cloud data refers to the point cloud data of the contour point cloud data in the world coordinate system. In some embodiments, the third point cloud data can be represented by the position coordinates of the contour point cloud data in the world coordinate system.

[0126] In some embodiments, the processor can convert the contour point cloud data from the vehicle body coordinate system to the world coordinate system based on the third conversion relationship to determine the third point cloud data.

[0127] The historical point cloud data refers to the point cloud data of the historical image data of the historical frame in the world coordinate system. The historical frame can refer to a frame before the current time. For example, the third point cloud data corresponds to the current time t, and at least one frame of historical point cloud data can correspond to t-1, t-2, …, t-2, which is before t-1.

[0128] In some embodiments, the processor can determine the risk area in various ways based on the third point cloud data and at least one frame of historical point cloud data. For example, the processor can compare the third point cloud data and at least one frame of historical point cloud data in the same world coordinate system, and determine the area that has changed as the risk area.

[0129] In some embodiments, as shown in FIG. 4, the processor can also determine the boundary movement direction 406 of the preset boundary of the obstacle based on the third point cloud data 403 and at least one frame of historical point cloud data 404, and determine the risk area 405 based on the vehicle motion direction 407 and the boundary movement direction 406. Figure 4

[0130] For more information about the risk area, please refer to the related description of Figure 2

[0131] ​​Obstacle refers to an object that affects the safe driving of the vehicle. For example, a person, another vehicle, etc.

[0132] Pre-set boundary refers to a boundary that can represent the position of the obstacle and is pre-set. For example, a boundary formed by the profile of one side of the obstacle.

[0133] Obstacle boundary refers to the edge line of the obstacle. In some embodiments, the obstacle boundary can be extracted from the point cloud data.

[0134] In some embodiments, the pre-set boundary can be the obstacle boundary that satisfies the pre-set distance condition with the vehicle motion direction.

[0135] Vehicle motion direction refers to the motion direction of the vehicle. For example, the vehicle is driving straight east, and the vehicle motion direction is a straight line east.

[0136] Pre-set distance condition refers to a pre-set distance condition between the obstacle and the vehicle motion direction. In some embodiments, the pre-set distance condition can be relatively close. Wherein, the relatively close distance represents that the distance between the pre-set boundary and the vehicle motion direction is closer than the distance between other boundaries of the obstacle and the vehicle motion direction, that is, the obstacle boundary that satisfies the pre-set distance condition is the obstacle boundary closer to the side of the vehicle motion direction.

[0137] In some embodiments of the present specification, by setting the pre-set distance condition, the appropriate pre-set boundary can be determined, which is beneficial to the determination of the boundary movement direction.

[0138] Boundary movement direction refers to the movement direction of the pre-set boundary. For example, the boundary movement direction can include approaching the vehicle motion direction, moving away from the vehicle motion direction, etc.

[0139] In some embodiments, the processor can observe the position changes of the plurality of pre-set boundaries over time based on the plurality of pre-set boundaries corresponding to the third point cloud data and at least one frame of historical point cloud data, and then determine the boundary movement direction. As shown in Figure 7 The blue straight line represents the pre-set boundary, the yellow line area represents the safe area, and the green ray represents the vehicle motion direction. The three pre-set boundaries from left to right are the pre-set boundaries at time t, t-1 and t-2, respectively. It can be determined that the boundary movement direction of the pre-set boundary is the direction represented by the blue ray, that is, approaching the vehicle motion direction.

[0140] In some embodiments, in response to the boundary movement direction being close to the vehicle motion direction, the processor can perform dynamic risk prompting and alarming.

[0141] In some embodiments, when the boundary movement direction is close to the vehicle motion direction (as shown in Figure 7When the boundary of the obstacle approaches the vehicle body, the processor can dynamically prompt and alarm the risk in various ways. For example, the processor can send a warning instruction to the vehicle to dynamically prompt and alarm the risk, and the vehicle can autonomously perform an avoidance operation (e.g., deceleration, parking, lane changing, etc.) in response to receiving the warning instruction.

[0142] In some embodiments of the present specification, when the boundary of the obstacle approaches the vehicle body, dynamically prompting and alarming the risk can give the vehicle time to make an avoidance response in advance to ensure safe driving of the vehicle.

[0143] In some embodiments, the processor can determine the risk area in various ways based on the vehicle body movement direction and the boundary movement direction. For example, the processor can determine the area around the intersection of the vehicle body movement direction and the boundary movement direction as the risk area.

[0144] In some embodiments of the present specification, considering the boundary movement direction of the obstacle can effectively reflect the movement of the obstacle, and accurately determining the risk area based on the vehicle body movement direction and the boundary movement direction can timely make a warning and effectively reduce the probability of a safety accident.

[0145] In some embodiments of the present specification, by converting the point cloud data to the world coordinate system, the map data, GPS positioning, and other information can be more accurately fused, thereby improving driving safety, and by combining historical point cloud data, the movement of the obstacle can be accurately reflected, which is beneficial to the determination of the risk area.

[0146] The above has described the basic concepts, and it is obvious that the above detailed disclosure is only an example for those skilled in the art, and does not limit the present specification. Although it is not explicitly stated here, those skilled in the art can make various modifications, improvements and corrections to the present specification. Such modifications, improvements and corrections are suggested in the present specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of the present specification.

[0147] Meanwhile, the present specification uses specific words to describe the embodiments of the present specification. As "one embodiment", "an embodiment", and / or "some embodiments" means a certain feature, structure or characteristic related to at least one embodiment of the present specification. Therefore, it should be emphasized and noted that the "an embodiment" or "one embodiment" or "one alternative embodiment" mentioned in different places in the present specification does not necessarily refer to the same embodiment. In addition, some features, structures or characteristics in one or more embodiments of the present specification can be properly combined.

[0148] Furthermore, the order of the processing elements and sequences described in this specification are not intended to be construed as a limitation, unless specifically stated, but are included to provide a complete description of one or more embodiments of the present specification. Regardless of the particular sequence of processing elements and sequences, however, the description herein of a process should be understood to include any and all combinations of one or more elements, and sequences that can be perceived as either open-ended or specific.

[0149] Similarly, it is to be noticed that the term "comprising", used in the description, should not be interpreted as limited to the means listed thereafter; it does not exclude other elements or steps. It is thus to be interpreted as specifying the presence of the stated features, integers, steps or components as referred to, but does not preclude the presence or addition of one or more other features, integers, steps or components, or groups thereof. Furthermore, to the extent that the term "comprising" is used in either the detailed description or the claims, such term is intended to be interpreted in the same manner as the term "including" and / or "constaining" to such extent.

[0150] Some embodiments use numerals to describe components, quantities of attributes. It should be understood that such numerals used in the description of embodiments are, in some examples, modified by the adjectives "about", "approximately", or "substantially". Unless otherwise stated, "about", "approximately", or "substantially" indicates that the stated numerical value is allowed ±20% variation. Accordingly, in some embodiments, numerical parameters in the specification and claims are approximations, and can vary depending on the desired characteristics set for each individual embodiment. In some embodiments, numerical parameters should be considered in the context of the number of significant digits and errors associated with them. Although the numerical ranges and parameters setting forth the broadest scope of some embodiments of the specification are approximations, in specific embodiments, these numerical values are set forth with a degree of precision that is not considered to be approximations.

[0151] Each patent, patent application, patent publication, and other material, articles, books, instructions, documents, that has been incorporated by reference into this specification, is hereby incorporated by reference herein, except to the extent that the incorporated material is inconsistent with the express disclosure herein. To the extent that any incorporated material contradicts or conflicts with the express disclosure herein, including defined terms, the instant disclosure controls. If not already incorporated by reference, the applicant hereby incorporates by reference the entire disclosure of each patent, patent application, and publication listed in the Priority Data section above.

[0152] Finally, it should be understood that the embodiments described herein are only given by way of example and that other modifications can occur to persons skilled in the art. Therefore, the scope of the present description is not intended to be limited to the embodiments described herein but is only limited by the claims that follow.

Claims

1. A method for protecting vehicle driving safety, characterized in that, The method includes: Acquire target image data captured by the target camera; Based on the target image data, determine the contour points of the object of interest; Based on the contour points and the target extrinsic parameters of the target camera, determine the contour point cloud data of the contour points in the vehicle coordinate system; Based on the contour point cloud data, safe areas and / or risk areas are determined.

2. The method as described in claim 1, characterized in that, The target camera is dynamically selected from at least one available camera based on the vehicle's mount posture, and the at least one available camera is mounted at at least one preset position on the vehicle mount.

3. The method as described in claim 1, characterized in that, The determination of safe areas and / or risk areas based on the contour point cloud data includes: Based on the first transformation relationship between the vehicle coordinate system and the polar coordinate system, the first point cloud data of the contour point cloud data in the polar coordinate system is determined; Based on the first point cloud data, the safe area is determined.

4. The method as described in claim 3, characterized in that, The step of determining the designated safe area based on the first point cloud data includes: Based on a preset angle, the first point cloud data is filtered to determine candidate point cloud data in the polar coordinate system; Based on the second transformation relationship between the polar coordinate system and the Cartesian coordinate system, the candidate point cloud data is determined as the second point cloud data in the Cartesian coordinate system. Based on the second point cloud data, the safe zone is determined.

5. The method as described in claim 1, characterized in that, The determination of safe areas and / or risk areas based on the contour point cloud data includes: Based on the third transformation relationship between the vehicle coordinate system and the world coordinate system, the third point cloud data of the contour point cloud data in the world coordinate system is determined. The risk area is determined based on the third point cloud data and at least one frame of historical point cloud data.

6. The method as described in claim 5, characterized in that, The determination of the risk area based on the third point cloud data and at least one frame of historical point cloud data includes: Based on the third point cloud data and the at least one frame of historical point cloud data, determine the boundary movement direction of the preset boundary of the obstacle; The risk area is determined based on the vehicle's direction of motion and the boundary's direction of movement.

7. The method as described in claim 6, characterized in that, The preset boundary is an obstacle boundary whose distance from the direction of vehicle movement meets a preset distance condition.

8. A vehicle driving safety protection system, characterized in that, The system includes: The image data acquisition module is configured to acquire target image data captured by the target camera. The contour point determination module is configured to determine the contour points of the object of interest based on the target image data; The point cloud data determination module is configured to determine the contour point cloud data of the contour points in the vehicle coordinate system based on the contour points and the target extrinsic parameters of the target camera. The region determination module is configured to determine safe regions and / or risk regions based on the contour point cloud data.

9. A vehicle driving safety protection device, characterized in that, The device includes at least one memory and at least one processor, the at least one memory being used to store computer instructions, and the at least one processor executing the computer instructions or parts thereof to implement the vehicle driving safety protection method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores computer instructions, and when the computer reads the computer instructions, the computer executes the vehicle driving safety protection method as described in any one of claims 1-7.

Images