Vehicle control method and device, vehicle and storage medium

Abstract

The invention provides a vehicle control method and device, a vehicle and a storage medium. The method comprises the following steps: acquiring a multi-frame depth image of an environment in front of a vehicle; based on each frame of the depth image, counting depth distribution information of different positions in the horizontal direction in the environment in front of the vehicle; and controlling the movement of the vehicle based on the difference of the depth distribution information of different positions in the horizontal direction corresponding to the multiple frames of depth images. According to the embodiment, under the condition that the detection device only detects part of the outline of the obstacle, the change condition of the obstacle in the front environment of the vehicle can be sensitively sensed, so that the vehicle can timely and accurately avoid the obstacle in the front environment, and the safety and reliability of the vehicle in the driving process are improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular, to a vehicle control method, device, vehicle and storage medium. Background Art

[0002] One of the difficulties of autonomous driving today is environmental perception, the most important of which is estimating the position and speed of surrounding vehicles, so as to plan and control the vehicle's driving trajectory. Vehicle perception in related technologies basically relies on neural network technology for identification.

[0003] However, the vehicle perception method based on neural networks can only accurately identify and detect when the detection device detects the complete outline of the vehicle. When the detection device only detects a partial outline of the vehicle, there will be missed detection and false detection. In the scenario where other vehicles cut in at the maximum distance, this missed detection and false detection will cause the vehicle to be unable to sense the vehicle cutting in at the lane in time, resulting in the danger of not planning to avoid it in time. Summary of the invention

[0004] In view of this, the present application provides a vehicle control method, device, vehicle and storage medium.

[0005] In a first aspect, an embodiment of the present application provides a vehicle control method, the method comprising:

[0006] Acquire multiple frames of depth images of the environment in front of the vehicle;

[0007] Based on each frame of the depth image, statistically analyzing the depth distribution information at different positions in the horizontal direction of the environment in front of the vehicle;

[0008] The movement of the vehicle is controlled based on the difference in depth distribution information at different positions in the horizontal direction corresponding to multiple frames of the depth image.

[0009] In a second aspect, an embodiment of the present application provides a vehicle control device, including:

[0010] one or more processors;

[0011] a memory for storing instructions executable by the processor;

[0012] The one or more processors execute the executable instructions individually or collectively to perform the method as described in the first aspect.

[0013] In a third aspect, an embodiment of the present application provides a vehicle, including:

[0014] body;

[0015] a power system, disposed in the fuselage, for driving the vehicle to move; and

[0016] The vehicle control device according to the second aspect is arranged in the fuselage.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores executable instructions, and when the executable instructions are executed by a processor, the method described in the first aspect is implemented.

[0018] The embodiment of the present application provides a vehicle control method, which controls the movement of the vehicle based on the difference in depth distribution information obtained by statistics from multiple frames of depth images. When the detection device only detects a partial outline of the obstacle, the depth distribution information related to it can also be counted. The depth distribution information can reflect the distance between the obstacles in different positions in the environment in front of the vehicle (including obstacles with partial or full outlines detected) and the vehicle, and then the difference in depth distribution information corresponding to the multiple frames of the depth images can reflect the change in the distance between the obstacle and the vehicle (that is, the change in the position of the obstacle). The vehicle movement is controlled based on the difference in depth distribution information, so that when the detection device only detects a partial outline of the obstacle, the change of the obstacle in the environment in front of the vehicle can be sensitively sensed, so that the vehicle can avoid the obstacles in the environment in front in a timely and accurate manner, and improve the safety and reliability of the vehicle during driving. Moreover, compared with the vehicle perception method based on neural networks in the related art, the method of counting depth distribution information in the embodiment of the present application is simpler and more intuitive, with less calculation and higher efficiency, and can meet the real-time requirements in certain scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 and are schematic diagrams showing a partial outline of an obstacle detected by a detection device according to an exemplary embodiment of the present application.

[0020] Figure 2 It is a schematic diagram showing a relatively complete outline of an obstacle detected by a detection device according to an exemplary embodiment of the present application.

[0021] Figure 3 It is a flowchart of a vehicle control method shown in an exemplary embodiment of the present application.

[0022] Figure 4 It is a schematic diagram showing an exemplary embodiment of the present application showing that the lateral direction of a depth image indicates the horizontal direction in the environment in front of a vehicle.

[0023] Figure 5 It is a schematic diagram showing an exemplary embodiment of the present application of segmenting a depth image to obtain a plurality of strip-shaped pixel areas.

[0024] Figure 6It is a schematic diagram of integerization and number statistics of different pixel values ​​in a strip pixel area shown in an exemplary embodiment of the present application.

[0025] Figure 7 It is a schematic diagram of a depth distribution diagram shown in an exemplary embodiment of the present application.

[0026] Figure 8 It is a schematic diagram showing a depth distribution diagram analogous to a bird's-eye view shown in an exemplary embodiment of the present application.

[0027] Fig. 9 It is a schematic diagram of performing motion estimation on depth distribution maps of two frames of depth images shown in an exemplary embodiment of the present application.

[0028] Fig.10 It is a schematic diagram of a navigation interface shown in an exemplary embodiment of the present application.

[0029] Fig.11 It is a structural diagram of a control device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0030] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0031] In the related art, the vehicle perception method based on neural network is used to identify the vehicle, such as Figure 1 As shown, when the detection device in the vehicle detects only a partial outline of the vehicle (part of the front of the vehicle on the right side), Figure 1 The detection device only detects a small part of the front of the vehicle. When this part of the detection data is passed to the vehicle perception method based on the neural network for processing and identification, the neural network does not consider this part of the front of the vehicle as a vehicle. The observation effect is not very good, and there are cases of missed detection and false picking. Figure 1 The upper picture in the figure is an image of the environment in front of the vehicle. Figure 1 The figure below is the depth image corresponding to the image.

[0032] In the scenario where other vehicles are cutting in at the maximum distance, this missed detection and false pick-up will cause the vehicle to be unable to sense the vehicle cutting in in time, resulting in the danger of not planning to avoid it in time. Figure 2 As shown, when the detection device detects a relatively complete outline of the vehicle and the neural network recognizes the presence of the vehicle based on the detection data of the detection device, it is too late to brake, which poses a safety hazard. Figure 2The upper picture in the figure is an image of the environment in front of the vehicle. Figure 2 The figure below is the depth image corresponding to the image.

[0033] Based on this, an embodiment of the present application provides a vehicle control method that can sensitively sense changes in obstacles in the environment ahead of the vehicle even when the detection device only detects a partial outline of the obstacle, so that the vehicle can avoid obstacles in the environment ahead in a timely and accurate manner, thereby improving the safety and reliability of the vehicle during driving.

[0034] In an exemplary embodiment, the vehicle can be a manned vehicle (such as an autonomous vehicle), an electric vehicle, a scooter, a balance vehicle, etc. The vehicle generally includes a chassis, a body, an engine, and electrical equipment. The engine is the power device of the vehicle, which is used to generate power; the chassis is used to support the engine and the body, and the chassis can drive the vehicle to move according to the power generated by the engine; the body is mounted on the frame of the chassis for the driver and passengers to ride or load cargo; the electrical equipment includes a power source and electrical equipment, for example, the power source includes a battery and a generator, and the electrical equipment includes an engine starting system or other electrical devices. Optionally, the vehicle also includes a detection device for sensing environmental information of the vehicle's surroundings. Optionally, the vehicle also includes an automatic driving system for assisting the driver in driving.

[0035] In some embodiments, the embodiments of the present application provide a vehicle control method that can be executed by a control device.

[0036] In one example, the control device may be an electronic device with data processing capabilities, such as an on-board terminal in a vehicle, or a server or terminal connected to the vehicle in communication. The control device includes at least a memory and a processor, and the memory stores executable instructions or computer programs for executing the vehicle control method. When the memory executes the executable instructions or computer programs, the vehicle control method provided in the embodiment of the present application can be implemented, and the obstacle in the environment ahead of the vehicle can be sensitively sensed even when the detection device only detects a partial outline of the obstacle, so that the vehicle can avoid the obstacle in the environment ahead in a timely and accurate manner.

[0037] In another example, the control device may be a computer chip or integrated circuit with data processing capability, such as a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a field-programmable gate array (FPGA), etc. The computer chip or integrated circuit may be disposed on the vehicle or a terminal connected to the vehicle for communication.

[0038] The control device communicates with the detection device, and examples of the type of communication between the control device and the detection device may include but are not limited to communication via the following methods: Internet, local area network (LAN), wide area network (WAN), Bluetooth, near field communication (NFC) technology, a network based on a mobile data protocol such as General Packet Radio Service (GPRS), GSM, Enhanced Data GSM Environment (EDGE), 3G, 4G or Long Term Evolution (LTE) protocol, infrared (IR) communication technology, and / or WiFi, and may be wireless, wired, or a combination thereof.

[0039] In one example, taking the control device as a vehicle-mounted terminal, the vehicle-mounted terminal can be connected to the detection device by wired communication to achieve rapid transmission of data detected by the detection device.

[0040] See also Figure 3 , Figure 3 A flow chart of a vehicle control method provided in an embodiment of the present application. The vehicle is provided with a detection device, and the method can be executed by a control device, the control device is in communication with the detection device, and the detection device is provided in a vehicle as an example for explanation, and the method includes:

[0041] In step S101 , multiple frames of depth images of the environment in front of the vehicle are acquired.

[0042] In step S102, based on each frame of the depth image, depth distribution information at different positions in the horizontal direction in the environment in front of the vehicle is counted.

[0043] In step S103, the movement of the vehicle is controlled based on the difference in depth distribution information at different positions in the horizontal direction corresponding to multiple frames of the depth image.

[0044] For example, the movement of the vehicle is controlled so that the vehicle avoids an obstacle in front of the vehicle.

[0045] In this embodiment, the vehicle movement is controlled based on the depth distribution information directly obtained from the depth image. When the detection device only detects the partial outline of the obstacle, the depth distribution information related to it can also be counted. The depth distribution information can reflect the distance between the obstacles in different positions in the environment in front of the vehicle (including obstacles with partial or full outlines detected) and the vehicle. Then, the difference in the depth distribution information corresponding to the multiple frames of the depth image can reflect the change in the distance between the obstacle and the vehicle (that is, the change in the position of the obstacle). The vehicle movement is controlled based on the difference in the depth distribution information, so that even when the detection device only detects the partial outline of the obstacle, the changes in the obstacles in the environment in front of the vehicle can be sensitively sensed, so that the vehicle can avoid the obstacles in the front environment in a timely and accurate manner, and improve the safety and reliability of the vehicle during driving. Moreover, compared with the vehicle perception method based on neural networks in the related art, the method of counting the depth distribution information in the embodiment of the present application is simpler and more intuitive, with less calculation and higher efficiency, and can meet the real-time requirements in certain scenarios.

[0046] In some embodiments, the vehicle is provided with a detection device; the multiple frames of depth images are obtained based on the detection of the detection device; the detection device includes but is not limited to a laser radar, a millimeter wave radar, a depth camera, a binocular vision sensor or a monocular vision sensor, etc. Multiple frames of depth images of the environment in front of the vehicle can be obtained based on the detection device, and the pixel values ​​of the pixels in the depth images are depth values, indicating the distances between different objects in the environment in front of the vehicle and the vehicle.

[0047] Exemplarily, taking laser radar and millimeter wave radar as examples, the depth image can be obtained by mapping the point cloud collected by the laser radar or millimeter wave radar from three-dimensional space to two-dimensional space.

[0048] Exemplarily, taking a binocular vision sensor as an example, the depth image may be a binocular vision sensor, where two cameras of the binocular vision sensor take two images at different angles at the same time, and then through the difference between the two images, as well as the position and angle relationship between the two cameras, the distance relationship between the scene and the camera can be calculated using trigonometric relationships, thereby obtaining a depth image. For example, a disparity map can generally be calculated from two images, and the pixel value of a pixel in the disparity map is the disparity value, which can reflect the position deviation of the pixels imaged by the two cameras for the same scene, and then a depth map is obtained through the formula depth = focal lens * baseline / disparity, where depth is the depth map; focal lens represents the focal length of the camera; baseline is the baseline distance, which can be understood as the binocular distance; disparity is the disparity map.

[0049] Based on the formula, it can be known that the depth map and the disparity map can be converted to each other. Therefore, the present application can also control the movement of the vehicle according to the processing result of the disparity map, and the embodiments of the present application do not impose any restrictions on this. For example, multiple frames of disparity maps can be obtained, and the distribution information of disparity values ​​at different positions in the horizontal direction in the environment in front of the vehicle can be counted according to each frame of disparity map. The movement of the vehicle is controlled according to the difference of the disparity value distribution information corresponding to the multiple frames of disparity maps, so that the vehicle avoids obstacles in the environment in front.

[0050] In some embodiments, taking into account the differences in the installation positions of the detection devices on the vehicle, sometimes the detection field of view of the detection device may not be directly in front of the vehicle, and it may have a certain inclination angle difference with the front of the vehicle. Therefore, based on the use requirements of the present application for depth images, the depth image used in the embodiment of the present application is a depth image after coordinate correction, and the lateral direction of the depth image after coordinate correction indicates the horizontal direction in the environment in front of the vehicle. For example, through a pre-calibration process, the conversion relationship between the detection device coordinate system and the vehicle coordinate system can be determined according to the installation position of the detection device on the vehicle. For example, the vehicle coordinate system takes the vehicle body as the origin, the vehicle's forward direction as the X-axis, the direction perpendicular to the forward direction (that is, the horizontal direction in the environment in front of the vehicle) as the Y-axis, and the gravity direction as the Z-axis. After acquiring the depth image collected by the detection device, the depth image can be coordinate corrected based on the conversion relationship to obtain the depth image in the vehicle coordinate system, such as Figure 4 As shown, the lateral direction of the depth image indicates the horizontal direction (Y-axis direction) in the environment in front of the vehicle.

[0051] For some possible implementations, see Figure 1 as well as Figure 2 Considering that the depth image collected by the detection device contains depth information of the sky and / or the ground in addition to the depth information of different obstacles and the vehicle, obstacle recognition processing for the depth information of the sky and / or the ground is obviously useless and increases the processing volume. Therefore, in order to improve processing efficiency, the pixels belonging to the sky and / or the pixels belonging to the ground in the depth image can be first identified, and then the depth values ​​belonging to the sky and / or the depth values ​​belonging to the ground can be removed. The depth image obtained after removing the depth values ​​belonging to the sky and / or the depth values ​​belonging to the ground is used to perform subsequent depth distribution information statistics and vehicle control processes, which is conducive to reducing processing volume and improving processing efficiency.

[0052] In some embodiments, after obtaining multiple frames of depth images of the environment in front of the vehicle, the detection device can count the depth distribution information of different positions in the horizontal direction in the environment in front of the vehicle based on each frame of the depth image, and the depth distribution information can reflect the distance between obstacles (including obstacles with partial or full outlines detected) at different positions in the environment in front of the vehicle and the vehicle. Wherein, the lateral direction of the depth image indicates the horizontal direction in the environment in front of the vehicle, and the depth values ​​of pixels at different positions along the lateral direction in the depth image are statistically analyzed respectively, so as to obtain the depth distribution information corresponding to each frame of the depth image.

[0053] For some possible implementations, see Figure 5 , Figure 5 The above picture is the depth image ( Figure 5 The control device can vertically adjust the depth image ( Figure 5 The columns of the image (in the lower figure) are cut to obtain a plurality of strip pixel areas, and different strip pixel areas can indicate different positions in the horizontal direction of the environment in front of the vehicle. Then, the control device counts the depth distribution information in each strip pixel area to determine the depth distribution information of different positions in the horizontal direction of the environment in front of the vehicle.

[0054] Wherein, the number of rows of the strip pixel area is the same as the number of rows of the depth image, and the column of the strip pixel area includes at least one column of the depth image. For example, for an 8*8 depth image, taking the case that the column of the strip pixel area includes 1 column of the depth image as an example, the column of the depth image is cut along the vertical direction to obtain 8 strip pixel areas; if the column of the strip pixel area includes 2 columns of the depth image, 4 strip pixel areas can be obtained. Wherein, the number of columns contained in the strip pixel area is related to the detection accuracy and the required computing power resources. The more columns the strip pixel area contains, the rougher the detection, the lower the detection accuracy, and the less computing power resources are required. Conversely, the fewer columns the strip pixel area contains, the finer the detection, the higher the detection accuracy, and the higher the required computing power resources; therefore, the number of columns of the strip pixel area can be determined according to the detection accuracy requirements and the actual computing power resources in the actual application scenario. In other application scenarios, the number of rows of the strip pixel area and the number of rows of the depth image may also be different, and this embodiment does not impose any restrictions on this.

[0055] In a possible implementation, when the control device counts the depth distribution information in each of the strip pixel areas, it may be to count the number of pixels corresponding to different depth values ​​in the strip pixel area. The number of pixels corresponding to different depth values ​​can reflect the distribution of distances between different obstacles and the vehicle. For example, if the depth values ​​of pixels in a strip pixel area including 24 pixels are counted, and there are 10 pixels with a depth value of A and 14 pixels with a depth value of B, it can be determined that there is a high probability that there are two obstacles with different distances from the vehicle at the position indicated by the strip pixel area in the horizontal direction of the environment in front of the vehicle.

[0056] Optionally, in order to facilitate the counting of the number of pixels corresponding to different depth values, before counting the depth distribution information in each of the strip pixel areas, the depth values ​​of the pixels in the strip pixel areas may be firstly integerized, so as to improve the subsequent statistical efficiency. The integerization method may be specifically set according to the actual application scenario; for example, integerization may be performed by rounding off; or depth values ​​in different ranges may be converted into different integers. In an example, for example, Figure 6 As shown, a strip pixel area containing 3 columns of pixels of a depth image is shown, one grid represents one pixel, and the value in the grid represents the pixel value of the pixel. In order to improve statistical efficiency, depth values ​​between 0 and 10 can be counted as 0; depth values ​​between 10 and 30 can be counted as 20; depth values ​​between 30 and 50 can be counted as 40; and so on, and finally the integer depth values ​​of 0, 20, 40, 60...120... are obtained. In the subsequent statistical process, the number of pixels with depth values ​​of 0, 20, 40, 60...120... are counted.

[0057] It is understandable that the depth distribution information may also be other information about the depth, and this embodiment does not impose any limitation on this. In another example, the depth distribution information in the strip pixel area may be the depth difference between each pixel in the strip pixel area and its neighboring pixels, and the pixels whose depth difference is less than the preset difference may be classified into one category, and the pixels in this category have a greater probability of pointing to the same obstacle.

[0058] In some embodiments, after counting the depth distribution information corresponding to multiple frames of depth images, the control device can determine the motion state of the obstacle in the environment in front of the vehicle based on the difference in depth distribution information at different positions in the horizontal direction corresponding to the multiple frames of the depth images. The difference in depth distribution information corresponding to the multiple frames of the depth images can reflect the change in the distance between the obstacle and the vehicle (that is, the change in the position of the obstacle). When the motion state of the vehicle is known, the motion state of the obstacle in the environment in front of the vehicle can be determined accordingly, and then the motion of the vehicle can be controlled according to the difference between the motion state of the obstacle and the motion state of the vehicle, so that when the detection device only detects a partial outline of the obstacle, the vehicle can also sensitively sense the changes in the obstacles in the environment in front of the vehicle based on the difference in the depth distribution information, so that the vehicle can avoid obstacles in the front environment in a timely and accurate manner, thereby improving the safety and reliability of the vehicle during driving.

[0059] Exemplarily, the depth distribution information at different positions in the horizontal direction of the environment in front of the vehicle includes a depth distribution map, and different columns of the depth distribution map correspond to different strip pixel areas; the pixel value of a pixel in the depth distribution map indicates: the number of pixels belonging to a preset depth value in the strip pixel area corresponding to the pixel. In other words, refer to Figure 7 The depth distribution diagram shown in the figure has multiple columns along the horizontal axis corresponding to multiple strip pixel areas of the depth image. The Nth column of the depth distribution diagram represents the Nth strip pixel area of ​​the depth image, where N is an integer greater than 0. The vertical axis of the depth distribution diagram represents different preset depth values. For example, the pixel value of the pixel with coordinate (1,0) in the depth distribution diagram represents that the number of pixels with a depth value of 0 in the first strip pixel area is 14. For example, the pixel value of the pixel with coordinate (3,40) represents that the number of pixels with a depth value of 40 in the third strip pixel area is 42. Figure 7 It can be seen that different pixels belonging to the same row in the depth distribution map correspond to the same preset depth value, and different pixels belonging to the same column in the depth distribution map correspond to different preset depth values.

[0060] In a possible implementation, the motion state of obstacles in the environment in front of the vehicle can be determined by image processing. When the depth distribution information at different positions in the horizontal direction in the environment in front of the vehicle is represented by a depth distribution map, the control device can perform motion estimation based on the depth distribution map corresponding to multiple frames of the depth image to determine the motion state of the obstacle. Compared with the vehicle perception method based on neural networks, this embodiment can use fewer computing resources and simple geometric operations to detect the motion state of obstacles around the vehicle, which is beneficial to improving detection efficiency.

[0061] In an example, here is a further explanation of the depth profile: see Figure 8 , the columns of the depth distribution map correspond to the strip pixel areas of the depth image one by one, and different strip pixel areas indicate different positions in the horizontal direction of the environment in front of the vehicle, and the vertical axis of the depth distribution map represents different preset depth values, and the different preset depth values ​​are arranged in order according to the depth value size (that is, the distance between the obstacle and the vehicle), for example, Figure 8 As shown in FIG. 1 , the depth distribution map is sorted from small to large according to the preset depth values. The horizontal axis of the depth distribution map can indicate the horizontal direction of the environment in front of the vehicle, and the vertical axis of the depth distribution map can indicate the distribution of obstacles at different distances from the vehicle in the horizontal direction of the environment in front of the vehicle. In this way, the depth distribution map can be compared to a bird's-eye view (BV map), such as Figure 8 As shown in the figure, the lateral direction of the bird's-eye view represents the horizontal direction of the environment in front of the vehicle, and the longitudinal distance represents the distance between the object in the environment in front of the vehicle and the vehicle; accordingly, the process of motion estimation of multiple frames of depth distribution map can also be understood by analogy as estimating the position change of an obstacle on the bird's-eye view at different times. Assuming that the time interval between two adjacent frames is t, and the position change of an obstacle on the depth distribution map is (x, y), the speed of the obstacle on the depth distribution map (analogous to the bird's-eye view) can be obtained, that is, (v x =x / t,v y =y / t).

[0062] For example, see Fig. 9 , the control device can obtain the sampling point in the depth image of the current frame, determine the pixel point of the sampling point on the depth distribution map corresponding to the depth image according to the strip pixel area and depth value corresponding to the sampling point; then perform optical flow tracking on the depth distribution map corresponding to the depth image of other frames to obtain the matching pixel point of the pixel point in other depth distribution maps; determine the speed and position of the obstacle according to the position difference between the strip pixel area corresponding to the matching pixel point and the strip pixel area corresponding to the pixel point. The depth image of the current frame and the depth image of the other frames can be adjacent frames, or can be two frames with a short acquisition time interval.

[0063] In some embodiments, after determining the motion state of the obstacle in the environment ahead of the vehicle, the control device can plan the driving path and / or speed of the vehicle according to the difference between the motion state of the obstacle and the motion state of the vehicle. Exemplarily, the control device can determine the space that the vehicle is about to occupy according to the motion state of the vehicle; predict whether the obstacle and the vehicle will interfere in the space that the vehicle is about to occupy according to the motion state of the obstacle, and if it is determined that the obstacle and the vehicle are about to interfere in the space that the vehicle is about to occupy, the vehicle is controlled to avoid the obstacle to ensure the safety and reliability of the vehicle during driving. Among them, "interference" means that the distance between the obstacle and the vehicle in the space that the vehicle is about to occupy is too close or there is a possibility of collision. In one example, the space that the obstacle is about to occupy can be determined according to the motion state of the obstacle, and whether interference occurs by the distance or overlap between the vehicle in the space that the obstacle is about to occupy and the space that the obstacle is about to occupy, and if the obstacle and the vehicle are about to interfere, the vehicle is controlled to avoid the obstacle.

[0064] For example, when it is predicted that the obstacle and the vehicle may interfere with each other in the space that the vehicle is about to occupy, the vehicle can be automatically controlled to slow down or brake so that the vehicle avoids the obstacle and ensures the safety and reliability of the vehicle during driving. While controlling the vehicle to slow down or brake, visual reminder information and / or auditory reminder information can also be output to remind the user that there is an obstacle invading the lane, so as to inform the user of the reason for the vehicle to slow down or brake.

[0065] In some embodiments, when the position of an obstacle in the environment in front of the vehicle can be determined, a mark indicating the obstacle can also be displayed around the model of the vehicle in a navigation interface that displays the model of the vehicle; wherein, as can be seen from the above description, the position of the obstacle in the environment in front of the vehicle can be determined based on the difference in depth distribution information at different positions in the horizontal direction corresponding to multiple frames of the depth image, and then the display position of the mark indicating the obstacle can be determined based on the position of the obstacle and a preset mapping relationship, and the mapping relationship indicates the conversion process from the three-dimensional space where the obstacle is located to the two-dimensional space where the navigation interface is located. In this embodiment, the display of visual marks can remind the user that there are obstacles that are too close or vehicles that are jamming in at the limit distance, so that the user can take timely measures to plan safe driving of the vehicle.

[0066] Optionally, in addition to displaying the mark indicating the obstacle, a voice prompt may be used to remind the user that there is an obstacle that is too close or a vehicle that is trying to squeeze in at the limit distance, so that the user can plan safe driving of the vehicle in a timely manner.

[0067] In some embodiments, the vehicle can also run a neural network model for detecting obstacles at the same time, and the neural network model is used to detect obstacles based on the detection data of the detection device; the method for detecting obstacles based on depth distribution information in the embodiment of the present application can complement the neural network model, and jointly control the operation of the vehicle based on the difference in the depth distribution information corresponding to multiple frames of the depth image and the obstacle detection results output by the neural network model, thereby further improving the safety and reliability of the vehicle during driving. Exemplarily, the motion state of obstacles in the environment in front of the vehicle can be determined based on the difference in depth distribution information at different positions in the horizontal direction corresponding to multiple frames of the depth image, and then the vehicle's driving trajectory and speed can be planned based on the difference in the motion state of obstacles in the environment in front of the vehicle, the obstacle detection results output by the neural network model, and the motion state of the vehicle, so as to drive safely.

[0068] Among them, the neural network model used to detect obstacles can also identify the type of obstacles, including but not limited to vehicles, pedestrians, traffic lights or trees, etc. The obstacle detection result output by the neural network model can include the type of obstacles, such as other vehicles around the vehicle. The method of motion estimation based on depth distribution information can determine the position and speed of the obstacle, but cannot determine the type of obstacle.

[0069] For example, when the position of an obstacle in front of the vehicle is determined based on the difference in depth distribution information of multiple frames, and the positions of other vehicles around the vehicle are determined based on a neural network model, a mark indicating the obstacle and the models of other vehicles detected by the neural network model can be displayed around the model of the vehicle in a navigation interface that displays the model of the vehicle. Since the type of obstacle cannot be determined by motion estimation based on depth distribution information, the display styles of the mark and the models of other vehicles are different. In one example, see Fig.10 For example, a mark 20 and other vehicle models 30 may be displayed around the vehicle model 10 .

[0070] In addition, when the relevant program of the neural network model used to detect obstacles runs abnormally, that is, the neural network model cannot be used, it is also possible to use the method provided in the embodiment of the present application as a backup, and control the movement of the vehicle based on the difference in the depth distribution information corresponding to multiple frames of the depth images to enable the vehicle to avoid obstacles in the front environment.

[0071] Accordingly, see Fig.11 The embodiment of the present application further provides a vehicle control device 20, including:

[0072] One or more processors 21;

[0073] A memory 22 for storing instructions executable by the processor 21;

[0074] The one or more processors 21 execute the executable instructions individually or collectively to perform:

[0075] Acquire multiple frames of depth images of the environment in front of the vehicle;

[0076] Based on each frame of the depth image, statistically analyzing the depth distribution information at different positions in the horizontal direction of the environment in front of the vehicle;

[0077] The movement of the vehicle is controlled based on the difference in depth distribution information at different positions in the horizontal direction corresponding to multiple frames of the depth image.

[0078] The processor 21 executes the executable instructions included in the memory 22. The processor 21 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0079] The memory 22 stores executable instructions of the vehicle control method, and the memory 22 may include at least one type of storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. Moreover, the device may cooperate with a network storage device that performs the storage function of the memory through a network connection. The memory 22 may be an internal storage unit of the vehicle control device 20, such as a hard disk or memory of the vehicle control device 20. The memory 22 may also be an external storage device of the vehicle control device 20, such as a plug-in hard disk equipped on the vehicle control device 20, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the memory 22 may also include both an internal storage unit of the vehicle control device 20 and an external storage device. The memory 22 is used to store executable instructions and other programs and data required by the vehicle control device 20. The memory 22 may also be used to temporarily store data that has been output or is to be output.

[0080] In some embodiments, the processor is also used to: cut the columns of the depth image along the vertical direction to obtain multiple strip pixel areas; different strip pixel areas indicate different positions in the horizontal direction in the environment in front of the vehicle; and count the depth distribution information in each strip pixel area.

[0081] In some embodiments, the depth distribution information in the strip pixel region includes the number of pixels corresponding to different depth values ​​in the strip pixel region.

[0082] In some embodiments, before counting the depth distribution information in each of the strip-shaped pixel regions, the processor is further configured to: perform integer processing on the depth values ​​of the pixels in the strip-shaped pixel regions, which is beneficial to improving statistical efficiency.

[0083] In some embodiments, the number of rows of the strip pixel regions is the same as the number of rows of the depth image, and the columns of the strip pixel regions include at least one column of the depth image.

[0084] In some embodiments, the depth distribution information at different positions in the horizontal direction in the environment in front of the vehicle includes a depth distribution map; different columns of the depth distribution map correspond to different strip pixel areas; the pixel value of a pixel in the depth distribution map indicates: the number of pixels belonging to a preset depth value in the strip pixel area corresponding to the pixel.

[0085] In some embodiments, different pixels belonging to the same row in the depth distribution map correspond to the same preset depth value, and different pixels belonging to the same column in the depth distribution map correspond to different preset depth values.

[0086] In some embodiments, the processor is also used to: determine the motion state of an obstacle in the environment in front of the vehicle based on the difference in depth distribution information at different positions in the horizontal direction corresponding to multiple frames of the depth images; and control the movement of the vehicle according to the difference between the motion state of the obstacle and the motion state of the vehicle.

[0087] In some embodiments, the depth distribution information at different positions in the horizontal direction in the environment in front of the vehicle includes a depth distribution map; the processor is also used to: perform motion estimation based on the depth distribution map corresponding to multiple frames of the depth image to determine the motion state of the obstacle.

[0088] In some embodiments, the processor is also used to: obtain sampling points in the depth image, determine the pixel points of the sampling points on the depth distribution map corresponding to the depth image according to the strip pixel area corresponding to the sampling points and the depth value; perform optical flow tracking on the depth distribution maps corresponding to other frame depth images to obtain matching pixel points of the pixel points in other depth distribution maps; determine the speed and position of the obstacle according to the position difference between the strip pixel area corresponding to the matching pixel point and the strip pixel area corresponding to the pixel point.

[0089] In some embodiments, the processor is also used to determine the space that the vehicle is about to occupy based on the movement state of the vehicle; if, based on the movement state of the obstacle, it is predicted that the obstacle and the vehicle will interfere in the space that is about to be occupied, the vehicle is controlled to avoid the obstacle.

[0090] In some embodiments, the processor is further configured to control the vehicle to decelerate or brake.

[0091] In some embodiments, the processor is also used to output visual reminder information and / or auditory reminder information to prompt the user that there is an obstacle invading the lane while controlling the vehicle to decelerate or brake.

[0092] In some embodiments, the processor is also used to display a mark indicating the obstacle around the model of the vehicle in a navigation interface displaying the model of the vehicle; wherein the display position of the mark is determined based on the difference in depth distribution information at different positions in the horizontal direction corresponding to multiple frames of the depth image.

[0093] In some embodiments, the vehicle is provided with a detection device; the vehicle also runs a neural network model for detecting obstacles, the neural network model is used to detect obstacles according to the detection data of the detection device; the processor is also used to control the movement of the vehicle according to the obstacle detection result output by the neural network model. The neural network model and the method of statistical depth distribution information are complementary.

[0094] In some embodiments, the processor is also used to display a mark indicating the obstacle and models of other vehicles detected by the neural network model around the model of the vehicle in a navigation interface displaying the model of the vehicle; wherein the display styles of the mark and the models of other vehicles are different.

[0095] In some embodiments, the vehicle is provided with a detection device; the multi-frame depth image is obtained based on detection by the detection device; the detection device includes at least one of the following: a lidar, a millimeter-wave radar, a depth camera, a binocular vision sensor, or a monocular vision sensor.

[0096] In some embodiments, the depth image is a depth image obtained by removing the depth value belonging to the sky and / or the depth value belonging to the ground, which is helpful to reduce the amount of data processing in subsequent processes and improve processing efficiency.

[0097] In some embodiments, the depth image is a depth image after coordinate correction, and the lateral direction of the depth image after coordinate correction indicates the horizontal direction in the environment in front of the vehicle.

[0098] Various embodiments described herein can be implemented using computer readable media such as computer software, hardware or any combination thereof. For hardware implementation, the embodiments described herein can be implemented using at least one of application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and electronic units designed to perform the functions described herein. For software implementation, embodiments such as processes or functions can be implemented with separate software modules that allow execution of at least one function or operation. Software code can be implemented by a software application (or program) written in any appropriate programming language, and the software code can be stored in a memory and executed by a controller.

[0099] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0100] Accordingly, an embodiment of the present application further provides a vehicle, including:

[0101] body;

[0102] a power system, disposed in the fuselage, for providing power to the vehicle;

[0103] And, provided in the fuselage such as Fig.11 The vehicle control device 20.

[0104] In some embodiments, the vehicle further includes a detection device, which includes at least one of the following: a laser radar, a millimeter wave radar, a depth camera, a binocular vision sensor, or a monocular vision sensor.

[0105] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by a processor of the device to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0106] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a terminal, enables the terminal to execute the above method.

[0107] It should be noted that the terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items.

[0108] It should be understood that, although the term first, second, third, etc. may be adopted in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of ... " or "when ... " or "in response to determination". The term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only include those elements, but also include other elements not clearly listed, or also include elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the statement "include one ..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.

[0109] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claim protection, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0110] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0111] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0112] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A vehicle control method, characterized in that: The method comprises: Acquire multiple frames of depth images of the environment in front of the vehicle; Based on each frame of the depth image, statistically analyzing the depth distribution information at different positions in the horizontal direction of the environment in front of the vehicle; The movement of the vehicle is controlled based on the difference in depth distribution information at different positions in the horizontal direction corresponding to multiple frames of the depth image.

2. The method according to claim 1, characterized in that The counting of depth distribution information at different positions in the horizontal direction of the environment in front of the vehicle based on each frame of the depth image includes: Cutting the columns of the depth image along the vertical direction to obtain a plurality of strip pixel regions; different strip pixel regions indicate different positions in the horizontal direction of the environment in front of the vehicle; The depth distribution information in each of the strip-shaped pixel regions is counted.

3. The method according to claim 2, characterized in that The depth distribution information in the strip pixel region includes the number of pixels corresponding to different depth values ​​in the strip pixel region.

4. The method according to claim 3, characterized in that Before counting the depth distribution information in each of the strip-shaped pixel regions, the method further includes: The depth values ​​of the pixels in the strip pixel area are integerized.

5. The method according to claim 2, characterized in that: The number of rows of the strip pixel region is the same as the number of rows of the depth image, and the column of the strip pixel region includes at least one column of the depth image.

6. The method according to any one of claims 2 to 5, characterized in that: The depth distribution information at different positions in the horizontal direction of the environment in front of the vehicle includes a depth distribution map; Different columns of the depth distribution map correspond to different strip-shaped pixel areas; The pixel value of a pixel in the depth distribution map indicates the number of pixels belonging to a preset depth value in the strip pixel area corresponding to the pixel.

7. The method according to claim 6, characterized in that Different pixels belonging to the same row in the depth distribution map correspond to the same preset depth value, and different pixels belonging to the same column in the depth distribution map correspond to different preset depth values.

8. The method according to any one of claims 1 to 6, characterized in that: The controlling the movement of the vehicle based on the difference in depth distribution information at different positions in the horizontal direction corresponding to the multiple frames of the depth images includes: Determining the motion state of the obstacle in the environment in front of the vehicle based on the difference in depth distribution information at different positions in the horizontal direction corresponding to the multiple frames of the depth image; The movement of the vehicle is controlled according to the difference between the movement state of the obstacle and the movement state of the vehicle.

9. The method according to claim 8, characterized in that The depth distribution information at different positions in the horizontal direction of the environment in front of the vehicle includes a depth distribution map; The determining the motion state of the obstacle in the environment in front of the vehicle based on the difference in depth distribution information at different positions in the horizontal direction corresponding to the multiple frames of the depth images includes: Motion estimation is performed based on the depth distribution maps corresponding to the multiple frames of the depth images to determine the motion state of the obstacle.

10. The method according to claim 9, characterized in that The performing motion estimation based on the depth distribution map corresponding to the multiple frames of the depth image to determine the motion state of the obstacle includes: Acquire a sampling point in the depth image, and determine a pixel point of the sampling point on a depth distribution map corresponding to the depth image according to a strip pixel area and a depth value corresponding to the sampling point; Perform optical flow tracking on the depth distribution maps corresponding to the depth images of other frames to obtain matching pixel points of the pixel points in the other depth distribution maps; The speed and position of the obstacle are determined according to the position difference between the strip pixel area corresponding to the matching pixel point and the strip pixel area corresponding to the pixel point.

11. The method according to claim 8, characterized in that The controlling the movement of the vehicle according to the difference between the movement state of the obstacle and the movement state of the vehicle comprises: determining a space that the vehicle is about to occupy based on the motion state of the vehicle; If it is predicted based on the movement state of the obstacle that the obstacle and the vehicle will interfere with each other in the space to be occupied, the vehicle is controlled to avoid the obstacle.

12. The method according to any one of claims 1 to 11, characterized in that: The controlling the movement of the vehicle comprises: Control the vehicle to decelerate or brake.

13. The method according to claim 12, characterized in that Also includes: While controlling the vehicle to decelerate or brake, visual reminder information and / or auditory reminder information is output to remind the user that there is an obstacle invading the lane.

14. The method according to any one of claims 1 to 13, characterized in that: Also includes: In a navigation interface displaying a model of the vehicle, a mark indicating the obstacle is displayed around the model of the vehicle; wherein the display position of the mark is determined based on the difference in depth distribution information at different positions in the horizontal direction corresponding to multiple frames of the depth image.

15. The method according to any one of claims 1 to 13, characterized in that: The vehicle is provided with a detection device; the vehicle also runs a neural network model for detecting obstacles, the neural network model is used to detect obstacles based on the detection data of the detection device; The controlling the movement of the vehicle further comprises: The movement of the vehicle is controlled according to the obstacle detection result output by the neural network model.

16. The method according to claim 15, characterized in that Also includes: In a navigation interface displaying a model of the vehicle, a mark indicating the obstacle and models of other vehicles detected by the neural network model are displayed around the model of the vehicle; wherein the display styles of the mark and the models of other vehicles are different.

17. The method according to any one of claims 1 to 16, characterized in that: The vehicle is provided with a detection device; the multiple frames of depth images are obtained based on detection by the detection device; The detection device includes at least one of the following: a laser radar, a millimeter wave radar, a depth camera, a binocular vision sensor or a monocular vision sensor.

18. The method according to any one of claims 1 to 16, characterized in that: The depth image is a depth image obtained by eliminating the depth value belonging to the sky and / or the depth value belonging to the ground.

19. The method according to any one of claims 1 to 16, characterized in that: The depth image is a depth image after coordinate correction, and the lateral direction of the depth image after coordinate correction indicates the horizontal direction in the environment in front of the vehicle.

20. A vehicle control device, characterized in that: include: one or more processors; a memory for storing instructions executable by the processor; The one or more processors execute the executable instructions individually or collectively to perform the method according to any one of claims 1 to 19.

21. A vehicle, characterized in that: include: body; A power system, disposed in the fuselage, for driving the vehicle to move; as well as A vehicle control device as claimed in claim 20 disposed in the fuselage.

22. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores executable instructions, and when the executable instructions are executed by a processor, the method according to any one of claims 1 to 19 is implemented.

Images