Vehicle door detection method, device, storage medium and automatic driving method

Abstract

The present invention relates to the field of autonomous driving technology, and discloses a vehicle door detection method, device, storage medium and autonomous driving method. The detection method comprises: acquiring an image of a target vehicle in the driving direction of the main vehicle, the image of the target vehicle comprising: current and previous frame images; obtaining a feature map of the image based on the acquired image; inputting the feature map into a classification network model and a detection network model to obtain a classification result and a detection result respectively; fusing the classification result and the detection result to obtain a door state detection result of the target vehicle. In this embodiment, the state of the door is determined by fusing the results of the classification network model and the detection network model, so that a more stable and accurate detection result can be obtained.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a vehicle door detection method, device, storage medium and autonomous driving method. Background Art

[0002] Autonomous driving (also known as automated driving or driving automation) means that the vehicle continuously performs some or all dynamic driving tasks in an automatic manner. The "Automobile Automated Driving Classification" (GB / T 40429-2021) divides the levels of autonomous driving into 0 to 5, with Level 0 being emergency assisted driving and Level 5 being fully autonomous driving.

[0003] During the driving process of an autonomous vehicle, the autonomous driving system not only needs to accurately locate the position and speed information of the surrounding vehicles, but also needs to obtain more detailed information such as the current light signals and door status of the surrounding vehicles to assist in judging their future behavior, so as to automatically control the vehicle for better route planning. For example, when the doors of surrounding vehicles open, it indicates that the vehicle is stationary and passengers may get off, so the vehicle needs to take a detour. Especially for the sudden opening of the door of the vehicle in front, if the door opening status of the vehicle in front is not detected in time, a collision or even more serious traffic accident will occur, posing a huge safety hazard.

[0004] However, in current autonomous driving systems, the status of the doors, such as whether they are open, is generally detected through the vehicle's radar point cloud contours. However, the solution based on lidar is expensive and easily affected by weather. At the same time, based on point cloud data, if the door is suddenly opened, it is impossible to detect the door status in time, so there are also safety risks.

[0005] In some other existing technologies, solutions for detecting the door status based on video images are proposed. However, these solutions are not proposed for autonomous driving. For example, some solutions are used for violation monitoring, such as monitoring whether passengers are getting on and off within the specified area. In these solutions, there are no special requirements for the timeliness and accuracy of door status recognition, so they are not directly applicable to autonomous driving. Summary of the invention

[0006] The main purpose of the present invention is to provide a vehicle door detection method, device, storage medium and automatic driving method, which can accurately detect the state of the door.

[0007] In order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention provides a vehicle door detection method, comprising: acquiring an image of a target vehicle in the driving direction of a main vehicle, the image of the target vehicle comprising: a current image of the target vehicle and an image of a previous frame; obtaining a feature map of the image of the target vehicle based on the image of the target vehicle; inputting the feature map into a classification network model and a detection network model respectively to obtain a classification result and a detection result respectively; obtaining a door state detection result of the target vehicle based on the classification result and the detection result.

[0008] Wherein, obtaining the image of the target vehicle in the driving direction of the main vehicle includes: based on the position information of the vehicles around the main vehicle detected by the radar, intercepting the image of the target vehicle from the environment video around the main vehicle taken by the on-board camera.

[0009] Wherein, obtaining the feature map of the image based on the image of the target vehicle includes: inputting the image of the target vehicle into a residual network model for image recognition, and obtaining the feature map from the output of the residual network model.

[0010] The classification result includes four parameters, which respectively indicate whether the door is open in the current image, whether the door is open on the left, whether the door is open on the right, and whether the trunk door is open.

[0011] The detection result is a multi-attribute detection frame, and the multi-attribute detection frame includes: a door frame, a door category, a confidence level, and left and right position information.

[0012] An embodiment of the present invention also provides an automatic driving method, which is characterized by comprising: using the above-mentioned vehicle door detection method to obtain a door status detection result of a target vehicle; and executing automatic driving according to the door status detection result.

[0013] An embodiment of the present invention also provides a vehicle door detection device, comprising: an image acquisition module, used to acquire an image of a target vehicle in the driving direction of a main vehicle, the image of the target vehicle comprising: a current image of the target vehicle and an image of a previous frame; a feature map extraction module, used to obtain a feature map of the image of the target vehicle based on the image of the target vehicle; a classification module, used to classify the feature map using a classification network model to obtain a classification result; a detection module, used to detect the door state in the feature map using a detection network model to obtain a detection result; and a fusion module, used to obtain a door state detection result of the target vehicle based on the classification result and the detection result.

[0014] An embodiment of the present invention further provides a vehicle door detection device, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor is used to execute the computer program stored in the memory to implement the above-mentioned vehicle door detection method.

[0015] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program can be executed by a processor to execute the above method.

[0016] Beneficial effects of the embodiments of the present invention:

[0017] In an embodiment of the present invention, a classification network model and a detection network model are first used to detect the door status of a target vehicle from the perspective of the entire image and the perspective of the door part, respectively, and then the detection results of the two models are fused to finally determine the status of the door (e.g., whether it is open, and the left and right position, etc.), thereby obtaining a more stable and accurate detection result. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic flow chart of an embodiment of a vehicle door detection method of the present invention;

[0019] Figure 2 yes Figure 1 A schematic flow chart of an embodiment of step S10;

[0020] Figure 3 yes Figure 1 A schematic flow chart of an embodiment of steps S12 to S16;

[0021] Figure 4 is a flowchart of an embodiment of the automatic driving method of the present invention;

[0022] Figure 5 It is a schematic structural diagram of an embodiment of a vehicle door detection device of the present invention. DETAILED DESCRIPTION

[0023] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention clearer and more understandable, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.

[0024] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present invention, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0026] The present invention is described below by way of embodiments with reference to the accompanying drawings.

[0027] like Figure 1 FIG. 2 is a flow chart of an embodiment of the vehicle door detection method of the present invention. In this embodiment, the door status (whether it is open, etc.) of other vehicles near the host vehicle can be accurately detected, so that automatic driving and other controls can be performed based on the detection result. For example, when the door of the vehicle on the right side in front is detected to be open, the driving path can be planned to avoid it.

[0028] like Figure 1 As shown, the vehicle door detection method includes the following steps:

[0029] Step S10: Acquire an image of a target vehicle in the driving direction of the host vehicle.

[0030] The main vehicle is, for example, a vehicle equipped with an automatic driving function, which is equipped with software and hardware systems supporting the automatic driving function, such as an automatic driving operating system, a laser radar, a camera, etc. The automatic driving here includes full automatic driving and assisted driving. For example, the automatic driving in this embodiment can be L0 to L5 automatic driving.

[0031] Among them, the target vehicle mainly refers to the vehicle in the driving direction of the main vehicle, such as the vehicle in front and the vehicle on the side. When this method is applied to autonomous driving, it mainly focuses on the vehicles in the adjacent lanes in front. For example, in autonomous driving, the main vehicle needs to pay attention to whether the doors of the vehicles in the adjacent lanes in front are open, whether passengers are getting off, etc., so as to perform evasive operations to avoid traffic accidents.

[0032] The image of the target vehicle at least includes: a current image of the target vehicle (such as an image at the current moment) and an image of the previous frame. In the embodiment of the present invention, at least two frames of images are selected as the image of the target vehicle, and the door opening of the target vehicle can be detected in time by using the difference between the two frames of images, thereby overcoming the defect that the solution based on point cloud data cannot detect the sudden opening of the door.

[0033] In addition, in addition to selecting two adjacent frames of images as the target vehicle image, the embodiment of the present invention can also select multiple consecutive frames (3 frames and above) of images as the target vehicle image. In addition, since the target vehicle image is a continuous image, the above-mentioned frame images can be extracted in combination with a time series model to save data processing and storage.

[0034] Among them, Figure 2As shown, step S10 can be implemented based on the following method. Specifically, Figure 2 As shown, in step S20, the vehicle-mounted camera is used to shoot the environmental video around the main vehicle. In step S22, the position information of the vehicles around the main vehicle is detected by radar. For example, the radar can detect the position (in front, behind, to the side, beside, etc.) of other vehicles in the detection range and the corresponding distance of the main vehicle. Combining this information, it can be determined which vehicles need special attention, for example, the vehicles in the right lane in front, which will be locked as target vehicles. Therefore, in step S24, according to the information detected in steps S20 and S22, the image of the target vehicle is intercepted from the captured environmental video.

[0035] Step S12: obtaining a feature map of the image of the target vehicle according to the image of the target vehicle.

[0036] Among them, in this step, the residual network (resnet) model can be used to obtain the feature map. Specifically, the image of the target vehicle is input into the residual network model for image recognition, and the above-mentioned feature map is obtained from the output of the residual network model. For example, the residual network model can be a resnet50 network or a resnet101 network. Among them, resnet is a convolutional neural network proposed by four scholars from Microsoft Research, which won the image classification and object recognition in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2015. The characteristics of the resnet network are that it is easy to optimize and can improve the accuracy by increasing the depth. The internal residual block uses a jump connection to alleviate the gradient vanishing problem caused by increasing the depth in the deep neural network.

[0037] Step S14: input the feature map into the classification network model and the detection network model to obtain the classification result and the detection result respectively.

[0038] Among them, in this embodiment, two network models are set up respectively, namely the classification network (ClassificationHead) model and the detection network (Detection Head) model. The classification network model is used to detect whether the door is open, which is a global feature, and the detection network model is used to detect the specific information of the door frame, focusing on local information.

[0039] Specifically, the classification network model detects the door state from the perspective of the entire image. Among them, the classification result contains four parameters. For example, the classification result is a four-dimensional vector V containing four parameters, and these four parameters respectively represent whether the door is open in the current image, whether the left door is open, whether the right door is open, and whether the trunk is open.

[0040] The detection network model detects the door state from the perspective of the door part in the entire image. For example, the detection network model detects with the door as the target feature. Among them, the detection result of the detection network model is a multi-attribute detection box, and the multi-attribute detection box includes: a door frame, the category of the door (door, trunk), a confidence level (a value between 0 and 1, indicating the possibility of the currently detected door frame existing, and the larger the value, the greater the possibility), and left and right position information.

[0041] Step S16: Obtain the detection result of the door state of the target vehicle according to the classification result and the detection result.

[0042] In step S16, by fusing the classification detection result and the network detection result of step S14, the final door state detection result can be obtained. For example, whether the door is open and the category of the opened door (left, right, trunk, etc.).

[0043] Among them, according to the classification result and the detection result, the final door state detection result can be obtained. For example, it can be: Assume that the scores based on classification and detection are score_c and score_d respectively. Then when score_d > 0.7, the door is open. When 0.5 < score_d < 0.7 and score_c > 0.5, the door is open, and other cases are closed.

[0044] Among them, an example of steps S12 to S16 is Figure 3 as shown. As Figure 3 shown, the images of the current and previous frames are input into the backbone network for feature extraction, and the backbone network can be resnet50. Then the extracted features are respectively input into the detection head and the classification head for door state recognition, and then the recognition results are fused to obtain the final door state.

[0045] In this embodiment, the classification network model and the detection network model are first used to detect the door state of the target vehicle from the global and local perspectives respectively, and then the detection results of the two models are fused to finally determine the state of the door (such as whether it is open and the left and right positions, etc.), so as to obtain a more stable and accurate detection result.

[0046] As Figure 4 shown, it is a schematic flowchart of the automatic driving method of the present invention. It includes the following steps:

[0047] Step S40: Detect the door status of the target vehicle to obtain the door status detection result. For example, the method as 1 is used for detection.

[0048] Step S42: Control the automatic driving according to the door status detection result.

[0049] In this step, the automatic driving control is performed according to the door state finally determined. For example, when the door of the right lane of the front door is detected to be open, an evasive action such as detouring is performed.

[0050] Since the method can accurately detect the door status of the target vehicle, the safety of autonomous driving can be guaranteed.

[0051] like Figure 5 FIG. 1 is a schematic diagram of the structure of an embodiment of a vehicle door detection device 5 of the present invention, which includes:

[0052] The image acquisition module 50 is used to acquire the image of the target vehicle in the driving direction of the main vehicle. The image of the target vehicle includes: the current image of the target vehicle and the image of the previous frame. The feature map extraction module 51 is used to obtain the feature map of the image of the target vehicle based on the image of the target vehicle. The classification module 52 is used to classify the feature map using a classification network model to obtain a classification result. The detection module 53 is used to detect the door state in the feature map using a detection network model to obtain a detection result. The fusion module 54 is used to obtain the door state detection result of the target vehicle based on the classification result and the detection result.

[0053] In addition, the present invention also provides a vehicle door detection device, including: a memory and a processor, the memory stores a computer program, and the processor is used to execute the computer program stored in the memory to implement the above-mentioned vehicle door detection method.

[0054] In addition, an embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program can be executed by a processor to implement the above-mentioned method, such as a vehicle door detection method or an automatic driving method.

[0055] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0056] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0057] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, controller, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0058] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A vehicle door detection method, characterized in that: include: Acquire an image of a target vehicle in the driving direction of the host vehicle, wherein the image of the target vehicle includes: a current image of the target vehicle and an image of a previous frame; According to the image of the target vehicle, a feature map of the image of the target vehicle is obtained; The feature map is input into a classification network model and a detection network model respectively, and a classification result and a detection result are obtained respectively; wherein the detection result is a multi-attribute detection frame, and the multi-attribute detection frame includes: a door frame, a category of a door, a confidence level, and left and right position information; wherein the classification network model detects whether the door is open from the perspective of the entire image; A door state detection result of the target vehicle is obtained according to the classification result and the detection result.

2. The vehicle door detection method according to claim 1, characterized in that: The step of acquiring an image of a target vehicle in the driving direction of the host vehicle comprises: According to the position information of the vehicles around the main vehicle detected by the radar, the image of the target vehicle is intercepted from the environmental video around the main vehicle taken by the on-board camera.

3. The vehicle door detection method according to claim 1, characterized in that: The step of obtaining a feature map of the image according to the image of the target vehicle includes: The image of the target vehicle is input into a residual network model for image recognition, and the feature map is obtained from the output of the residual network model.

4. The vehicle door detection method according to claim 1, characterized in that: The classification result includes four parameters, which respectively indicate whether the door is open in the current image, whether the door is open on the left, whether the door is open on the right, and whether the trunk door is open.

5. An automatic driving method, characterized in that: include: Using the vehicle door detection method according to any one of claims 1 to 4 to obtain a door state detection result of the target vehicle; Automatic driving is performed according to the door status detection result.

6. A vehicle door detection device, characterized in that: include: An image acquisition module is used to acquire an image of a target vehicle in the driving direction of the host vehicle, wherein the image of the target vehicle includes: a current image of the target vehicle and an image of a previous frame; A feature map extraction module, used for obtaining a feature map of the image of the target vehicle according to the image of the target vehicle; A classification module, used to classify the feature graph using a classification network model to obtain a classification result; A detection module, used to detect the door state in the feature map using a detection network model to obtain a detection result; wherein the detection result is a multi-attribute detection frame, and the multi-attribute detection frame includes: a door frame, a door category, a confidence level, and left and right position information; wherein the classification network model detects whether the door is open from the perspective of the entire image; A fusion module is used to obtain a door status detection result of the target vehicle according to the classification result and the detection result.

7. The vehicle door detection device according to claim 6, characterized in that: The classification result includes four parameters, which respectively indicate whether the door is open in the current image, whether the door is open on the left, whether the door is open on the right, and whether the trunk door is open.

8. A vehicle door detection device, comprising: A memory and a processor, characterized in that the memory stores a computer program, and the processor is used to execute the computer program stored in the memory to implement the method according to any one of claims 1 to 4.

9. A computer-readable storage medium storing a computer program, characterized in that: The computer program can be executed by a processor to implement the method according to any one of claims 1 to 5.

Images