Two-stage-based truck detection method, electronic device, and storage medium

Abstract

The application provides a two-stage-based truck detection method, an electronic device and a storage medium, including: obtaining a 3D point cloud of a to-be-predicted task; extracting a first feature map of the 3D point cloud based on a 3D backbone network, and inputting the first feature map into a 2D backbone network for feature training to obtain a second feature map; obtaining a truck-class whole-vehicle detection box of the second feature map based on a first detection head network; inputting the truck-class whole-vehicle detection box and a corresponding third feature map into a second detection head network for feature training to obtain a truck-head detection box and a truck-body detection box corresponding to the truck-class whole-vehicle detection box; and determining a separation state between a truck head and a truck body of a corresponding truck based on the truck-head detection box and the truck-body detection box. The application uses a two-stage network to obtain a truck-head detection box and a truck-body detection box, efficiently predicts the separation state of the truck head and the truck body, and achieves the technical effect of improving the accuracy of truck detection.

Description

Technical Field

[0001] This application relates to the field of target detection technology, and in particular to a two-stage truck detection method, electronic device, and storage medium. Background Technology

[0002] With the development of computer technology and the widespread application of computer vision principles, the use of computer image processing technology for real-time target research is becoming increasingly popular. Dynamic real-time target positioning has broad application value in intelligent transportation systems, intelligent monitoring systems, and surgical instrument positioning in medical navigation surgery.

[0003] In intelligent transportation systems, especially for the currently popular autonomous driving technology, three-dimensional (3D) object detection has become an indispensable task, attracting widespread attention from industry and academia. However, in related technologies, vehicle object detection focuses more on the detection accuracy of the entire target vehicle, typically outputting only a single overall bounding box. But for trucks, especially heavy trucks and other large vehicles, the front and body of the truck may separate when turning. Using the overall bounding box cannot accurately capture the separation of the truck's front and body, thus failing to accurately determine its direction of travel, which could potentially lead to safety hazards. Summary of the Invention

[0004] In view of the above problems, this application provides a two-stage truck detection method, electronic device and storage medium to solve the safety hazards caused by the current target detection algorithm's inability to accurately obtain the separation of the truck's front and body and its driving direction.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] In a first aspect, embodiments of this application provide a two-stage truck detection method, comprising:

[0007] Obtain the 3D point cloud of the task to be predicted;

[0008] The first feature map of the 3D point cloud is extracted based on the 3D backbone network, and the first feature map is input into the 2D backbone network for feature training to obtain the second feature map.

[0009] The truck-type vehicle detection bounding box is obtained based on the second feature map obtained by the first detection head network;

[0010] The whole vehicle detection box of the truck class and its corresponding third feature map are input into the second detection head network for feature training to obtain the front detection box and body detection box corresponding to the whole vehicle detection box of the truck class; and, based on the front detection box and the body detection box, the separation state between the front and body of the corresponding truck is determined.

[0011] In one implementation, obtaining the truck-type vehicle detection bounding box from the second feature map based on the first detection head network includes:

[0012] The prediction box of the second feature map and its corresponding classification confidence are obtained based on the first detection head network, and the whole vehicle detection box of the truck class is obtained based on the prediction box and the corresponding classification confidence.

[0013] In one implementation, the task to be predicted carries truck-type and non-truck-type objects, and the prediction box includes a whole vehicle prediction box for truck-type objects and a whole prediction box for non-truck-type objects.

[0014] The step of obtaining the whole vehicle detection box for the truck class based on the predicted box and the corresponding classification confidence includes:

[0015] Remove all non-truck category overall prediction boxes;

[0016] Based on the classification confidence level of the whole vehicle prediction box for the truck class, whole vehicle prediction boxes with classification confidence levels within a preset classification confidence level range are selected and used as whole vehicle detection boxes for the truck class.

[0017] In one embodiment, the second detection head network is a RoI detection head network, which includes RoI pooling layers and convolutional layers.

[0018] The step of inputting the whole vehicle detection bounding box of the truck type and its corresponding third feature map into the second detection head network for feature training includes:

[0019] The RoI pooling layer is used to perform pooling operations on the whole vehicle detection bounding box of the truck class and its corresponding third feature map to obtain the features corresponding to the image in the whole vehicle detection bounding box of the truck class; and the convolutional layer is used to train the features corresponding to the image in the whole vehicle detection bounding box.

[0020] In one embodiment, determining the separation state between the cab and body of the corresponding truck based on the cab detection frame and the body detection frame includes:

[0021] Obtain the deviation angle between the front detection frame and the body detection frame;

[0022] If the deviation angle is greater than a preset threshold, it is determined that the truck's front and body are separated, as indicated by the front detection frame and the body detection frame.

[0023] In one embodiment, after determining the separation state between the cab and body of the corresponding truck based on the cab detection frame and the body detection frame, the method further includes:

[0024] Obtain the deviation direction between the front detection frame and the body detection frame, and display a prompt message in the deviation direction. The prompt message is used to indicate the turning trend of the truck.

[0025] In one implementation, after acquiring the 3D point cloud of the task to be predicted and before extracting the first feature map of the 3D point cloud based on the 3D backbone network, the method further includes:

[0026] The 3D point cloud is voxelized to obtain a voxelized 3D point cloud.

[0027] Extracting the first feature map of the 3D point cloud based on a 3D backbone network includes: extracting a voxelized 3D point cloud based on a 3D backbone network; wherein the 3D backbone network is a sparse convolutional network.

[0028] In one implementation, the task to be predicted carries both truck-type and non-truck-type objects, and the prediction bounding box includes a whole vehicle prediction bounding box for truck-type objects and a whole prediction bounding box for non-truck-type objects.

[0029] The method further includes: detecting the positional state between trucks and non-trucks based on the whole vehicle detection frame of the truck type, the front detection frame and the body detection frame, and the whole detection frame of the non-truck type.

[0030] According to another aspect of this application, a two-stage truck inspection device is provided, comprising:

[0031] The acquisition module is configured to acquire the 3D point cloud of the task to be predicted;

[0032] The extraction module is configured to extract a first feature map of the 3D point cloud based on a 3D backbone network, and input the first feature map into a 2D backbone network for feature training to obtain a second feature map.

[0033] The overall bounding box detection module is configured to detect the predicted bounding box and the corresponding classification confidence of the second feature map based on the first detection head network;

[0034] The acquisition module is further configured to acquire a whole vehicle detection box for the truck class based on the prediction box and the corresponding classification confidence.

[0035] The vehicle front and body detection module is configured to input the whole vehicle detection box of the truck type and the corresponding third feature map into the second detection head network for feature training, so as to obtain the vehicle front detection box and body detection box corresponding to the whole vehicle detection box of the truck type.

[0036] The post-processing module is configured to predict the separation state between the front and body of the corresponding truck based on the front detection frame and the body detection frame.

[0037] According to another aspect of this application, an electronic device is provided, comprising: a processor and a memory;

[0038] The memory stores computer-executed instructions;

[0039] The processor executes computer execution instructions stored in the memory, causing the electronic device to perform the two-stage truck detection method.

[0040] According to the two-stage truck detection method, electronic device, and storage medium provided in the embodiments of this application, a 3D backbone network and a 2D backbone network are used to generate two-dimensional feature maps from 3D point clouds, improving the accuracy of the two-dimensional feature maps. Then, a first detection head network is used to detect the truck-type vehicle detection boxes in the two-dimensional feature maps. In the second detection head network, feature training is performed based on the vehicle detection boxes and the corresponding third feature maps to obtain the truck front detection boxes and vehicle body detection boxes corresponding to the vehicle detection boxes. Based on the obtained truck front detection boxes and vehicle body detection boxes, the separation state between the truck front and the vehicle body is predicted. This process, by first predicting the truck-type vehicle boxes and then further predicting the truck front and vehicle body based on the prediction results of the truck vehicle boxes, effectively improves the detection accuracy of the truck front and vehicle body. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of a truck detection frame in related technologies;

[0042] Figure 2a This is one of the schematic diagrams of the truck detection frame in the embodiments of this application;

[0043] Figure 2b This is the second schematic diagram of the truck detection frame in the embodiments of this application;

[0044] Figure 3 This is a schematic diagram of a possible scenario according to an embodiment of this application;

[0045] Figure 4 One of the flowcharts for a two-stage truck detection method provided in this application embodiment;

[0046] Figure 5This is a second schematic flowchart of a two-stage truck detection method in an embodiment of this application;

[0047] Figure 6 A flowchart illustrating another two-stage truck detection method provided in this application embodiment;

[0048] Figure 7 A flowchart illustrating another two-stage truck detection method provided in this application embodiment;

[0049] Figure 8 A schematic diagram of a two-stage truck detection device provided in an embodiment of this application;

[0050] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0051] In autonomous driving, 3D object detection is an essential task that has attracted widespread attention from industry and academia. Various techniques have been employed to improve the overall detection accuracy of target vehicles. For example, in single-level detectors, VoxelNet is used to voxelize the point cloud and a voxel feature encoding layer is proposed to extract point features from each voxel, generating fixed-length features for batch training. PointPillar technology utilizes pseudo-images for 3D object detection, dividing the point cloud into multiple pillars instead of voxels for feature extraction, and then using a 2D convolutional detection structure for object detection. SECOND technology uses sparse convolution to replace traditional 3D convolution. TANet (Triple Attention Net) proposes a triple attention module embedded in voxel feature extraction and combines it with a proposed cascaded thinning network. Recently, Point-GNN (Point-Graph Neural Network) has proposed a graph neural network to extract point features and achieved good performance. 3DSSD (3D Single Shot MultiBox) has also been developed. Detector (3D single-stage detection) proposes a fusion sampling strategy that combines feature-based and point-based farthest-point sampling to achieve better classification performance; and SA-SSD (StructureAware-SSD, structure-aware single-stage detection) proposes an auxiliary network parallel to a sparse convolutional network to regress the bounding box center and semantic class of each point using interpolated voxel features. However, none of the above techniques for 3D object detection methods consider the separate detection of different parts of the same object.

[0052] For trucks, especially heavy-duty trucks, there is a significant deviation between the front of the truck and the body when turning, such as... Figure 1 As shown, if only one labeled ground truth bounding box (the dashed box in the figure) is used to represent the current turning truck, the following problems will occur: it is impossible to know that the truck's front and body have separated, which will not accurately determine the current driving direction and is not conducive to the prediction of the driving trajectory; it is unsafe, that is, pedestrians may be very close to the front of the truck, but far from the labeled ground truth bounding box, and at this time, pedestrians are very likely to collide with the front of the truck; in addition, the network prediction is unstable, that is, multiple trucks facing different directions may be represented by the same labeled ground truth bounding box, causing ambiguity during training.

[0053] In view of this, embodiments of this application provide a two-stage truck detection method, electronic device, and storage medium. It utilizes a 3D backbone network and a 2D backbone network to generate two-dimensional feature maps from 3D point clouds, improving the accuracy of the two-dimensional feature maps. Then, a first detection head network is used to detect the truck-type vehicle detection boxes from the two-dimensional feature maps. Further feature training is performed based on a second detection head network to obtain the truck front detection box and the truck body detection box corresponding to the vehicle detection box. Finally, based on the obtained truck front detection box and truck body detection box, the separation state between the truck front and the truck body is predicted. This process, by first predicting the truck-type vehicle frame and then further predicting the truck front and truck body based on the prediction results, effectively improves the detection accuracy of the truck front and truck body.

[0054] like Figure 2a and Figure 2b The diagram illustrates two scenarios: a truck's cab and body are in a connected state and a connected state, respectively. Compared to related technologies, this method effectively predicts the connection state between the truck's cab and body during truck detection. This allows for more accurate prediction of the truck's turning behavior, facilitating trajectory prediction. Furthermore, detecting the connection state allows for more precise prediction of the position of non-truck objects relative to the truck, such as pedestrians. When a pedestrian is positioned in front of the truck, they are closer to the labeled ground truth bounding box, making it easier to predict whether a collision is imminent. Additionally, for trucks facing multiple directions, when network prediction is unstable, the presence of separate bounding boxes for the cab and body effectively reduces training ambiguity.

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar components or components having the same or similar functions throughout. The described embodiments are some, but not all, embodiments of this application. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0056] Figure 3 This is a schematic diagram of a possible scenario provided for an embodiment of this application, such as... Figure 3 As shown, the system includes a terminal device 310, a server 320, and a lidar (not shown). In some embodiments, the terminal 310 provides the server 320 with a pre-trained target detection model. For example, the target detection model includes a 3D backbone network, a 2D backbone network, a first detection head network, and a second detection head network. The lidar provides the server 320 with 3D point cloud data of the area to be predicted. The server 320 performs predictions on the task based on the data provided by the terminal 310 and the lidar 330. Optionally, during the prediction process, the server 320 undertakes the main computational work, and the terminal 310 undertakes the secondary computational work; or, the server 320 undertakes the secondary computational work, and the terminal 310 undertakes the main computational work; or, the server 320 or the terminal 310 can each independently perform computational work.

[0057] Among them, the terminal device 310 may include, but is not limited to, computers, smartphones, tablets, e-book readers, Moving Picture Experts Group Audio Layer III (MP3) players, Moving Picture Experts Group Audio Layer IV (MP4) players, portable computers, in-vehicle computers, wearable devices, desktop computers, set-top boxes, smart TVs, etc.

[0058] Server 320 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0059] Optionally, the number of terminals 310 or servers 320 may be more or less, and this application embodiment does not limit this. In some embodiments, the terminals 310 and servers 320 may also act as nodes in a blockchain system, synchronizing the prediction results with other nodes in the blockchain to enable the widespread application of the prediction results.

[0060] The above provides a brief illustration of the scenario of this application. The following section describes its application... Figure 3 Taking server 320 as an example, this application will explain in detail the two-stage truck detection method provided in the embodiments of this application.

[0061] Please refer to Figure 4 , Figure 4 This is a flowchart illustrating one of the truck detection methods provided in this application, the method including steps S401-S405.

[0062] It is understood that, in this embodiment, "truck" generally refers to a vehicle that can be separated from its body, where body separation means that the cab and body are flexibly connected.

[0063] Step S401: Obtain the 3D point cloud of the task to be predicted.

[0064] The task to be predicted can be a task to be predicted in a certain region. The region to be predicted can include road scenes, trucks, non-trucks, and pedestrians, etc., and the task to be predicted is the truck in that region.

[0065] Specifically, in this embodiment, a lidar is used to detect the 3D point cloud of the task to be predicted, and the 3D point cloud of the task to be predicted obtained by the lidar is obtained based on the communication interface between the two.

[0066] LiDAR (Light Detection and Ranging) is a radar system that uses laser beams to detect the position, velocity, and other characteristics of targets. Its working principle involves emitting a detection signal (laser beam) towards the target, then comparing the received signal reflected back from the target (target echo) with the emitted signal. After appropriate processing, information about the target can be obtained, such as its distance, azimuth, altitude, velocity, attitude, and even shape. This allows for the detection and identification of targets such as aircraft and vehicles. The data acquired by lidar scanning is called lidar point cloud data; the 3D point cloud data collected in this embodiment is obtained through lidar.

[0067] Step S402: Extract the first feature map of the 3D point cloud based on the 3D backbone network, and input the first feature map into the 2D backbone network for feature training to obtain the second feature map.

[0068] In this embodiment, the second feature map obtained by using a two-stage network formed between the 3D backbone network and the 2D backbone network to extract features from the 3D point cloud is more accurate than the feature map obtained by training a single-stage network, laying the foundation for accurate identification of truck frames.

[0069] In one embodiment, before step S402, the method further includes: voxelizing the 3D point cloud to obtain a voxelized 3D point cloud; step S402 specifically involves: extracting the voxelized 3D point cloud based on a 3D backbone network; wherein the 3D backbone network is a sparse convolutional network.

[0070] Combination Figure 5 As shown, the 3D point cloud is first voxelized, ensuring that the voxelized point cloud data is stored in an ordered manner in memory, which helps reduce random memory access and increases data processing efficiency. The voxelized 3D point cloud is then fed into a 3D backbone network for feature extraction, resulting in a two-dimensional feature map (the first feature map). Specifically, the 3D feature network can employ a sparse convolutional network to extract features from sparse voxels. The sparse convolutional network can consist of four modules, each composed of a {2,2,3,3} submanifold sparse convolutional layer and a single sparse convolutional layer. The sparse convolutional layer is used as the last layer of each module, downsampling the 3D feature map by a factor of 2. Finally, the sparse voxel features are superimposed on the z-axis (i.e., a bird's-eye view projection) to obtain the first feature map. Furthermore, the 2D feature map is input into a 2D backbone network, which consists of multiple 2D convolutional layers. First, a set of 2D convolutions, for example, four convolutional layers, is input. Then, a convolution with stride=2 is used to double the feature channels and halve the width and height. This is then followed by another set of 2D convolutions, also containing four convolutional layers. Finally, the feature maps obtained from the two sets of convolutional layers are concatenated to output the final 2D feature map (the second feature map), which is used in the detection head network in subsequent steps.

[0071] Step S403: Obtain the truck-type vehicle detection box based on the second feature map using the first detection head network.

[0072] It is understood that a detection head network is a network containing a detection head, which can be a neural network in an object detection algorithm. Specifically, the detection head in the neural network is used to predict the type and location of the target (bounding boxes). After the 2D image is input into the detection head network, the detection head outputs the corresponding 3D predicted box and the corresponding classification confidence score. In this embodiment, the first detection head network and the second detection head network are only used to distinguish similar objects and have no other meaning. They can have the same network structure or different network structures.

[0073] In one implementation, step S403 specifically involves: obtaining the predicted bounding box of the second feature map and its corresponding classification confidence based on the first detection head network, and obtaining the whole vehicle detection bounding box of the truck class based on the predicted bounding box and the corresponding classification confidence.

[0074] In this embodiment, the task to be predicted is to carry truck-type and non-truck-type objects. The prediction bounding boxes include whole-vehicle prediction boxes for truck-type objects and overall prediction boxes for non-truck-type objects. Each type of prediction box has its corresponding classification confidence level. It can be understood that classification confidence level is the level of confidence in classification; for each 3D prediction box, there is a corresponding classification confidence level. The higher the classification confidence level, the more accurate the classification result. In this embodiment, the classification confidence level can be used to classify the prediction boxes, remove duplicate prediction boxes, and filter out useless prediction boxes.

[0075] In a further embodiment, to improve the efficiency and accuracy of obtaining whole vehicle prediction boxes for trucks, this embodiment removes all whole prediction boxes for non-truck categories and further filters them based on the corresponding classification confidence scores of the whole vehicle prediction boxes for trucks. The process of obtaining whole vehicle detection boxes for trucks based on the prediction boxes and their corresponding classification confidence scores includes the following steps:

[0076] Remove all non-truck category overall prediction boxes;

[0077] Based on the classification confidence level of the whole vehicle prediction box for the truck class, whole vehicle prediction boxes with classification confidence levels within a preset classification confidence level range are selected and used as whole vehicle detection boxes for the truck class.

[0078] In one implementation, vehicle prediction frames that do not fall within a preset classification confidence interval are considered useless and can be filtered out. It should be noted that those skilled in the art can adaptively set the preset classification confidence interval based on existing technology and practical applications.

[0079] Step S404: Input the whole vehicle detection box of the truck type and the corresponding third feature map into the second detection head network for feature training to obtain the front detection box and body detection box corresponding to the whole vehicle detection box of the truck type.

[0080] In one implementation, in order to improve the accuracy of the vehicle front detection box and the vehicle body detection box, the second detection head network is a RoI (Region of Interest) detection head network, which includes RoI pooling layers and convolutional layers.

[0081] The step of inputting the whole vehicle detection bounding box of the truck type and the corresponding third feature map into the second detection head network for feature training (step S404) specifically includes the following steps:

[0082] The RoI pooling layer is used to pool the vehicle detection bounding boxes of the truck class to obtain the features corresponding to the images in the vehicle detection bounding boxes of the truck class; and the convolutional layer is used to train the features corresponding to the images in the vehicle detection bounding boxes.

[0083] In this embodiment, the RoI pooling layer is used to pool the image in the vehicle detection box to remove redundant information, compress features, simplify network complexity, reduce computation, and reduce memory consumption, thereby obtaining the features corresponding to the image in the vehicle detection box. Then, the features corresponding to the image in the vehicle detection box are trained in the convolutional layer to output the predicted front detection box and its corresponding front confidence score, body detection box and its corresponding body confidence score. In some embodiments, the front detection box and body detection box can be further filtered based on the front confidence score and body confidence score to obtain more accurate truck detection results.

[0084] It is understandable that the third feature map is the clipping feature map corresponding to the whole vehicle frame in the second feature map. In some embodiments, in order to improve the detection accuracy of the front and body detection boxes, the difference between the results obtained by the first and second detection head networks and the position of the real boxes is calculated, and gradient backpropagation training is performed to finally obtain a converged network that can stably output obstacle boxes and front and body boxes. Here, the real boxes are the manually labeled boxes.

[0085] Step S405: Determine the separation state between the front and body of the corresponding truck based on the front detection frame and the body detection frame.

[0086] Compared to related technologies, this embodiment extracts features from 3D point clouds using a 3D backbone network and a 2D backbone network to obtain a more accurate two-dimensional feature map. The two-dimensional feature map is then sequentially fed into a first detection head network to obtain the truck's whole vehicle detection frame, and a second detection head network to obtain the truck's front detection frame and body detection frame corresponding to the truck's whole vehicle detection frame. The obtained front and body detection frames are used to predict the separation state between the truck's front and body, which can effectively predict the truck's front and body separation state, thereby further accurately predicting the truck's turning state, facilitating the prediction of its driving trajectory, and effectively improving road prediction safety.

[0087] Please refer to Figure 6 , Figure 6 This is a flowchart illustrating another two-stage truck detection method provided in this application embodiment. Based on the above embodiment, to further determine the separation device between the truck cab and the truck body, this embodiment obtains the deviation angle between the cab detection frame and the truck body detection frame to achieve accurate prediction of the separation state between the truck cab and the truck body. Specifically, the step of determining the separation state between the truck cab and the truck body based on the cab detection frame and the truck body detection frame (step S405) is further divided into steps S405a and S405b, and step S601 is added.

[0088] Step S405a: Obtain the deviation angle between the front detection frame and the body detection frame;

[0089] Step S405b: If the deviation angle is greater than a preset threshold, then it is determined that the truck's front and body are separated, as indicated by the truck front detection frame and the truck body detection frame.

[0090] In practical applications, the truck cab and body are usually connected by a movable joint. When turning, the truck cab and body are generally at a large angle. However, during road travel, the truck may not be turning, but rather there may be some angular deviation between the cab and body due to the driver's operating habits. Such angular deviations may affect road safety or be normal deviations that do not affect surrounding vehicles. To distinguish between turning deviations, deviations affecting road safety, and deviations that do not affect road safety, this embodiment obtains the deviation distance between the cab detection frame and the body detection frame and compares it with a preset threshold. When the preset threshold is greater than a certain threshold, it is determined that the truck cab and body have separated. Based on this separation state, further maintenance of road safety is achieved without affecting the driver's normal operation.

[0091] The preset threshold is set based on angular deviations that do not affect road safety. Those skilled in the art can specifically set the preset threshold in conjunction with practical applications and existing technologies. It is understood that when the front detection frame and the body detection frame are in a straight-line connection state (…),… Figure 2a The deviation angle is 0°.

[0092] Furthermore, in order to improve road safety during truck operation, after determining the separation state between the truck's front and body based on the truck front detection frame and the truck body detection frame (step S201), step S601 is also included.

[0093] Step S601: Obtain the deviation direction between the front detection frame and the body detection frame, and display a prompt message in the deviation direction. The prompt message is used to indicate the turning trend of the truck.

[0094] In one example, if the deviation is to the right, a message indicating that the truck is about to turn right can be displayed. In some examples, to improve road safety, information related to the driving speed, such as prompts to accelerate, can also be displayed.

[0095] Please refer to Figure 7 , Figure 7 This is a flowchart illustrating another truck detection method provided in this application. Based on the above embodiments, to improve the detection accuracy of truck-type and non-truck-type objects and further enhance road safety, this embodiment first obtains the front detection frame and body detection frame of truck-type and non-truck-type objects, and then uses the whole vehicle detection frame of truck-type and the overall detection frame of non-truck-type objects to detect the positional state between truck-type and non-truck-type objects. Specifically, the task to be predicted carries truck-type and non-truck-type objects, and the prediction frame includes the whole vehicle prediction frame of truck-type and the overall prediction frame of non-truck-type objects. After obtaining the front detection frame and body detection frame corresponding to the whole vehicle detection frame of truck-type (i.e., step S404), the method further includes step S701.

[0096] In this embodiment, non-truck objects may include cars, bicycles, etc. In some embodiments, they may also include pedestrians. When performing target detection, the overall prediction box of the pedestrian is obtained to achieve target detection between pedestrians and trucks.

[0097] Step S701: Detect the positional state between trucks and non-trucks based on the truck-type vehicle detection frame, the truck front detection frame, the vehicle body detection frame, and the non-truck-type overall detection frame.

[0098] In one possible implementation, taking a truck and a non-truck (e.g., a car) on a road as an example, the relative positional relationship between the truck and the non-truck is obtained using the truck's whole vehicle detection box and the non-truck's overall prediction box. Then, the positional orientation of the two vehicles is detected based on the truck's front detection box and the non-truck's overall detection box. For example, the relative position of the two vehicles on the road is first obtained based on the truck's whole vehicle detection box and the non-truck's overall prediction box. When the truck's front detection box and the non-truck's overall prediction box are at right angles, the positional state between the truck and the non-truck is detected as a vertical positional orientation.

[0099] In some embodiments, the positional states between different trucks can be predicted more accurately based on the whole vehicle detection frame, the truck front detection frame, and the truck body detection frame. Based on the truck front detection frames, not only the position between trucks can be predicted, but also the driving direction between different trucks. Compared to related technologies, where a single detection frame is typically used to label the detection between trucks and non-trucks, the detection of the position, distance, and state between the truck front and non-trucks is ambiguous. This embodiment uses a vehicle body-separated detection frame recognition method when detecting trucks, which can more accurately locate the positional state and direction between trucks and non-trucks.

[0100] In some embodiments, considering that duplicate (3D) prediction boxes may be output in the detection head network, in order to improve the accuracy of the prediction boxes, this embodiment first removes duplicate 3D prediction boxes. Before proceeding with the steps of obtaining the vehicle front detection box and the vehicle body detection box, it is first determined whether there are duplicate prediction boxes in the 3D prediction boxes. If there are no duplicate prediction boxes, then step S404 is executed to input the whole vehicle detection box of the truck type into the second detection head network for feature training.

[0101] Among them, repeated prediction boxes are those that include multiple prediction boxes for the same object. For example, a truck may include multiple prediction boxes for the whole vehicle.

[0102] Furthermore, for duplicate prediction boxes, a non-maximum suppression algorithm can be used to remove duplicate prediction boxes, and the removed prediction boxes can be input into the second detection head network for feature training.

[0103] In this embodiment, Non-Maximum Suppression (NMS) suppresses low-scoring predicted boxes. It involves two metrics: Intersection over Union (IOU), which is the ratio of the area of ​​the overlapping region of two boxes to the area of ​​the union of the two boxes, and the predicted box score. If multiple predicted boxes predict the same object, and if the IOU value of two boxes is greater than a threshold (typically 0.5 and 0.7), then the NMS algorithm will discard the low-scoring predicted box.

[0104] It is worth noting that non-maximum suppression (NMS) is a common method for removing duplicate bounding boxes because the network will eventually generate duplicate prediction boxes. However, this embodiment considers that the truck body and cab bounding boxes are additional attributes of the overall truck bounding box. If NMS is applied directly, it may cause the cab and body bounding boxes to mismatch, meaning that the cab or body bounding box corresponding to the same overall truck bounding box may be filtered out. To solve this problem, this embodiment only applies NMS to the overall truck bounding box, and then decodes the corresponding cab and body bounding boxes from the network for the remaining overall truck bounding boxes. In other words, in this embodiment, the 3D prediction box of the truck is equivalent to the overall truck bounding box.

[0105] Furthermore, to facilitate the identification of truck-type and non-truck-type objects, this embodiment uniquely labels truck-type objects. Specifically, after obtaining the front detection box and body detection box corresponding to the whole vehicle detection box of the truck-type object (step S404), the method further includes:

[0106] A first annotation is applied to the truck-type vehicle inspection frame, the matching front inspection frame, and the body inspection frame. In this embodiment, the first annotation can be a special color; for example, red is used for all truck-type inspection frames. In some embodiments, other methods can be used to adaptively set the first annotation, and this embodiment does not particularly limit this approach.

[0107] Please refer to Figure 8 , Figure 8 This is a schematic diagram of a two-stage truck detection device provided in an embodiment of this application. The truck detection device includes an acquisition module 81, an extraction module 82, a detection module 83, a training module 83, and a determination module 84.

[0108] The acquisition module 81 is configured to acquire the 3D point cloud of the task to be predicted;

[0109] The extraction module 82 is configured to extract a first feature map of the 3D point cloud based on a 3D backbone network, and input the first feature map into a 2D backbone network for feature training to obtain a second feature map.

[0110] The overall bounding box detection module 83 is configured to detect the predicted bounding box and the corresponding classification confidence of the second feature map based on the first detection head network;

[0111] The acquisition module 81 is further configured to acquire a whole vehicle detection box for trucks based on the prediction box and the corresponding classification confidence.

[0112] The vehicle front and body detection module 83 is configured to input the whole vehicle detection box of the truck type and the corresponding third feature map into the second detection head network for feature training, so as to obtain the vehicle front detection box and body detection box corresponding to the whole vehicle detection box of the truck type.

[0113] The post-processing module 84 is configured to predict the separation state between the front and body of the corresponding truck based on the front detection frame and the body detection frame.

[0114] This application also provides an electronic device, such as... Figure 9 As shown, it includes: processor 91 and memory 92;

[0115] The memory 92 stores computer-executed instructions;

[0116] The processor 91 executes computer execution instructions stored in the memory 92, causing the electronic device to perform the two-stage truck detection method.

[0117] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the two-stage truck detection method described above.

[0118] This application provides an embodiment that will be understood by those skilled in the art: all or some steps, systems, and devices in the methods disclosed above, functional modules / units, can be implemented as software, firmware, hardware, and suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0119] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer.

[0120] Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0121] In the description of the embodiments of this application, the term "and / or" merely indicates a relationship describing the associated objects, meaning that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the term "at least one" indicates any combination of at least two of a plurality of options, for example, including at least one of A, B, and C, which can represent any one or more elements selected from a set including communication between A, B, and C. Furthermore, the term "multiple" means two or more, unless otherwise precisely specified.

[0122] In the description of the embodiments of this application, the terms "first," "second," "third," "fourth," etc. (if present) are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A two-stage truck detection method, characterized in that, include: Obtain the 3D point cloud of the task to be predicted; The first feature map of the 3D point cloud is extracted based on the 3D backbone network, and the first feature map is input into the 2D backbone network for feature training to obtain the second feature map. The truck-type vehicle detection bounding box is obtained based on the second feature map obtained by the first detection head network; The whole vehicle detection box of the truck class and its corresponding third feature map are input into the second detection head network for feature training to obtain the front detection box and body detection box corresponding to the whole vehicle detection box of the truck class; and, based on the front detection box and the body detection box, the separation state between the front and body of the corresponding truck is determined; the third feature map is the clipping feature map corresponding to the whole vehicle frame in the second feature map; The step of determining the separation state between the cab and body of the corresponding truck based on the cab detection frame and the body detection frame includes: Obtain the deviation angle between the front detection frame and the body detection frame; If the deviation angle is greater than a preset threshold, it is determined that the truck's front and body are separated, as indicated by the front detection frame and the body detection frame. Obtain the deviation direction between the front detection frame and the body detection frame, and display a prompt message in the deviation direction. The prompt message is used to indicate the turning trend of the truck.

2. The method according to claim 1, characterized in that, The truck-class vehicle detection bounding boxes obtained based on the second feature map using the first detection head network include: The predicted bounding box of the second feature map and its corresponding classification confidence are obtained based on the first detection head network, and the whole vehicle detection box of the truck class is obtained based on the predicted bounding box and the corresponding classification confidence.

3. The method according to claim 2, characterized in that, The task to be predicted carries truck-type and non-truck-type objects, and the prediction box includes the whole vehicle prediction box for truck-type objects and the overall prediction box for non-truck-type objects. The step of obtaining the whole vehicle detection box for the truck class based on the predicted box and the corresponding classification confidence includes: Remove all non-truck category overall prediction boxes; Based on the classification confidence level of the whole vehicle prediction box for the truck class, whole vehicle prediction boxes with classification confidence levels within a preset classification confidence level range are selected and used as whole vehicle detection boxes for the truck class.

4. The method according to claim 1, characterized in that, The second detection head network is a Region of Interest (RoI) detection head network, which includes RoI pooling layers and convolutional layers. The step of inputting the whole vehicle detection bounding box of the truck type and its corresponding third feature map into the second detection head network for feature training includes: The RoI pooling layer is used to perform pooling operations on the whole vehicle detection bounding box of the truck class and its corresponding third feature map to obtain the features corresponding to the image in the whole vehicle detection bounding box of the truck class; and the convolutional layer is used to train the features corresponding to the image in the whole vehicle detection bounding box.

5. The method according to claim 1, characterized in that, After acquiring the 3D point cloud of the task to be predicted, and before extracting the first feature map of the 3D point cloud based on the 3D backbone network, the process also includes: The 3D point cloud is voxelized to obtain a voxelized 3D point cloud. Extracting the first feature map of the 3D point cloud based on a 3D backbone network includes: extracting a voxelized 3D point cloud based on a 3D backbone network; wherein the 3D backbone network is a sparse convolutional network.

6. The method according to claim 3, characterized in that, The task to be predicted carries both truck-type and non-truck-type objects, and the prediction bounding boxes include whole-vehicle prediction bounding boxes for truck-type objects and overall prediction bounding boxes for non-truck-type objects. The method further includes: detecting the positional state between trucks and non-trucks based on the whole vehicle detection frame of the truck type, the front detection frame and the body detection frame, and the whole detection frame of the non-truck type.

7. A truck inspection device based on a two-stage process, characterized in that, include: The acquisition module is configured to acquire the 3D point cloud of the task to be predicted; The extraction module is configured to extract a first feature map of the 3D point cloud based on a 3D backbone network, and input the first feature map into a 2D backbone network for feature training to obtain a second feature map. The overall bounding box detection module is configured to detect the predicted bounding box and the corresponding classification confidence of the second feature map based on the first detection head network; The acquisition module is further configured to acquire a whole vehicle detection box for the truck class based on the prediction box and the corresponding classification confidence. The vehicle front and body frame detection module is configured to input the whole vehicle detection frame of the truck type and its corresponding third feature map into the second detection head network for feature training, so as to obtain the vehicle front detection frame and body detection frame corresponding to the whole vehicle detection frame of the truck type; the third feature map is the clipping feature map corresponding to the whole vehicle frame in the second feature map; The post-processing module is configured to determine the separation state between the front and body of the corresponding truck based on the front detection frame and the body detection frame. The post-processing module is specifically configured to obtain the deviation angle between the front detection frame and the body detection frame; if the deviation angle is greater than a preset threshold, it is determined that the front and body of the truck corresponding to the front detection frame and the body detection frame are separated. Obtain the deviation direction between the front detection frame and the body detection frame, and display a prompt message in the deviation direction. The prompt message is used to indicate the turning trend of the truck.

8. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the electronic device to perform the two-stage truck detection method according to any one of claims 1-6.

Images