Method, apparatus, device and medium for detecting dropped goods during transportation

Abstract

Embodiments of the present disclosure disclose a method and device for detecting dropped goods during transportation, electronic equipment and a storage medium. The method comprises: during driving of a vehicle, acquiring initial point clouds collected by a vehicle-mounted multi-line laser radar; extracting candidate points belonging to each target from the initial point clouds in combination with attribute information of the laser radar; based on each group of target candidate points extracted, extracting line features respectively and calculating feature confidence of the line features; fusing line features belonging to the same target collected by different laser radars together to obtain feature scores corresponding to each target respectively; and determining whether the detected target is goods dropped from the vehicle according to the feature scores corresponding to each target respectively in combination with multi-frame tracking technology. The present disclosure realizes accurate detection of dropped goods during transportation.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of unmanned driving, and particularly relates to a detection method, device, equipment and medium for dropped goods in a transportation process. BACKGROUND

[0002] In recent years, unmanned transportation vehicles begin to play an increasingly important role in logistics transportation in scenarios such as airports and factories due to their cost and efficiency advantages.

[0003] During the running process of the unmanned transportation vehicle, the transported goods may sometimes fall off the unmanned transportation vehicle. Airport logistics explicitly requires the transported goods to be delivered to the corresponding aircraft in a complete state, and factories also require the transported goods to be transported to the corresponding material area in a complete state. If the goods fall off during the transportation process and are not found, a series of problems will be caused.

[0004] Therefore, it is necessary to accurately detect and identify the goods that fall on the ground in order to handle them in a timely manner. SUMMARY

[0005] To solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present disclosure provide a detection method, device, equipment and medium for dropped goods in a transportation process, which realizes the purpose of stably detecting the goods that fall off in the transportation process.

[0006] In a first aspect, the embodiments of the present disclosure provide a detection method for dropped goods in a transportation process, which comprises:

[0007] During the running process of the vehicle, an initial point cloud collected by a vehicle-mounted multi-line laser radar is acquired;

[0008] Candidate points respectively belonging to each target are extracted from the initial point cloud in combination with attribute information of the laser radar, and a plurality of groups of target candidate points are obtained;

[0009] Based on each group of target candidate points extracted, line features are respectively extracted in combination with the appearance features of the goods transported by the vehicle, and a feature confidence of the extracted line features is calculated;

[0010] Based on the extracted line features and the corresponding feature confidence, line features belonging to the same target collected by different laser radars are fused together, and a feature score corresponding to each target is obtained, the feature score being used to represent the credibility of the corresponding target being dropped goods;

[0011] According to the feature score corresponding to each target, it is determined whether the detected target is the goods that fall off from the vehicle in combination with a multi-frame tracking technology.

[0012] In a second aspect, the embodiments of the present disclosure further provide a device for detecting dropped goods in a transportation process, the device comprising:

[0013] an acquisition module, configured to acquire initial point clouds collected by a vehicle-mounted multi-line laser radar during driving of a vehicle;

[0014] a first extraction module, configured to extract candidate points respectively belonging to each target from the initial point clouds in combination with attribute information of the laser radar, to obtain a plurality of groups of target candidate points;

[0015] a second extraction module, configured to extract line features in combination with appearance features of goods transported by the vehicle based on the extracted groups of target candidate points, and to calculate feature confidence of the extracted line features;

[0016] a fusion module, configured to fuse line features belonging to the same target collected by different laser radars based on the extracted line features and corresponding feature confidence, to obtain feature scores respectively corresponding to each target, the feature scores being used to represent credibility of the corresponding target being dropped goods;

[0017] a recognition module, configured to determine whether the detected target is dropped goods from the vehicle according to the feature scores respectively corresponding to each target in combination with multi-frame tracking technology.

[0018] In a third aspect, the embodiments of the present disclosure further provide an electronic device, which comprises one or more processors, a storage device configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method for detecting dropped goods in a transportation process as described above.

[0019] In a fourth aspect, the embodiments of the present disclosure further provide a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the method for detecting dropped goods in a transportation process as described above is implemented.

[0020] The method for detecting fallen goods in a transportation process provided by the embodiments of the present disclosure comprises the following steps: acquiring initial point clouds collected by a vehicle-mounted multi-line laser radar during the driving of a vehicle; extracting candidate points belonging to each target from the initial point clouds in combination with attribute information of the laser radar to obtain multiple groups of target candidate points; extracting line features in combination with the appearance features of the goods transported by the vehicle based on each group of target candidate points, and calculating the feature confidence of the extracted line features; fusing the line features belonging to the same target collected by different laser radars based on the extracted line features and the corresponding feature confidence to obtain feature scores corresponding to each target, wherein the feature scores are used to represent the credibility of the corresponding target being fallen goods; and determining whether the detected target is the goods fallen from the vehicle in combination with the feature scores corresponding to each target and a multi-frame tracking technology, so as to realize the purpose of stably detecting the fallen goods in the transportation process. BRIEF DESCRIPTION OF DRAWINGS

[0021] The above and other features, advantages, and aspects of the present disclosure will become more apparent by describing in detail the embodiments thereof with reference to the attached drawings. Throughout the drawings, the same or similar reference numerals refer to the same or similar elements. It should be understood that the drawings are schematic, and the original and elements are not necessarily drawn according to the scale.

[0022] Figure 1 A flowchart of a method for detecting fallen goods in a transportation process in the embodiments of the present disclosure;

[0023] Figure 2 A schematic diagram of a multi-line laser radar in the embodiments of the present disclosure;

[0024] Figure 3 A schematic diagram of a circular circular buffer in the embodiments of the present disclosure;

[0025] Figure 4 A flowchart of a method for detecting fallen goods in a transportation process in the embodiments of the present disclosure;

[0026] Figure 5 A structural schematic diagram of a device for detecting fallen goods in a transportation process in the embodiments of the present disclosure;

[0027] Figure 6 A structural schematic diagram of an electronic device in the embodiments of the present disclosure. DETAILED DESCRIPTION

[0028] Embodiments of the present disclosure will be described below in greater detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be implemented in various forms and should not be interpreted as being limited to the embodiments set forth herein, but rather these embodiments are provided so as to more completely and thoroughly understand the present disclosure. It is understood that the drawings and embodiments of the present disclosure are merely for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0029] It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0030] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0031] Figure 1 A flowchart of a method for detecting dropped goods in a transportation process in an embodiment of the present disclosure. The method can be performed by a device for detecting dropped goods in a transportation process, which can be implemented in software and / or hardware, and can be configured in an electronic device. As shown in Figure 1 The method can specifically include the following steps:

[0032] S110, acquiring an initial point cloud collected by a vehicle-mounted multi-line laser radar during vehicle driving.

[0033] The vehicle-mounted multi-line laser radar can be a multi-line mechanical rotating laser radar. At present, automatic driving vehicles are generally equipped with this type of laser radar. Therefore, for an automatic driving vehicle, a laser radar does not need to be additionally arranged to implement the technical solution of the embodiment of the present disclosure, and the existing laser radar of the automatic driving vehicle can be directly reused, which can improve the implementability of the technical solution of the embodiment of the present disclosure, reduce the implementation cost, and have strong practicability.

[0034] It can be understood that the scanning area of the vehicle-mounted multi-line laser radar should include the rear area of the vehicle to scan the goods dropped from the vehicle, so as to identify and detect based on the point cloud obtained by scanning.

[0035] S120, extracting candidate points respectively belonging to each target from the initial point cloud in combination with attribute information of the laser radar to obtain a plurality of groups of target candidate points.

[0036] The attribute information of the laser radar can include its horizontal resolution, the lowest beam depression angle, etc. The lowest beam depression angle can be referred to as Figure 2A schematic diagram of a multi-line laser radar is shown, where the angle β is the depression angle of the lowest line beam.

[0037] Exemplarily, the attribute information of the laser radar is combined to extract candidate points belonging to each target from the initial point cloud, to obtain a plurality of groups of target candidate points, including the following steps 121-124:

[0038] 121. Extract the interest point cloud of the region of interest from the initial point cloud based on the coordinate system of the laser radar itself.

[0039] Exemplarily, the interest point cloud of the region of interest can be extracted from the initial point cloud based on a preset extraction condition. The preset extraction condition includes a distance less than a certain value from the laser radar and / or being scanned by a laser beam capable of scanning the ground.

[0040] The interest point cloud of the region of interest extracted from the initial point cloud based on the preset extraction condition can be realized by the following operations: first, the initial point cloud is subjected to a first screening, for example, an x value and a y value are limited, and a rectangular region composed of the x value and the y value is determined as the region of interest, and the point cloud therein is determined as the interest point cloud. The x value and the y value can be determined according to the installation angle of the laser radar, and the purpose is to roughly screen out the point cloud most likely to belong to the goods falling from the vehicle, to exclude interference point clouds, to reduce the amount of point clouds participating in subsequent calculations, and to achieve the purpose of improving calculation efficiency and calculation accuracy.

[0041] Further, the point cloud obtained by the first screening can also be subjected to a second screening by layer (points scanned by the same laser beam are points on a layer, i.e. points on a single layer). Specifically, points on a set layer are usually selected, which means points scanned by a set laser beam. The set laser beam refers to a laser beam capable of scanning the ground, which can be determined according to the installation angle of the laser radar.

[0042] 122. For the points on a single layer in the interest point cloud, the horizontal projection distance difference of the current point and its adjacent previous point to the laser radar is calculated, where the current point is any point on a single layer in the interest point cloud.

[0043] Wherein, the points on a single layer refer to points scanned by the same laser beam. The points on a single layer are arranged in order according to the scanning direction of the laser radar. In the coordinate system of the laser radar itself, the position of the laser radar is the coordinate origin. Assuming that the coordinates of the current point are (x k , y k ), and the coordinates of the previous point adjacent to the current point are (x k-1 , y k-1), the distance difference between the current point and its adjacent previous point to the horizontal projection of the laser radar is calculated based on the following formula:

[0044]

[0045] wherein dist k represents the distance difference.

[0046] 123. Determine the category attribute of the current point according to the distance difference and the attribute information of the laser radar, wherein the category attribute comprises a continuous point, a boundary point and a noise point.

[0047] For example, if the absolute value of the distance difference is greater than a first threshold value, it is determined that the category attribute of the current point is a boundary point;

[0048] If the absolute value of the distance difference is less than a second threshold value, it is determined that the category attribute of the current point is a continuous point;

[0049] If the distance difference is neither greater than the first threshold value nor less than the second threshold value, it is determined that the category attribute of the current point is a noise point;

[0050] wherein the first threshold value is greater than the second threshold value, the first threshold value is determined based on the depression angle of the lowest beam of the laser radar, and the second threshold value is determined based on the horizontal resolution of the laser radar, and the attribute information comprises the depression angle and the horizontal resolution.

[0051] Generally, the category attribute of the current point is determined by the following expression:

[0052]

[0053] wherein DP represents that the category attribute is a boundary point, CP represents that the category attribute is a continuous point, NP represents that the category attribute is a noise point, d thres is the first threshold value determined based on the depression angle of the lowest beam of the laser radar, and c thres is the second threshold value determined based on the horizontal resolution of the laser radar.

[0054] By determining the category attribute of the current point according to the distance difference between the current point and its adjacent previous point to the horizontal projection of the laser radar, the influence caused by the different angles of different layers in the vertical direction can be eliminated, and the determination accuracy of the category attribute is improved.

[0055] 124. Determine candidate points belonging to each target respectively according to the category attribute of each point in the interest point cloud, the corresponding distance difference and the appearance feature of the goods transported by the vehicle, wherein the candidate points belonging to each target respectively constitute a plurality of groups of target candidate points.

[0056] For example, the points on a single layer in the interest point cloud are traversed in a clockwise direction, and a set of points that all meet the following requirements is determined as a group of target candidate points:

[0057] The category attribute of the first current point P m is a boundary point, the first current point P m is adjacent to a point whose height is less than the ground height threshold, and the first current point P m is adjacent to a point whose horizontal projection distance difference to the laser radar is less than 0, and the first current point P m is determined as the starting point of a group of target candidate points.

[0058] The category attribute of the second current point P n is a boundary point, the second current point P n is adjacent to a point whose height is less than the ground height threshold, and the second current point P n is adjacent to a point whose horizontal projection distance difference to the laser radar is greater than 0, and the second current point P n is determined as the ending point of a group of target candidate points.

[0059] The category attribute of the point between the first current point P m and the second current point P n is a continuous point or a boundary point, and the proportion of the boundary point is less than the proportion threshold.

[0060] The Euclidean distance between the first current point P m and the second current point P n is greater than the lower threshold and less than the upper threshold, and the lower threshold and the upper threshold are determined according to the appearance characteristics of the goods transported by the vehicle.

[0061] The point set {P m …P n-1} that all meet the above requirements is determined as a group of target candidate points, and multiple groups of target candidate points are obtained by traversing each layer in the interest point cloud.

[0062] S130, based on the extracted groups of target candidate points, the appearance characteristics of the goods transported by the vehicle are combined to extract line features respectively, and the feature confidence of the extracted line features is calculated.

[0063] For the ground box type target, since the adjacent surfaces are perpendicular to each other, the laser radar point cloud hitting the surface will form obvious line features on a single layer. When the target distance is far, the feature is in a straight line type; when the target distance is close, an "L" type feature formed by two perpendicular line segments may appear. Based on this, line feature extraction is performed, and whether the detected target is a box type cargo falling from the vehicle is identified according to the extracted line features.

[0064] In the method, line feature extraction is performed for each group of target candidate points. Considering that the surface of the container is not necessarily flat, the point cloud ranging of the radar also has a certain error, and thus the point cloud data has a certain noise, the RANSAC method is used to extract the line feature. The line feature of the point cloud is irrelevant to the height, and in order to reduce the calculation amount, the point cloud is projected into the vehicle coordinate system (i.e., the x-y coordinate system) when the line feature is extracted.

[0065] For example, S130 specifically includes the following sub-steps 131-134:

[0066] 131. For each group of target candidate points, the current group of target candidate points is projected into the vehicle coordinate system, RANSAC fitting is performed on the points in the vehicle coordinate system, an inlier set and an outlier set are obtained, and when the proportion of the points in the inlier set to all points exceeds a proportion threshold, a line segment composed of the points in the inlier set is determined as a main line segment. The current group is any one of the groups.

[0067] 132. RANSAC fitting is continued on the points in the outlier set to determine whether a line segment can be obtained again, and the line segment obtained again is determined as a secondary line segment.

[0068] 133. If the secondary line segment is obtained, an included angle between the main line segment and the secondary line segment is determined, if the included angle meets a right angle requirement, the main line segment and the secondary line segment are determined as a group of “L” type line features, if the included angle does not meet the right angle requirement, the secondary line segment is discarded and the main line segment is reserved as a “one” type line feature. If the secondary line segment is not obtained, the main line segment is determined as a “one” type line feature.

[0069] The included angle meeting the right angle requirement can be specifically represented by the following expression:

[0070]

[0071] wherein, represents a vector of the main line segment, represents a vector of the secondary line segment, and a_thres is an allowed angle error threshold.

[0072] 134. For the “one” type line feature FL x , the feature confidence is calculated as follows:

[0073]

[0074] For the “L” type line feature FR x , the feature confidence is calculated as follows:

[0075]

[0076]

[0077] wherein, C L (FL x ) denotes the feature confidence of the “I”-shaped line feature FL x , C R (FR x ) denotes the feature confidence of the “L”-shaped line feature FR x , N(P) denotes the number of elements of the set P, P{IN1} x denotes the inlier set, P{OUT1} x denotes the outlier set, N(P{IN1} x ) denotes the number of elements in the inlier set P{IN1} x , N(P{OUT1} x ) denotes the number of elements in the outlier set P{OUT1} x , denotes the vector of the primary line segment, denotes the vector of the secondary line segment, and a is the angle deviation penalty weight, which can be set to 3 according to experience.

[0078] S140, based on the extracted line features and the corresponding feature confidences, fusing the line features belonging to the same target collected by different laser radars together to obtain feature scores corresponding to each target respectively, wherein the feature scores are used to represent the credibility of the corresponding target being a fallen cargo.

[0079] For example, S140 specifically includes the following sub-steps 141-144:

[0080] 141, converting the point cloud corresponding to the extracted line features to a unified vehicle coordinate system.

[0081] 142, performing Euclidean distance clustering on the point cloud of each line feature in the vehicle coordinate system to obtain a line feature set, a reference center corresponding to each set, and a reference slope corresponding to each set.

[0082] wherein each line feature set corresponds to a clustering cluster, the corresponding reference center refers to the class center of the clustering cluster, and the corresponding reference slope is the average value of the slope of each line feature in the set.

[0083] 143. Based on the reference slope, determine whether the line features in the line feature set meet the parallel or perpendicular requirements, delete the line features that do not meet the parallel or perpendicular requirements, and determine whether the distance between the geometric center of the point cloud corresponding to the line feature in the line feature set and the reference center exceeds a threshold, delete the line features that exceed the threshold, and obtain the line feature set {FL1,…,FL} belonging to the same target. m ,FR1,…,FR n}, where FL m For the m-th detected "I"-shaped line feature, FR n This is the nth "L"-shaped line feature detected.

[0084] Specifically, the algorithm verifies whether the "L"-shaped line features in each line feature set meet the perpendicularity requirement, and whether the "I"-shaped line features in each line feature set meet the parallelism requirement. "L"-shaped line features that do not meet the perpendicularity requirement are removed from their respective line feature sets, and "I"-shaped line features that do not meet the parallelism requirement are also removed. Simultaneously, for each line feature set, the algorithm also verifies whether the distance between the geometric center of the point cloud corresponding to each line feature and the corresponding reference center exceeds a threshold. Line features exceeding the threshold are removed. Finally, a line feature set {FL1,…,FL} belonging to the same target is obtained. m ,FR1,…,FR n}

[0085] 144. Based on the set of line features {FL1,…,FL} belonging to the same target m ,FR1,…,FR n The feature score corresponding to the target is determined by the following formula:

[0086]

[0087] Where score represents the feature score corresponding to the target, ω L The basic weight for the "I"-type line characteristic can empirically be taken as 1, ω R The basic weight for the "L"-shaped line feature can empirically be taken as 8, ω R Greater than ω L C L (FL m ) indicates the characteristic of the "I" type line FL m Feature confidence, C R (FR n ) indicates the "L" shaped line characteristic FR n The feature confidence level.

[0088] S150, determine whether the detected target is the cargo dropped from the vehicle according to the feature score corresponding to each target respectively in combination with the multi-frame tracking technology.

[0089] Illustratively, S150 includes the following sub-steps 151-152:

[0090] 151, match the same target in the front and rear frames based on the position in the global coordinate system, if the matching fails, determine that the tracking target appears once lost, and when the number of consecutive lost times exceeds a number threshold, delete the tracking target from the tracking queue.

[0091] 152, set a ring-shaped circular buffer with a preset length for each target on the tracking in the tracking queue, each buffer stores the feature score of a single frame of the corresponding target, when the number of frames of the target on the tracking exceeds a frame number threshold, and the feature score of the current frame is greater than a score threshold, and the sum of the feature scores stored in the ring-shaped buffer is greater than a total threshold, determine that the target on the tracking is the cargo dropped from the vehicle.

[0092] It can be understood that false detection is prone to occur when detection is performed based on a single frame of point cloud, and the application adopts a multi-frame tracking manner to improve the accuracy of detection. A good prior knowledge is that the ground cargo box is a static target, so the design of the tracker is relatively simple, and the matching of the targets in the front and rear frames is completed according to the position in the global coordinate system. When the target is not matched, it is considered that the tracking target appears once lost, and when the number of consecutive lost times exceeds a certain number of frames, the target on the tracking is deleted from the tracking queue.

[0093] Each target on the tracking maintains a ring-shaped circular buffer with a length of w (i.e. a preset value), such as a ring-shaped circular buffer shown in a schematic diagram of a ring-shaped circular buffer as shown in Figure 3 , which is a ring-shaped circular buffer with a length of 6, and each buffer stores the feature score of a single frame of the target in history. First, the values inside each buffer are initialized to 0 and stored in a clockwise order, and according to the detection result of each frame, the feature score corresponding to the kth frame is denoted as S k , which is added to the corresponding buffer, when the number of frames k of the tracking exceeds w, the ring-shaped circular buffer is full, S k , which covers S k-w , and the cycle is repeated. The obstacle category discrimination formula is as follows:

[0094]

[0095] , where S k is the feature score stored in the xth buffer, and T k is the number of effective frames on the tracking at the kth frame. When T kexceeds a tracking threshold track thres, and a feature score S k is greater than score thres, and a total feature score sum score recorded in a circular buffer is greater than a threshold sum thres, it is determined that the corresponding obstacle target belongs to the ground box category, i.e., the corresponding obstacle target is a cargo dropped from the vehicle.

[0096] Correspondingly, the method further comprises:

[0097] When it is determined that the tracked target is the cargo dropped by the vehicle, at least the picture captured by the vehicle-mounted camera and the current position information are uploaded to the cloud for event early warning.

[0098] Specifically, when the unmanned vehicle detects an obstacle on the ground during driving and identifies that the obstacle type is a ground box, a ground box discovery event is generated at the vehicle end, and the picture of the corresponding region of the vehicle-mounted camera, position information, etc. are packaged, the event and the packaged data are uploaded to the cloud through the vehicle-cloud link, and the corresponding alarm and prompt are issued on the cloud operation interface, the event result can be viewed by the operation personnel, and it can be judged whether it is a real dropped box through the picture, and a corresponding manual intervention processing process is started.

[0099] The technical scheme of the embodiment of the application can accurately identify the dropped ground box in the unmanned logistics scene of an airport, a factory building, etc., avoid a series of problems caused by the omission of cargo and materials, and help improve the operation efficiency of the entire unmanned logistics transportation. The technical scheme of the embodiment of the application only relies on a laser radar, which is a widely used sensor in unmanned driving, does not rely on a camera, a millimeter wave sensor, etc., and has a relatively low requirement on the performance of the laser radar. In the case of a relatively sparse line bundle of the laser radar (for example, a 16-line bundle laser radar), or in the case of a small size of the detected cargo, a relatively high detection accuracy can also be obtained through the technical scheme of the embodiment of the application. The technical scheme of the embodiment of the application has a low requirement on the hardware of the unmanned vehicle and has a wide application. Through the adjustment of parameters, deployment in a new scene can be completed in a short time and at a low cost. Compared with a data-driven algorithm, a large amount of manpower and material resources do not need to be spent to collect data and train a model, and the efficiency is higher.

[0100] In summary, on the basis of the above-mentioned technical scheme of the embodiment, reference is made to a detection method flowchart of a dropped cargo in a transportation process as shown in Figure 4 , which specifically comprises: laser radar point cloud input-candidate obstacle point extraction-candidate line feature extraction-multi-radar line feature fusion-multi-frame tracking judgment category-identification result reporting cloud event processing.

[0101] Figure 5 ​It is a structural schematic diagram of a detection device for fallen goods in a transportation process in an embodiment of the present disclosure. As shown in the figure: the device comprises: an acquisition module 510, configured to acquire an initial point cloud collected by a vehicle-mounted multi-line laser radar during vehicle driving; Figure 5

[0102] A first extraction module 520 is configured to extract candidate points respectively belonging to each target from the initial point cloud in combination with attribute information of the laser radar, to obtain multiple groups of target candidate points;

[0103] A second extraction module 530 is configured to extract line features in combination with appearance features of goods transported by the vehicle based on each group of target candidate points extracted, and calculate feature confidence of the extracted line features;

[0104] A fusion module 540 is configured to fuse line features belonging to the same target collected by different laser radars based on the extracted line features and corresponding feature confidence, to obtain feature scores respectively corresponding to each target, the feature scores being used to represent credibility of the corresponding target being fallen goods;

[0105] An identification module 550 is configured to determine whether the detected target is a fallen good from the vehicle according to the feature scores respectively corresponding to each target in combination with multi-frame tracking technology.

[0106] Further, the first extraction module 520 comprises: an extraction unit configured to extract an interest point cloud of a region of interest from the initial point cloud based on a coordinate system of the laser radar itself;

[0107] A first calculation unit is configured to calculate a horizontal projection distance difference between a current point and a previous point adjacent to the current point to the laser radar for a point on a single layer in the interest point cloud, the current point being any point on the single layer in the interest point cloud;

[0108] A first determination unit is configured to determine a category attribute of the current point according to the distance difference and the attribute information of the laser radar, the category attribute comprising a continuous point, a boundary point and a noise point;

[0109] A second determination unit is configured to determine candidate points respectively belonging to each target according to the category attribute of each point in the interest point cloud, the corresponding distance difference and the appearance features of the goods transported by the vehicle, wherein the candidate points respectively belonging to each target constitute multiple groups of target candidate points.

[0110] Further, the horizontal projection distance difference is determined based on the following formula:

[0111]

[0112] Wherein, dist k ​represents the horizontal projection distance difference, (x k , y k ) represents the coordinates of the current point in the laser radar self coordinate system, (x k-1 , y k-1 ) represents the coordinates of the previous point adjacent to the current point in the laser radar self coordinate system;

[0113] Further, the first determination unit is specifically configured to: if the absolute value of the distance difference is greater than a first threshold value, determine that the category attribute of the current point is a boundary point;

[0114] if the absolute value of the distance difference is less than a second threshold value, determine that the category attribute of the current point is a continuous point;

[0115] if the distance difference is neither greater than the first threshold value nor less than the second threshold value, determine that the category attribute of the current point is a noise point;

[0116] wherein the first threshold value is greater than the second threshold value, the first threshold value is determined based on the depression angle of the lowest beam of the laser radar, the second threshold value is determined based on the horizontal resolution of the laser radar, and the attribute information includes the depression angle and the horizontal resolution.

[0117] Further, the second determination unit is specifically configured to: traverse the points on a single layer in the interest point cloud in a clockwise direction, and a set of points meeting the following requirements is determined as a group of target candidate points:

[0118] the category attribute of a first current point P m is a boundary point, the height of the previous N continuous points before the first current point P m is less than a ground height threshold value, and the horizontal projection distance difference between the first current point P m and the previous point adjacent thereto to the laser radar is less than 0, it is determined that the first current point P m is a starting point of a group of target candidate points;

[0119] the category attribute of a second current point P n is a boundary point, the height of the next N continuous points after the second current point P n is less than a ground height threshold value, and the horizontal projection distance difference between the second current point P n and the previous point adjacent thereto to the laser radar is greater than 0, it is determined that the second current point P n is a termination point of a group of target candidate points;

[0120] the category attribute of the points between the first current point P m and the second current point P n is a continuous point or a boundary point, and the proportion of the boundary points is less than a proportion threshold value.

[0121] the first current point P m and the second current point P n is greater than a lower threshold and less than an upper threshold, the lower threshold and the upper threshold being determined according to an appearance feature of the goods transported by the vehicle;

[0122] The point set {P m … P n-1} that meets the above requirements is determined as a group of target candidate points, and multiple groups of target candidate points are obtained by traversing each layer in the point cloud of interest.

[0123] Further, the second extraction module 530 includes a fitting unit configured to project each group of target candidate points into the vehicle coordinate system respectively, perform RANSAC fitting on the points in the vehicle coordinate system, obtain an inlier set and an outlier set, and when the proportion of points in the inlier set to all points exceeds a proportion threshold, determine a line segment composed of the points in the inlier set as a main line segment; and continue to perform RANSAC fitting on the points in the outlier set to determine whether a line segment can be obtained again, and determine the line segment obtained again as a secondary line segment.

[0124] A third determination unit is configured to, if the secondary line segment is obtained, determine an included angle between the main line segment and the secondary line segment, and if the included angle meets a right angle requirement, determine the main line segment and the secondary line segment as a group of "L" type line features, and if the included angle does not meet the right angle requirement, discard the secondary line segment and retain the main line segment as a "I" type line feature.

[0125] If the secondary line segment is not obtained, the main line segment is determined as a "I" type line feature.

[0126] A second calculation unit is configured to calculate the feature confidence, wherein for the "I" type line feature FL x , the calculation method of the feature confidence is as follows:

[0127]

[0128] For the "L" type line feature FR x , the calculation method of the feature confidence is as follows:

[0129]

[0130]

[0131] wherein C L (FL x ) represents the feature confidence of the "I" type line feature FL x , and C R (FR x) represents the "L" type line feature FR x x represents the set of inner points, P{OUT1} x represents the set of outer points, N(P{IN1} x ) represents the number of elements in the set of inner points P{IN1} x x ) represents the number of elements in the set of outer points P{OUT1} x represents the vector of the main line segment, represents the vector of the secondary line segment, and α is the angle deviation penalty weight.

[0132] Further, the fusion module 540 comprises a conversion unit configured to convert the point cloud corresponding to the extracted line feature into a unified vehicle coordinate system;

[0133] a clustering unit configured to perform Euclidean distance clustering on the point cloud of each line feature in the vehicle coordinate system to obtain a line feature set, a reference center corresponding to each set, and a reference slope corresponding to each set;

[0134] a screening unit configured to determine whether the line features in the line feature set meet the parallel or perpendicular requirements based on the reference slope, delete the line features that do not meet the parallel or perpendicular requirements, and determine whether the distance between the geometric center of the point cloud corresponding to the line feature in the line feature set and the reference center exceeds a threshold value, delete the line features that exceed the threshold value, and obtain the line feature set {FL1,…,FL m ,FR1,…,FR n} belonging to the same target, wherein FL m is the mth "I" type line feature detected, and FR n is the nth "L" type line feature detected;

[0135] a third calculation unit configured to determine the feature score corresponding to the target based on the line feature set {FL1,…,FL m ,FR1,…,FR n} belonging to the same target by the following formula:

[0136]

[0137] wherein score represents the feature score corresponding to the target, ω L is the basic weight of the "I" type line feature, ω R is the basic weight of the "L" type line feature, ω R is greater than ω L , C L (FL​​​m ) indicates the characteristic of the "I" type line FL m Feature confidence, C R (FR n ) indicates the "L" shaped line characteristic FR n The feature confidence level.

[0138] Furthermore, the identification module 550 includes: a tracking unit, used to match the same target in previous and next frames based on position in the global coordinate system; if the matching fails, it is determined that the tracked target has been lost once; when the number of consecutive losses exceeds the number threshold, the tracked target is deleted from the tracking queue.

[0139] The identification unit is used to set up a circular buffer of a preset length for each target being tracked in the tracking queue. Each buffer stores the feature score of a single frame of the corresponding target. When the number of frames of the target being tracked exceeds the frame number threshold, and the feature score of the current frame is greater than the score threshold, and the sum of the feature scores stored in the circular buffer is greater than the total threshold, then the target being tracked is determined to be the cargo that fell off the vehicle.

[0140] Correspondingly, the device also includes an early warning module, which, when it is determined that the target being tracked is cargo that has fallen from the vehicle, uploads at least the image captured by the vehicle-mounted camera and the current location information to the cloud for event warning.

[0141] The detection device for detecting goods falling during transportation provided in this embodiment can perform the steps in the detection method for detecting goods falling during transportation provided in this embodiment, and can obtain the same beneficial effects, which will not be repeated here.

[0142] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. See below for details. Figure 6 It shows a schematic diagram of a structure suitable for implementing the electronic device 500 in the embodiments of this disclosure. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0143] like Figure 6As shown, the electronic device 500 can include a processing device (e.g., a central processor, a graphics processor, etc.) 501 that can perform various appropriate actions and processes to implement the methods of embodiments as described in the present disclosure according to programs stored in a read-only memory (ROM) 502 or loaded into a random access memory (RAM) 503 from a storage device 508. Various programs and data required by the electronic device 500 for its operation are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0144] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts, thereby implementing the method of detecting dropped goods in transportation as described above. In such embodiments, the computer program can be downloaded and installed from a network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the methods of embodiments of the present disclosure are performed.

[0145] It should be noted that the computer readable medium described above in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer readable medium can be transmitted by any suitable medium, including but not limited to a wire, a cable, an RF (radio frequency) or the like, or any suitable combination of the above.

[0146] The computer readable medium described above can be contained in the electronic device described above; or can exist separately and not be assembled into the electronic device. The computer readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to: during vehicle driving, acquire an initial point cloud collected by a vehicle-mounted multi-line laser radar; extract candidate points respectively belonging to each target from the initial point cloud in combination with attribute information of the laser radar, to obtain multiple groups of target candidate points; based on each group of target candidate points extracted, in combination with an appearance feature of a cargo transported by the vehicle, extract line features respectively, and calculate a feature confidence of the extracted line features; based on the extracted line features and the corresponding feature confidence, fuse line features belonging to the same target collected by different laser radars together, to obtain feature scores respectively corresponding to each target, the feature scores being used to represent a credibility of the corresponding target being a fallen cargo; and determine whether the detected target is a cargo fallen from the vehicle according to the feature scores respectively corresponding to each target in combination with a multi-frame tracking technology.

[0147] Optionally, when the one or more programs are executed by the electronic device, the electronic device can further perform other steps as described in the above embodiments.

[0148] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more of: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0149] The above description is merely illustrative of the exemplary embodiments of this disclosure and the principles of the technology involved. It is understood that modifications and variations of the disclosed embodiments are possible, and also fall within the scope of the disclosure, which is defined by the appended claims. For example, the disclosed features can be combined in any combination, and are not limited to the combinations explicitly disclosed herein.

Claims

1. A method for detecting goods that have fallen during transportation, characterized in that, The method includes: During vehicle operation, the initial point cloud is acquired through onboard multi-line lidar. By combining the attribute information of the lidar, candidate points belonging to each target are extracted from the initial point cloud to obtain multiple sets of target candidate points; the attribute information of the lidar includes horizontal resolution and the depression angle of the lowest beam. Based on the extracted target candidate points, line features are extracted in combination with the appearance features of the goods transported by the vehicle, and the feature confidence of the extracted line features is calculated. For each group of target candidate points, the current group of target candidate points is projected onto the vehicle coordinate system. RANSAC fitting is performed on the points in the vehicle coordinate system to obtain the set of interior points and the set of exterior points. When the proportion of points in the interior point set to all points exceeds the proportion threshold, the line segment composed of the points in the interior point set is determined as the main line segment. RANSAC fitting is performed on the points in the set of external points to determine whether a line segment can be obtained again. The line segment obtained again is determined as the secondary line segment. If a secondary line segment is obtained, the included angle between the primary line segment and the secondary line segment is determined. If the included angle meets the right angle requirement, the primary line segment and the secondary line segment are determined as a set of "L" shaped line features. If the included angle does not meet the right angle requirement, the secondary line segment is discarded, and the primary line segment is retained as a "I" shaped line feature. If the secondary line segment is not obtained, the primary line segment is identified as a "I" type line feature; Based on the extracted line features and the corresponding feature confidence scores, the line features belonging to the same target collected by different lidars are fused together to obtain feature scores corresponding to each target. The feature scores are used to characterize the credibility of the corresponding target being the dropped cargo. Based on the feature scores corresponding to each target and combined with multi-frame tracking technology, it is determined whether the detected target is cargo that fell from the vehicle.

2. The method according to claim 1, characterized in that, The step involves extracting candidate points belonging to each target from the initial point cloud by combining the attribute information of the lidar, thereby obtaining multiple sets of target candidate points, including: The region of interest point cloud is extracted from the initial point cloud based on the self-coordinate system of the lidar. For a point on a single layer in the point cloud of interest, calculate the difference in horizontal projection distance from the current point to the lidar between the current point and its adjacent previous point, where the current point is any point on a single layer of the point cloud of interest; The category attribute of the current point is determined based on the distance difference and the attribute information of the lidar. The category attribute includes continuous point, boundary point and noise point. Candidate points belonging to each target are determined based on the category attribute of each point in the point cloud of interest, the corresponding distance difference, and the appearance characteristics of the goods transported by the vehicle. The candidate points belonging to each target constitute multiple sets of target candidate points.

3. The method according to claim 2, characterized in that, The calculation of the horizontal projection distance difference between the current point and its adjacent previous point to the lidar includes: The horizontal projection distance difference is determined based on the following formula: ; Among them, dist k This represents the difference in horizontal projection distance, (x) k y k (x) represents the coordinates of the current point in the lidar's own coordinate system. k-1 y k-1 () represents the coordinates of the previous point adjacent to the current point in the lidar's own coordinate system; Determining the category attribute of the current point based on the distance difference and the attribute information of the lidar includes: If the absolute value of the distance difference is greater than the first threshold, then the category attribute of the current point is determined to be a boundary point; If the absolute value of the distance difference is less than the second threshold, then the category attribute of the current point is determined to be a continuous point; If the distance difference is neither greater than the first threshold nor less than the second threshold, then the category attribute of the current point is determined to be a noise point; Wherein, the first threshold is greater than the second threshold, the first threshold is determined based on the depression angle of the lowest beam of the lidar, the second threshold is determined based on the horizontal resolution of the lidar, and the attribute information includes the depression angle and the horizontal resolution.

4. The method according to claim 2, characterized in that, The step involves determining candidate points belonging to each target based on the category attribute of each point in the point cloud of interest, the corresponding distance difference, and the appearance characteristics of the goods transported by the vehicle. These candidate points constitute multiple sets of target candidate points, including: For each point in a single layer of the interest point cloud, traversing clockwise, the set of points that satisfy all of the following requirements is determined as a group of target candidate points: First current point P m The category attribute is the boundary point, the first current point P. m The heights of the preceding N consecutive points are all less than the ground height threshold, and the first current point P m If the horizontal projection distance difference between the previous adjacent point and the lidar is less than 0, then the first current point P is determined. m The starting point of a set of target candidate points; Second current point P n The category attribute is the boundary point, and the second current point P n The heights of the subsequent N consecutive points are all less than the ground height threshold, and the second current point P n If the difference in horizontal projection distance from the preceding adjacent point to the lidar is greater than 0, then the second current point P is determined. n The endpoint of a set of target candidate points; First current point P m With the second current point P n The category attribute of the points between them is either continuous points or boundary points, and the proportion of boundary points is less than the proportion threshold; First current point P m With the second current point P n The Euclidean distance between them is greater than the lower threshold and less than the upper threshold, wherein the lower threshold and the upper threshold are determined based on the appearance characteristics of the goods transported by the vehicle; The point set {P} that satisfies the above requirements m …P n-1 A set of target candidate points is identified, and multiple sets of target candidate points are obtained by traversing each layer in the point cloud of interest.

5. The method according to claim 1, characterized in that, Based on the extracted target candidate points, line features are extracted in combination with the appearance features of the goods transported by the vehicle, and the feature confidence of the extracted line features is calculated, including: For the characteristics of the "I" type line The method for calculating feature confidence is as follows: ; For the "L" shaped line characteristics The method for calculating feature confidence is as follows: ; ; in, Indicates the characteristics of the "I" type line Feature confidence, Indicates the characteristics of an "L" shaped line The feature confidence score, N(P), represents the number of elements in set P. Denotes the set of interior points. Describe the set of exterior points. Describe the set of interior points The number of elements in Describe the set of exterior points The number of elements in The vector representing the main line segment. Let α be the vector representing the sub-segment, and α be the weight of the angle deviation penalty.

6. The method according to claim 1, characterized in that, Based on the extracted line features and their corresponding feature confidence levels, line features belonging to the same target acquired by different LiDARs are fused together to obtain feature scores corresponding to each target, including: The point cloud corresponding to the extracted line features is transformed into a unified vehicle coordinate system; In the vehicle coordinate system, Euclidean distance clustering is performed on the point cloud of each line feature to obtain the line feature set, the reference center corresponding to each set, and the reference slope corresponding to each set; Based on the reference slope, determine whether the line features in the line feature set meet the parallel or perpendicular requirements, and delete line features that do not meet the parallel or perpendicular requirements. Also, determine whether the distance between the geometric center of the point cloud corresponding to a line feature in the line feature set and the reference center exceeds a threshold, and delete line features that exceed the threshold, thus obtaining a line feature set belonging to the same target. },in, For the m-th detected "I"-shaped line feature, This is the nth "L"-shaped line feature detected. Based on the set of line features belonging to the same target { The feature score corresponding to the target is determined by the following formula: ; Wherein, "score" represents the feature score corresponding to the target. The basic weights for the "I" type line characteristics, The basic weights for the "L" shaped line characteristics Greater than , Indicates the characteristics of the "I" type line Feature confidence, Indicates the characteristics of an "L" shaped line The feature confidence level.

7. The method according to claim 1, characterized in that, The step of determining whether a detected target is cargo that fell from the vehicle based on the feature scores corresponding to each target and multi-frame tracking technology includes: In the global coordinate system, the same target in the previous and next frames is matched based on its position. If the match fails, it is determined that the tracked target has been lost once. When the number of consecutive losses exceeds the threshold, the tracked target is removed from the tracking queue. For each target tracked in the tracking queue, a circular buffer with a preset length is set. Each buffer stores the feature score of a single frame of the corresponding target. When the number of frames of the tracked target exceeds the frame number threshold, and the feature score of the current frame is greater than the score threshold, and the sum of the feature scores stored in the circular buffer is greater than the total threshold, then the tracked target is determined to be the cargo that fell off the vehicle. Correspondingly, the method also includes: When the target of the tracking is determined to be cargo that has fallen from the vehicle, at least the images captured by the vehicle's onboard camera and the current location information should be uploaded to the cloud for event alert.

8. A detection device for goods falling during transportation, characterized in that, include: The acquisition module is used to acquire the initial point cloud collected by the vehicle-mounted multi-line lidar during vehicle operation; The first extraction module is used to extract candidate points belonging to each target from the initial point cloud by combining the attribute information of the lidar, and obtain multiple sets of target candidate points; the attribute information of the lidar includes horizontal resolution and the depression angle of the lowest beam. The second extraction module is used to extract line features based on the extracted target candidate points and the appearance features of the goods transported by the vehicle, and to calculate the feature confidence of the extracted line features. For each group of target candidate points, the current group of target candidate points is projected onto the vehicle coordinate system, and RANSAC fitting is performed on the points in the vehicle coordinate system to obtain the set of interior points and the set of exterior points. When the proportion of points in the interior point set to all points exceeds the proportion threshold, the line segment composed of the points in the interior point set is determined as the main line segment. RANSAC fitting is performed on the points in the set of external points to determine whether a line segment can be obtained again. The line segment obtained again is determined as the secondary line segment. If a secondary line segment is obtained, the included angle between the primary line segment and the secondary line segment is determined. If the included angle meets the right angle requirement, the primary line segment and the secondary line segment are determined as a set of "L" shaped line features. If the included angle does not meet the right angle requirement, the secondary line segment is discarded, and the primary line segment is retained as a "I" shaped line feature. If the secondary line segment is not obtained, the primary line segment is identified as a "I" type line feature; The fusion module is used to fuse the line features of the same target collected by different lidars together based on the extracted line features and the corresponding feature confidence scores, so as to obtain the feature scores corresponding to each target. The feature scores are used to characterize the confidence that the corresponding target is the fallen cargo. The identification module is used to determine whether the detected target is cargo that fell from the vehicle, based on the feature scores corresponding to each target and multi-frame tracking technology.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images