Foreign matter detection method and device

Abstract

The invention relates to a foreign matter detection method and device. The method comprises the steps of obtaining vehicle driving data of a target vehicle and environment point cloud data of an environment where the target vehicle is located in the driving process of the target vehicle, performing fusion processing on the environment point cloud data by adopting the vehicle driving data to obtain target environment data, and obtaining a point cloud reconstruction model according to the target environment data and the point cloud reconstruction model. A foreign matter detection result of the environment where the target vehicle is located is determined, the point cloud reconstruction model is used for reconstructing environment data not containing foreign matter point cloud data, the point cloud reconstruction model is obtained through training of initial environment data and sample driving data of the sample vehicle, and the initial environment data does not contain the foreign matter point cloud data. By adopting the method, the accuracy of foreign matter detection can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a foreign object detection method and device, which can be applied to intelligent vehicles. Background Art

[0002] With the rapid development of technologies such as unmanned driving and intelligent robots, more and more driving tasks will be handed over to machines for decision-making and processing. Among them, vehicle foreign object detection is one of the important driving tasks. The existing foreign object detection method generally collects data on the road through the laser radar deployed on the vehicle, and then compares the collected data with the preset foreign object data to obtain the detection result.

[0003] However, since there are many types of foreign objects that may appear in real driving scenarios, the existing foreign object detection methods need to enumerate all types of foreign objects in advance, which increases the difficulty of collecting preset foreign object data and thus reduces the accuracy of foreign object detection. Summary of the invention

[0004] Based on this, it is necessary to provide a foreign object detection method and device that can improve the accuracy of foreign object detection in response to the above technical problems.

[0005] In a first aspect, the present application provides a foreign body detection method, comprising:

[0006] During the driving process of the target vehicle, the vehicle driving data of the target vehicle and the environmental point cloud data of the environment where the target vehicle is located are obtained;

[0007] The vehicle driving data is used to fuse the environmental point cloud data to obtain the target environmental data;

[0008] According to the target environment data and the point cloud reconstruction model, the foreign object detection result of the target vehicle's environment is determined; wherein, the point cloud reconstruction model is used to reconstruct the environment data that does not contain foreign object point cloud data, and the point cloud reconstruction model is trained using the initial environment data and sample driving data of the sample vehicle, and the initial environment data does not contain foreign object point cloud data.

[0009] In one embodiment, determining a foreign object detection result of an environment in which a target vehicle is located according to target environment data and a point cloud reconstruction model includes:

[0010] The target environment data is input into the point cloud reconstruction model to obtain the reconstructed environment data; wherein the reconstructed environment data does not include the foreign body point cloud data; based on the target environment data and the reconstructed environment data, the reconstruction error corresponding to each environment point cloud in the target environment data is determined; based on the reconstruction error corresponding to the target environment data and each environment point cloud, the foreign body detection result of the environment in which the target vehicle is located is determined.

[0011] In one embodiment, determining a foreign object detection result of an environment in which a target vehicle is located according to target environment data and a reconstruction error corresponding to each environment point cloud includes:

[0012] The spatial area of ​​the target vehicle's environment is evenly divided to obtain unit areas; the number of point clouds in each unit area is determined based on the target environment data; the foreign object detection result of each unit area is determined based on the number of point clouds in each unit area and the reconstruction error corresponding to the environmental point cloud in each unit area; the foreign object detection result of the target vehicle's environment is determined based on the foreign object detection result of each unit area.

[0013] In one embodiment, the foreign body detection result of each unit area is determined according to the number of point clouds in each unit area and the reconstruction error corresponding to the environmental point cloud in each unit area, including:

[0014] For each unit area, the existence of objects in the unit area is determined based on the quantity threshold and the number of point clouds in the unit area; the abnormality of objects in the unit area is determined based on the reconstruction error corresponding to each environmental point cloud in the unit area; the foreign body detection result of the unit area is determined based on the existence of objects and the abnormality of objects.

[0015] In one embodiment, determining the abnormality of an object in a unit area according to the reconstruction error corresponding to each environmental point cloud in the unit area includes:

[0016] According to the reconstruction errors corresponding to the environmental point clouds in the unit area, the average reconstruction error corresponding to the unit area is determined; according to the error threshold and the average reconstruction error, the abnormality of the object in the unit area is determined.

[0017] In one embodiment, the vehicle driving data is used to fuse the environmental point cloud data to obtain the target environmental data, including:

[0018] The vehicle driving data is used to perform motion compensation processing on the environmental point cloud data to obtain compensated environmental data; the compensated environmental data is filtered to obtain target environmental data.

[0019] In one embodiment, the environmental point cloud data includes three-dimensional coordinate data and reflection intensity data of each environmental point cloud; filtering the compensated environmental data to obtain target environmental data includes:

[0020] According to the configuration information of the acquisition device corresponding to the environmental point cloud data, the three-dimensional coordinate threshold and the reflection intensity threshold are determined; the three-dimensional coordinate threshold and the reflection intensity threshold are used to eliminate abnormal point clouds from the compensated environmental data to obtain first filtered data; and the target environmental data are determined based on the first filtered data.

[0021] In one embodiment, determining target environment data according to the first filtered data includes:

[0022] According to the position characteristics of each environmental point cloud in the first filtered data, the first filtered data is processed to eliminate abnormal point clouds to obtain second filtered data; and the second filtered data is processed by voxel filtering to obtain target environmental data.

[0023] In one embodiment, the point cloud reconstruction model is trained in the following manner:

[0024] Acquire initial environmental data and sample driving data of a sample vehicle; use the sample driving data to process the initial environmental data to obtain sample environmental data; input the sample environmental data into a variational autoencoder model to obtain sample reconstruction data; determine a model reconstruction loss and a model divergence loss based on the sample environmental data and the sample reconstruction data; use the model reconstruction loss and the model divergence loss to train the variational autoencoder model to obtain a point cloud reconstruction model.

[0025] In one embodiment, the variational autoencoder model includes an encoder, a sampling layer, and a decoder, and the sample environment data is input into the variational autoencoder model to obtain sample reconstruction data, including:

[0026] The sample environment data is input into the encoder to obtain the mean parameter and variance parameter corresponding to the sample environment data output by the encoder after encoding the sample environment data; the mean parameter and variance parameter are input into the sampling layer to obtain the environment data features output by the sampling layer after sampling the mean parameter and variance parameter; the environment data features are input into the decoder to obtain the sample reconstruction data output by the decoder after decoding the environment data features.

[0027] In a second aspect, the present application also provides a foreign body detection device, comprising:

[0028] A data acquisition module is used to acquire the vehicle driving data of the target vehicle and the environmental point cloud data of the environment in which the target vehicle is located during the driving process of the target vehicle;

[0029] A data processing module is used to fuse the environmental point cloud data with the vehicle driving data to obtain the target environmental data;

[0030] The foreign object detection module is used to determine the foreign object detection result of the target vehicle's environment based on the target environment data and the point cloud reconstruction model; wherein the point cloud reconstruction model is used to reconstruct the environment data that does not contain the foreign object point cloud data, and the point cloud reconstruction model is trained using the initial environment data and sample driving data of the sample vehicle, and the initial environment data does not contain the foreign object point cloud data.

[0031] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0032] During the driving process of the target vehicle, the vehicle driving data of the target vehicle and the environmental point cloud data of the environment where the target vehicle is located are obtained;

[0033] The vehicle driving data is used to fuse the environmental point cloud data to obtain the target environmental data;

[0034] According to the target environment data and the point cloud reconstruction model, the foreign object detection result of the target vehicle's environment is determined; wherein, the point cloud reconstruction model is used to reconstruct the environment data that does not contain foreign object point cloud data, and the point cloud reconstruction model is trained using the initial environment data and sample driving data of the sample vehicle, and the initial environment data does not contain foreign object point cloud data.

[0035] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0036] During the driving process of the target vehicle, the vehicle driving data of the target vehicle and the environmental point cloud data of the environment where the target vehicle is located are obtained;

[0037] The vehicle driving data is used to fuse the environmental point cloud data to obtain the target environmental data;

[0038] According to the target environment data and the point cloud reconstruction model, the foreign object detection result of the target vehicle's environment is determined; wherein, the point cloud reconstruction model is used to reconstruct the environment data that does not contain foreign object point cloud data, and the point cloud reconstruction model is trained using the initial environment data and sample driving data of the sample vehicle, and the initial environment data does not contain foreign object point cloud data.

[0039] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0040] During the driving process of the target vehicle, the vehicle driving data of the target vehicle and the environmental point cloud data of the environment where the target vehicle is located are obtained;

[0041] The vehicle driving data is used to fuse the environmental point cloud data to obtain the target environmental data;

[0042] According to the target environment data and the point cloud reconstruction model, the foreign object detection result of the target vehicle's environment is determined; wherein, the point cloud reconstruction model is used to reconstruct the environment data that does not contain foreign object point cloud data, and the point cloud reconstruction model is trained using the initial environment data and sample driving data of the sample vehicle, and the initial environment data does not contain foreign object point cloud data.

[0043] The above-mentioned foreign object detection method and device obtain the vehicle driving data of the target vehicle and the environmental point cloud data of the target vehicle's environment during the target vehicle's driving process, and use the vehicle driving data to fuse the environmental point cloud data to obtain the target environmental data; then, based on the point cloud reconstruction model trained by using the initial environmental data that does not contain the foreign object point cloud data and the sample driving data, and the target environmental data, reconstruct the environmental data that does not contain the foreign object point cloud data, and then determine the foreign object detection result of the target vehicle's environment. Compared with the related art, in which the point cloud data collected by the laser radar is directly used for foreign object detection, the above-mentioned method introduces a point cloud reconstruction model constructed based on a variational autoencoder model, reconstructs the target environmental data by using the point cloud reconstruction model, and determines the foreign object detection result according to the reconstruction result, without the need to collect all the point cloud data corresponding to the foreign objects in advance, which not only improves the efficiency of foreign object detection processing, but also improves the accuracy of foreign object detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 is an application environment diagram of a foreign body detection method in an embodiment;

[0046] Figure 2 Schematic diagram of a foreign body detection method in one embodiment;

[0047] Figure 3 A schematic diagram of a process for determining a foreign body detection result in one embodiment;

[0048] Figure 4 A schematic diagram of a process for determining a foreign body detection result in another embodiment;

[0049] Figure 5 A schematic diagram of a process for determining a foreign body detection result in another embodiment;

[0050] Figure 6 A schematic diagram of a process for determining target environment data in one embodiment;

[0051] Figure 7 A schematic diagram of a process for determining target environment data in another embodiment;

[0052] Figure 8 A schematic diagram of a process for determining target environment data in yet another embodiment;

[0053] Fig. 9 is a schematic diagram of a variational autoencoder in one embodiment;

[0054] Fig.10 is a schematic flow chart of a foreign body detection method in another embodiment;

[0055] Fig.11 is a structural block diagram of a foreign body detection device in one embodiment;

[0056] Fig.12 is a structural block diagram of a foreign body detection device in another embodiment;

[0057] Fig.13 is a structural block diagram of a foreign body detection device in yet another embodiment;

[0058] Fig.14 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] The foreign body detection method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the acquisition device 102 communicates with the vehicle terminal 104 through the network. The data storage system can store data that the vehicle terminal 104 needs to process, such as vehicle driving data. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. For example, during the driving of the target vehicle, the vehicle terminal 104 obtains the vehicle driving data of the target vehicle and the environmental point cloud data of the target vehicle's environment through the acquisition device 102, and then uses the vehicle driving data to fuse the environmental point cloud data to obtain the target environment data; finally, according to the target environment data and the point cloud reconstruction model, the foreign object detection result of the target vehicle's environment is determined.

[0061] With the rapid development of technologies such as unmanned driving and intelligent robots, more and more driving tasks will be handed over to machines for decision-making and processing. Among them, vehicle foreign object detection is one of the important driving tasks. The existing foreign object detection method generally collects data on the road through the laser radar deployed on the vehicle, and then compares the collected data with the preset foreign object data to obtain the detection result.

[0062] However, since there are many types of foreign objects that may appear in real driving scenarios, the existing foreign object detection methods need to enumerate all types of foreign objects in advance, which increases the difficulty of collecting preset foreign object data and thus reduces the accuracy of foreign object detection.

[0063] Based on this, in an exemplary embodiment, Figure 2 As shown, a foreign body detection method is provided, which is applied to Figure 1 The vehicle-mounted terminal in the example is used as an example to illustrate, and the specific steps include:

[0064] S201, during the driving process of the target vehicle, obtaining vehicle driving data of the target vehicle and environmental point cloud data of the environment in which the target vehicle is located.

[0065] Among them, the target vehicle is an intelligent vehicle with foreign object detection needs; the vehicle driving data refers to the data generated by the target vehicle during the driving process, for example, it may include but is not limited to speed, acceleration data, position and posture data, etc.; the environmental point cloud data is the relevant data of the point cloud of the environment in which the target vehicle is located, for example, it may include but is not limited to coordinate data, etc.

[0066] Optionally, during the driving process of the target vehicle, the environmental point cloud data of the environment in which the target vehicle is located can be collected using a collection device deployed on the target vehicle; then, the vehicle driving data of the target vehicle can be obtained based on the inertial sensor inside the vehicle.

[0067] For example, a laser radar device deployed on a vehicle can be used to collect environmental point cloud data of the vehicle's environment; then, a three-axis accelerometer and a three-axis gyroscope inside the vehicle can be used to obtain the vehicle's inertial sensor (Inertial Measurement Unit, IMU) data as vehicle driving data.

[0068] S202, using the vehicle driving data to fuse the environmental point cloud data to obtain target environmental data.

[0069] Among them, the target environment data is the point cloud data after fusion processing.

[0070] Optionally, in order to ensure the reliability of the point cloud of the target vehicle's environment, the vehicle driving data can be used to process each environmental point cloud in the environmental point cloud data, thereby obtaining the target environmental data. For example, the vehicle driving data and the environmental point cloud data are simultaneously input into a trained point cloud processing model, and the point cloud processing model outputs the target environmental data based on the vehicle driving data, the environmental point cloud data, and the model parameters.

[0071] Alternatively, in order to ensure the reliability of the fusion processing, the vehicle driving data and the environmental point cloud data collected in the same time period may be aligned in time according to the data collection frequency of the vehicle driving data and the data collection frequency of the environmental point cloud data.

[0072] For example, if the data collection frequency of vehicle driving data is 100 Hz and the data collection frequency of environmental point cloud data is 10 Hz, it proves that while collecting 1 frame of environmental point cloud data, 10 frames of vehicle driving data will be collected. Therefore, 1 frame of environmental point cloud data under timestamp A can be aligned with 10 consecutive frames of vehicle driving data starting with timestamp A.

[0073] Furthermore, since point cloud data is collected for a certain area while the vehicle is in motion, it is easy for there to be a position difference between the collected point cloud data and the actual point cloud data. Therefore, for the vehicle driving data and environmental point cloud data collected within the same time period, motion compensation processing can be performed on the environmental point cloud data based on the vehicle driving data to obtain target environmental data, so as to restore the collected environmental point cloud data to the real point cloud data corresponding to the environment where the target vehicle is located.

[0074] S203, determining a foreign object detection result of the environment in which the target vehicle is located according to the target environment data and the point cloud reconstruction model.

[0075] Among them, the point cloud reconstruction model is used to reconstruct environmental data that does not contain foreign object point cloud data. The point cloud reconstruction model is trained using the initial environmental data and sample driving data of the sample vehicle. The initial environmental data does not contain foreign object point cloud data; the foreign object point cloud data is the point cloud data corresponding to foreign objects in the driving environment, for example, it can be the point cloud data corresponding to garbage on the lane; the foreign object detection result is the detection result of whether the environment in which the target vehicle is located contains foreign objects.

[0076] In order to ensure the reliability of foreign object detection, in this embodiment, it is necessary to pre-train a point cloud reconstruction model using sample driving data and initial environmental data that does not contain foreign object point cloud data, so that the point cloud reconstruction model can learn the data potential representation corresponding to the normal environment point cloud that does not contain foreign objects, and then the trained point cloud reconstruction model can reconstruct the point cloud data corresponding to the normal environment based on the input point cloud data.

[0077] Based on this, we can judge whether the point cloud reconstruction model has eliminated part of the input point cloud data based on the difference between the input point cloud data and the reconstructed point cloud data. If so, it proves that the part of the input point cloud data eliminated by the point cloud reconstruction model is foreign point cloud data, that is, there are foreign objects in the environment corresponding to the input point cloud data; if not, it proves that the point cloud reconstruction model has not eliminated the point cloud data, that is, there are no foreign objects in the environment corresponding to the input point cloud data.

[0078] For example, when the input point cloud data B includes foreign bodies B1 and B2, the input point cloud data B is input into the point cloud reconstruction model, and the point cloud reconstruction model can output output point cloud data B' that does not include foreign bodies B1 and B2.

[0079] Optionally, the target environment data can be input into the point cloud reconstruction model, and the point cloud reconstruction model outputs output data corresponding to the target environment data based on the target environment data and model parameters; subsequently, the point cloud difference between the target environment data and the output data can be used to determine whether there are foreign objects in the environment where the target vehicle is located, the location of the foreign objects, and the appearance information of the foreign objects and other foreign object detection results.

[0080] For example, if the difference value between the target environment data and the output data is greater than a preset difference threshold, it is determined that there is foreign matter in the environment where the target vehicle is located. If the difference value between the target environment data and the output data is less than or equal to the preset difference threshold, it is determined that there is no foreign matter in the environment where the target vehicle is located.

[0081] Furthermore, when there are foreign objects in the environment where the target vehicle is located, the location of the foreign objects and their appearance information can be displayed through the display device associated with the vehicle-mounted terminal for the driver in the vehicle to check in time. At the same time, the audio equipment in the vehicle can be used to play a prompt sound to remind the driver; accordingly, the location of the foreign object can be sent to the vehicle's control system so that the control system can automatically avoid the foreign object according to its location.

[0082] In the above foreign object detection method, the vehicle driving data of the target vehicle and the environmental point cloud data of the target vehicle's environment are obtained during the target vehicle's driving process, and the environmental point cloud data are fused with the vehicle driving data to obtain the target environmental data; then, the environmental data that does not contain the foreign object point cloud data is reconstructed based on the point cloud reconstruction model trained with the initial environmental data that does not contain the foreign object point cloud data and the sample driving data, as well as the target environmental data, to determine the foreign object detection result of the target vehicle's environment. Compared with the related art, in which the point cloud data collected by the laser radar is directly used for foreign object detection, the above method introduces a point cloud reconstruction model constructed based on a variational autoencoder model, reconstructs the target environmental data by using the point cloud reconstruction model, and determines the foreign object detection result based on the reconstruction result, without the need to collect all the point cloud data corresponding to the foreign objects in advance, which not only improves the efficiency of foreign object detection processing, but also improves the accuracy of foreign object detection.

[0083] In order to ensure the accuracy of the foreign body detection result, based on the above embodiment, in this embodiment, an optional method for determining the foreign body detection result is provided, such as Figure 3 As shown, the specific steps include:

[0084] S301, inputting target environment data into the point cloud reconstruction model to obtain reconstructed environment data.

[0085] The reconstructed environment data is point cloud data output by the point cloud reconstruction model. Furthermore, the reconstructed environment data does not include foreign body point cloud data.

[0086] Optionally, the target environment data may be input into a point cloud reconstruction model, and the point cloud reconstruction model removes foreign point cloud data contained in the target environment data according to the target environment data and model parameters, and then outputs the reconstructed environment data.

[0087] S302, determining the reconstruction error corresponding to each environmental point cloud in the target environmental data according to the target environmental data and the reconstructed environmental data.

[0088] The environmental point cloud is each point cloud in the target environmental data; and the reconstruction error is the error between each environmental point cloud in the target environmental data and the reconstructed environmental data.

[0089] Optionally, for each environment point cloud, a reconstruction error corresponding to the environment point cloud in the target environment data may be determined according to differences in each dimension between the environment point cloud in the target environment data and the reconstructed environment data.

[0090] For example, when the environmental point cloud data includes point cloud data in the three-dimensional coordinate dimension and point cloud data in the reflection intensity dimension, for each environmental point cloud, the position reconstruction error and reflectivity reconstruction error corresponding to the environmental point cloud can be determined based on the target environmental data and the reconstructed environmental data; then, the sum of the position reconstruction error and the reflectivity reconstruction error corresponding to the environmental point cloud can be used as the reconstruction error corresponding to the environmental point cloud.

[0091] S303, determining a foreign object detection result of the environment in which the target vehicle is located according to the target environment data and the reconstruction error corresponding to each environment point cloud.

[0092] Optionally, based on the target environment data, the abnormal information corresponding to each environment point cloud can be determined according to the reconstruction error corresponding to each environment point cloud; then, the foreign object detection result of the environment in which the target vehicle is located is determined in combination with the abnormal information corresponding to each environment point cloud.

[0093] For example, the target environment data and the reconstruction errors corresponding to each environment point cloud can be input into a trained first abnormality determination model, and the first abnormality determination model outputs the foreign object detection result of the environment in which the target vehicle is located based on the target environment data, the reconstruction errors corresponding to each environment point cloud and the model parameters.

[0094] In the embodiment of the present application, a reconstruction error corresponding to each environmental point cloud is introduced, and the accuracy of the foreign object detection result can be ensured by determining the foreign object detection result based on the reconstruction error corresponding to each environmental point cloud.

[0095] In order to ensure the accuracy of the foreign body detection result, based on the above embodiment, in this embodiment, another optional method for determining the foreign body detection result is provided, such as Figure 4 As shown, the specific steps include:

[0096] S401, evenly divide the spatial area of ​​the environment where the target vehicle is located to obtain unit areas.

[0097] The unit area is a spatial area of ​​standard size within the spatial area of ​​the environment where the target vehicle is located.

[0098] Optionally, the spatial area of ​​the environment where the target vehicle is located may be evenly divided based on preset size information to obtain unit areas.

[0099] For example, in order to ensure the reliability of vehicle operation, the spatial area S of the target vehicle's environment can be set to a three-dimensional space with the target vehicle's location as the center, 120 meters in front of the vehicle, 30 meters behind, 3 meters on each side, and a height range of [-2.5, 3.5] meters; then, the spatial area S can be evenly divided with a unit side length of 0.25 meters to obtain each unit area.

[0100] S402: Determine the number of point clouds in each unit area according to the target environment data.

[0101] Optionally, the location information of each environment point cloud in the target environment data can be compared with the location range of each unit area, and the number of point clouds in each unit area can be determined based on the comparison result.

[0102] S403, determining the foreign body detection result of each unit area according to the number of point clouds in each unit area and the reconstruction error corresponding to the environmental point cloud in each unit area.

[0103] Optionally, for each unit area, it is possible to determine whether there is an object in the unit area based on the number of point clouds in the unit area, and to determine whether there is an abnormality in the unit area based on the reconstruction error corresponding to the environmental point cloud in the unit area, thereby determining the foreign object detection result for the unit area.

[0104] For example, for each unit area, the number of point clouds in the unit area and the reconstruction error corresponding to the environmental point cloud can be input into the second abnormality determination model, and the second abnormality determination model outputs the foreign body detection result of the unit area based on the number of point clouds, the reconstruction error corresponding to each environmental point cloud and the model parameters.

[0105] S404, determining the foreign object detection result of the environment where the target vehicle is located according to the foreign object detection result of each unit area.

[0106] Optionally, the foreign object detection results of each unit area can be combined to determine the foreign object detection results of the environment where the target vehicle is located. Exemplarily, based on the foreign object detection results of each unit area, abnormal areas with abnormal conditions can be extracted from each unit area; then, based on the location information of each abnormal area, the environmental point clouds in each abnormal area can be spliced ​​to obtain the foreign object detection results of the environment where the target vehicle is located.

[0107] Alternatively, the foreign object detection results of each abnormal area can be directly input into a trained third abnormality determination model, and the third abnormality determination model outputs the foreign object detection results of the target vehicle's environment based on the foreign object detection results of each abnormal area and model parameters.

[0108] In the embodiment of the present application, by determining the foreign object detection result of each unit area one by one, and then determining the foreign object detection result of the environment in which the target vehicle is located, the accuracy of the foreign object detection result can be guaranteed.

[0109] In order to ensure the accuracy of the foreign body detection result of the unit area, based on the above embodiment, in this embodiment, another optional method for determining the foreign body detection result is provided, such as Figure 5 As shown, the specific steps include:

[0110] S501 : for each unit area, determine the existence of objects in the unit area according to a quantity threshold and the number of point clouds in the unit area.

[0111] The quantity threshold is the number of point clouds used to measure whether an object exists. For example, the quantity threshold can be set to 15; the object existence status is used to characterize whether an object exists in a unit area.

[0112] It is understandable that, since the number of point clouds can reflect the existence of objects, for each unit area, the existence of objects in the unit area can be determined based on the relationship between the number threshold and the number of point clouds in the unit area.

[0113] For example, for each unit area, when the number of point clouds in the unit area is greater than or equal to the quantity threshold, it proves that there is an object in the unit area; when the number of point clouds in the unit area is less than the quantity threshold, it proves that there is no object in the unit area.

[0114] S502, determining abnormal conditions of objects in the unit area according to reconstruction errors corresponding to each environmental point cloud in the unit area.

[0115] Optionally, for each unit area, the overall reconstruction anomaly in the unit area may be determined based on the reconstruction errors corresponding to the environmental point clouds in the unit area, and further the anomaly of the objects in the unit area may be determined.

[0116] Exemplarily, based on the reconstruction errors corresponding to each environmental point cloud in the unit area, an average reconstruction error corresponding to the unit area is determined; based on the error threshold and the average reconstruction error, the abnormality of the object in the unit area is determined.

[0117] Among them, the reconstruction error average is the mean of the reconstruction errors corresponding to each environmental point cloud in the unit area, which can characterize the overall reconstruction anomaly in the unit area; the error threshold is the reconstruction error value used to measure whether it is an abnormal object. For example, the error threshold can be set to 0.65; the object anomaly is used to characterize whether there is a foreign object in the unit area.

[0118] Optionally, for each unit area, the reconstruction error average value corresponding to the unit area can be calculated based on the reconstruction errors corresponding to each environmental point cloud in the unit area; then, the abnormality of the object in the unit area can be determined based on the relationship between the error threshold and the reconstruction error average value.

[0119] For example, for each unit area, when the average reconstruction error in the unit area is greater than or equal to the error threshold, it proves that there is an abnormal object in the unit area; when the average reconstruction error in the unit area is less than the error threshold, it proves that there is no abnormal object in the unit area.

[0120] S503, determining the foreign body detection result of the unit area according to the existence of the object and the abnormality of the object.

[0121] It is understandable that in actual applications, small foreign objects such as flying insects may be collected, which may cause the average reconstruction error in the unit area to be greater than the error threshold. Therefore, it is necessary to comprehensively consider the object existence and object abnormality of each unit area to determine the foreign object detection result of the unit area.

[0122] Exemplarily, for each unit area, when there is no object in the unit area, it is not necessary to determine whether there is no abnormal object in the unit area based on the abnormal situation of the object in the unit area; when there is an object in the unit area, it is necessary to determine whether there is an abnormal object in the unit area based on the abnormal situation of the object in the unit area.

[0123] In the embodiment of the present application, by determining the foreign object detection result of the unit area based on the object information in two dimensions, namely, the object existence condition and the object abnormality condition, the accuracy of the foreign object detection result of the unit area can be ensured.

[0124] In order to ensure the accuracy of the target environment data, based on the above embodiment, in this embodiment, an optional method for determining the target environment data is provided, such as Figure 6 As shown, the specific steps include:

[0125] S601, using vehicle driving data, performing motion compensation processing on environmental point cloud data to obtain compensated environmental data.

[0126] The compensated environment data is the environment point cloud data obtained after motion compensation processing.

[0127] It is understandable that since it is necessary to collect point cloud data in real time and 360 degrees during the movement of the target vehicle, there will be a certain degree of error between the collected environmental point cloud data and the actual scene. Based on this, it is necessary to perform motion compensation processing on the environmental point cloud data based on the vehicle driving data to obtain compensated environmental data.

[0128] Exemplarily, a single frame of environmental point cloud data can be time-aligned with the vehicle driving data according to the data timestamp based on the data collection frequency of the vehicle driving data and the data collection frequency of the environmental point cloud data; then, motion compensation processing is performed based on the aligned data.

[0129] For example, the data acquisition frequency of vehicle driving data is 100 Hz, and the data acquisition frequency of environmental point cloud data is 10 Hz. Therefore, 1 frame of environmental point cloud data corresponds to 10 frames of vehicle driving data, that is, 1 frame of environmental point cloud data and 10 frames of vehicle driving data can be used as a compensation association group.

[0130] Then, for each compensation association group, the acceleration and angular velocity in the 10 frames of vehicle driving data can be integrated respectively to obtain the position change of the subsequent 9 frames relative to the first frame. And posture change R, where the posture change R can be expressed in the form of a rotation matrix; for 1 frame of environmental point cloud data in the compensation association group, it can be sliced ​​according to different sampling rotation angles to obtain 10 sector area point clouds.

[0131] Furthermore, according to the vehicle driving data of the first frame and the position change and posture change obtained by integration of the subsequent 9 frames, motion compensation can be performed on the 10 sector area point clouds respectively, and the compensated 10 sector point clouds can be merged into 1 frame of compensated environment data. The compensation formula can be , is the rotation matrix representing the vehicle posture change corresponding to the i-th frame, is the position change of the i-th frame relative to the first frame, R i is the posture change of the i-th frame relative to the 1st frame; P i is the environmental point cloud data in the i-th sector area; It is the compensation environment data in the i-th sector area after motion compensation.

[0132] S602, filtering the compensated environment data to obtain target environment data.

[0133] Optionally, in order to filter out abnormal point clouds in the compensated environmental data, the compensated environmental data may be input into a trained first filtering model, and the first filtering model outputs the target environmental data based on the compensated environmental data and model parameters.

[0134] In the embodiment of the present application, by using vehicle driving data to perform motion compensation processing and filtering processing on environmental point cloud data, the accuracy of the determined target environmental data can be guaranteed.

[0135] In order to ensure the reliability of filtering, based on the above embodiment, in this embodiment, the environmental point cloud data includes the three-dimensional coordinate data and reflection intensity data of each environmental point cloud; further, an optional method for determining the target environmental data is provided, such as Figure 7 As shown, the specific steps include:

[0136] S701, determining a three-dimensional coordinate threshold and a reflection intensity threshold according to configuration information of a collection device corresponding to environmental point cloud data.

[0137] Among them, the configuration information is the basic configuration parameters of the acquisition device, such as the available acquisition range and laser reflection frequency; the three-dimensional coordinate threshold is a value used to measure the distance of the point cloud position. For example, the three-dimensional coordinate threshold can be set to x-axis coordinate threshold = 300, y-axis coordinate threshold = 300 and z-axis coordinate threshold = 100; the reflection intensity threshold is a value used to measure the reflection intensity of the point cloud.

[0138] It is understandable that, since the acquisition range of the acquisition device is limited, the three-dimensional coordinate threshold and the reflection intensity threshold for eliminating abnormal environment point clouds can be determined based on the configuration information of the acquisition device.

[0139] For example, for the three-dimensional coordinate threshold, if the configuration information of the acquisition device is that it can only accurately collect environmental point cloud data within a radius of 30m, then the environmental point cloud located outside 30m may be collected incorrectly. At this time, in order to filter out the environmental point cloud outside 30m, the coordinate thresholds of the x-axis and y-axis can be configured based on the 30m parameter.

[0140] S702, using a three-dimensional coordinate threshold and a reflection intensity threshold to perform abnormal point cloud elimination processing on the compensated environment data to obtain first filtered data.

[0141] The first filtered data is data obtained by processing the compensated environment data.

[0142] Optionally, for any environmental point cloud in the compensated environmental data, if the coordinate data in any dimension in the environmental point cloud is greater than the coordinate threshold in the dimension, the environmental point cloud is directly eliminated as an abnormal environmental point cloud.

[0143] Subsequently, for any of the remaining environmental point clouds, if the reflection intensity data of the environmental point cloud is greater than the reflection intensity threshold, the environmental point cloud is removed as an abnormal environmental point cloud. Finally, the point cloud data corresponding to the final remaining environmental point cloud can be used as the first filtering data.

[0144] S703: Determine target environment data according to the first filtered data.

[0145] Optionally, in order to further ensure the accuracy of the target environment data, the first filtered data may be input into a trained second filtering model, and the second filtering model outputs the target environment data based on the first filtered data and model parameters.

[0146] In the embodiment of the present application, by adopting a three-dimensional coordinate threshold and a reflection intensity threshold, the compensated environmental data is processed to remove abnormal point clouds, and then the target environmental data is determined, thereby ensuring the accuracy of the target environmental data.

[0147] In order to further ensure the reliability of filtering, based on the above embodiment, in this embodiment, another optional method for determining the target environment data is provided, such as Figure 8 As shown, the specific steps include:

[0148] S801, performing abnormal point cloud elimination processing on the first filtered data according to the position characteristics of each environmental point cloud in the first filtered data to obtain second filtered data.

[0149] The second filtered data is point cloud data obtained by processing the first filtered data.

[0150] Optionally, for any environmental point cloud in the first filtered data, the neighboring point clouds of the environmental point cloud can be determined from other environmental point clouds based on the three-dimensional coordinate data of the point cloud; then, the distance standard deviation corresponding to each neighboring point cloud is determined based on the average distance L between each neighboring point cloud and the environmental point cloud. .

[0151] Furthermore, according to the distance standard deviation corresponding to each neighboring point cloud, the point cloud that is relatively isolated relative to the environment point cloud in each neighboring point cloud is removed as an abnormal point cloud to obtain the second filtered data. For example, the formula L> , to determine whether the neighboring point cloud is an isolated point cloud.

[0152] S802, performing voxel filtering on the second filtered data to obtain target environment data.

[0153] Among them, voxel filtering is a filtering processing method that selects a point cloud from each voxel space to replace all point clouds in the voxel. Furthermore, a denser point cloud can choose a smaller voxel size, while a sparser point cloud requires a larger voxel size. For example, the size of the voxel space can be set to 0.05.

[0154] Optionally, the size of the voxel space can be determined based on the configuration information of the acquisition device (the number of laser radar lines); then, the area corresponding to the second filtered data is divided into a regular cubic grid according to the size of the voxel space; then, for each voxel space, the three-dimensional coordinate data and reflection intensity data of all the environmental point clouds in the voxel space are averaged to obtain the environmental point cloud data corresponding to the voxel space.

[0155] Furthermore, by combining the environmental point cloud data corresponding to each voxel space, the target environmental data can be obtained.

[0156] In the embodiment of the present application, the second filtered data is obtained by eliminating isolated point clouds, and the second filtered data is subjected to voxel filtering to obtain target environment data, thereby ensuring the accuracy of the target environment data.

[0157] In order to ensure the reliability of the point cloud reconstruction model, based on the above embodiment, in this embodiment, an optional method for training the point cloud reconstruction model is provided, which specifically includes the following steps:

[0158] The first step is to obtain the initial environment data and sample driving data of the sample vehicle. The sample vehicle is the vehicle from which the sample data is collected; the initial environment data is the point cloud data of the environment in which the sample vehicle is located, and the initial environment data does not contain foreign body point cloud data; the sample driving data is the form data of the sample vehicle.

[0159] It is understandable that in order to enable the point cloud reconstruction model to learn the data characteristics of the normal lane environment, it is necessary to use the acquisition equipment on the sample vehicle to collect point cloud data that does not contain foreign objects as the initial environmental data; at the same time, the inertial sensor inside the sample vehicle is used to obtain the sample driving data.

[0160] In the second step, the sample driving data is used to process the initial environment data to obtain sample environment data, wherein the sample environment data is the point cloud data obtained after processing.

[0161] Optionally, the sample driving data may be used to perform motion compensation processing on the initial environmental data to obtain sample compensated data; then, referring to the above steps, the sample compensated data may be subjected to abnormal point cloud removal processing and voxel filtering data to obtain sample environmental data.

[0162] The third step is to input the sample environment data into the variational autoencoder model to obtain sample reconstruction data.

[0163] Among them, the variational autoencoder is a semi-supervised learning neural network model that combines the ideas of autoencoders and variational inference. It can learn the potential representation of data and generate new data samples with diversity.

[0164] Optionally, the sample environment data may be input into a variational autoencoder model, and the variational autoencoder model outputs sample reconstruction data based on the sample environment data and model parameters.

[0165] The fourth step is to determine the model reconstruction loss and model divergence loss based on the sample environment data and the sample reconstruction data.

[0166] Among them, the model reconstruction loss is used to measure the difference between the model's output and the original input; the model divergence loss is used to measure the difference between the model's potential representation and the prior distribution.

[0167] Optionally, the model reconstruction loss and the model divergence loss may be determined based on the difference between the sample environment data and the sample reconstruction data.

[0168] In the fifth step, the variational autoencoder model is trained using model reconstruction loss and model divergence loss to obtain a point cloud reconstruction model.

[0169] It can be understood that the variational autoencoder model can be optimized based on the model reconstruction loss and the model divergence loss until the model reconstruction loss and the model divergence loss of the variational autoencoder model are minimized at the same time. At this time, the point cloud reconstruction model can be obtained.

[0170] In an embodiment of the present application, the variational autoencoder model is trained by using sample data that does not contain foreign matter to obtain a point cloud reconstruction model, which enables the point cloud reconstruction model to learn the potential representation of the normal road environment so that the point cloud reconstruction model can reconstruct the point cloud data under the normal road environment.

[0171] In order to ensure the accuracy of sample reconstruction data, based on the above embodiment, in this embodiment, the variational autoencoder model includes an encoder, a sampling layer and a decoder. Further, an optional method for determining sample reconstruction data is provided, which specifically includes the following steps:

[0172] The first step is to input the sample environment data into the encoder to obtain the mean parameter and variance parameter corresponding to the sample environment data output by the encoder after encoding the sample environment data.

[0173] Among them, the encoder can be composed of a convolutional neural network and a single-layer fully connected neural network. Furthermore, the encoder is used to map the sample environment data into parameters (mean parameter and variance parameter) of the probability distribution.

[0174] Optionally, the sample environment data can be input into the encoder, which encodes each point cloud data in the sample environment data and maps them to mean parameters and variance parameters respectively, so as to output the mean parameters and variance parameters corresponding to each point cloud data in the sample environment data.

[0175] For example, refer to Fig.10 , each point cloud data x1, x2 and x3 can be input into the encoders G1 and G2 in the variational autoencoder model respectively, and the encoders G1 and G2 respectively map each point cloud data x1, x2 and x3 to the mean parameter and variance parameter, and then output mean 1 and variance 1, mean 2 and variance 2, and mean 3 and variance 3.

[0176] The second step is to input the mean parameter and the variance parameter into the sampling layer to obtain the environmental data features outputted after the sampling layer performs sampling processing on the mean parameter and the variance parameter, wherein the environmental data features are data features corresponding to the sample environmental data.

[0177] Optionally, the mean parameter and variance parameter corresponding to each point cloud data may be input into a sampling layer, and the sampling layer performs feature sampling processing on the mean parameter and variance parameter to output the environmental data features corresponding to each point cloud data.

[0178] For example, refer to Fig.10 , the corresponding normal distribution 1 can be constructed based on mean 1 and variance 1, the corresponding normal distribution 2 can be constructed based on mean 2 and variance 2, and the corresponding normal distribution 3 can be constructed based on mean 3 and variance 3; then, the sampling layer can be used to sample normal distribution 1, normal distribution 2, and normal distribution 3 respectively to obtain environmental data features z1, z2, and z3.

[0179] The third step is to input the environmental data features into the decoder to obtain sample reconstruction data output by the decoder after decoding the environmental data features.

[0180] Among them, the decoder can be composed of a convolutional neural network and a single-layer fully connected neural network.

[0181] Optionally, the environmental data features corresponding to each point cloud data may be input into a decoder, and the decoder decodes the environmental data features corresponding to each point cloud data respectively to output sample reconstruction data containing each reconstructed point cloud.

[0182] For example, refer to Fig.10 , the environmental data features z1, z2 and z3 can be input into the decoder F, and the decoder F decodes the environmental data features z1, z2 and z3 respectively to obtain the reconstructed point clouds in the sample reconstructed data , and .

[0183] In an embodiment of the present application, the sample environment data is processed by using the encoder, sampling layer and decoder in the variational autoencoder model to obtain sample reconstruction data, which can ensure the accuracy of the sample reconstruction data.

[0184] Fig.10 FIG. 2 is a flow chart of a foreign body detection method in another embodiment. Based on the above embodiment, this embodiment provides an optional example of a foreign body detection method. Fig.10 The specific implementation process is as follows:

[0185] S1001, using the initial environment data and sample driving data associated with the sample vehicle that do not contain foreign object point cloud data, to train the variational autoencoder model to obtain a point cloud reconstruction model.

[0186] Specifically, the initial environmental data and sample driving data of the sample vehicle are obtained; the sample driving data is used to process the initial environmental data to obtain sample environmental data; the sample environmental data is input into the variational autoencoder model to obtain sample reconstruction data; the model reconstruction loss and the model divergence loss are determined according to the sample environmental data and the sample reconstruction data; the variational autoencoder model is trained using the model reconstruction loss and the model divergence loss to obtain a point cloud reconstruction model.

[0187] Exemplarily, the variational autoencoder model includes an encoder, a sampling layer and a decoder. The sample environment data is input into the encoder to obtain the mean parameter and variance parameter corresponding to the sample environment data output by the encoder after encoding the sample environment data; the mean parameter and variance parameter are input into the sampling layer to obtain the environment data features output by the sampling layer after sampling the mean parameter and variance parameter; the environment data features are input into the decoder to obtain the sample reconstruction data output by the decoder after decoding the environment data features.

[0188] S1002, during the driving process of the target vehicle, obtaining vehicle driving data of the target vehicle and environmental point cloud data corresponding to the environment where the target vehicle is located, including three-dimensional coordinate data and reflection intensity data of each environmental point cloud.

[0189] S1003, using the vehicle driving data, performing motion compensation processing on the environmental point cloud data to obtain compensated environmental data.

[0190] S1004, determining a three-dimensional coordinate threshold and a reflection intensity threshold according to configuration information of a collection device corresponding to the environmental point cloud data.

[0191] S1005, using a three-dimensional coordinate threshold and a reflection intensity threshold to perform abnormal point cloud elimination processing on the compensated environment data to obtain first filtered data.

[0192] S1006: According to the position characteristics of each environmental point cloud in the first filtered data, the first filtered data is subjected to abnormal point cloud elimination processing to obtain second filtered data.

[0193] S1007, performing voxel filtering on the second filtered data to obtain target environment data.

[0194] S1008, inputting the target environment data into the point cloud reconstruction model to obtain reconstructed environment data.

[0195] Among them, the reconstructed environment data does not include foreign body point cloud data.

[0196] S1009, determining the reconstruction error corresponding to each environmental point cloud in the target environmental data according to the target environmental data and the reconstructed environmental data.

[0197] S1010, evenly divide the spatial area of ​​the environment where the target vehicle is located to obtain unit areas.

[0198] S1011, determining the number of point clouds in each unit area according to the target environment data.

[0199] S1012, determining the foreign body detection result of each unit area according to the number of point clouds in each unit area and the reconstruction error corresponding to the environmental point cloud in each unit area.

[0200] Optionally, for each unit area, the existence of objects in the unit area is determined based on the quantity threshold and the number of point clouds in the unit area; the abnormality of objects in the unit area is determined based on the reconstruction error corresponding to each environmental point cloud in the unit area; and the foreign object detection result of the unit area is determined based on the existence of the object and the abnormality of the object.

[0201] Furthermore, according to the reconstruction errors corresponding to the environmental point clouds in the unit area, an average reconstruction error corresponding to the unit area is determined; and according to the error threshold and the average reconstruction error, the abnormality of the object in the unit area is determined.

[0202] S1013, determining the foreign object detection result of the environment where the target vehicle is located according to the foreign object detection result of each unit area.

[0203] The specific process of the above S1001-S1013 can be found in the description of the above method embodiment. The implementation principle and technical effect are similar and will not be repeated here.

[0204] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0205] Based on the same inventive concept, the embodiment of the present application also provides a foreign body detection device for implementing the foreign body detection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more foreign body detection device embodiments provided below can refer to the limitations of the foreign body detection method above, and will not be repeated here.

[0206] In an exemplary embodiment, Fig.11 As shown, a foreign body detection device 1 is provided, comprising: a data acquisition module 10, a data processing module 20 and a foreign body detection module 30, wherein:

[0207] The data acquisition module 10 is used to acquire the vehicle driving data of the target vehicle and the environmental point cloud data of the environment in which the target vehicle is located during the driving process of the target vehicle;

[0208] The data processing module 20 is used to fuse the environmental point cloud data with the vehicle driving data to obtain the target environmental data;

[0209] The foreign object detection module 30 is used to determine the foreign object detection result of the environment in which the target vehicle is located based on the target environment data and the point cloud reconstruction model; wherein the point cloud reconstruction model is used to reconstruct the environment data that does not contain the foreign object point cloud data, and the point cloud reconstruction model is trained using the initial environment data and sample driving data of the sample vehicle, and the initial environment data does not contain the foreign object point cloud data.

[0210] In an exemplary embodiment, Fig.12 As shown, the foreign object detection module 30 includes:

[0211] The point cloud reconstruction unit 31 is used to input the target environment data into the point cloud reconstruction model to obtain the reconstructed environment data; wherein the reconstructed environment data does not include the foreign body point cloud data;

[0212] An error determination unit 32, configured to determine a reconstruction error corresponding to each environmental point cloud in the target environmental data according to the target environmental data and the reconstructed environmental data;

[0213] The foreign object detection unit 33 is used to determine the foreign object detection result of the environment where the target vehicle is located according to the target environment data and the reconstruction error corresponding to each environment point cloud.

[0214] In an exemplary embodiment, the foreign matter detection unit 33 includes:

[0215] The first subunit is used to evenly divide the spatial area of ​​the environment where the target vehicle is located to obtain each unit area;

[0216] The second subunit is used to determine the number of point clouds in each unit area based on the target environment data;

[0217] The third subunit is used to determine the foreign body detection result of each unit area according to the number of point clouds in each unit area and the reconstruction error corresponding to the environmental point cloud in each unit area;

[0218] The fourth subunit is used to determine the foreign object detection result of the environment where the target vehicle is located according to the foreign object detection result of each unit area.

[0219] In an exemplary embodiment, the third subunit is specifically configured to:

[0220] For each unit area, the existence of objects in the unit area is determined based on the quantity threshold and the number of point clouds in the unit area; the abnormality of objects in the unit area is determined based on the reconstruction error corresponding to each environmental point cloud in the unit area; the foreign body detection result of the unit area is determined based on the existence of objects and the abnormality of objects.

[0221] In an exemplary embodiment, the third subunit is further configured to:

[0222] According to the reconstruction errors corresponding to the environmental point clouds in the unit area, the average reconstruction error corresponding to the unit area is determined; according to the error threshold and the average reconstruction error, the abnormality of the object in the unit area is determined.

[0223] In an exemplary embodiment, Fig.13 As shown, the data processing module 20 includes:

[0224] The compensation processing unit 21 is used to use the vehicle driving data to perform motion compensation processing on the environmental point cloud data to obtain compensated environmental data;

[0225] The filtering processing unit 22 is used to filter the compensated environment data to obtain target environment data.

[0226] In an exemplary embodiment, the environmental point cloud data includes three-dimensional coordinate data and reflection intensity data of each environmental point cloud; the filtering processing unit 22 includes:

[0227] A fifth subunit is used to determine a three-dimensional coordinate threshold and a reflection intensity threshold according to configuration information of a collection device corresponding to the environmental point cloud data;

[0228] A sixth subunit is used to use a three-dimensional coordinate threshold and a reflection intensity threshold to perform abnormal point cloud elimination processing on the compensation environment data to obtain first filtered data;

[0229] The seventh subunit is used to determine target environment data according to the first filtered data.

[0230] In an exemplary embodiment, the seventh subunit is specifically configured to:

[0231] According to the position characteristics of each environmental point cloud in the first filtered data, the first filtered data is processed to eliminate abnormal point clouds to obtain second filtered data; and the second filtered data is processed by voxel filtering to obtain target environmental data.

[0232] In an exemplary embodiment, the foreign matter detection device 1 further includes a model training module, wherein the model training module is specifically used for:

[0233] Acquire initial environmental data and sample driving data of a sample vehicle; use the sample driving data to process the initial environmental data to obtain sample environmental data; input the sample environmental data into a variational autoencoder model to obtain sample reconstruction data; determine a model reconstruction loss and a model divergence loss based on the sample environmental data and the sample reconstruction data; use the model reconstruction loss and the model divergence loss to train the variational autoencoder model to obtain a point cloud reconstruction model.

[0234] In one embodiment, the variational autoencoder model includes an encoder, a sampling layer, and a decoder. Furthermore, the model training module is also used to:

[0235] The sample environment data is input into the encoder to obtain the mean parameter and variance parameter corresponding to the sample environment data output by the encoder after encoding the sample environment data; the mean parameter and variance parameter are input into the sampling layer to obtain the environment data features output by the sampling layer after sampling the mean parameter and variance parameter; the environment data features are input into the decoder to obtain the sample reconstruction data output by the decoder after decoding the environment data features.

[0236] Each module in the above-mentioned foreign body detection device can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0237] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.14As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a foreign body detection method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0238] Those skilled in the art will understand that Fig.14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0239] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0240] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0241] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0242] It should be noted that the data involved in this application (including but not limited to vehicle driving data and environmental point cloud data, etc.) are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0243] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0244] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0245] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A foreign body detection method, characterized in that: The method comprises: During the driving process of the target vehicle, obtaining vehicle driving data of the target vehicle and environmental point cloud data of the environment in which the target vehicle is located; Using the vehicle driving data to fuse the environmental point cloud data to obtain target environmental data; According to the target environment data and the point cloud reconstruction model, the foreign object detection result of the environment in which the target vehicle is located is determined; wherein the point cloud reconstruction model is used to reconstruct environmental data that does not contain foreign object point cloud data, and the point cloud reconstruction model is trained using the initial environment data and sample driving data of a sample vehicle, and the initial environment data does not contain foreign object point cloud data.

2. The method according to claim 1, characterized in that The step of determining the foreign object detection result of the environment in which the target vehicle is located according to the target environment data and the point cloud reconstruction model includes: Inputting the target environment data into a point cloud reconstruction model to obtain reconstructed environment data; wherein the reconstructed environment data does not include foreign body point cloud data; Determining, according to the target environment data and the reconstructed environment data, a reconstruction error corresponding to each environment point cloud in the target environment data; The foreign object detection result of the environment in which the target vehicle is located is determined according to the target environment data and the reconstruction errors corresponding to each environment point cloud.

3. The method according to claim 2, characterized in that The determining of the foreign object detection result of the environment in which the target vehicle is located according to the target environment data and the reconstruction error corresponding to each environment point cloud includes: Evenly dividing the spatial area of ​​the environment where the target vehicle is located to obtain unit areas; Determine the number of point clouds in each unit area according to the target environment data; Determine the foreign body detection result of each unit area according to the number of point clouds in each unit area and the reconstruction error corresponding to the environmental point cloud in each unit area; According to the foreign object detection result of each unit area, the foreign object detection result of the environment where the target vehicle is located is determined.

4. The method according to claim 3, characterized in that: The foreign body detection result of each unit area is determined according to the number of point clouds in each unit area and the reconstruction error corresponding to the environmental point cloud in each unit area, including: For each unit area, determining the existence of objects in the unit area according to a quantity threshold and the number of point clouds in the unit area; Determining abnormal conditions of objects within the unit area according to reconstruction errors corresponding to each environmental point cloud within the unit area; The foreign object detection result of the unit area is determined according to the existence of the object and the abnormality of the object.

5. The method according to claim 4, characterized in that The determining, according to the reconstruction errors corresponding to the environmental point clouds in the unit area, the abnormality of the object in the unit area comprises: Determine an average reconstruction error value corresponding to the unit area according to the reconstruction errors corresponding to each environmental point cloud in the unit area; The abnormality of the object in the unit area is determined according to the error threshold and the reconstruction error average value.

6. The method according to claim 1, characterized in that The step of fusing the environmental point cloud data with the vehicle driving data to obtain target environmental data includes: Using the vehicle driving data, motion compensation processing is performed on the environmental point cloud data to obtain compensated environmental data; The compensated environment data is filtered to obtain target environment data.

7. The method according to claim 6, characterized in that The environmental point cloud data includes three-dimensional coordinate data and reflection intensity data of each environmental point cloud; The filtering process of the compensated environment data to obtain target environment data includes: Determine a three-dimensional coordinate threshold and a reflection intensity threshold according to configuration information of a collection device corresponding to the environmental point cloud data; Using the three-dimensional coordinate threshold and the reflection intensity threshold, performing abnormal point cloud elimination processing on the compensated environment data to obtain first filtered data; Target environment data is determined according to the first filtered data.

8. The method according to claim 7, characterized in that The step of determining target environment data according to the first filtered data includes: According to the position characteristics of each environmental point cloud in the first filtered data, the first filtered data is subjected to abnormal point cloud elimination processing to obtain second filtered data; The second filtered data is subjected to voxel filtering to obtain target environment data.

9. The method according to any one of claims 1 to 8, characterized in that The point cloud reconstruction model is trained in the following way: Obtaining initial environmental data and sample driving data of a sample vehicle; Processing the initial environment data using the sample driving data to obtain sample environment data; Inputting the sample environment data into a variational autoencoder model to obtain sample reconstruction data; Determining a model reconstruction loss and a model divergence loss according to the sample environment data and the sample reconstruction data; The variational autoencoder model is trained using the model reconstruction loss and the model divergence loss to obtain the point cloud reconstruction model.

10. The method according to claim 9, characterized in that The variational autoencoder model includes an encoder, a sampling layer and a decoder, and the sample environment data is input into the variational autoencoder model to obtain sample reconstruction data, including: Inputting the sample environment data into the encoder to obtain a mean parameter and a variance parameter corresponding to the sample environment data outputted by the encoder after encoding the sample environment data; Inputting the mean parameter and the variance parameter into the sampling layer to obtain the environmental data features outputted after the sampling layer performs sampling processing on the mean parameter and the variance parameter; The environmental data feature is input into the decoder to obtain sample reconstruction data output by the decoder after the decoder performs decoding processing on the environmental data feature.

11. A foreign body detection device, characterized in that: The device comprises: A data acquisition module, used to acquire vehicle driving data of the target vehicle and environmental point cloud data of the environment in which the target vehicle is located during the driving process of the target vehicle; A data processing module, used to fuse the environmental point cloud data with the vehicle driving data to obtain target environmental data; A foreign object detection module is used to determine the foreign object detection result of the environment in which the target vehicle is located based on the target environment data and the point cloud reconstruction model; wherein the point cloud reconstruction model is used to reconstruct environmental data that does not contain foreign object point cloud data, and the point cloud reconstruction model is trained using initial environmental data and sample driving data of a sample vehicle, and the initial environmental data does not contain foreign object point cloud data.

Images