Background object detection method, device, equipment and autonomous driving vehicle

Abstract

The present disclosure provides a method, device, equipment and autonomous driving vehicle for detecting background objects, which relate to the fields of artificial intelligence technology, and in particular to autonomous driving, high-precision maps, and environmental perception technology. The specific implementation scheme is as follows: the detection device obtains first point cloud data and a target image, the first point cloud data includes point cloud data within a target area, and the target image is an image of the target area. The detection device processes the target image to determine a first image background object set. The detection device processes the first point cloud data to determine a first point cloud foreground object set and ground point cloud data. The detection device filters the first point cloud foreground object set and the ground point cloud data from the first point cloud data to determine a first point cloud background object set. Then, the detection device filters the point cloud objects in the first point cloud background object set based on the first image background object set to obtain a second point cloud background object set.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence technology, in particular to the fields of autonomous driving, high-precision mapping, and environmental perception technology, and specifically to a background object detection method, device, equipment, and autonomous driving vehicle. Background Art

[0002] With the development of autonomous driving technology, vehicles can automatically avoid objects on the road by identifying them. Generally, there may be various objects on the road, such as movable objects (which can be called foreground objects) and stationary objects (which can be called background objects). Moreover, background objects may affect the driving of autonomous vehicles. Currently, vehicles can detect objects through sensors (such as lidar sensors, visual sensors, millimeter-wave radar sensors, etc.) to determine whether the object is a background object, and then adopt corresponding autonomous driving strategies to avoid the object. Summary of the Invention

[0003] The present disclosure provides a background object detection method, apparatus, device, and autonomous driving vehicle.

[0004] In a first aspect, the present disclosure provides a method for detecting a background object, comprising:

[0005] A detection device for background objects (which may be referred to as a detection device for short) obtains first point cloud data and a target image, wherein the first point cloud data includes point cloud data within a target area, and the target image is an image of the target area. Thereafter, the detection device processes the target image to determine a first image background object set. Next, the detection device processes the first point cloud data to determine a first point cloud foreground object set and ground point cloud data. Then, the detection device filters the first point cloud foreground object set and the ground point cloud data from the first point cloud data to determine a first point cloud background object set. Then, based on the first image background object set, the detection device filters the point cloud objects in the first point cloud background object set to obtain a second point cloud background object set, which is the intersection of the first image background object set and the first point cloud background object set.

[0006] In a second aspect, the present disclosure provides a background object detection device, comprising: an acquisition unit, configured to acquire first point cloud data and a target image, wherein the first point cloud data comprises point cloud data within a target area, the target image is an image of the target area, and the first point cloud data comprises: a first point cloud foreground object set, a first point cloud background object set, and ground point cloud data. A processing unit, configured to process the target image and determine the first image background object set. The processing unit is further configured to process the first point cloud data and determine the first point cloud foreground object set and the ground point cloud data. The processing unit is further configured to filter the first point cloud foreground object set and the ground point cloud data from the first point cloud data to determine the first point cloud background object set. The processing unit is further configured to filter the point cloud objects in the first point cloud background object set based on the first image background object set to obtain a second point cloud background object set, wherein the second point cloud background object set is the intersection of the first image background object set and the first point cloud background object set.

[0007] In a third aspect, the present disclosure provides an electronic device, comprising:

[0008] at least one processor; and

[0009] A memory communicatively connected to at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of the methods in the first aspect.

[0010] In a fourth aspect, the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, comprising:

[0011] The computer instructions are used to cause a computer to execute any one of the methods in the first aspect.

[0012] In a fifth aspect, the present disclosure provides a computer program product, comprising:

[0013] A computer program, wherein the computer program executes any one of the methods in the first aspect when executed by a processor.

[0014] In a sixth aspect, the present disclosure provides an autonomous driving vehicle, comprising:

[0015] Electronic equipment.

[0016] The technology according to the present disclosure solves the problem of low accuracy in detecting background objects.

[0017] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0019] Figure 1 is a flowchart of a background object detection method provided by an embodiment of the present disclosure;

[0020] Figure 2 is an example diagram of a target area provided by an embodiment of the present disclosure;

[0021] Figure 3 is an example diagram of determining background objects provided by an embodiment of the present disclosure;

[0022] Figure 4 is a flowchart of another background object detection method provided by an embodiment of the present disclosure;

[0023] Figure 5 is a flowchart of a method for updating an object provided by an embodiment of the present disclosure;

[0024] Figure 6 is a flowchart of another background object detection method provided by an embodiment of the present disclosure;

[0025] Figure 7 is a flowchart of a method for updating an object provided by an embodiment of the present disclosure;

[0026] Figure 8 is a flowchart of another background object detection method provided by an embodiment of the present disclosure;

[0027] Figure 9 is a schematic diagram of a flow chart of updating a grid provided by an embodiment of the present disclosure;

[0028] Figure 10 is a flowchart of another background object detection method provided by an embodiment of the present disclosure;

[0029] Figure 11 is a structural diagram of a background object detection device provided by an embodiment of the present disclosure;

[0030] Figure 12 It is a block diagram of an electronic device for a background object detection method provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0033] Before introducing the background object detection method of the embodiment of the present disclosure in detail, the application scenario of the embodiment of the present disclosure is first introduced.

[0034] First, the application scenarios of the embodiments of the present disclosure are introduced.

[0035] On public roads, as well as in closed or semi-enclosed areas, the road surface is prone to a variety of objects, including movable objects (referred to as foreground objects) and stationary objects (referred to as background objects). Common background objects include branches, stones, fallen leaves, plastic bags, paper scraps, and objects dropped during transportation. For autonomous or unmanned vehicles, these background objects can affect their operation and pose a safety hazard.

[0036] With the development of autonomous driving technology, vehicles can automatically avoid objects on the road by identifying them. Currently, vehicles use sensors (such as lidar sensors, visual sensors, millimeter-wave radar sensors, etc.) to detect objects to determine whether they are background objects, and then adopt corresponding autonomous driving strategies to avoid the objects.

[0037] Commonly used sensors include lidar, vision sensors, and millimeter-wave radar sensors. Lidar sensors can acquire point cloud data, containing information such as the 3D position and shape of an object, and they achieve high accuracy in obtaining position. Lidar sensors are insensitive to lighting conditions, but are susceptible to weather and environmental influences, such as rain or dust, which can easily introduce noise. Vision sensors can acquire image data, which includes information such as an object's color, shape, and texture, and they offer high classification accuracy. However, vision sensors are susceptible to lighting and weather conditions. For example, rain can cause fogging of the camera lens, resulting in blurred images, while strong or weak light conditions can reduce image clarity. Millimeter-wave radar sensors can acquire point cloud data, containing information such as the 3D position and velocity of an object. Compared to lidar sensors, millimeter-wave radar sensors have lower position accuracy, and compared to vision sensors, millimeter-wave radar sensors have lower classification accuracy. However, millimeter-wave radar sensors are less affected by lighting, weather, and environmental dust, which means they have less impact on perception and detection.

[0038] In summary, current sensors may be affected by environmental factors when acquiring object data, resulting in inaccurate object data, which in turn affects the accuracy of detecting whether an object is a background object.

[0039] To address the aforementioned issues, embodiments of the present disclosure provide a background object detection method for use in background object detection scenarios. In this method, a background object detection device (hereinafter referred to as a detection device) may acquire first point cloud data and a target image. The first point cloud data includes point cloud data within a target area, the target image is an image of the target area, and the first point cloud data includes: a first point cloud foreground object set, a first point cloud background object set, and ground point cloud data. The detection device may then process the target image to determine a first image background object set. Furthermore, the detection device may process the first point cloud data to determine a first point cloud foreground object set and ground point cloud data. The detection device may then filter the first point cloud foreground object set and the ground point cloud data from the first point cloud data to determine a first point cloud background object set. The detection device may then filter the point cloud objects in the first point cloud background object set based on the first image background object set to obtain a second point cloud background object set, where the second point cloud background object set is the intersection of the first image background object set and the first point cloud background object set.

[0040] Based on the above technical solution, the detection device can separately acquire first point cloud data of a target area and a target image of the target area. The detection device then processes the target image to obtain a first image background object set. Furthermore, the detection device can process the first point cloud data to determine a first point cloud foreground object set and ground point cloud data. The detection device can then filter the first point cloud foreground object set and the ground point cloud data from the first point cloud data to obtain a first point cloud background object set. Because the accuracy of background objects identified based on the image is high and the position information of background objects obtained from the point cloud data is relatively accurate, the detection device can filter the point cloud objects in the first point cloud background object set based on the first image background object set to obtain a second point cloud background object set. Specifically, the second point cloud background object set is the intersection of the first image background object set and the first point cloud background object set. This not only utilizes the image to increase the probability that point cloud objects in the second point cloud background object set are background objects, thereby improving the accuracy of background object detection using point cloud data, but also utilizes the position information of the point cloud data to ensure the positional accuracy of identified background objects.

[0041] The background object detection method provided by the present disclosure may be implemented by a background object detection device, which may be a motor vehicle. Furthermore, the device may be a central processing unit (CPU) of the motor vehicle, or a detection module in the motor vehicle for detecting background objects.

[0042] Motor vehicles may include sensors for sensing their surroundings. These sensors may include one or more of the following: visual cameras, infrared cameras, ultrasonic sensors, millimeter-wave radar sensors, and lidar sensors. Different sensors offer varying detection accuracy and range. Cameras may be mounted on the front, rear, or other locations of the vehicle. Visual cameras can capture real-time images of the vehicle's interior and exterior and present them to the driver and / or passengers. Furthermore, by analyzing the images captured by the visual cameras, information such as traffic light indications, intersection conditions, and the operating status of other vehicles can be obtained. Infrared cameras can detect objects in night vision conditions. Ultrasonic sensors can be mounted on all sides of the vehicle, leveraging the strong directionality of ultrasonic waves to measure the distance of external objects from the vehicle. Millimeter-wave radar sensors can be mounted on the front, rear, or other locations of the vehicle, leveraging the properties of electromagnetic waves to measure the distance of external objects from the vehicle. Lidar sensors can be mounted on the front, rear, or other locations of the vehicle to detect object edges and shapes, enabling object recognition and tracking. Radar devices can also measure changes in the speed of the vehicle and moving objects due to the Doppler effect.

[0043] In an embodiment of the present disclosure, a motor vehicle is equipped with at least one visual sensor for collecting images and at least one point cloud sensor for collecting point cloud data. The point cloud sensor may include at least one of the following: a millimeter-wave radar sensor, a lidar sensor, or the like.

[0044] Optionally, the detection device may be an electronic device. The electronic device may perform wired / wireless communication with the vehicle. The motor vehicle may transmit the collected data (such as image data and point cloud data) to the electronic device, which then performs the background object detection method provided in the embodiments of the present disclosure.

[0045] It should be noted that the embodiments of the present disclosure do not limit electronic devices. The electronic devices in the embodiments of the present disclosure may include tablet computers, mobile phones, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, cellular phones, personal digital assistants (PDAs), augmented reality (AR) and virtual reality (VR) devices, in-vehicle devices, servers, and other devices. The embodiments of the present disclosure do not impose any particular limitations on the specific form of the electronic devices.

[0046] In the embodiment of the present disclosure, the method for detecting background objects performed by a detection device is taken as an example to illustrate the method for detecting background objects provided by the embodiment of the present disclosure.

[0047] like Figure 1 As shown, a method for detecting background objects provided by an embodiment of the present disclosure includes:

[0048] S101. A detection device obtains first point cloud data and a target image.

[0049] The first point cloud data includes point cloud data within the target area, and the target image is an image of the target area.

[0050] It should be noted that the embodiment of the present disclosure does not limit the setting method of the target area. The target area can be set by a developer, or the target area can be set by a detection device.

[0051] In one possible design, the target area can be set by a developer. The detection device stores preset area information, and the detection device can obtain the first point cloud data and the target image based on the preset area information.

[0052] Exemplarily, the preset area information may be: 60 meters in front of the vehicle, 10 meters behind the vehicle, and 5 meters on each side.

[0053] It should be noted that the faster the current driving speed, the larger the emergency zone of the motor vehicle may be. Therefore, if the target area is fixed, the faster the current driving speed of the motor vehicle, the smaller the emergency zone reserved for the motor vehicle may be, making it impossible for the motor vehicle to avoid the obstacle in time.

[0054] In another possible design, the target area can be set by a detection device. The detection device can obtain the current speed of the vehicle. The detection device can then determine the target area based on the current speed of the vehicle. The current speed is inversely proportional to the area of the target area.

[0055] That is, the faster the current driving speed, the smaller the target area, and the slower the current driving speed, the larger the target area.

[0056] It is understood that the detection device can improve the safety of motor vehicle driving by determining the target area based on the current driving speed. In addition, it can also reduce the amount of data processed and improve resource utilization while ensuring safety.

[0057] In an embodiment of the present disclosure, the first point cloud data includes: a first point cloud foreground object set, a first point cloud background object set, and ground point cloud data. The first point cloud foreground object set includes: point cloud data of at least one foreground object, and the first point cloud background object set includes: point cloud data of at least one background object.

[0058] It should be noted that in the embodiments of this disclosure, foreground objects are movable objects. For example, foreground objects may include people, cars, small animals, etc. Background objects may be stationary objects. For example, stationary objects may include traffic cones, triangles, rocks, and guardrails in construction or accident scenes. The embodiments of this disclosure do not limit the type and form of background objects.

[0059] In a possible implementation, a point cloud sensor and a visual sensor are installed in the detection device. The detection device can collect first point cloud data through the point cloud sensor and collect target images through the visual sensor.

[0060] In another possible implementation, the detection apparatus may receive the first point cloud data and the target image from other devices.

[0061] It should be noted that the embodiments of the present disclosure are not limited to other devices. For example, other devices may be communication devices of motor vehicles, mobile phones, servers, etc.

[0062] It should be noted that when a motor vehicle is traveling on a road, the area acquired by the detection device may include not only the road area but also areas outside the road. This may increase the amount of data processed by the detection device and occupy the processing resources of the detection device.

[0063] In some embodiments, the detection device may acquire data of a first area, and the data of the first area may include: point cloud data of the first area, where the first area includes a target area.

[0064] It should be noted that the embodiment of the present disclosure does not limit the manner in which the detection device acquires the data of the first area. For example, the detection device may collect the data of the first area through a sensor.

[0065] The detection device can then perform boundary detection on the data of the first region to obtain boundary point cloud data. The detection device can then determine first point cloud data based on the position information of the boundary point cloud data. The first point cloud data is point cloud data within the boundary point cloud data of the first region.

[0066] It should be noted that the detection device can obtain boundary point cloud data by referring to conventional methods for determining road boundaries, and the present disclosure is not limited to this. For example, the detection device can identify boundary objects (such as guardrails) from point cloud data to obtain boundary point cloud data. For another example, the detection device can determine boundary point cloud data based on a high-precision map.

[0067] For example, Figure 2 As shown, area 201 is a first area, and area 201 includes: guardrail 202, guardrail 203, object 204, and area 205. Area 205 is a first area between the right side of guardrail 202 and the left side of guardrail 203.

[0068] S102: The detection device processes the target image to determine a first image background object set.

[0069] In one possible implementation, the detection device may process the target image based on a first image background detection algorithm, determine at least one background object image from the target image, and use the at least one background object image as a first image background object set. The first image background object set includes the at least one background object image.

[0070] It should be noted that the embodiment of the present disclosure does not limit the first image background detection algorithm. For example, the image background detection algorithm can be an algorithm based on a convolutional neural network.

[0071] For example, Figure 3As shown, if the image 301 includes trees, street lights and cars, the trees in the detection box (such as a 2D box) 302 and the street lights in the detection box 303 are background objects.

[0072] Optionally, the detection device may also obtain position information of each detection frame, where the position information of the detection frame is used to indicate the position of the background object.

[0073] In another possible implementation, the detection device may input the target image into a trained image background detection model to obtain a first image background object set, where the image background detection model is constructed based on the first image background detection algorithm.

[0074] It should be noted that, for the method of training the image background object detection model by the detection device, reference can be made to the model training method in conventional technology, which will not be described in detail here.

[0075] S103: The detection device processes the first point cloud data to determine a first point cloud foreground object set and ground point cloud data.

[0076] In a possible implementation, the detection device may process the first point cloud data based on a point cloud foreground detection algorithm and a ground detection algorithm, respectively, to determine a first point cloud foreground object set and ground point cloud data.

[0077] It should be noted that the disclosed embodiments do not limit the point cloud foreground detection algorithm and the ground detection algorithm. For example, the point cloud foreground detection algorithm can be a convolutional neural network algorithm (such as PointRCNN, VoxNet), PointNet (point cloud network) algorithm, PointNet++, etc. For another example, the ground detection algorithm can be a ground fitting algorithm.

[0078] S104: The detection device filters the first point cloud foreground object set and the ground point cloud data from the first point cloud data to determine a first point cloud background object set.

[0079] In one possible implementation, the detection device may delete the point cloud data and ground point cloud data in the first point cloud foreground object set from the first point cloud data to obtain fourth point cloud data, where the fourth point cloud data is point cloud data of background objects in the target area, and the first point cloud background object set includes the fourth point cloud data.

[0080] That is, the fourth point cloud data is the point cloud data in the first point cloud data except the point cloud data in the first point cloud foreground object set and the ground point cloud data.

[0081] It is understandable that since the first point cloud data includes: a first point cloud foreground object set, a first point cloud background object set, and ground point cloud data, after the detection device filters the point cloud data in the first point cloud foreground object set and the ground point cloud data from the first point cloud data, the point cloud data in the first point cloud background object set can be obtained.

[0082] S105 . The detection device filters the point cloud objects in the first point cloud background object set based on the first image background object set to obtain a second point cloud background object set.

[0083] The first image background object set includes: a second point cloud background object set, and the second point cloud background object set is the intersection of the first image background object set and the first point cloud background object set.

[0084] In one possible implementation, a detection device may detect the first point cloud background object set based on a point cloud background detection algorithm to determine at least one point cloud background object. Furthermore, the detection device may obtain background objects from the first image background object set. The detection device may then compare each point cloud background object in the first point cloud background object set with background objects in the first image background object set, and select point cloud background objects in the first point cloud background object set that are identical to background objects in the first image background object set as the second point cloud background object set.

[0085] For example, if the first point cloud background object set includes rocks, trees, billboards, and beverage boxes, and the first image background object set includes rocks, billboards, and beverage boxes, then the second point cloud background object set includes rocks, billboards, and beverage boxes.

[0086] In another possible implementation, for each point cloud data item in the first point cloud background object set, the detection device may perform a first operation to determine a second point cloud background object set from the first point cloud background object set. The first operation includes: the detection device obtains first position information for a first sub-point cloud data item. The first sub-point cloud data item is any point cloud data item in the first point cloud background object set, and the first position information indicates the position of the first sub-point cloud data item in the point cloud coordinate system. The detection device may then convert the first position information into second position information based on parameters of the point cloud sensor, where the second position information indicates the position of the first sub-point cloud data item in the world coordinate system. The detection device may then convert the second position information into third position information, where the third position information indicates the position of the first sub-point cloud data item in the target image coordinate system. The detection device may then determine whether the third position information is within the image area of the sub-background object, which is the background object corresponding to the first sub-point cloud data item. If the third position information is within the image area of the sub-background object, the detection device determines that the first sub-point cloud data item is a point cloud data item in the second point cloud background object set.

[0087] For example, combined Figure 3 If the third position information of the first sub-point cloud data is located within the detection frame 302, the detection device determines that the first sub-point cloud data is point cloud data in the second point cloud background object set.

[0088] Based on the above technical solution, the detection device can separately acquire first point cloud data of a target area and a target image of the target area. The detection device then processes the target image to obtain a first image background object set. Furthermore, the detection device can process the first point cloud data to determine a first point cloud foreground object set and ground point cloud data. The detection device can then filter the first point cloud foreground object set and the ground point cloud data from the first point cloud data to obtain a first point cloud background object set. Because the accuracy of background objects identified based on the image is high and the position information of background objects obtained from the point cloud data is relatively accurate, the detection device can filter the point cloud objects in the first point cloud background object set based on the first image background object set to obtain a second point cloud background object set. Specifically, the second point cloud background object set is the intersection of the first image background object set and the first point cloud background object set. This not only utilizes the image to increase the probability that point cloud objects in the second point cloud background object set are background objects, thereby improving the accuracy of background object detection using point cloud data, but also utilizes the position information of the point cloud data to ensure the positional accuracy of identified background objects.

[0089] It should be noted that point cloud sensors (such as LiDAR sensors) are susceptible to weather and environmental influences, generating noise. In poor weather conditions, the point cloud data collected by the point cloud sensor may be inaccurate, resulting in reduced accuracy in detecting foreground objects, which in turn affects the accuracy of identifying background objects.

[0090] In some embodiments, the detection device may determine the point cloud foreground objects based on the image foreground objects.

[0091] like Figure 4 FIG. 1 is another background object detection method provided by an embodiment of the present disclosure, the method comprising:

[0092] S401: The detection device obtains first point cloud data and a target image.

[0093] It should be noted that, for a detailed introduction to how the detection device acquires the first point cloud data and the target image, reference may be made to the description in S101 , which will not be elaborated herein.

[0094] S402: The detection device processes the target image to determine a first image background object set and a first image foreground object set.

[0095] In one possible implementation, the detection device may process the target image based on a first image foreground detection algorithm, determine at least one foreground object image from the target image, and use the at least one foreground object image as a first image foreground object set. The first image foreground object set includes the at least one foreground object image.

[0096] It should be noted that the disclosed embodiments do not limit the first image foreground detection algorithm. For example, the image foreground detection algorithm may be an algorithm based on a convolutional neural network. The process by which the detection device processes the target image to determine the first image background object set can be found in the description of S102 and is not further described here.

[0097] In some embodiments, the detection device processes the first point cloud data to determine the first point cloud foreground object set, which may include S403-S404.

[0098] S403: The detection device performs foreground object detection on the first point cloud data to determine a second point cloud foreground object set.

[0099] In a possible implementation, the detection device may process the first point cloud data based on a point cloud foreground detection algorithm to determine a second point cloud foreground object set.

[0100] S404: The detection device filters the point cloud objects in the second point cloud foreground object set based on the first image foreground object set to obtain a first point cloud foreground object set.

[0101] The first point cloud foreground object set is the intersection of the first image foreground object set and the second point cloud foreground object set.

[0102] In one possible implementation, the detection device may compare each point cloud foreground object in the second point cloud foreground object set with the foreground objects in the first image foreground object set, and take the point cloud foreground objects in the second point cloud foreground object set that are identical to the foreground objects in the first image foreground object set as the first point cloud foreground object set.

[0103] For example, if the first image foreground object set includes: kitten, puppy, and car, and the second point cloud foreground object set includes: kitten and car, then the first point cloud foreground object set includes: kitten and car.

[0104] In another possible implementation, the detection device may perform a second operation for each point cloud data item in the second point cloud foreground object set to determine the first point cloud foreground object set from the second point cloud foreground object set. The second operation includes: the detection device obtains fourth position information of the second sub-point cloud data item. The second sub-point cloud data item is any point cloud data item in the second point cloud foreground object set, and the fourth position information indicates the position of the second sub-point cloud data item in the point cloud coordinate system. The detection device may then convert the fourth position information into fifth position information based on parameters of the point cloud sensor, where the fifth position information indicates the position of the second sub-point cloud data item in the world coordinate system. The detection device may then convert the fifth position information into sixth position information, where the sixth position information indicates the position of the second sub-point cloud data item in the target image coordinate system. The detection device may then determine whether the sixth position information item is located within the image region of the sub-foreground object, which is a foreground object in the first image foreground object set corresponding to the second sub-point cloud data item. If the sixth position information item is located within the image region of the sub-foreground object, the detection device determines that the second sub-point cloud data item is point cloud data in the first point cloud foreground object set.

[0105] Optionally, for each second sub-point cloud data, if the sixth position information is located within the image area of the sub-foreground object, the detection device may fuse the second sub-point cloud data with the image of the sub-foreground object to obtain updated second sub-point cloud data and an updated image of the sub-foreground object.

[0106] For example, Figure 5As shown, the detection device obtains a point cloud foreground object set and an image foreground object set. Afterwards, for each second sub-point cloud data, the detection device converts the position information of the second sub-point cloud data (such as the point cloud data of object a) into a position in the image coordinate system. Then, for each detection frame of the object in the image foreground object set, the detection device determines whether the point cloud data of object a is within the detection frame of the object (such as the detection frame of object b). If the point cloud data of object a is not within the detection frame of object b, the detection device deletes the point cloud data of object a from the point cloud foreground object set. If the point cloud data of object a is within the detection frame of object b, the detection device can fuse the point cloud data of object a and the image of object b to obtain updated point cloud data of object a and updated image of object b.

[0107] It is understandable that the detection device processes the target image to determine the first image background object set and the first image foreground object set. Furthermore, foreground object detection is performed on the first point cloud data to determine the second point cloud foreground object set. Then, since the accuracy of the foreground objects identified based on the image is high and the position information of the foreground objects obtained from the point cloud data is relatively accurate, the detection device can filter the point cloud objects in the second point cloud foreground object set based on the first image foreground object set to obtain the first point cloud foreground object set, that is, the first point cloud foreground object set is the intersection between the first image foreground object set and the second point cloud foreground object set. In this way, not only can the image be used to increase the probability that the point cloud objects in the first point cloud foreground object set are foreground objects, thereby improving the accuracy of detecting foreground objects through point cloud data; but also the position information of the point cloud data can be used to ensure the position accuracy of the identified foreground objects.

[0108] S405: The detection device processes the first point cloud data to determine ground point cloud data.

[0109] It should be noted that the way in which the detection device processes the first point cloud data and determines the ground point cloud data can be referred to the introduction to S103 and will not be described in detail here.

[0110] S406: The detection device filters the first point cloud foreground object set and the ground point cloud data from the first point cloud data to determine a first point cloud background object set.

[0111] It should be noted that the detection device filters the first point cloud foreground object set and the ground point cloud data from the first point cloud data to determine the first point cloud background object set. Please refer to the introduction of S104 and will not be repeated here.

[0112] S407 : The detection device filters the point cloud objects in the first point cloud background object set based on the first image background object set to obtain a second point cloud background object set.

[0113] It should be noted that the detection device filters the point cloud objects in the first point cloud background object set based on the first image background object set to obtain the second point cloud background object set. Please refer to the introduction of S105 and will not be repeated here.

[0114] It should be noted that visual sensors are easily affected by lighting and weather, which can cause image blur and reduced clarity, thus affecting the accuracy of detecting foreground and background objects. For example, an object may be detected as both a foreground object and a background object.

[0115] like Figure 6 FIG. 1 is another background object detection method provided by an embodiment of the present disclosure, the method comprising:

[0116] S501: The detection device obtains first point cloud data and a target image.

[0117] It should be noted that, for a detailed introduction to how the detection device acquires the first point cloud data and the target image, reference may be made to the description in S101 , which will not be elaborated herein.

[0118] In some embodiments, the detection device may determine the first image background object set and the first image foreground object set through S502 - S507 .

[0119] S502: The detection device performs foreground object detection on the target image to determine a second image foreground object set.

[0120] In one possible implementation, the detection device may process the target image based on the first image foreground detection algorithm, determine at least one foreground object image from the target image, and use the at least one foreground object image as the second image foreground object set. The second image foreground object set includes the at least one foreground object image.

[0121] S503: The detection device performs background object detection on the target image to determine a second image background object set.

[0122] In one possible implementation, the detection device may process the target image based on the first image background detection algorithm, determine at least one background object image from the target image, and use the at least one background object image as the second image background object set. The second image background object set includes the at least one background object image.

[0123] S504: The detection device determines whether there is a target object in the foreground object set of the second image.

[0124] The target object is an object that exists in both the second image foreground object set and the second image background object set.

[0125] For example, if the foreground object set of the second image includes: kitten, puppy, and car, and the background object set of the second image includes: kitten, stone, and guardrail, then the target object is the kitten.

[0126] In one possible implementation, the detection device may compare each object in the foreground object set of the second image with an object in the background object set of the second image. If an object in the background object set of the second image is identical to an object in the foreground object set of the second image, the detection device determines that the target object is present in the foreground object set of the second image.

[0127] In another possible implementation, for each object in the foreground object set of the second image, the detection device may determine whether a target object exists in the foreground object set of the second image based on a third operation. The third operation may include: the detection device determining a target overlapping area based on the position information of the detection frame of the first object and the position information of the detection frame of each object in the background object set of the second image, where the target overlapping area is the overlapping area between the detection frame of the first object and the detection frame of the second object, where the first object is any object in the foreground object set of the second image, and the second object is any object in the background object set of the second image, and one target overlapping area corresponds to one second object. Subsequently, if the ratio of the target overlapping area to the first area is greater than a preset overlapping threshold, the detection device determines that a target object exists in the foreground object set of the second image, where the first area is the area of the smallest detection frame between the detection frame of the first object and the detection frame of the second object.

[0128] For example, suppose the foreground object set of the second image includes object a, and the background object set of the second image includes object b. The detection frame of object a overlaps with the detection frame of object b, and the overlapping area is 9. If the area of the detection frame of object a is 10, and the area of the detection frame of object b is 11, then the first area is 10. If the preset overlap threshold is 0.8, then the ratio of the target overlapping area to the first area is 0.9, indicating that objects a and b are target objects.

[0129] It can be understood that when the detection frames of objects in the foreground object set and the background object set overlap and the overlapping area is large, it means that the objects in the two sets are located close to each other and are the same object.

[0130] In some embodiments, if the target object exists in the foreground object set of the second image, S505 is executed.

[0131] In some embodiments, if there is no target object in the second image foreground object set, the detection device uses the second image foreground object set as the first image foreground object set and the second image background object set as the first image background object set.

[0132] It is understandable that if the target object does not exist in the second image foreground object set, it means that the second image foreground object set and the second image background object set do not have the same object. In this case, there is no need to update the second image foreground object set and the second image background object set.

[0133] S505: The detection device re-checks whether the target object is a background object.

[0134] In one possible implementation, the detection device may detect the target object based on a second image background detection algorithm to determine whether the target object is a background object, wherein the second image background detection algorithm is different from the first image background detection algorithm.

[0135] In another possible implementation, the detection device may detect the target object based on a second image foreground detection algorithm to determine whether the target object is a foreground object. If the target object is not a foreground object, the detection device determines that the target object is a background object. The second image foreground detection algorithm is different from the first image foreground detection algorithm.

[0136] Optionally, if the foreground target is not a foreground object, the detection device may obtain a confidence level that the target object is a foreground object and a confidence level that the target object is a background object. If the confidence level that the target object is a foreground object is less than the confidence level that the target object is a background object, the detection device determines that the target object is a background object. If the confidence level that the target object is a foreground object is greater than the confidence level that the target object is a background object, the detection device determines that the target object is a foreground object.

[0137] It should be noted that the detection device can obtain confidence when detecting the target object using the image foreground detection algorithm / image background detection algorithm.

[0138] In some embodiments, if the target object is a background object, the detection device executes S506. If the target object is a foreground object, the detection device executes S507.

[0139] It should be noted that, for each target object, the detection device can execute S505.

[0140] S506: The detection device deletes the target object from the second image foreground object set to obtain the first image foreground object set.

[0141] S507: The detection device deletes the target object from the second image background object set to obtain a first image background object set.

[0142] For example, Figure 7 As shown, the detection device can obtain an image foreground object set and an image background object set. Afterwards, for each object in the image foreground object set (such as object A), the detection device can determine the overlapping area between object A and an object in the image background object set (such as object B). For each overlapping area between object A and object B, the detection device can determine whether the ratio between the overlapping area and the first area is greater than a preset overlapping threshold. If the ratio between the overlapping area and the first area is less than the preset overlapping threshold, the detection device does not process it. If the ratio between the overlapping area and the first area is greater than the preset overlapping threshold, the detection device re-checks whether object A (or object B) is a background object. If object B is a background object, the detection device deletes object A from the image foreground object set. If object B is a foreground object, the detection device deletes object B from the image background object set.

[0143] Based on the above technical solution, the detection device can perform foreground object detection and background object detection on the target image separately to determine the second image foreground object set and the second image background object set. Then, if the target object exists in the second image foreground object set, and the target object is an object that exists in both the second image foreground object set and the second image background object set, it means that the detection result is inaccurate, and the detection device can re-check whether the target object is a background object. If the target object is a background object, the detection device deletes the target object from the second image foreground object set to obtain the first image foreground object set. If the target object is a foreground object, the detection device deletes the target object from the second image background object set to obtain the first image background object set. In this way, the accuracy of the objects in the first image foreground object set and the first image background object set can be further improved.

[0144] S508: The detection device performs foreground object detection on the first point cloud data to determine a second point cloud foreground object set.

[0145] It should be noted that, for a detailed introduction of the detection device performing foreground object detection on the first point cloud data and determining the second point cloud foreground object set, reference may be made to the description in S403 , which will not be elaborated here.

[0146] S509: The detection device filters the point cloud objects in the second point cloud foreground object set based on the first image foreground object set to obtain a first point cloud foreground object set.

[0147] It should be noted that, for a specific introduction to the detection device screening the point cloud objects in the second point cloud foreground object set based on the first image foreground object set to obtain the first point cloud foreground object set, please refer to the description in S404 and will not be repeated here.

[0148] S510: The detection device processes the first point cloud data to determine ground point cloud data.

[0149] It should be noted that, for a detailed description of how the detection device processes the first point cloud data to determine the ground point cloud data, reference may be made to the description in S405 , which will not be elaborated here.

[0150] S511. The detection device filters the first point cloud foreground object set and the ground point cloud data from the first point cloud data to determine the first point cloud background object set.

[0151] It should be noted that, for the specific introduction of the detection device filtering the first point cloud foreground object set and the ground point cloud data from the first point cloud data to determine the first point cloud background object set, please refer to the description in S511 and will not be repeated here.

[0152] S512: The detection device filters the point cloud objects in the first point cloud background object set based on the first image background object set to obtain a second point cloud background object set.

[0153] It should be noted that, for the specific introduction of the detection device screening the point cloud objects in the first point cloud background object set based on the first image background object set to obtain the second point cloud background object set, please refer to the description in S512, which will not be repeated here.

[0154] like Figure 8 FIG. 1 is another background object detection method provided by an embodiment of the present disclosure, the method comprising:

[0155] S701: The detection device obtains first point cloud data and a target image.

[0156] It should be noted that, for a detailed introduction to how the detection device acquires the first point cloud data and the target image, reference may be made to the description in S101 , which will not be elaborated herein.

[0157] S702: The detection device processes the target image to determine a first image background object set.

[0158] It should be noted that, for the specific introduction of the detection device processing the target image and determining the first image background object set, reference may be made to the description in the above embodiments (such as S102, S402, S502-S507), which will not be repeated here.

[0159] S703: The detection device processes the first point cloud data to determine a first point cloud foreground object set.

[0160] It should be noted that, for the specific introduction of the detection device processing the target image and determining the first image background object set, reference may be made to the description in the above embodiments (such as S103 and S404), which will not be repeated here.

[0161] In some embodiments, the detection device may determine ground point cloud data through S704 - S706 .

[0162] S704: The detection device filters the first point cloud foreground object set and the floating point cloud data from the first point cloud data to determine the second point cloud data.

[0163] S705: The detection device updates the point cloud data in the preset grid based on the second point cloud data to generate an updated preset grid.

[0164] The preset grid corresponds to the target area, and the height information of the point cloud data in the updated preset grid is greater than the height information of the point cloud data in the preset grid before the update.

[0165] In one possible implementation, a detection device stores a preset grid containing height information of point cloud data. The detection device may project the second point cloud data onto the preset grid and update the height information of the point cloud data in the preset grid. The updated height information of the point cloud data in the preset grid is the sum of the height information of the second point cloud data and the height information of the point cloud data in the preset grid.

[0166] For example, if the height information of point cloud data A in the preset grid is 1, and the detection device reacquires point cloud data A, and the height information of point cloud data A is 1.1, then the height information of point cloud data A in the updated preset grid is 2.1.

[0167] Optionally, if the detection device does not store a preset grid, the detection device may construct a preset grid based on the second point cloud data. Thereafter, whenever the detection device obtains the second point cloud data, the preset grid may be updated.

[0168] For example, Figure 9As shown, the detection device acquires point cloud data. Afterwards, the detection device can determine whether the current moment is the initial moment (i.e., the moment when the point cloud data is acquired for the first time) (or, the detection device can determine whether there is historical point cloud data or a preset grid). If it is the initial moment, the detection device divides the two-dimensional grid according to the area corresponding to the point cloud data and generates a grid (also called a grid map). Afterwards, the detection device projects the point cloud data onto the grid and saves the height information of the point cloud data. If the current moment is not the initial moment, the range of the grid is updated according to the area corresponding to the point cloud data. Afterwards, the detection device projects the point cloud data onto the updated grid and updates the height information of the point cloud data.

[0169] It's important to note that the grid size is related to the measurement accuracy of the point cloud reaching the sensor point. As the detection device moves, the distance between the device and the object changes, and the height information of the point cloud data corresponding to the object collected by the detection device will also change accordingly. Furthermore, some point cloud data may not be available in real time due to long distances.

[0170] Optionally, the preset grid stores information about historical point cloud data, where the historical point cloud data is point cloud data of the target area acquired before the current acquisition of the second point cloud data. The detection device may update the information about the point cloud data in the preset grid based on the historical point cloud data.

[0171] Exemplarily, the detection device may fill in information of the second point cloud data, such as location information, based on the historical point cloud data.

[0172] S706 : The detection device performs ground detection on the point cloud data in the updated preset grid based on the height information of the point cloud data in the updated preset grid to determine the ground point cloud data.

[0173] In one possible implementation, the detection device performs ground detection on the point cloud data in the updated preset grid based on the height information of the point cloud data in the updated preset grid and a ground detection algorithm to determine the ground point cloud data. The detection device can also obtain position information of the ground point cloud data.

[0174] It is understood that the detection device filters the first point cloud foreground object set from the first point cloud data to determine the second point cloud data. The detection device can then update the point cloud data in the preset grid based on the second point cloud data to generate an updated preset grid. The preset grid corresponds to the target area, and the height information of the point cloud data in the updated preset grid is greater than the height information of the point cloud data in the preset grid before the update. Because the height information of the point cloud data in the updated preset grid is greater, the height information of each point cloud data item varies significantly. Therefore, the detection device can determine the ground point cloud data based on the height information of the point cloud data in the updated preset grid, thereby improving the accuracy of detecting the ground point cloud data.

[0175] S707: The detection device filters the first point cloud foreground object set and the ground point cloud data from the first point cloud data to determine a first point cloud background object set.

[0176] It should be noted that, for the specific introduction of the detection device filtering the first point cloud foreground object set and the ground point cloud data from the first point cloud data to determine the first point cloud background object set, please refer to the description in S104 and will not be repeated here.

[0177] In an embodiment of the present disclosure, the detection device may filter the ground point cloud data from the second point cloud data to determine the first point cloud background object set.

[0178] S708: The detection device filters the point cloud objects in the first point cloud background object set based on the first image background object set to obtain a second point cloud background object set.

[0179] It should be noted that, for the specific introduction of the detection device screening the point cloud objects in the first point cloud background object set based on the first image background object set to obtain the second point cloud background object set, please refer to the description in S105, which will not be repeated here.

[0180] It should be noted that the first point cloud data may contain some floating point cloud data with relatively high altitude information, and this floating point cloud data will not affect the driving of the motor vehicle (such as a plastic bag in the air). Therefore, if the determined first point cloud background object set includes floating point cloud data, the motor vehicle may need to formulate an autonomous driving strategy based on the floating point cloud data, wasting processing resources.

[0181] In some embodiments, the height information of the point cloud data in the first point cloud background object set is less than a preset height threshold. The detection device can determine floating point cloud data from the first point cloud data based on the preset height threshold. The floating point cloud data is point cloud data in the first point cloud data whose height information is greater than the preset height threshold. The detection device can then filter the first point cloud foreground object set, the ground point cloud data, and the floating point cloud data from the first point cloud data to determine the first point cloud background object set.

[0182] Optionally, before S704, the detection device may determine floating point cloud data from the first point cloud data based on a preset height threshold. Thereafter, S705 may include: filtering the first point cloud foreground object set and the floating point cloud data from the first point cloud data to determine second point cloud data.

[0183] It is understood that the height information of the point cloud data in the first point cloud background object set is less than the preset height threshold. In this way, the motor vehicle can avoid formulating an autonomous driving strategy based on objects that do not affect the vehicle's autonomous driving, saving processing resources.

[0184] It should be noted that the point cloud data lacks a lot of information such as color, shape, and texture.

[0185] In some embodiments, the detection device may perform fusion processing on the point cloud data in the second point cloud background object set based on the first image background object set to obtain a third point cloud background object set.

[0186] In one possible implementation, for each point cloud data in the second point cloud background object set, the detection device can fuse the point cloud data with the image corresponding to the point cloud data to obtain fused point cloud data, and the third point cloud background object set includes: each point cloud data in the fused second point cloud background object set.

[0187] It should be noted that the method by which the detection device fuses the point cloud data in the second point cloud background object set based on the first image background object set can refer to conventional methods for fusing image and point cloud data, and is not limited here. For example, the fusion process can be performed based on a Kalman filter.

[0188] Optionally, the detection device may perform a fusion process on the first image background object set based on the second point cloud background object set to obtain a fused first image background object set.

[0189] It is understood that the detection device fuses the point cloud data in the second point cloud background object set based on the first image background object set to obtain a third point cloud background object set. Because image objects contain richer information such as color, shape, and texture than point cloud data, the point cloud data in the third point cloud background object set obtained by the detection device is richer in information, thereby improving object recognition accuracy.

[0190] The following describes the embodiments of the present disclosure with reference to specific examples. Figure 10 As shown, the detection device can acquire point cloud data and image data. The detection device can then first extract point cloud data within area a, and then extract point cloud data within the boundary based on the boundary information (i.e., the point cloud data of the target area). Furthermore, the detection device can perform foreground object detection based on the point cloud data to obtain a point cloud foreground object set a. Simultaneously, the detection device can perform foreground object detection and background object detection on the image data, filtering and updating the image data based on the detection results to obtain an image foreground object set a and an image background object set a. The detection device can then perform filtering and updating based on the point cloud foreground object set a and the image background object set a to obtain a point cloud foreground object set b and an image background object set b. The detection device then filters the point cloud foreground object set b and point cloud data with height information greater than a preset height threshold from the point cloud data of the target area to obtain point cloud data a. The detection device can then filter the ground point cloud data based on point cloud data a (alternatively, before filtering the ground point cloud data, the detection device can first construct / update a grid and then filter the ground point cloud data) and fill the remaining point cloud data to obtain a point cloud background object set a. Then, the detection device can fuse the point cloud background object set a and the image background object set b to obtain fused point cloud data. In addition, the detection device can also obtain the fused image background object set b.

[0191] The above mainly introduces the solution provided by the embodiment of the present disclosure from the perspective of computer equipment. It can be understood that in order to realize the above functions, the computer equipment includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the steps of the detection method of the background object of each example described in the embodiment disclosed in the present disclosure, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present disclosure.

[0192] The embodiments of the present disclosure can divide the detection method of background objects into functional modules or functional units according to the above method examples. For example, each functional module or functional unit can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules or functional units. Among them, the division of modules or units in the embodiments of the present disclosure is schematic and is only a logical functional division. In actual implementation, there may be other division methods.

[0193] like Figure 11 FIG. 1 is a schematic diagram of a background object detection device according to an embodiment of the present disclosure. The background object detection device may include: an acquisition unit 1001 and a processing unit 1002 .

[0194] The acquisition unit 1001 is used to acquire first point cloud data and a target image, wherein the first point cloud data includes point cloud data within a target area, and the target image is an image of the target area. The processing unit 1002 is used to process the target image and determine a first image background object set. The processing unit 1002 is also used to process the first point cloud data and determine a first point cloud foreground object set and ground point cloud data. The processing unit 1002 is also used to filter the first point cloud foreground object set and the ground point cloud data from the first point cloud data to determine a first point cloud background object set. The processing unit 1002 is also used to filter the point cloud objects in the first point cloud background object set based on the first image background object set to obtain a second point cloud background object set, wherein the second point cloud background object set is the intersection of the first image background object set and the first point cloud background object set.

[0195] Optionally, processing unit 1002 is further configured to process the target image to determine a first image foreground object set. Processing unit 1002 is further configured to perform foreground object detection on the first point cloud data to determine a second point cloud foreground object set. Processing unit 1002 is further configured to filter point cloud objects in the second point cloud foreground object set based on the first image foreground object set to obtain a first point cloud foreground object set, where the first point cloud foreground object set is the intersection of the first image foreground object set and the second point cloud foreground object set.

[0196] Optionally, the processing unit 1002 is further configured to perform foreground object detection on the target image to determine a second image foreground object set. The processing unit 1002 is further configured to perform background object detection on the target image to determine a second image background object set. The processing unit 1002 is further configured to re-verify whether the target object is a background object if the target object exists in the second image foreground object set, and the target object is an object that exists in both the second image foreground object set and the second image background object set. The processing unit 1002 is further configured to delete the target object from the second image foreground object set to obtain a first image foreground object set if the target object is a background object. The processing unit 1002 is further configured to delete the target object from the second image background object set to obtain a first image background object set if the target object is a foreground object.

[0197] Optionally, processing unit 1002 is further configured to filter the first point cloud foreground object set from the first point cloud data to determine second point cloud data. Processing unit 1002 is further configured to update the point cloud data in the preset grid based on the second point cloud data to generate an updated preset grid, wherein the preset grid corresponds to the target area, and the height information of the point cloud data in the updated preset grid is greater than the height information of the point cloud data in the preset grid before the update. Processing unit 1002 is further configured to perform ground detection on the point cloud data in the updated preset grid based on the height information of the point cloud data in the updated preset grid to determine ground point cloud data.

[0198] Optionally, height information of the point cloud data in the first point cloud background object set is less than a preset height threshold.

[0199] Optionally, the processing unit 1002 is further configured to perform fusion processing on the point cloud data in the second point cloud background object set based on the first image background object set to obtain a third point cloud background object set.

[0200] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0201] Figure 12 A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0202] like Figure 12 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. Various programs and data required for the operation of the device 1100 can also be stored in the RAM 1103. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0203] Various components in device 1100 are connected to I / O interface 1105, including an input unit 1106, such as a keyboard and mouse; an output unit 1107, such as various types of displays and speakers; a storage unit 1108, such as a magnetic disk and optical disk; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0204] The computing unit 1101 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as the background object detection method. For example, in some embodiments, the background object detection method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the background object detection method described above can be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute the background object detection method in any other appropriate manner (for example, by means of firmware).

[0205] According to an embodiment of the present disclosure, the present disclosure also provides an autonomous driving vehicle, including an electronic device.

[0206] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0207] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0208] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0209] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0210] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0211] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0212] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0213] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for detecting background objects, comprising: Acquire first point cloud data and a target image, wherein the first point cloud data includes point cloud data within a target area, and the target image is an image of the target area; Processing the target image to determine a first image background object set; processing the first point cloud data to determine a first point cloud foreground object set and ground point cloud data in the first point cloud data; filtering the first point cloud foreground object set and the ground point cloud data from the first point cloud data to determine a first point cloud background object set; Based on the first image background object set, the point cloud objects in the first point cloud background object set are screened to obtain a second point cloud background object set, where the second point cloud background object set is the intersection of the first image background object set and the first point cloud background object set.

2. The method according to claim 1, further comprising: Processing the target image to determine a first image foreground object set; The step of processing the first point cloud data to determine the first point cloud foreground object set includes: performing foreground object detection on the first point cloud data to determine a second point cloud foreground object set; The point cloud objects in the second point cloud foreground object set are screened based on the first image foreground object set to obtain the first point cloud foreground object set, where the first point cloud foreground object set is the intersection of the first image foreground object set and the second point cloud foreground object set.

3. The method according to claim 2, wherein: The first image background object set and the first image foreground object set are determined by the following process: Performing foreground object detection on the target image to determine a foreground object set of a second image; Performing background object detection on the target image to determine a second image background object set; If the target object exists in the second image foreground object set, rechecking whether the target object is a background object, the target object being an object that exists in both the second image foreground object set and the second image background object set; If the target object is a background object, deleting the target object from the second image foreground object set to obtain the first image foreground object set; If the target object is a foreground object, the target object is deleted from the second image background object set to obtain the first image background object set.

4. The method according to any one of claims 1 to 3, wherein The ground point cloud data is determined through the following process: filtering the first point cloud foreground object set from the first point cloud data to determine second point cloud data; updating the point cloud data in the preset grid based on the second point cloud data to generate an updated preset grid, wherein the preset grid corresponds to the target area, and height information of the point cloud data in the updated preset grid is greater than height information of the point cloud data in the preset grid before the update; Based on the height information of the point cloud data in the updated preset grid, ground detection is performed on the point cloud data in the updated preset grid to determine the ground point cloud data. 5 . The method according to claim 1 , wherein height information of point cloud data in the first point cloud background object set is less than a preset height threshold.

6. The method according to any one of claims 1 to 5, wherein The method further comprises: Based on the first image background object set, point cloud data in the second point cloud background object set is fused to obtain a third point cloud background object set.

7. A background object detection device, comprising: an acquisition unit, configured to acquire first point cloud data and a target image, wherein the first point cloud data includes point cloud data within a target area, and the target image is an image of the target area; a processing unit, configured to process the target image and determine a first image background object set; The processing unit is further configured to process the first point cloud data to determine a first point cloud foreground object set and ground point cloud data in the first point cloud data; The processing unit is further configured to filter the first point cloud foreground object set and the ground point cloud data from the first point cloud data to determine a first point cloud background object set; The processing unit is further used to filter the point cloud objects in the first point cloud background object set based on the first image background object set to obtain a second point cloud background object set, where the second point cloud background object set is the intersection of the first image background object set and the first point cloud background object set.

8. The device according to claim 7, wherein The processing unit is further configured to process the target image to determine a first image foreground object set; The processing unit is further configured to perform foreground object detection on the first point cloud data to determine a second point cloud foreground object set; The processing unit is further used to filter the point cloud objects in the second point cloud foreground object set based on the first image foreground object set to obtain the first point cloud foreground object set, where the first point cloud foreground object set is the intersection of the first image foreground object set and the second point cloud foreground object set.

9. The device according to claim 8, wherein The processing unit is further configured to perform foreground object detection on the target image to determine a set of foreground objects for the second image; The processing unit is further configured to perform background object detection on the target image to determine a second image background object set; The processing unit is further configured to re-verify whether the target object is a background object if the target object exists in the second image foreground object set, the target object being an object that exists in both the second image foreground object set and the second image background object set; The processing unit is further configured to, if the target object is a background object, delete the target object from the second image foreground object set to obtain the first image foreground object set; The processing unit is further configured to, if the target object is a foreground object, delete the target object from the second image background object set to obtain the first image background object set.

10. The device according to any one of claims 7 to 9, wherein: The processing unit is further configured to filter the first point cloud foreground object set from the first point cloud data to determine second point cloud data; The processing unit is further configured to update the point cloud data in the preset grid based on the second point cloud data to generate an updated preset grid, wherein the preset grid corresponds to the target area, and the height information of the point cloud data in the updated preset grid is greater than the height information of the point cloud data in the preset grid before the update; The processing unit is further configured to perform ground detection on the point cloud data in the updated preset grid based on the height information of the point cloud data in the updated preset grid to determine the ground point cloud data.

11. The device according to any one of claims 7 to 10, wherein height information of point cloud data in the first point cloud background object set is less than a preset height threshold.

12. The device according to any one of claims 7 to 11, wherein: The processing unit is further configured to perform fusion processing on the point cloud data in the second point cloud background object set based on the first image background object set to obtain a third point cloud background object set.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

15. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

16. An autonomous driving vehicle comprising the electronic device according to claim 13.

Images