Target detection method and device, equipment and medium

Abstract

The embodiment of the invention discloses a target detection method and device, equipment and a medium, and the method comprises the steps: obtaining a vehicle detection result obtained through the detection of a vehicle sensor, and obtaining a roadside detection result obtained through the detection of a roadside sensor; constructing an object distribution diagram corresponding to the vehicle detection result and an object distribution diagram corresponding to the roadside detection result according to objects and object relationships respectively included in the vehicle detection result and the roadside detection result; matching the object distribution diagrams to obtain a matching result, and determining common visible objects among the object distribution diagrams according to the matching result; and fusing the vehicle detection result and the roadside detection result according to the common visible object to obtain a target detection result under a vehicle view angle. According to the technical scheme, the visual field of the vehicle can be expanded, the sensing range of the vehicle is widened, and the sensing limitation of a single vehicle sensor is avoided.

Description

Technical Field

[0001] The present application relates to the field of computer and communication technology, and in particular to a target detection method, a target detection device, an electronic device, and a computer-readable storage medium. Background Art

[0002] In driving scenarios, the rapid development of vehicle assisted driving systems provides important support for drivers. However, the current vehicle driving assistance systems mainly rely on information fed back by sensors on the vehicle to perceive the environment, which has some limitations to a certain extent. Taking the current common configuration as an example, vehicles are usually equipped with radars that can detect the distance between the vehicle in front and the vehicle itself, as well as cameras for detecting surrounding objects. Although these sensors provide environmental perception capabilities to a certain extent, the perception capabilities of a single vehicle's sensors have their limitations. Summary of the invention

[0003] The embodiments of the present application provide a target detection method, a target detection device, an electronic device, a computer-readable storage medium, and a computer program product, which expand the field of view of a vehicle, improve the perception range of the vehicle, and avoid the perception limitations of a single vehicle sensor.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0005] In a first aspect, an embodiment of the present application provides a target detection method, including: obtaining a vehicle detection result obtained by a vehicle sensor, and obtaining a roadside detection result obtained by a roadside sensor; constructing an object distribution map corresponding to the vehicle detection result and an object distribution map corresponding to the roadside detection result according to the objects and object relationships respectively contained in the vehicle detection result and the roadside detection result; matching the object distribution maps to obtain a matching result, and determining the common visible objects between the object distribution maps according to the matching result; fusing the vehicle detection result and the roadside detection result according to the common visible objects to obtain a target detection result from the vehicle perspective.

[0006] In the second aspect, an embodiment of the present application also provides a target detection device, which includes: an acquisition module, used to acquire vehicle detection results obtained by vehicle sensors, and to acquire roadside detection results obtained by roadside sensors; a construction module, used to construct an object distribution map corresponding to the vehicle detection results and an object distribution map corresponding to the roadside detection results according to the objects and object relationships respectively contained in the vehicle detection results and the roadside detection results; a matching module, used to match the object distribution maps to obtain matching results, and determine the common visible objects between the object distribution maps according to the matching results; and a fusion module, used to fuse the vehicle detection results and the roadside detection results according to the common visible objects to obtain target detection results from the vehicle perspective.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising one or more processors; and a storage device for storing one or more computer programs, so that when the one or more computer programs are executed by the one or more processors, the electronic device implements the target detection method as described above.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor of an electronic device, the electronic device executes the target detection method as described above.

[0009] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, wherein the computer program is stored in a computer-readable storage medium, and a processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the target detection method as described above.

[0010] In the technical solution provided in the embodiments of the present application, the vehicle detection result obtained by the vehicle sensor is obtained, and the roadside detection result obtained by the roadside sensor is obtained, and the object distribution map corresponding to the detection result is constructed through the objects and object relationships contained in the detection result, and then the common visible objects between the object distribution maps are determined through the matching results of the object distribution maps, and then the vehicle detection result and the roadside detection result are fused through the common visible objects to obtain the target detection result from the vehicle perspective; that is, in the technical solution provided by the present application, on the one hand, the vehicle detection result detected by the vehicle sensor and the roadside detection result detected by the roadside sensor are fused to expand the vehicle's field of view and improve the vehicle's perception range to avoid the perception limitations of a single vehicle sensor; on the other hand, by constructing a graph and performing graph matching to determine the common visible objects, the detection results are fused based on the common visible objects, so that the fusion of the detection results is more accurate, thereby ensuring a more accurate environmental perception capability of the vehicle.

[0011] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a schematic diagram of an implementation environment involved in this application;

[0013] Figure 2 It is a schematic diagram of an implementation environment involved in this application;

[0014] Figure 3 is a flow chart of a target detection method shown in an exemplary embodiment of the present application;

[0015] Figure 4 is a schematic diagram of a detection result and an object distribution diagram shown in an exemplary embodiment of the present application;

[0016] Figure 5 is a flow chart of another target detection method shown in an exemplary embodiment of the present application;

[0017] Figure 6 is a flow chart of another target detection method shown in an exemplary embodiment of the present application;

[0018] Figure 7 is a flow chart of another target detection method shown in an exemplary embodiment of the present application;

[0019] Figure 8 is a flow chart of another target detection method shown in an exemplary embodiment of the present application;

[0020] Fig. 9 is a flow chart of another target detection method shown in an exemplary embodiment of the present application;

[0021] Fig.10 is a flow chart of another target detection method shown in an exemplary embodiment of the present application;

[0022] Fig.11 is a flow chart of another target detection method shown in an exemplary embodiment of the present application;

[0023] Fig.12 is a flow chart of another target detection method shown in an exemplary embodiment of the present application;

[0024] Fig.13 is a flow chart of a target detection method shown in another exemplary embodiment of the present application;

[0025] Fig.14 is a flow chart of another target detection method shown in another exemplary embodiment of the present application;

[0026] Fig.15 is a flow chart of another target detection method shown in another exemplary embodiment of the present application;

[0027] Fig.16 is a schematic diagram of a target detection method shown in another exemplary embodiment of the present application;

[0028] Fig.17 is a structural block diagram of a target detection device shown in an exemplary embodiment of the present application;

[0029] Fig.18 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0030] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0031] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0032] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0033] It should also be noted that the "multiple" mentioned in this application refers to two or more than two. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship.

[0034] The technical solutions of the embodiments of the present application can also be applied to the field of intelligent transportation. In the field of transportation, the Intelligent Vehicle Infrastructure Cooperative System (IVICS), referred to as the vehicle-infrastructure cooperative system, is a development direction of the Intelligent Traffic System (ITS). The vehicle-infrastructure cooperative system adopts advanced wireless communications and new generation Internet technologies to implement all-round dynamic real-time information interaction between vehicles and roads, and carries out active vehicle safety control and road cooperative management based on the collection and integration of dynamic traffic information in all time and space, fully realizing the effective coordination of people, vehicles and roads, ensuring traffic safety, and improving traffic efficiency, thereby forming a safe, efficient and environmentally friendly road traffic system.

[0035] Based on the intelligent vehicle-road cooperative system, in an embodiment of the present application, the vehicle interacts with the road dynamic real-time information to obtain relevant information.

[0036] The technical solutions of the embodiments of the present application can also be applied in the field of autonomous driving.

[0037] The following is a detailed introduction to the technical solutions of the embodiments of the present application:

[0038] See also Figure 1 , Figure 1 1 is a schematic diagram of an implementation environment involved in the present application. The implementation environment includes a vehicle 10 and a roadside unit 20, wherein the vehicle 10 and the roadside unit 20 communicate with each other via a wireless network; wherein the roadside unit 20 is deployed on both sides of the road, and the vehicle 10 and the roadside unit 20 are each provided with one or more sensors 30, which can collect corresponding data, and by processing the corresponding data, the detection results of various objects on the road can be obtained.

[0039] The vehicle 10 obtains a vehicle detection result obtained by a vehicle sensor and a roadside detection result obtained by a roadside sensor, and constructs an object distribution map corresponding to the vehicle detection result and an object distribution map corresponding to the roadside detection result according to the objects and object relationships respectively contained in the vehicle detection result and the roadside detection result; matches the object distribution maps to obtain a matching result, and determines the common visible objects between the object distribution maps according to the matching result; and fuses the vehicle detection result and the roadside detection result according to the common visible objects to obtain a target detection result from the vehicle perspective.

[0040] See also Figure 2 , Figure 2 This is a schematic diagram of another implementation environment involved in the present application, where the implementation environment includes a vehicle 10 , a roadside unit 20 , and a cloud server 40 .

[0041] The vehicle 10 detects various objects on the road through the sensors 30 provided thereon, and obtains a vehicle detection result. The roadside unit 20 detects various objects on the road through the sensors 30 provided thereon, and obtains a roadside detection result.

[0042] The vehicle 10 sends the vehicle detection result to the cloud server 40 , and the roadside unit 20 sends the roadside detection result to the cloud server 40 .

[0043] The cloud server 40 constructs object distribution maps based on the vehicle detection results and the roadside detection results respectively; matches the object distribution maps to obtain matching results, and determines the common visible objects between the object distribution maps based on the matching results; fuses the vehicle detection results and the roadside detection results based on the common visible objects to obtain the target detection results from the vehicle's perspective, and sends the target detection results to the vehicle.

[0044] In other embodiments of the present application, vehicle sensors and roadside sensors may collect sensor data, and the vehicle and roadside units may send the collected sensor data to a cloud server. The cloud server may obtain vehicle detection results and roadside detection results based on the corresponding sensor data, and then fuse the vehicle detection results and roadside detection results, and finally send the target detection results to the vehicle.

[0045] Figure 2 In the real-time environment shown, the system architecture based on cloud computing is very advantageous when facing tasks with high computing power requirements.

[0046] The aforementioned roadside unit 20 can be an intelligent lamp post equipped with sensors and computing units, traffic lights and road signs, etc., deployed on both sides of the road; the aforementioned vehicle 10 can be an unmanned vehicle, a fuel vehicle, an electric vehicle, a battery vehicle and other drivable vehicles; the cloud server 40 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network) and basic cloud computing services such as big data and intelligent platforms, and no restrictions are imposed here.

[0047] It should be noted that in the specific implementation of the present application, if the vehicle detection results and / or roadside detection results are related to objects, when the embodiments of the present application are applied to specific products or technologies, it is necessary to obtain the permission or consent of the object, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0048] The following is a detailed description of various implementation details of the technical solution of the embodiment of the present application:

[0049] like Figure 3 As shown, Figure 3 is a flowchart of a target detection method shown in an embodiment of the present application, which can be applied to Figure 1 or Figure 2 In the implementation environment shown, the method can be executed by a vehicle or a cloud server, or can be executed by a vehicle and a cloud server together. In the embodiment of the present application, the method is executed by a vehicle as an example for explanation. The target detection method can include steps S310 to S340, which are described in detail as follows:

[0050] S310: Acquire vehicle detection results obtained by vehicle sensors, and acquire roadside detection results obtained by roadside sensors.

[0051] In an embodiment of the present application, one or more sensors are configured on the vehicle, and the configured sensors can obtain vehicle information in real time, such as position, direction, driving distance, speed and acceleration, etc.; the configured sensors can also obtain external environment information of the vehicle, such as the position of pedestrians and other vehicles on the road; the information obtained by the sensors can obtain vehicle detection results, where, if the vehicle is configured with multiple sensors, the multiple sensors include at least two different types of sensors; similarly, one or more sensors are configured on the roadside unit, which can detect vehicles and pedestrians on the road to obtain roadside detection results.

[0052] In one example, the vehicle detection results and the roadside detection results are both the results of three-dimensional target detection, that is, the vehicle detection results and the roadside detection results include the position, size and category information of the object in three-dimensional space, and the category information includes vehicles, pedestrians, obstacles, etc.

[0053] In one example, the sensors include but are not limited to lidar, ordinary cameras, infrared sensors, etc.; the sensors configured on the roadside unit may be the same as or different from the sensors configured on the vehicle, which is not limited here.

[0054] S320: construct an object distribution map corresponding to the vehicle detection result and an object distribution map corresponding to the roadside detection result according to the objects and object relationships respectively included in the vehicle detection result and the roadside detection result.

[0055] As described above, the vehicle detection result includes the position, size and category information of the detected object in the three-dimensional space, and multiple objects and object relationships of multiple objects can be obtained. The object relationship includes spatial relationship and semantic relationship. The spatial relationship, such as adjacent, overlapping, etc., can be obtained through the position and size of the object in the three-dimensional space. The semantic relationship, such as the same category, can be obtained through the category information of the object in the three-dimensional space. Therefore, the object distribution map corresponding to the vehicle detection result can be constructed based on the objects and object relationships contained in the vehicle detection result; similarly, the roadside detection result includes the position, size and category information of the object, and then the relationship between the objects is determined, and then the object distribution map corresponding to the roadside detection result is constructed based on the objects and object relationships.

[0056] In one example, the object distribution graph uses objects as nodes and object relationships as edges. Figure 4 As shown, the vehicle detection result of vehicle 401 includes vehicle 402 and vehicle 403, and the corresponding object distribution graph 410 is based on vehicle 402 and vehicle 403 as nodes, and the relationship between vehicle 402 and vehicle 403 as edges; the roadside detection result of roadside unit 404 includes vehicle 402, vehicle 403 and pedestrian 405, and the corresponding object distribution graph 420 is based on vehicle 402, vehicle 403 and pedestrian 405 as nodes, and the relationship between vehicle 402, vehicle 403 and pedestrian 405 as edges. If the detection result also includes the running direction of the object, the node also includes the moving direction.

[0057] S330: Match the object distribution graphs to obtain matching results, and determine the common visible objects between the object distribution graphs according to the matching results.

[0058] In the embodiment of the present application, the object distribution graphs may be matched to obtain a matching result, which indicates whether the object distribution graphs are similar, and the similar and different contents in the object distribution graphs.

[0059] According to the matching results, the common visible objects between the object distribution maps can be determined, and the common visible objects are objects existing in all the object distribution maps; for example Figure 4 In the figure, object distribution map 410 and object distribution map 420 include the same objects, namely vehicle 401 and vehicle 402. The same objects have the same distribution positions in the object distribution map corresponding to the vehicle detection result and the object distribution map corresponding to the roadside detection result. For example, the relative direction and position between vehicle 401 and vehicle 402 are the same in object distribution map 410 and object distribution map 420.

[0060] In one example, after the matching result indicates that the similarity between the object distribution graphs reaches a preset similarity threshold, similar objects between the object distribution graphs are determined and the similar objects are taken as common visible objects.

[0061] S340: Fusing the vehicle detection result and the roadside detection result according to the common visible object to obtain the target detection result from the vehicle perspective.

[0062] In an embodiment of the present application, after determining the common visible objects, the detection results can be fused according to the common visible objects to obtain the target detection results from the vehicle's perspective, that is, the roadside detection results are supplemented to the vehicle detection results to supplement the vehicle's field of view.

[0063] In one example, the vehicle detection result and the roadside detection result may be aligned with the common visible object as the origin, that is, the object distribution map corresponding to the vehicle detection result and the object distribution map corresponding to the roadside detection result are overlapped with the common visible object as the origin.

[0064] In one example, the positions of invisible objects in vehicle detection results or roadside detection results are determined based on the positions of common visible objects, and then the corresponding invisible positions are added to the reference map, so that each object is fused into the reference map, and then the target detection results from the vehicle perspective are generated based on the fused reference map.

[0065] In an embodiment of the present application, a vehicle detection result obtained by a vehicle sensor is obtained, and a roadside detection result obtained by a roadside sensor is obtained, and an object distribution map corresponding to the detection result is constructed through the objects and object relationships contained in the detection result, and then the common visible objects between the object distribution maps are determined through the matching results of the object distribution maps, and then the vehicle detection result and the roadside detection result are fused through the common visible objects to obtain the target detection result from the vehicle perspective; that is, in the technical solution provided by the present application, on the one hand, the vehicle detection result detected by the vehicle sensor and the roadside detection result detected by the roadside sensor are fused to expand the field of view of the vehicle and improve the perception range of the vehicle to avoid the perception limitations of a single vehicle sensor; on the other hand, by constructing a graph and performing graph matching to determine the common visible objects, the detection results are fused based on the common visible objects, so that the fusion of the detection results is more accurate, thereby ensuring a more accurate environmental perception capability of the vehicle.

[0066] In one embodiment of the present application, another target detection method is provided, which can be applied to Figure 1 or Figure 2 In the implementation environment shown, the method can be executed by the vehicle or the cloud server, or by the vehicle and the cloud server together. The method is described by taking the vehicle as an example. Figure 5 As shown in Figure 3 Based on S310 to S340 shown in Figure 3The step S330 shown in is expanded into steps S510 to S540. The vehicle detection result in S310 includes vehicle detection results obtained by multiple sensors respectively, that is, one sensor corresponds to one vehicle detection result, and one vehicle detection result corresponds to one object distribution map; steps S510 to S540 are described in detail as follows:

[0067] S510: Select a target sensor from multiple sensors, and use an object distribution map corresponding to a vehicle detection result obtained by the target sensor as a reference map.

[0068] In an embodiment of the present application, in order to facilitate the matching of the object distribution map, an object distribution map can be selected as a reference map, and then the reference map is matched with other distribution maps in sequence to avoid duplicate matching; wherein, the final fused detection result is applied to the vehicle, so a target sensor can be randomly selected from multiple sensors of the vehicle, or a target sensor can be selected from multiple sensors according to the detection accuracy of each sensor; for example, multiple sensors include a lidar, an ordinary camera and an infrared sensor, then the lidar is selected as the target sensor, and then the object distribution map corresponding to the vehicle detection result corresponding to the lidar is used as the reference map.

[0069] S520: Match the reference image with other object distribution images, and obtain the matching degree between the reference image and other object distribution images.

[0070] Among them, other object distribution maps include object distribution maps corresponding to roadside detection results obtained by roadside sensors, and object distribution maps corresponding to vehicle detection results obtained by other sensors except the target sensor among multiple sensors. Continuing with the above example, if the object distribution map corresponding to the vehicle detection result corresponding to the laser radar is used as the reference map, the object distribution maps corresponding to the ordinary camera and infrared camera of the vehicle are used as other object distribution maps, and the object distribution map corresponding to the roadside detection result is used as other object distribution map.

[0071] In one example, the reference image is matched with each other object distribution image in turn to obtain the matching degree between the reference image and the other distribution images. The higher the matching degree, the more similar the reference image and the other distribution images are.

[0072] S530: Determine the matching result between the reference image and other object distribution images according to the matching degree.

[0073] In an embodiment of the present application, the matching result between the reference image and the other object distribution images is determined based on the matching degree. If the matching degree is greater than a preset matching degree threshold, the matching result between the reference image and the other object distribution images is determined to be similar; otherwise, the matching result is determined to be dissimilar.

[0074] S540: Determine the common visible objects between the reference image and other object distribution images according to the matching results.

[0075] It should be noted that Figure 5 For other detailed descriptions of steps S310 to S320 and S340 shown in Figure 3 The steps S310 to S320 and S340 shown are not described in detail here.

[0076] In an embodiment of the present application, the vehicle detection result includes vehicle detection results obtained by multiple sensors respectively, and the vehicle's perception capability is further expanded by multiple sensors. In addition, a target sensor is selected from the multiple sensors of the vehicle, and the object distribution map corresponding to the target sensor is used as a reference map, and then matched with other object distribution maps. On the one hand, the orderliness of the map matching is guaranteed, and on the other hand, it is convenient to directly apply the fused target detection results to the vehicle later.

[0077] The present invention provides another target detection method, which can be applied to Figure 1 or Figure 2 In the implementation environment shown, the method can be executed by the vehicle or the cloud server, or by the vehicle and the cloud server together. The method is described by taking the vehicle as an example. Figure 6 As shown in Figure 5 Based on the above, S520 is expanded to S610-S630. The object distribution graph is constructed with objects as nodes and object relationships as edges. Steps S610-S630 are described in detail as follows:

[0078] S610: Match the reference map with other object distribution maps.

[0079] S620, calculating at least one of the category similarity of node pairs formed by nodes in the reference graph and nodes in other object distribution graphs, the bounding box position similarity of the node pairs, and the bounding box size similarity of the node pairs to generate a matching degree between the nodes.

[0080] In an embodiment of the present application, when performing graph matching, matching is performed on a node-by-node basis, and a node in a reference graph and a node in another object distribution graph can form a node pair, wherein any node in the reference graph and any node in another object distribution graph can form a node pair, and it is necessary to calculate at least one of the category similarity of the node pair, the bounding box position similarity of the node pair, and the bounding box size similarity of the node pair.

[0081] Among them, the category similarity of the node pair refers to the similarity between the category of the node in the benchmark graph and the category of the node in other object distribution graphs, where the categories include vehicles (including bicycles, electric vehicles, etc.), pedestrians and obstacles. Only when the categories are the same can it be said that the node pair is matched successfully.

[0082] The bounding box of a node pair is used to describe the shape of the node. Therefore, the position similarity of the node pair can be determined by calculating the position similarity of the node pair's bounding box, and the shape similarity of the node pair can be determined by calculating the size similarity of the node pair's bounding box.

[0083] like Figure 4 , the category and bounding box similarity of the node 402 (vehicle 402) in the object distribution map 410 and the node 402 (vehicle 402) in the object distribution map 420 are calculated to generate a matching degree between the node 402 (vehicle 402) in the object distribution map 410 and the node 402 (vehicle 402) in the object distribution map 420.

[0084] In one example, when generating the degree of matching between nodes, the category similarities of the two nodes are first compared. After the category similarity is greater than the preset category similarity threshold, other factors, such as position and size similarity, are considered to comprehensively determine the accuracy of the matching degree. For example, after the category similarity is greater than the preset category similarity threshold, the average of the category similarity, bounding box position similarity, and bounding box size similarity is used as the node matching degree, or the category similarity, bounding box position similarity, and bounding box size similarity are weighted and summed to obtain the node matching degree; if the category similarity is less than the preset category similarity threshold, the node matching degree is directly determined to be 0.

[0085] S630: Calculate at least one of category similarity, distance similarity, and angle similarity between node pairs in the reference graph and node pairs in other object distribution graphs to generate a matching degree between edges.

[0086] In an embodiment of the present application, when performing graph matching, matching is also performed using edges as units, and the edges are represented by node pairs in the two graphs respectively; that is, the node pairs in the reference graph refer to the node pairs formed by the two nodes in the reference graph. Similarly, the node pairs in other object distribution graphs are formed by the two nodes in the reference graph.

[0087] The distance / angle similarity between node pairs refers to the similarity between the distance / angle between two nodes in the base graph and the distance / angle between two nodes in other object distribution graphs.

[0088] In one example, if at least two of the category similarity, distance similarity, and angle similarity between node pairs are calculated, the average of the at least two similarities can be used as the degree of matching between edges, or weights can be assigned to the category similarity, distance similarity, and angle similarity, and a weighted sum is performed based on the corresponding weights and the corresponding similarities to obtain the degree of matching between edges.

[0089] like Figure 4 In the figure, object distribution graph 410 is used as the reference graph, and object distribution graph 420 is used as the other distribution graphs. The node pairs of the reference graph are: node 402-node 403, and the node pairs of the other distribution graphs are: node 402-node 403. Assuming that the category of node 402-node 403 in the reference graph is vehicle-pedestrian, and the category of node 402-node 403 in the other object distribution graph is vehicle-vehicle, it means that the category similarity of the two node pairs is 50%; assuming that the distance between node 402-node 403 in the reference graph is x, and the distance between node 402-node 403 in the other object distribution graph is x, it means that the distance similarity of the two node pairs is 100%, that is, the smaller the distance difference, the greater the distance similarity, and then the matching degree between the edges is determined by combining the category similarity and the distance similarity.

[0090] It should be noted that Figure 6 For a detailed description of steps S310 to S320, S510, S530 to S540, and S340, see Figure 5 The steps S310 to S320, S510, S530 to S540, and S340 shown are not repeated here.

[0091] In an embodiment of the present application, when performing graph matching, not only the matching degree between nodes is considered, but also the matching degree between edges is considered, so that the graph matching is more accurate. The accuracy of the matching degree between nodes is guaranteed by at least one of the category similarity, bounding box position similarity, and bounding box size similarity of node pairs. The accuracy of the matching degree between edges is guaranteed by at least one of the category similarity, distance similarity, and angle similarity between node pairs in the reference graph and node pairs in other object distribution graphs.

[0092] The present application also provides another target detection method, which can be applied to Figure 1 or Figure 2 In the implementation environment shown, the method can be executed by the vehicle or the cloud server, or by the vehicle and the cloud server together. The method is described by taking the vehicle as an example. Figure 7 As shown in Figure 5 Based on the above, S340 is expanded to S710 to S720. Steps S710 to S720 are described in detail as follows:

[0093] S710 . For each common visible object, extract vertices of a bounding box where the common visible object is located from the reference map and the other object distribution map respectively, and generate two point sets according to the extracted vertices.

[0094] In an embodiment of the present application, the bounding box is a box of a regular shape. Taking a rectangular box as an example, for each common visible object, the vertices of the bounding box where the common visible object is located are extracted from the reference map and the other object distribution map, that is, the four vertices of the rectangular box are extracted. The vertices can be represented as two-dimensional points with x and y coordinates.

[0095] The four vertices of the bounding box where the common visible object in the base image is located are used to generate a point set, and the four vertices of the bounding box where the common visible object in the other object distribution image is located are used to generate a point set. In one example, the point set is a matrix, and each column of the matrix corresponds to the x and y coordinates of a vertex.

[0096] It can be understood that due to the difference in sensor types or sensor deployment locations between the reference map and other object distribution maps, the spatial coordinates of the two generated point sets are different.

[0097] S720, calculating a transformation matrix between the two point sets, and aligning the two point sets according to the transformation matrix, so as to fuse the vehicle detection result corresponding to the reference map with the detection result corresponding to the other object distribution map.

[0098] In an embodiment of the present application, the transformation matrix between two point sets can be calculated by a linear least squares problem, and the two point sets can be aligned according to the transformation matrix, which contains information such as rotation, scaling and translation. The spatial coordinates of one point set can be transformed into the spatial coordinates of another point set, so that the two point sets are aligned to the same coordinate space, and then the vehicle detection results corresponding to the baseline map are fused with the detection results corresponding to other object distribution maps.

[0099] It should be noted that Figure 7 For other detailed descriptions of steps S310-S320, S510-S540 shown in Figure 5 Steps S310-S320 and S510-S540 shown in FIG. 1 are not described in detail here.

[0100] In the embodiment of the present application, two point sets are generated by using the vertices of the bounding boxes of the common visible objects from the reference image and the distribution image of other objects, and then the transformation matrix of the two point sets is calculated to achieve point set alignment, thereby ensuring the accuracy and reliability of the fusion of the detection results.

[0101] In one embodiment of the present application, another target detection method is also provided. The target detection method can be applied to Figure 1 or Figure 2 In the implementation environment shown, the method can be executed by the vehicle or the cloud server, or by the vehicle and the cloud server together. The method is described by taking the vehicle as an example. Figure 8 As shown in Figure 7 Based on the information shown in , S720 is expanded to S810 to S830; steps S810 to S830 are described in detail as follows:

[0102] S810: Calculate the covariance matrix between the two point sets, and perform singular value decomposition on the covariance matrix.

[0103] In the embodiment of the present application, the covariance matrix reflects the linear correlation between vertices. The covariance matrix is ​​subjected to singular value decomposition (SVD), which is a matrix decomposition method that can decompose any m×n matrix A into the product of three matrices U, Σ and V^T, where U is an m×m orthogonal matrix, whose column vectors are called left singular vectors; Σ is an m×n diagonal matrix, whose diagonal elements are called singular values; V^T is an n×n orthogonal matrix, whose row vectors are called right singular vectors.

[0104] In one example, before calculating the covariance matrix, the means of the vertex coordinates may be calculated respectively, and then the corresponding mean may be subtracted from each vertex coordinate so that the center point of the data is the origin.

[0105] S820, calculating a rotation matrix according to the left and right singular vector matrices obtained by singular value decomposition, and calculating a translation vector according to the rotation matrix and the vertices in the two point sets.

[0106] A rotation matrix R is obtained by multiplying the first two columns of U and V^T; the rotation matrix describes the rotation relationship between the vertices in the other object distribution map and the vertices in the reference map; a translation vector t is obtained by calculating the vertex mean in the reference map minus the rotation matrix R multiplied by the vertex mean in the other object distribution map; the translation vector t describes the translation relationship of the vertices in the other object distribution map relative to the reference map.

[0107] S830: transform one point set to another point set according to the rotation matrix and the translation vector to align the two point sets.

[0108] Apply the rotation matrix R and translation vector t to all vertices in the other object distribution map to align them with the vertices in the base map.

[0109] It should be noted that Figure 8 For other detailed descriptions of steps S310 to S320, S510 to S540, and S710 shown in Figure 7The steps S310 to S320, S510 to S540, and S710 shown are not repeated here.

[0110] In the embodiment of the present application, the linear correlation between vertices is reflected by the covariance matrix, and then the rotation matrix and translation vector are calculated by SVD decomposition. The two point sets are aligned to the same coordinate space through rotation and translation, thereby ensuring the accuracy and reliability of the point set alignment.

[0111] In one embodiment of the present application, another target detection method is also provided. The target detection method can be applied to Figure 1 or Figure 2 In the implementation environment shown, the method can be executed by the vehicle or the cloud server, or by the vehicle and the cloud server together. The method is described by taking the vehicle as an example. Fig. 9 As shown in Figure 3 Based on the above, S910 to S920 are added before S320. Steps S910 to S920 are described in detail as follows:

[0112] S910: Acquire prior map information of the area where the vehicle is located, where the prior map information includes at least one of a scene structure, a road network, and a building layout.

[0113] In an embodiment of the present application, the detection results obtained by the sensor may be erroneous or spatially inconsistent. Therefore, before constructing the object distribution map, it is necessary to optimize the vehicle detection results and the roadside detection results so that the detection results are more consistent with the spatial distribution in the actual scene.

[0114] It is necessary to first obtain the prior map information of the area where the vehicle is located, wherein the prior map information can be extracted from a map server according to the area where the vehicle is located, and the map server pre-stores the prior map information of various areas.

[0115] The prior map information includes at least one of the structure of the scene, the road network and the building layout, wherein the structure of the scene refers to the description of the spatial layout and structure of the entire scene in the map, including the position, shape and relationship of elements such as roads, buildings and natural features; the road network refers to the part of the map that describes the road system, including information such as road connections, intersections, lanes, lane markings and the like; the building layout includes the location, orientation and spacing of buildings and facilities.

[0116] S920. Constrain the positions and movement ranges of objects respectively included in the vehicle detection results and the roadside detection results according to the prior map information, so as to filter out detection results that are inconsistent with the prior map information.

[0117] In an embodiment of the present application, the vehicle detection result includes the object, the position of the object, and the direction of movement of the object. The movement range of the object can be determined based on the category of the object and the position and direction of movement of the object. For example, the movement range of a pedestrian is the pedestrian road with the current position of the pedestrian as the origin and the direction of movement as the extension.

[0118] In one example, for each object included in the vehicle detection result, the possible location range of the object is determined based on the prior map information. The possible location range can be determined based on the projection position of the object on the map, the connectivity between the object and the road network, and the relationship between the object and the surrounding buildings. Based on the position and motion access of the object in the vehicle detection result, the location range of the object in the vehicle detection result can be determined, and then the location range of the object in the vehicle detection result is matched with the possible location range in the map. According to the matching degree, it is determined whether the detection result of the object is filtered. Among them, if the matching degree between the location of the object included in the vehicle detection result and the possible location in the map is lower than a certain threshold, the detection result for the object can be filtered.

[0119] In one example, the prior map information can be matched with the position and motion range of the object contained in the vehicle detection result to determine whether the position and motion range of the object conform to at least one of the structure, road network and building layout of the scene. For example, if the object contained in the vehicle detection result is a vehicle, and the position and motion range of the object are consistent with the road network, it means that it conforms to the road network and is consistent with the prior map information; if it does not conform, it means that it is inconsistent with the prior map information, and the detection result for the object does not conform to the actual situation, and the detection result for the object needs to be filtered.

[0120] It should be noted that Fig. 9 For more detailed description of steps S310 to S340 shown in Figure 3 The steps S310 to S340 shown are not described in detail here.

[0121] In an embodiment of the present application, the vehicle detection results and the roadside detection results are constrained by the prior map information of the area where the vehicle is located to filter out the detection results that are inconsistent with the prior map information, so that the detection results are more consistent with the spatial distribution in the actual scene.

[0122] In one embodiment of the present application, another target detection method is also provided. The target detection method can be applied to Figure 1 or Figure 2 In the implementation environment shown, the method can be executed by the vehicle or the cloud server, or by the vehicle and the cloud server together. The method is described by taking the vehicle as an example. Fig.10 As shown in Figure 3Based on the information shown in , S310 is expanded to S1010~S1040. S1010~S1040 are described in detail as follows:

[0123] S1010: Acquire vehicle detection results obtained by vehicle sensors.

[0124] The vehicle detection result includes the vehicle detection results at the current moment obtained by detection of multiple different types of sensors, such as the vehicle radar detection result at the current moment obtained by lidar detection, and the vehicle camera detection result at the current moment obtained by camera detection.

[0125] S1020: Obtain roadside detection results at historical moments obtained by roadside sensors, and determine the sensor type corresponding to the roadside detection results at historical moments.

[0126] In an embodiment of the present application, the vehicle detection results and the roadside detection results may be out of sync due to different detection times. Therefore, in an embodiment of the present application, the vehicle detection results at the current moment are used as a reference, and the roadside sensor detection is obtained to obtain the roadside detection results at historical moments. The historical moment is the most recent historical moment relative to the current moment, and the sensor type corresponding to the roadside detection results at the historical moment is determined.

[0127] In one example, after each detection by the roadside sensor, the latest roadside detection result sent by the roadside unit can be received and stored; after the vehicle sensor performs detection at the current moment, the roadside detection result at the most recent historical moment is extracted.

[0128] S1030. Select a reference detection result from vehicle detection results at the current moment corresponding to a plurality of different types of sensors according to the determined sensor type.

[0129] The vehicle detection result includes the vehicle detection results at the current moment obtained by detecting multiple different types of sensors respectively, and the vehicle detection result of the same sensor type as the determined one is selected from the multiple vehicle detection results at the current moment as the reference detection result. For example, if the sensor type corresponding to the roadside detection result at the historical moment is a laser radar, the vehicle radar detection result at the current moment obtained by the laser radar detection is used as the reference detection result.

[0130] S1040: Predict the roadside detection result at the current moment based on the roadside detection results at historical moments and the reference detection results.

[0131] For the same sensor type, the roadside detection results of the roadside unit at the current moment can be predicted based on the roadside detection results of the roadside unit at the historical moment and the reference detection results of the vehicle at the current moment, so as to achieve time synchronization of the detection results at the current moment.

[0132] It should be noted that Fig.10 For more detailed description of steps S320 to S340 shown in Figure 3 Steps S320 to S340 are shown and will not be described in detail here.

[0133] In an embodiment of the present application, the roadside detection result at the current moment is predicted by using several roadside detection results at historical moments obtained by roadside sensor detection and the vehicle reference detection results of the corresponding sensor type at the current moment, so as to avoid the situation where the roadside detection result and the vehicle detection result are out of synchronization in time, or data is lost during the data transmission process.

[0134] In one embodiment of the present application, another target detection method is also provided. The target detection method can be applied to Figure 1 or Figure 2 In the implementation environment shown, the method can be executed by the vehicle or the cloud server, or by the vehicle and the cloud server together. The method is described by taking the vehicle as an example. Fig.11 As shown in Fig.10 Based on the above, S1040 is expanded to S1110 to S1130. Steps S1110 to S1130 are described in detail as follows:

[0135] S1110. Obtain the time interval corresponding to the reference detection result and the roadside detection result at the historical moment.

[0136] In an embodiment of the present application, the difference between the current time corresponding to the reference detection result and the historical time corresponding to the roadside detection result at the historical time is taken as the time interval.

[0137] S1120. Predict the position and speed of the object included in the roadside detection results at the current moment based on the linear motion model, the roadside detection results at the historical moments, and the time interval.

[0138] Among them, the linear motion model is a model that assumes that an object moves at a constant speed in space. The linear motion model can be expressed by the following formula:

[0139] x t =x t-1 +v t-1 Δt; where x t is the position of the object at time t, v t-1 is the velocity of the object at time t-1, x t-1 is the position of the object at time t-1, and Δt is the time interval.

[0140] Therefore, the position and speed of the object at the current moment can be calculated based on the linear motion model, the position of the object included in the roadside detection results at historical moments, and the time interval.

[0141] In one example, before predicting the position and speed of the object at the current moment, the category of the object needs to be determined first. If the category is an obstacle, there is no need to predict the position and speed of the object at the current moment.

[0142] S1130. Generate a roadside detection result at the current moment according to the position and speed of the object at the current moment.

[0143] In an embodiment of the present application, the roadside result at the current moment is generated based on the position and speed of the object at the current moment, combined with the category and size of the object.

[0144] In one example, after predicting the position and speed of the object at the current moment, the predicted detection result may be corrected according to the detection results of other sensors for the object at the current moment.

[0145] It should be noted that Fig.11 For other detailed descriptions of steps S1010 to S1030 and S320 to S340 shown in Fig.10 The steps S1010 to S1030 and S320 to S340 shown are not described in detail here.

[0146] In an embodiment of the present application, the position and speed of the object at the current moment are predicted by a linear motion model, roadside detection results at historical moments, and time intervals, and then the roadside detection results at the current moment are generated, thereby ensuring the accuracy of the roadside detection results generated at the current moment.

[0147] The present application embodiment provides another target detection method, which can be applied to Figure 1 or Figure 2 In the implementation environment shown, the method can be executed by a vehicle or a cloud server, or by both a vehicle and a cloud server. The method is described by taking the vehicle as an example. Figures 3 to 11 Based on the information shown in , S310 is expanded to S1210 to S1240. Wherein, the roadside sensor includes a roadside camera, and the vehicle sensor includes a vehicle-mounted camera, such as Fig.12 As shown, S1210 to S1240 are described in detail as follows:

[0148] S1210. Obtain a depth probability distribution map of the roadside camera based on image data collected by the roadside camera, where the depth probability distribution map is used to represent the probability distribution of depth information corresponding to each pixel point.

[0149] In an embodiment of the present application, a roadside camera and a vehicle-mounted camera collect image data from different perspectives of the same scene. The image data collected by the roadside camera can be used to extract features through a convolutional neural network to obtain a depth probability distribution map of the roadside camera, wherein the convolutional neural network can be any network that can output a depth probability distribution, such as a network based on a convolutional neural network (CNN) and a conditional random field (CRF).

[0150] The depth probability distribution map is used to characterize the probability distribution of the depth information corresponding to each pixel. For example, if an element is [0.1, 0.2, 0.3, 0.4], then the probability that the depth value corresponding to the pixel is between [0, 1, 2, 3] meters is 0.1, 0.2, 0.3, and 0.4 respectively.

[0151] S1220. Obtain a target depth probability distribution map corresponding to each pixel point based on a fusion result of the depth probability distribution map of the roadside camera and the depth probability distribution map of the vehicle-mounted camera.

[0152] It can be understood that the depth probability distribution map of the vehicle-mounted camera can also be obtained through S1210. In order to improve the accuracy of depth estimation, in the embodiment of the present application, it is necessary to fuse the depth probability distribution map of the roadside camera and the depth probability distribution map of the vehicle-mounted camera, that is, to combine the depth information shared by the two cameras to obtain a more accurate and robust depth probability distribution map, that is, to obtain the target depth probability distribution map corresponding to each camera; wherein, the fusion can be achieved by methods such as weighted average or maximum a posteriori (MAP).

[0153] S1230 , converting the target depth probability distribution map corresponding to each pixel into a voxel feature of each pixel, where the voxel feature represents the position and feature of the pixel in three-dimensional space.

[0154] In an embodiment of the present application, it is necessary to convert the target depth probability distribution map corresponding to each pixel point into a voxel feature of each pixel point, that is, to map the depth probability distribution of each camera into a voxel network in a three-dimensional space to obtain a voxel feature, which represents the position and characteristics of the pixel point in the three-dimensional space, so as to facilitate subsequent three-dimensional target detection.

[0155] In one example, converting the target depth probability distribution map into a voxel feature can be, for each pixel p, finding the depth value d corresponding to the maximum probability in its depth probability distribution map; converting the two-dimensional coordinates (u, v) and depth value d of pixel p into three-dimensional coordinates (x, y, z) according to the intrinsic and extrinsic parameters of the camera; mapping the three-dimensional coordinates (x, y, z) to a voxel unit v according to the resolution and range of the voxel feature, and accumulating the attributes of the pixel (such as color, normal vector, etc.) into the voxel unit; repeating the above steps until all pixels are converted into voxel features.

[0156] S1240: Perform target detection according to voxel features to obtain roadside detection results, and obtain vehicle detection results obtained by vehicle sensor detection.

[0157] In an embodiment of the present application, target detection is performed based on voxel features to obtain roadside detection results, such as converting voxel features into feature maps through a feature encoding network, and performing feature extraction on the feature maps to obtain feature vectors, detecting objects in voxel features based on the feature vectors, and obtaining detection information of the objects, and detecting object trajectories based on a Kalman filter and a Hungarian algorithm to obtain roadside detection results.

[0158] It should be noted that Fig.12 For more detailed description of steps S320 to S340 shown in Figure 3 Steps S320 to S340 are shown and will not be described in detail here.

[0159] In an embodiment of the present application, by combining the depth probability distribution map of the roadside camera and the depth probability distribution of the vehicle-mounted camera, a more accurate and robust target depth probability distribution map is obtained, and the target depth probability distribution map is converted into voxel features to facilitate subsequent three-dimensional target detection, making the roadside detection results more accurate.

[0160] The present application embodiment provides another target detection method, which can be applied to Figure 1 or Figure 2 In the implementation environment shown, the method can be executed by the vehicle or the cloud server, or by the vehicle and the cloud server together. The method is described by taking the vehicle as an example. Fig.13 As shown in Fig.12 Based on the results shown in Fig.12 The S1220 shown is expanded into steps S1310 to S1330. Steps S1310 to S1330 are described in detail as follows:

[0161] S1310. According to the depth probability distribution map of the roadside camera, obtain target depth information whose uncertainty of target pixel points is lower than a preset threshold.

[0162] In the embodiment of the present application, for each pixel, it is necessary to first calculate the uncertainty value of the depth information of each pixel. The uncertainty value of the depth information represents the degree of dispersion of the depth probability distribution corresponding to the pixel. The lower the uncertainty value, the more concentrated the depth probability distribution is, and the more reliable the depth corresponding to the pixel is. The depth uncertainty value of each pixel can be calculated by information entropy, variance, etc.

[0163] In order to further enhance the accuracy of depth estimation, for a target pixel point, which may be any pixel point, target depth information of the target pixel point whose uncertainty is lower than a preset threshold is obtained, and the preset threshold can be flexibly adjusted according to actual conditions.

[0164] S1320, obtaining a pixel point corresponding to the target pixel point in the depth probability distribution map of the vehicle-mounted camera.

[0165] S1330 , fusing the target depth information of the target pixel and the depth information of the reference pixel whose uncertainty is lower than a preset threshold, to obtain a target depth probability distribution map corresponding to the target pixel.

[0166] For the target pixel point, it is necessary to find a reference pixel point corresponding to the target pixel point in the depth probability distribution map of the vehicle-mounted camera.

[0167] The depth information of the reference pixel whose uncertainty is lower than a preset threshold is added to the target depth information of the target pixel to increase the depth information corresponding to the target pixel, and obtain the target depth probability distribution of the target pixel. Repeat S1310 to S1330 to obtain a depth probability distribution map of each pixel, thereby realizing the use of depth information at the same position under different viewing angles to enhance the accuracy of depth estimation.

[0168] It should be noted that Fig.13 For a detailed description of S1210, S1230~S1240, S320~S340 shown in Fig.12 S1210, S1230~S1240, S320~S340 shown in are not repeated here.

[0169] In an embodiment of the present application, target depth information of a pixel whose uncertainty is lower than a preset threshold is obtained and fused with the depth information of a corresponding reference pixel whose uncertainty is lower than a preset threshold to increase the depth information corresponding to the pixel, thereby enhancing the accuracy of depth estimation by utilizing the depth information at the same position under different viewing angles.

[0170] The present application embodiment provides another target detection method, which can be applied to Figure 1 or Figure 2 In the implementation environment shown, the method can be executed by the vehicle or the cloud server, or can be executed by the vehicle and the cloud server together, such as Fig.14 As shown in Fig.12 Based on the Fig.12 Take S1240 in the example of being expanded into steps S1410 to S1440. Steps S1410 to S1440 are described in detail as follows:

[0171] S1410: Obtain two corresponding pixel points in the image data of the roadside camera and the image data of the vehicle-mounted camera.

[0172] S1420 , calculating voxel feature consistency according to feature similarity of voxel features corresponding to two pixel points.

[0173] In an embodiment of the present application, it is necessary to calculate the voxel feature consistency of a pixel point under different viewing angles. For two corresponding pixels, the voxel features corresponding to the two pixels can be compared, and the feature similarity of the two pixels at different depth positions can be calculated to obtain the voxel feature consistency. The higher the consistency, the more similar the voxel features of the two pixels are, that is, the more consistent the depth distribution of the two pixels is, so as to realize the possibility of constraining the depth by using the voxel features of the same position under different viewing angles.

[0174] S1430 , generating a coordinated depth distribution for each pixel point according to the voxel feature consistency, and estimating optimal depth information corresponding to each pixel point according to the coordinated depth distribution.

[0175] For each pixel, the target depth probability distribution of the pixel is updated according to the consistency of its voxel features, so that the voxel features corresponding to the depth positions with higher consistency have higher probabilities, and the voxel features corresponding to the depth positions with lower consistency have lower depth probabilities, thereby obtaining a coordinated depth distribution, which can optimize the depth estimation results by utilizing the consistency of voxel features at the same position under different perspectives.

[0176] In an embodiment of the present application, after the collaborative depth distribution of each pixel is obtained, the optimal depth information corresponding to each pixel can be estimated according to the collaborative depth distribution, such as the optimal depth information being the depth value corresponding to the highest probability.

[0177] S1440 , converting the optimal depth information into target voxel features, and performing target detection according to the target voxel features to obtain a roadside detection result.

[0178] By converting the optimal depth information into voxel features again, three-dimensional target detection can be performed to obtain roadside detection results.

[0179] If the vehicle sensor is a vehicle-mounted camera, the process of obtaining the vehicle detection result corresponding to the vehicle-mounted camera is similar to the roadside detection result corresponding to the roadside camera. For details, see Figure 12 to Figure 14 .

[0180] It should be noted that Fig.14 For more detailed information about S1210~S1230, S320~S340, see Fig.12 S1210~S1230, S320~S340 shown are not repeated here.

[0181] In an embodiment of the present application, the voxel feature consistency is calculated by the feature similarity of two corresponding pixel points, and a collaborative depth distribution is generated based on the voxel feature consistency to optimize the depth estimation result by utilizing the voxel feature consistency at the same position under different viewing angles, and finally converted into voxel features, making target detection more accurate and reliable.

[0182] The present application embodiment provides another target detection method, which can be applied to Figure 1 or Figure 2 In the implementation environment shown, the method can be executed by a vehicle or a cloud server, or by both a vehicle and a cloud server. The method is described by taking the vehicle as an example. Figures 3 to 11 Based on the example shown in , S310 is expanded to S1510 to S1530. Among them, the roadside sensor includes a roadside laser radar, such as Fig.15 As shown, S1510~S1530 are introduced in detail as follows:

[0183] S1510: Convert the point cloud data collected by the roadside laser radar into a feature map, and perform feature extraction on the feature map to obtain a feature vector.

[0184] It is understandable that the roadside laser radar collects point cloud data, which is a collection of points in three-dimensional space, each point having attributes such as x, y, z coordinates and reflection intensity. In the embodiment of the present application, the point cloud data can be converted into a feature map through a feature encoding network, and the feature map can be extracted through a convolutional neural network, such as compressing the feature map into a low-dimensional vector through multiple convolutional layers and pooling layers to extract high-level semantic information.

[0185] S1520. Detect objects in the feature map according to the feature vector, and obtain detection information of the objects.

[0186] In an embodiment of the present application, the low-dimensional vector is restored into a high-resolution feature map through multiple layers of deconvolution layers and upsampling layers, and the spatial information is restored. The object of each frame in the feature map obtained by detection is output through a fully connected layer, and the detection information of the object includes the category of the object, the position and size of the three-dimensional bounding box, the heading angle and the confidence score.

[0187] S1530: Detect the trajectory of the object according to the detection information of the object to obtain a roadside detection result.

[0188] In an embodiment of the present application, a roadside detection result can be obtained by detecting the motion trajectory of the object based on a Kalman filter, a Hungarian algorithm and the detection information of the object, wherein the roadside detection result includes the detection information of the object and the motion trajectory of the object.

[0189] For example, a Kalman filter is initialized for each object to estimate its state vector, which includes information such as the object's position, size, heading angle, and velocity. The Kalman filter has two main steps: prediction and update. The prediction step predicts the state of the current frame based on the state and dynamic model of the previous frame. The update step calculates the Kalman gain based on the observed data and predicted state of the current frame, and uses it to correct the predicted state to obtain the optimal state estimate.

[0190] A bipartite graph is built for each frame, where nodes are divided into two categories: detection results and following results. The detection result is the detection information of the object, and the following result is the state vector output by the Kalman filter. The weight of the edge is calculated based on the similarity between the two nodes, which can be determined by factors such as position, size, heading angle, category, etc. The goal is to find a maximum weight match, that is, a match with the largest sum of edge weights, which can be achieved using the Hungarian algorithm.

[0191] Update the following result and Kalman filter according to the matching result. If a detection result matches a following result, update the following result and Kalman filter according to the detection result, and keep the following object ID unchanged. If a detection result does not match any following result, create a new following result and Kalman filter, and assign a new following object ID. If a following result does not match any detection result, it is marked as lost and deleted if it is not matched again within a certain period of time. The above process completes the multi-target tracking process within one frame.

[0192] It should be noted that Fig.15 For more detailed description of steps S320 to S340 shown in Figure 3 Steps S320 to S340 are shown and will not be described in detail here.

[0193] In an embodiment of the present application, point cloud data is converted into a feature map and feature extraction is performed to obtain a feature vector, so as to detect objects in the feature map and the detection information of the objects, and then the trajectory of the objects is detected, so that the roadside detection results are more comprehensive and reliable.

[0194] For ease of understanding, the present application embodiment also provides a target detection method, which is optimized for the three-dimensional target detection task in the process of autonomous driving, such as Fig.16 As shown, the three steps of single sensor detection, multi-sensor frame synchronization and multi-sensor detection result fusion are mainly used to provide autonomous driving vehicles with more comprehensive environmental perception capabilities, more accurate three-dimensional target detection, a wider perception range and faster perception result fusion; among them, the vehicle is deployed with on-board cameras and on-board lidars, and the roadside units are deployed with roadside cameras and roadside lidars to help the vehicle expand its perception range.

[0195] For single lidar detection, three-dimensional target detection is performed on the identified three-dimensional point cloud through a single lidar. For example, objects are independently extracted from the point clouds collected by the lidar on the roadside unit and the vehicle based on the 3D object detector. For each frame, the 3D object detector outputs a tuple, which includes the category of each object in the frame, the position and size of the three-dimensional bounding box, the heading angle and the confidence score. The tuple output by the 3D object detector is input into a 3D multi-target follower, and each object in the frame is followed by the 3D multi-target follower. The representation of each object trajectory is to add the speed and following ID to the object detection result.

[0196] For target detection with a single camera, the problem is to perform three-dimensional target detection when there is only two-dimensional information. The embodiment of the present application proposes to use a collaborative depth estimation method, but for a monocular system, it is also difficult to obtain the depth (distance) information of the scene and objects; while a binocular system can solve the problem of obtaining depth information, but the amount of calculation is very large. Since cameras are deployed on the roadside unit and the vehicle respectively; and the correct depth of the pixel point taken by one camera should have consistency in spatial characteristics with the corresponding pixel point taken by another camera from multiple perspectives, the depth information with low uncertainty estimated by the two cameras can be shared with each other. The uncertainty here refers to the uncertainty of the depth probability distribution estimated for a certain pixel point of the two-dimensional image. The lower the uncertainty, the more reliable the depth estimated by this pixel point is, and the more worthy of reference for other cameras; then, after converting this two-dimensional image into voxel features through the depth probability distribution of each pixel point, the voxel feature consistency of all pixels corresponding to a certain pixel point of the two-dimensional image under different perspectives is calculated, and the coordinated depth distribution of the pixel point is obtained. The higher the consistency, the greater the depth probability of the pixel point; then, after estimating the depth, converting it back to voxel features again, the three-dimensional target detection and tracking performed by the previous lidar can be performed.

[0197] It should be noted that in the embodiments of the present application, after the roadside unit and the vehicle each complete their own three-dimensional target detection sensor detection results locally, what is transmitted is only the sensor detection results, rather than the three-dimensional point cloud with a very large amount of data, which achieves faster perception result fusion.

[0198] After the roadside unit obtains the lidar detection results and the camera detection results, the detection results need to be sent to the vehicle for fusion. However, before fusion, another problem that needs to be solved is that the frames obtained by the roadside unit may not always be aligned with the vehicle frames in time, that is, when the vehicle wants to fuse multiple frames later, the sampling time of these frames is inconsistent. This is because the standard sampling rate of the lidar is 10Hz. When the roadside unit and the vehicle sampling are not synchronized, time inconsistency will occur. In addition, even if the two frames are aligned when sampling, data packet loss may occur when the roadside unit transmits data to the vehicle; therefore, the embodiment of the present application needs to achieve multi-sensor frame synchronization, taking the vehicle's lidar detection result as the benchmark, and sending the roadside lidar detection result to the vehicle each time the roadside unit performs lidar sampling, so that the vehicle always has the latest lidar detection result of the roadside unit. After the vehicle performs lidar sampling, the linear motion model is used to predict the lidar detection result of the roadside unit at this moment based on the stored last lidar detection result of the roadside unit; for the camera detection result, since the camera shooting also has a frame rate limitation, a similar approach is adopted to store the latest camera detection result and use the linear motion model for prediction, so as to obtain the synchronized detection results of the vehicle and roadside unit's respective lidars and cameras.

[0199] The vehicle fuses the detection results of multiple sensors and constructs an object distribution map based on the detection results of each sensor. The nodes of the map correspond to each detected object, and the edges represent the relationship between objects, such as the object distribution map corresponding to the vehicle laser radar detection results, the object distribution map corresponding to the vehicle camera detection results, the object distribution map corresponding to the roadside laser radar detection results, and the object distribution map corresponding to the roadside camera detection results. After the object distribution map is constructed, the object distribution map corresponding to the vehicle laser radar detection results is used as the reference map, and the graph is matched with the other three object distribution maps respectively. The node matching is determined by calculating the matching degree of the categories of the two detected objects and the position and size of their respective bounding boxes, while the edge matching is determined by matching the category pairs of the two pairs of detected objects and the matching degree of the distance and angle between the two pairs of detected objects.

[0200] After the base image is matched with a certain image, the common visible objects between the two images are obtained. Object alignment is performed based on the common visible objects, that is, two small point sets are formed by the bounding box vertices of the common visible objects in the two images. For example, the bounding box vertices of the common visible objects in the base image form a small point set, and the bounding box vertices of the common visible objects in a distribution image of other objects form another small point set. The two small point sets are used to align the detection results corresponding to the two base images and the distribution of the other objects. The two small point sets can be aligned based on the linear least squares problem using singular value decomposition (SVD). After the base image is aligned with the other two images, the 3D target detection results of all the vehicles that have been finally fused are obtained.

[0201] According to the technical solution provided by the embodiment of the present application, objects outside the vehicle's line of sight can be detected from the point cloud data of the roadside unit, and the vehicle's field of view is well supplemented by integrating the scene perception results of the roadside unit into the vehicle's view in real time, thereby increasing the vehicle's perception range; this real-time perception fusion capability greatly improves the vehicle's perception capability and perception accuracy, and greatly improves the accuracy and reliability of many autonomous driving tasks (such as path planning, positioning and navigation); in addition, the influence of the time error between subsequent frames to be fused is reduced through multi-sensor inter-frame synchronization, making the fusion result more accurate and the three-dimensional target detection more precise and effective; only the sensor detection result is transmitted, rather than the three-dimensional point cloud and image data with a very large amount of data, so that the intermediate transmission delay is reduced and a faster perception result fusion is achieved; the camera's ability to recognize color and texture is used to enhance the object detection capability of the lidar. Specifically, when using lidar for 3D object detection, for distant objects or small objects, the number of reflection points will be very small. At this time, the camera can recognize them instead, and then the vehicle can also perceive these distant objects or small objects through fusion.

[0202] The device embodiment of the present application is introduced, which can be used to execute the target detection method in the above embodiment of the present application. For details not disclosed in the device embodiment of the present application, please refer to the above embodiment of the target detection method of the present application.

[0203] The present application embodiment provides a target detection device, such as Fig.17 As shown, the device comprises:

[0204] An acquisition module 1710 is used to acquire a vehicle detection result obtained by a vehicle sensor and a roadside detection result obtained by a roadside sensor;

[0205] A construction module 1720, configured to construct an object distribution map corresponding to the vehicle detection result and an object distribution map corresponding to the roadside detection result according to the objects and object relationships respectively included in the vehicle detection result and the roadside detection result;

[0206] A matching module 1730 is used to match the object distribution graphs to obtain matching results, and determine common visible objects between the object distribution graphs according to the matching results;

[0207] The fusion module 1740 is used to fuse the vehicle detection result and the roadside detection result according to the common visible object to obtain the target detection result from the vehicle perspective.

[0208] In one embodiment of the present application, based on the aforementioned scheme, the vehicle detection result includes vehicle detection results respectively detected by multiple sensors; the matching module 1730 is further used to select a target sensor from the multiple sensors, and use the object distribution map corresponding to the vehicle detection result detected by the target sensor as a reference map; match the reference map with other object distribution maps, and obtain the degree of matching between the reference map and the other object distribution maps, wherein the other object distribution maps include the object distribution map corresponding to the roadside detection result detected by the roadside sensor, and the object distribution map corresponding to the vehicle detection result detected by other sensors among the multiple sensors except the target sensor; determine the matching result between the reference map and the other object distribution maps according to the degree of matching.

[0209] In one embodiment of the present application, based on the aforementioned scheme, the object distribution graph is constructed with the objects as nodes and the object relationships as edges; the matching module 1730 is further used to calculate at least one of the category similarity of node pairs formed by the nodes in the reference graph and the nodes in the other object distribution graphs, the bounding box position similarity of the node pairs, and the bounding box size similarity of the node pairs to generate the degree of matching between the nodes; calculate at least one of the category similarity, distance similarity, and angle similarity between the node pairs in the reference graph and the node pairs in the other object distribution graphs to generate the degree of matching between the edges.

[0210] In one embodiment of the present application, based on the aforementioned scheme, the fusion module 1740 is further used to extract, for each common visible object, the vertices of the bounding box where the common visible object is located from the reference map and the other object distribution map, respectively, and generate two point sets based on the extracted vertices; calculate the transformation matrix between the two point sets, and align the two point sets according to the transformation matrix to fuse the vehicle detection result corresponding to the reference map with the detection result corresponding to the other object distribution map.

[0211] In one embodiment of the present application, based on the aforementioned scheme, the fusion module 1740 is further used to calculate the covariance matrix between the two point sets, and perform singular value decomposition on the covariance matrix; calculate the rotation matrix based on the left and right singular vector matrices obtained by the singular value decomposition, and calculate the translation vector based on the rotation matrix and the vertices in the two point sets; transform one point set to another point set based on the rotation matrix and the translation vector to align the two point sets.

[0212] In one embodiment of the present application, based on the aforementioned scheme, the device also includes a constraint module, which is used to obtain prior map information of the area where the vehicle is located, the prior map information including at least one of the structure of the scene, the road network and the building layout; according to the prior map information, the position and motion range of the objects respectively included in the vehicle detection result and the roadside detection result are constrained to filter out the detection results that are inconsistent with the prior map information.

[0213] In one embodiment of the present application, based on the aforementioned scheme, the vehicle detection result includes the vehicle detection result at the current moment obtained by respectively detecting multiple different types of sensors; the acquisition module 1710 is further used to obtain the roadside detection results at historical moments obtained by detecting the roadside sensor, and determine the sensor type corresponding to the roadside detection results at the historical moments; select a reference detection result from the vehicle detection results at the current moment corresponding to the multiple different types of sensors according to the determined sensor type; and predict the roadside detection result at the current moment based on the roadside detection results at the historical moments and the reference detection result.

[0214] In one embodiment of the present application, based on the aforementioned scheme, the acquisition module 1710 is further used to obtain the time interval corresponding to the reference detection result and the roadside detection result at the historical moment; predict the position and speed of the object contained in the roadside detection result at the historical moment at the current moment according to the linear motion model, the roadside detection result at the historical moment and the time interval; generate the roadside detection result at the current moment according to the position and speed of the object at the current moment.

[0215] In one embodiment of the present application, based on the aforementioned scheme, the roadside sensor includes a roadside camera, and the vehicle sensor includes a vehicle-mounted camera; the acquisition module 1710 is further used to obtain a depth probability distribution map of the roadside camera based on the image data collected by the roadside camera, and the depth probability distribution map is used to characterize the probability distribution of the depth information corresponding to each pixel; based on the fusion result of the depth probability distribution map of the roadside camera and the depth probability distribution map of the vehicle-mounted camera, a target depth probability distribution map corresponding to each pixel is obtained; the target depth probability distribution map corresponding to each pixel is converted into a voxel feature of each pixel, and the voxel feature characterizes the position and characteristics of the pixel in three-dimensional space; target detection is performed according to the voxel feature to obtain a roadside detection result.

[0216] In one embodiment of the present application, based on the aforementioned scheme, the acquisition module 1710 is further used to obtain target depth information of the target pixel whose uncertainty is lower than a preset threshold according to the depth probability distribution map of the roadside camera; obtain the reference pixel corresponding to the target pixel in the depth probability distribution map of the vehicle-mounted camera; and fuse the target depth information of the target pixel and the depth information of the reference pixel whose uncertainty is lower than a preset threshold to obtain a target depth probability distribution map corresponding to the target pixel.

[0217] In one embodiment of the present application, based on the aforementioned scheme, the acquisition module 1710 is further used to acquire two corresponding pixel points in the image data of the roadside camera and the image data of the vehicle-mounted camera; calculate the voxel feature consistency based on the feature similarity of the voxel features corresponding to the two pixel points respectively; generate a collaborative depth distribution for each pixel point based on the voxel feature consistency, and estimate the optimal depth information corresponding to each pixel point based on the collaborative depth distribution; convert the optimal depth information into a target voxel feature, and perform target detection based on the target voxel feature to obtain the roadside detection result.

[0218] In one embodiment of the present application, based on the aforementioned scheme, the roadside sensor includes a roadside laser radar; the acquisition module 1710 is further used to convert the point cloud data collected by the roadside laser radar into a feature map, and perform feature extraction on the feature map to obtain a feature vector; detect the object in the feature map according to the feature vector, and obtain the detection information of the object; detect the trajectory of the object according to the detection information of the object to obtain the roadside detection result.

[0219] It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here.

[0220] An embodiment of the present application also provides an electronic device, comprising one or more processors and a storage device, wherein the storage device is used to store one or more computer programs, and when the one or more computer programs are executed by one or more processors, the electronic device implements the target detection method as described above.

[0221] Fig.18 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0222] It should be noted that Fig.18 The computer system 1800 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0223] like Fig.18 As shown, the computer system 1800 includes a processor (Central Processing Unit, CPU) 1801, which can perform various appropriate actions and processes according to the program stored in the read-only memory (Read-Only Memory, ROM) 1802 or the program loaded from the storage part 1808 to the random access memory (Random Access Memory, RAM) 1803, such as executing the method in the above embodiment. In RAM 1803, various programs and data required for system operation are also stored. CPU 1801, ROM 1802 and RAM 1803 are connected to each other through bus 1804. Input / output (Input / Output, I / O) interface 1805 is also connected to bus 1804.

[0224] In some embodiments, the following components are connected to the I / O interface 1805: an input section 1806 including a keyboard, a mouse, etc.; an output section 1807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1808 including a hard disk, etc.; and a communication section 1809 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1809 performs communication processing via a network such as the Internet. A drive 1810 is also connected to the I / O interface 1805 as needed. A removable medium 1811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1810 as needed, so that a computer program read therefrom is installed into the storage section 1808 as needed.

[0225] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1809, and / or installed from a removable medium 1811. When the computer program is executed by a processor (CPU) 1801, various functions defined in the system of the present application are executed.

[0226] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (Erasable Programmable Read Only Memory), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0227] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and a computer program.

[0228] The units or modules involved in the embodiments described in this application may be implemented by software or hardware, and the units or modules described may also be set in a processor. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves.

[0229] Another aspect of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the method described above is implemented. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0230] Another aspect of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the electronic device executes the method described above in each of the above embodiments.

[0231] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0232] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0233] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person of ordinary skill in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A target detection method, characterized in that: include: Obtain vehicle detection results obtained by vehicle sensors, and obtain roadside detection results obtained by roadside sensors; Constructing an object distribution map corresponding to the vehicle detection result and an object distribution map corresponding to the roadside detection result according to the objects and object relationships respectively included in the vehicle detection result and the roadside detection result; Matching the object distribution graphs to obtain a matching result, and determining common visible objects between the object distribution graphs according to the matching result; The vehicle detection result and the roadside detection result are fused according to the common visible object to obtain a target detection result from a vehicle perspective.

2. The method according to claim 1, characterized in that The vehicle detection result includes vehicle detection results respectively detected by a plurality of sensors; the matching result obtained by matching the object distribution map includes: Selecting a target sensor from the multiple sensors, and using an object distribution map corresponding to a vehicle detection result obtained by the target sensor as a reference map; Matching the reference map with other object distribution maps, and obtaining a matching degree between the reference map and the other object distribution maps, wherein the other object distribution maps include an object distribution map corresponding to a roadside detection result obtained by the roadside sensor, and an object distribution map corresponding to a vehicle detection result obtained by other sensors among the multiple sensors except the target sensor; According to the matching degree, a matching result between the reference image and the other object distribution images is determined.

3. The method according to claim 2, characterized in that The object distribution graph is constructed with the objects as nodes and the object relationships as edges; and obtaining the matching degree between the reference graph and other object distribution graphs includes: Calculating at least one of a category similarity of a node pair formed by a node in the reference graph and a node in the other object distribution graph, a bounding box position similarity of the node pair, and a bounding box size similarity of the node pair, so as to generate a matching degree between the nodes; At least one of category similarity, distance similarity and angle similarity between the node pairs in the reference graph and the node pairs in the other object distribution graph is calculated to generate a matching degree between the edges.

4. The method according to claim 2, characterized in that: The fusing the vehicle detection result and the roadside detection result according to the common visible object comprises: For each common visible object, extracting vertices of a bounding box where the common visible object is located from the reference map and the other object distribution map respectively, and generating two point sets according to the extracted vertices; A transformation matrix between the two point sets is calculated, and the two point sets are aligned according to the transformation matrix to fuse the vehicle detection result corresponding to the reference map with the detection result corresponding to the other object distribution map.

5. The method according to claim 4, characterized in that The calculating a transformation matrix between the two point sets and aligning the two point sets according to the transformation matrix comprises: Calculating a covariance matrix between the two point sets, and performing singular value decomposition on the covariance matrix; Calculating a rotation matrix according to the left and right singular vector matrices obtained by singular value decomposition, and calculating a translation vector according to the rotation matrix and the vertices in the two point sets; One point set is transformed to another point set according to the rotation matrix and the translation vector to align the two point sets.

6. The method according to claim 1, characterized in that Before constructing the object distribution map corresponding to the vehicle detection result and the object distribution map corresponding to the roadside detection result according to the objects and object relationships respectively included in the vehicle detection result and the roadside detection result, the method further includes: Acquire prior map information of the area where the vehicle is located, the prior map information including at least one of a structure of a scene, a road network, and a building layout; The positions and movement ranges of the objects respectively included in the vehicle detection result and the roadside detection result are constrained according to the prior map information to filter out the detection results inconsistent with the prior map information.

7. The method according to claim 1, characterized in that The vehicle detection result includes vehicle detection results at the current moment obtained by detection by multiple different types of sensors respectively; The obtaining of the roadside detection result obtained by the roadside sensor comprises: Obtaining a roadside detection result at a historical moment detected by the roadside sensor, and determining a sensor type corresponding to the roadside detection result at the historical moment; Selecting a reference detection result from the vehicle detection results at the current moment corresponding to the multiple different types of sensors according to the determined sensor type; The roadside detection result at the current moment is predicted based on the roadside detection result at the historical moment and the reference detection result.

8. The method according to claim 7, characterized in that The predicting the roadside detection result at the current moment according to the roadside detection result at the historical moment and the reference detection result includes: Obtaining the time interval corresponding to the reference detection result and the roadside detection result at the historical moment; Predicting the position and speed of the object included in the roadside detection results at the historical moment at the current moment according to the linear motion model, the roadside detection results at the historical moment and the time interval; The roadside detection result at the current moment is generated according to the position and speed of the object at the current moment.

9. The method according to any one of claims 1 to 8, characterized in that: The roadside sensor includes a roadside camera, and the vehicle sensor includes a vehicle-mounted camera; The obtaining of the roadside detection result obtained by the roadside sensor comprises: Acquire a depth probability distribution map of the roadside camera according to the image data collected by the roadside camera, wherein the depth probability distribution map is used to represent the probability distribution of the depth information corresponding to each pixel point; Obtaining a target depth probability distribution map corresponding to each pixel point according to a fusion result of the depth probability distribution map of the roadside camera and the depth probability distribution map of the vehicle-mounted camera; Converting the target depth probability distribution map corresponding to each pixel into a voxel feature of each pixel, wherein the voxel feature represents the position and feature of the pixel in three-dimensional space; Target detection is performed according to the voxel features to obtain a roadside detection result.

10. The method according to claim 9, characterized in that The method of obtaining a target depth probability distribution map corresponding to each pixel point based on a fusion result of the depth probability distribution map of the roadside camera and the depth probability distribution map of the vehicle-mounted camera includes: According to the depth probability distribution map of the roadside camera, obtaining target depth information of target pixel points whose uncertainty is lower than a preset threshold; Obtain a reference pixel point corresponding to the target pixel point in the depth probability distribution map of the vehicle-mounted camera; The target depth information of the target pixel and the depth information of the reference pixel whose uncertainty is lower than a preset threshold are fused to obtain a target depth probability distribution map corresponding to the target pixel.

11. The method according to claim 9, characterized in that The performing target detection according to the voxel features to obtain a roadside detection result includes: Acquire two corresponding pixel points in the image data of the roadside camera and the image data of the vehicle-mounted camera; Calculating voxel feature consistency according to feature similarities of voxel features corresponding to the two pixel points; Generating a coordinated depth distribution for each pixel according to the voxel feature consistency, and estimating optimal depth information corresponding to each pixel according to the coordinated depth distribution; The optimal depth information is converted into target voxel features, and target detection is performed according to the target voxel features to obtain the roadside detection result.

12. The method according to any one of claims 1 to 8, characterized in that: The roadside sensor includes a roadside laser radar; the step of obtaining a roadside detection result obtained by the roadside sensor includes: Converting the point cloud data collected by the roadside laser radar into a feature map, and performing feature extraction on the feature map to obtain a feature vector; Detecting an object in the feature map according to the feature vector, and acquiring detection information of the object; The trajectory of the object is detected according to the detection information of the object to obtain the roadside detection result.

13. A target detection device, characterized in that: include: An acquisition module is used to acquire vehicle detection results obtained by vehicle sensors and roadside detection results obtained by roadside sensors; A construction module, used to construct an object distribution map corresponding to the vehicle detection result and an object distribution map corresponding to the roadside detection result according to the objects and object relationships respectively included in the vehicle detection result and the roadside detection result; A matching module, configured to match the object distribution graphs to obtain matching results, and determine common visible objects between the object distribution graphs according to the matching results; A fusion module is used to fuse the vehicle detection result and the roadside detection result according to the common visible object to obtain the target detection result from the vehicle perspective.

14. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to execute the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of an electronic device, the electronic device executes the method according to any one of claims 1 to 12.

Images