Training method of fusion perception model, fusion perception method, equipment and medium

Abstract

The invention provides a training method of a fusion perception model, a fusion perception method, equipment and a medium, and relates to the technical field of image processing, and the method comprises the steps: obtaining a preset roadside scene data set; according to the historical roadside images of the at least two directions of each intersection and the camera parameters of the at least two directions, an initial fusion perception model is adopted for perception, and a target detection three-dimensional bounding box of the at least one preset traffic participation object is obtained; and according to the real three-dimensional bounding box of the at least one preset traffic participation object and the target detection three-dimensional bounding box of each preset traffic participation object, training the initial fusion perception model to obtain a target fusion perception model. According to the method, the waste of computing resources is reduced, and meanwhile, the accuracy of identifying the traffic participating objects is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a training method for a fusion perception model, a fusion perception method, a device, and a medium. Background Art

[0002] With the rapid development of artificial intelligence (AI), image-based object recognition has gained widespread application in fields such as autonomous driving, intelligent transportation, and security monitoring. In this area, accurate and reliable object recognition is crucial for ensuring traffic safety and improving traffic efficiency. Algorithms based on pure visual perception, which construct bird's-eye-view scene representations from two-dimensional camera images, have become a research hotspot in the field of object recognition.

[0003] Currently, this approach relies primarily on estimating the depth of objects in an image, projecting 3D point clouds onto a 2D image plane, and utilizing dense attention mechanisms for feature extraction and fusion. For example, depth estimation methods provide scene depth information, while projection-based methods associate 3D point cloud data with 2D images. Attention mechanisms effectively model long-range dependencies in images, thereby improving object recognition accuracy.

[0004] However, end-to-end object recognition methods based on neural network multi-view bird's-eye view feature fusion are highly sensitive to changes in camera configurations, making it difficult to transfer perception models between different platforms and significantly degrading model performance. Consequently, in practical applications, separate models must be trained for each specific intersection, consuming significant computing resources and hindering widespread application and deployment. Summary of the Invention

[0005] The purpose of this application is to address the deficiencies in the above-mentioned existing technologies and provide a training method, a fusion perception method, a device and a medium for a fusion perception model, so as to reduce the waste of computing resources and improve the accuracy of identifying traffic participants.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of the present application are as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for training a fusion perception model, the method comprising:

[0008] Acquiring a preset roadside scene dataset, wherein the preset roadside scene dataset includes: sample roadside scene images of at least one intersection, and the sample roadside scene images of each intersection include: historical roadside images in at least two directions of each intersection, camera parameters in the at least two directions, and a real three-dimensional bounding box of at least one preset traffic participant object at each intersection;

[0009] Based on the historical roadside images of the at least two directions of each intersection and the camera parameters of the at least two directions, an initial fusion perception model is used to perform perception to obtain a target detection three-dimensional bounding box of the at least one preset traffic participant;

[0010] The initial fusion perception model is trained according to the real three-dimensional bounding box of the at least one preset traffic participant object and the target detection three-dimensional bounding boxes of each preset traffic participant object to obtain a target fusion perception model.

[0011] Optionally, the initial fusion perception model includes: an initial recognition model and an initial fusion model; the initial fusion perception model is used for perception based on the historical roadside images in the at least two directions of each intersection and the camera parameters in the at least two directions to obtain a target detection three-dimensional bounding box of the at least one preset traffic participant, including:

[0012] Based on the historical roadside images in the at least two directions of each intersection and the camera parameters in the at least two directions, the initial recognition model is used to perform recognition to obtain a plurality of detected three-dimensional bounding boxes and descriptor information of the plurality of detected three-dimensional bounding boxes;

[0013] According to the descriptor information of the multiple detected three-dimensional bounding boxes, the multiple detected three-dimensional bounding boxes are fused using the initial fusion model to obtain a target three-dimensional detection box of the at least one preset traffic participant object.

[0014] Optionally, the training of the initial fusion perception model based on the real three-dimensional bounding box of the at least one preset traffic participant object and the target detection three-dimensional bounding boxes of each preset traffic participant object to obtain the target fusion perception model includes:

[0015] The initial recognition model and the initial fusion model are trained based on the multiple detected three-dimensional bounding boxes, the descriptor information of the multiple detected three-dimensional bounding boxes, the target three-dimensional detection box of the at least one preset traffic participant object, and the real three-dimensional bounding box of the at least one preset traffic participant object to obtain the target fusion perception model having the target recognition model and the target fusion model.

[0016] Optionally, the training of the initial recognition model and the initial fusion model based on the multiple detected 3D bounding boxes, the descriptor information of the multiple detected 3D bounding boxes, the target 3D detection box of the at least one preset traffic participant object, and the real 3D bounding boxes of each preset traffic participant object to obtain the target fusion perception model having the target recognition model and the target fusion model includes:

[0017] Calculating a first loss parameter based on the multiple detected three-dimensional bounding boxes and the real three-dimensional bounding boxes of the corresponding preset traffic participant objects having the same object identifier;

[0018] Calculating a second loss parameter based on the descriptor information of the multiple detected three-dimensional bounding boxes and the descriptor information of other detected three-dimensional bounding boxes having the same object identifier;

[0019] Calculating a third loss parameter based on the target three-dimensional detection frame of the at least one preset traffic participant object and the real three-dimensional bounding boxes of the preset traffic participant objects;

[0020] Calculating a target loss parameter according to the first loss parameter, the second loss parameter, and the third loss parameter;

[0021] The initial recognition model and the initial fusion model are trained according to the target loss parameter to obtain the target fusion perception model having the target recognition model and the target fusion model.

[0022] Optionally, the step of performing recognition using the initial recognition model based on the sample roadside images in the at least two directions of each intersection and the camera parameters in the at least two directions to obtain a plurality of detected three-dimensional bounding boxes and descriptor information of the plurality of detected three-dimensional bounding boxes includes:

[0023] Based on the sample roadside images in at least two directions, the camera parameters in at least two directions, and the timestamp information of each sample roadside image, the initial recognition model is used for recognition to obtain the multiple detected three-dimensional bounding boxes, the descriptor information of the multiple detected three-dimensional bounding boxes, and the timestamp information of the image corresponding to each detected three-dimensional bounding box; wherein each detected three-dimensional bounding box carries the semantic category and heading angle of the corresponding traffic participant object, and the descriptor information of each detected three-dimensional bounding box stores the high-dimensional feature information of the perspective of the corresponding traffic participant object.

[0024] Optionally, before obtaining the preset roadside scene dataset, the method further includes:

[0025] Acquire a historical three-dimensional point cloud image of each intersection at the same time as the historical roadside image;

[0026] Perform three-dimensional detection on the historical three-dimensional point cloud image to obtain a true three-dimensional bounding box of the at least one preset traffic participant object.

[0027] In a second aspect, another embodiment of the present application provides a fusion perception method, the method comprising:

[0028] Acquire roadside images to be detected at a target intersection in at least two directions;

[0029] According to the camera parameters of the at least two directions of the target intersection and the roadside images to be detected in the at least two directions, a pre-trained target fusion perception model is used for perception to obtain the detection results of traffic participants at the target intersection, wherein the target fusion perception model is a model trained using the training method of any fusion perception model described in the first aspect above.

[0030] Optionally, the target fusion perception model includes: a target recognition model and a target fusion model;

[0031] The method of obtaining a detection result of a traffic participant at the target intersection by using a pre-trained target fusion perception model to perceive the target intersection based on the camera parameters in the at least two directions and the roadside images to be detected in the at least two directions includes:

[0032] Based on the roadside images to be detected in the at least two directions of the target intersection and the camera parameters in the at least two directions, the target recognition model is used to perform recognition to obtain a plurality of initial three-dimensional bounding boxes and descriptor information of the plurality of initial three-dimensional bounding boxes;

[0033] According to the descriptor information of the multiple initial three-dimensional bounding boxes, the target fusion model is used to fuse the multiple initial three-dimensional bounding boxes to obtain the traffic participant object detection result.

[0034] In a third aspect, another embodiment of the present application provides a training device for a fusion perception model, the device comprising:

[0035] A first acquisition module is configured to acquire a preset roadside scene dataset, wherein the preset roadside scene dataset includes: a sample roadside scene image of at least one intersection, and the sample roadside scene image of each intersection includes: historical roadside images in at least two directions of each intersection, camera parameters in the at least two directions, and a real three-dimensional bounding box of at least one preset traffic participant object at each intersection;

[0036] a first perception module, configured to perform perception using an initial fusion perception model based on historical roadside images in the at least two directions of each intersection and camera parameters in the at least two directions, to obtain a three-dimensional bounding box for target detection of the at least one preset traffic participant;

[0037] A training module is used to train the initial fusion perception model based on the real three-dimensional bounding box of the at least one preset traffic participant object and the target detection three-dimensional bounding box of each preset traffic participant object to obtain a target fusion perception model.

[0038] In a fourth aspect, another embodiment of the present application provides a fusion sensing device, comprising:

[0039] A second acquisition module is used to acquire roadside images to be detected at the target intersection in at least two directions;

[0040] The second perception module is used to perceive the target intersection based on the camera parameters in at least two directions and the roadside images to be detected in at least two directions, using a pre-trained target fusion perception model to obtain the detection results of traffic participants at the target intersection, wherein the target fusion perception model is a model trained using the training method of any fusion perception model described in the first aspect above.

[0041] In the fifth aspect, another embodiment of the present application provides a computer device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the method described in any one of the first and second aspects above.

[0042] In a sixth aspect, another embodiment of the present application provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in any one of the first and second aspects above are executed.

[0043] The beneficial effects of this application are:

[0044] The present application provides a training method, a fusion perception method, a device, and a medium for a fusion perception model, which obtains a preset roadside scene data set, and uses an initial fusion perception model for perception based on historical roadside images from at least two directions of each intersection and camera parameters from at least two directions to obtain a target detection three-dimensional bounding box of at least one preset traffic participant. The initial fusion perception model is trained based on the real three-dimensional bounding box of at least one preset traffic participant and the target detection three-dimensional bounding box of each preset traffic participant to obtain a target fusion perception model. In the present application, there is no need to train the model separately for each specific intersection. It is only necessary to obtain historical roadside images from at least two directions of each intersection and camera parameters from at least two directions to determine the target detection three-dimensional bounding box of at least one preset traffic participant. In other words, the model in the present application is a universal model that can be applied to every intersection, avoiding the need to design and train a model for each intersection, saving a large amount of computing resources, and improving the accuracy of traffic participant recognition through historical roadside images from at least two directions. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 A flowchart of a training method for a fusion perception model provided in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of a process for determining a three-dimensional bounding box for target detection in a training method for a fusion perception model provided in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of a process for determining a target fusion perception model in another fusion perception model training method provided in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of the process of determining a true three-dimensional bounding box in a training method for a fusion perception model provided in this application;

[0050] Figure 5 A flowchart of a fusion perception method provided in an embodiment of the present application;

[0051] Figure 6 A schematic diagram of a process for determining traffic participants in a fusion perception method provided in an embodiment of the present application;

[0052] Figure 7 A schematic diagram of a fusion perception of a roadside image to be detected provided in an embodiment of the present application;

[0053] Figure 8 A schematic diagram of the structure of a training device for a fusion perception model provided in an embodiment of the present application;

[0054] Figure 9 A schematic diagram of the structure of a fusion sensing device provided in an embodiment of the present application;

[0055] Figure 10 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of illustration and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can be implemented out of sequence, and steps without logical context can be reversed or implemented simultaneously. In addition, those skilled in the art, under the guidance of the contents of this application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0057] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0058] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0059] To clearly describe the method provided in the embodiment of the present application, the training method of the fusion perception model provided in the embodiment of the present application is described below with reference to a plurality of drawings. Figure 1 A flow chart of a training method for a fusion perception model provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the method includes:

[0060] Step 101: Obtain a preset roadside scene dataset.

[0061] The preset roadside scene dataset includes sample roadside scene images of at least one intersection. Each sample roadside scene image of each intersection includes historical roadside images from at least two directions of the intersection, camera parameters from at least two directions, and a true three-dimensional bounding box of at least one preset traffic participant at each intersection. The sample roadside scene images of each intersection contain the preset traffic participant.

[0062] Optionally, the preset roadside scene can be a crossroads, a T-junction or a complex intersection, etc., and the embodiments of the present application do not impose any restrictions on this. The preset traffic participants can be motor vehicles, pedestrians or non-motor vehicles, etc., and the embodiments of the present application do not impose any restrictions on this. The sample roadside scene image of each intersection is a two-dimensional image, which is obtained by setting a two-dimensional camera at the intersection of the preset roadside scene. When it is necessary to obtain historical roadside images of at least two directions of each intersection, two-dimensional cameras are set in two directions of the intersection, and the images of the intersection are collected by the two two-dimensional cameras at the same time to obtain historical roadside images of at least two directions of each intersection, and the camera parameters corresponding to the historical roadside images are obtained, wherein the camera parameters include: camera intrinsic parameters and camera extrinsic parameters. The camera intrinsic parameters are used to describe the geometric properties and imaging characteristics of the camera itself, and the camera extrinsic parameters describe the position and direction of the camera in three-dimensional space. The three-dimensional scene corresponding to the historical roadside image can be determined by the camera parameters and the historical roadside image. After the historical roadside image is obtained, the historical roadside image is cropped according to the preset size so that the size of the historical roadside image is consistent.

[0063] Among them, the real three-dimensional bounding box of the preset traffic participant object can be (x, y, z, l, w, h, heading), x is the coordinate of the center point of the bounding box in the horizontal axis direction, y is the coordinate of the center point of the bounding box in the vertical axis direction, z is the coordinate of the center point of the bounding box in the depth axis direction, l is the length of the bounding box in the horizontal axis direction, w is the length of the bounding box in the vertical axis direction, h is the length of the bounding box in the depth axis direction, and heading is the orientation or rotation angle of the bounding box.

[0064] Step 102: Based on historical roadside images in at least two directions of each intersection and camera parameters in at least two directions, an initial fusion perception model is used for perception to obtain a target detection three-dimensional bounding box of at least one preset traffic participant.

[0065] Among them, the initial fusion perception model is used to obtain the target detection three-dimensional bounding box of the preset traffic participant at each intersection based on the historical roadside images in at least two directions at the same time of each intersection and the camera parameters in at least two directions.

[0066] Optionally, historical roadside images and camera parameters for each intersection are obtained from at least two directions at the same time. The historical roadside images and camera parameters corresponding to each historical roadside image are input into an initial fusion perception model. The initial fusion perception model is used for perception to obtain a 3D bounding box for target detection of at least one predetermined traffic participant in the historical roadside images from one direction, thereby obtaining a 3D bounding box for target detection of at least one predetermined traffic participant at each intersection. The at least one predetermined traffic participant may appear in any one of the historical roadside images from the at least two directions, or in both of the historical roadside images from the at least two directions.

[0067] Step 103: Train the initial fusion perception model based on the real three-dimensional bounding box of at least one preset traffic participant object and the target detection three-dimensional bounding boxes of each preset traffic participant object to obtain a target fusion perception model.

[0068] Optionally, the initial fusion perception model is trained based on the real three-dimensional bounding box of at least one preset traffic participant object and the target detection three-dimensional bounding box of each preset traffic participant object to obtain a target fusion perception model, so that the target fusion perception model can detect traffic participant objects in roadside scenes more accurately.

[0069] For example, a loss function is calculated based on the real three-dimensional bounding box of at least one preset traffic participant object and the target detection three-dimensional bounding box of each preset traffic participant object, and the initial fusion perception model is trained according to the loss function to obtain a target fusion perception model.

[0070] In an embodiment of the present application, a preset roadside scene data set is obtained, and an initial fusion perception model is used for perception based on historical roadside images from at least two directions of each intersection and camera parameters from at least two directions to obtain a target detection three-dimensional bounding box of at least one preset traffic participant object. The initial fusion perception model is trained based on the real three-dimensional bounding box of at least one preset traffic participant object and the target detection three-dimensional bounding box of each preset traffic participant object to obtain a target fusion perception model. In the present application, there is no need to train the model separately for each specific intersection. It is only necessary to obtain historical roadside images from at least two directions of each intersection and camera parameters from at least two directions to determine the target detection three-dimensional bounding box of at least one preset traffic participant object. In other words, the model in the present application is a universal model that can be applied to every intersection, avoiding the need to train the model for each intersection, saving a lot of computing resources, and improving the accuracy of traffic participant object recognition through historical roadside images from at least two directions.

[0071] Based on the above embodiment, the initial fusion perception model includes: an initial recognition model and an initial fusion model. This application also provides a process for determining a target detection three-dimensional bounding box in a training method for a fusion perception model. Figure 2 A flow chart of determining a three-dimensional bounding box for target detection in a training method for a fusion perception model provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, in the above step 102, based on the historical roadside images in at least two directions of each intersection and the camera parameters in at least two directions, an initial fusion perception model is used for perception to obtain a target detection 3D bounding box of at least one preset traffic participant, including:

[0072] Step 201: Based on historical roadside images in at least two directions of each intersection and camera parameters in at least two directions, an initial recognition model is used to perform recognition to obtain multiple detected three-dimensional bounding boxes and descriptor information of the multiple detected three-dimensional bounding boxes.

[0073] The detected three-dimensional bounding box can be (x, y, z, l, w, h, heading), where x is the coordinate of the center point of the bounding box in the horizontal axis direction, y is the coordinate of the center point of the bounding box in the vertical axis direction, z is the coordinate of the center point of the bounding box in the depth axis direction, l is the length of the bounding box in the horizontal axis direction, w is the length of the bounding box in the vertical axis direction, h is the length of the bounding box in the depth axis direction, and heading is the orientation or rotation angle of the bounding box. The descriptor information is used to describe the bird's-eye view information of the preset traffic participants in the detected three-dimensional bounding box. For example, when the preset traffic participant is a vehicle, the descriptor information can be features such as the contour edge of the vehicle, the position and shape of the wheels, etc. The initial recognition model can be a model built based on a neural network for identifying traffic participant objects in the image based on the image and camera parameters, and obtaining the detected three-dimensional bounding box corresponding to the traffic participant and the descriptor information corresponding to the detected three-dimensional bounding box.

[0074] Optionally, the detected 3D bounding box is a 3D bounding box of a pre-set traffic participant, and the descriptor information of the detected 3D bounding box is bird's-eye view description information of the pre-set traffic participant. When the historical roadside image includes multiple pre-set traffic participants, multiple detected 3D bounding boxes and multiple descriptor information of the detected 3D bounding boxes can be obtained from the historical roadside image.

[0075] Step 202: Based on the descriptor information of the multiple detected three-dimensional bounding boxes, an initial fusion model is used to fuse the multiple detected three-dimensional bounding boxes to obtain a target three-dimensional detection box of at least one preset traffic participant object.

[0076] Among them, the initial fusion object is the terminology built based on the Transformer model to fuse the detected three-dimensional bounding box to obtain the model of the target three-dimensional detection box.

[0077] Optionally, if the descriptor information of the detected three-dimensional bounding boxes is the same, it means that the preset traffic participant objects in the detected three-dimensional bounding boxes are the same, then the detected three-dimensional bounding boxes with the same descriptor information are fused to obtain the target three-dimensional detection box of the preset traffic participant object.

[0078] Optionally, if the descriptor information of multiple detected three-dimensional bounding boxes is different, the initial fusion model is used to fuse the multiple detected three-dimensional bounding boxes to obtain target three-dimensional detection boxes of multiple preset traffic participating objects, wherein the target three-dimensional detection boxes are the same as the multiple detected three-dimensional bounding boxes identified by the initial recognition model.

[0079] In an embodiment of the present application, an initial recognition model is used for recognition, resulting in multiple detected 3D bounding boxes and descriptor information for each of the multiple detected 3D bounding boxes. This allows accurate identification of traffic participants at each intersection. Based on the descriptor information for the multiple detected 3D bounding boxes, an initial fusion model is used to fuse the multiple detected 3D bounding boxes to obtain a target 3D detection box for at least one preset traffic participant object. This avoids duplicate recognition of traffic participants due to their appearance in multiple roadside images, thereby improving the accuracy of traffic participant recognition.

[0080] Based on the above embodiment, the present application further provides a process for determining a target fusion perception model in a fusion perception model training method. In the above step 103, the initial fusion perception model is trained based on the real 3D bounding box of at least one preset traffic participant object and the target detection 3D bounding box of each preset traffic participant object to obtain the target fusion perception model, including:

[0081] Based on multiple detected three-dimensional bounding boxes, descriptor information of multiple detected three-dimensional bounding boxes, a target three-dimensional detection box of at least one preset traffic participant object, and a real three-dimensional bounding box of at least one preset traffic participant object, an initial recognition model and an initial fusion model are trained to obtain a target fusion perception model having a target recognition model and a target fusion model.

[0082] Optionally, the accuracy of the initial recognition model's recognition of the 3D bounding boxes is determined based on multiple detected 3D bounding boxes and the actual 3D bounding boxes of the preset traffic participants within the multiple detected 3D bounding boxes. Based on the descriptor information of the multiple detected 3D bounding boxes, it is determined whether the descriptor information of the detected 3D bounding boxes corresponding to the same preset traffic participant is consistent, thereby determining the accuracy of the initial recognition model's recognition of the descriptor information corresponding to the 3D bounding boxes. Furthermore, the accuracy of the initial fusion perception model's fusion of the target 3D detection box corresponding to the preset traffic participant is determined based on the target 3D detection box of at least one preset traffic participant and the actual 3D bounding box of at least one preset traffic participant. Thus, the initial recognition model and the initial fusion model are trained to obtain the target fusion perception model comprising the target recognition model and the target fusion model.

[0083] In an embodiment of the present application, the initial recognition model and the initial fusion model are trained to obtain a target fusion perception model having a target recognition model and a target fusion model. This application can ensure the accuracy of the target recognition model in detecting the three-dimensional bounding box of traffic participants in historical roadside images, and improve the accuracy of the target fusion model in detecting the three-dimensional bounding box fusion.

[0084] Based on the above embodiment, the present application also provides another process for determining a target fusion perception model in a training method of a fusion perception model. Figure 3 A flow chart of determining a target fusion perception model in another fusion perception model training method provided in an embodiment of the present application is shown as follows: Figure 3 As shown, in the above, based on multiple detected 3D bounding boxes, multiple descriptor information of the detected 3D bounding boxes, at least one target 3D detection box of a preset traffic participant object, and the real 3D bounding boxes of each preset traffic participant object, the initial recognition model and the initial fusion model are trained to obtain a target fusion perception model having a target recognition model and a target fusion model, including:

[0085] Step 301: Calculate a first loss parameter based on a plurality of detected three-dimensional bounding boxes and real three-dimensional bounding boxes of corresponding preset traffic participant objects having the same object identifier.

[0086] Each preset traffic participant object has a unique identifier. When the same preset traffic participant object appears in different roadside images, the corresponding identifiers are the same.

[0087] Optionally, a first loss parameter is calculated based on a plurality of detected three-dimensional bounding boxes and real three-dimensional bounding boxes of corresponding preset traffic participant objects having the same object identifier using a target detection loss calculation method.

[0088] Step 302: Calculate a second loss parameter based on the descriptor information of the multiple detected three-dimensional bounding boxes and the descriptor information of other detected three-dimensional bounding boxes with the same object identifier.

[0089] Optionally, a third loss value is calculated using a similarity algorithm based on the descriptor information of the multiple detected three-dimensional bounding boxes and the descriptor information of other detected three-dimensional bounding boxes with the same object identifier.

[0090] Step 303: Calculate a third loss parameter based on the target three-dimensional detection box of at least one preset traffic participant object and the real three-dimensional bounding box of each preset traffic participant object.

[0091] Optionally, a third loss parameter is calculated based on a target three-dimensional detection frame of at least one preset traffic participant object and a real three-dimensional bounding box of each preset traffic participant object using a target detection loss calculation method.

[0092] Step 304: Calculate the target loss parameter based on the first loss parameter, the second loss parameter, and the third loss parameter.

[0093] Optionally, determine the first weight corresponding to the first loss parameter, the second weight corresponding to the second loss parameter, and the third weight corresponding to the third loss function, and perform weighted calculations on the first loss parameter, the second loss parameter, and the third loss parameter, respectively, to obtain the weighted first loss parameter, the second loss parameter, and the third loss parameter, respectively, and determine the target loss parameter based on the weighted first loss parameter, the second loss parameter, and the third loss parameter.

[0094] Step 305: Train the initial recognition model and the initial fusion model according to the target loss parameter to obtain a target fusion perception model having the target recognition model and the target fusion model.

[0095] Optionally, the initial recognition model and the initial fusion model are tuned based on the target loss parameter to minimize the target loss parameter, resulting in a target fusion perception model comprising the target recognition model and the target fusion model. Specifically, gradients can be calculated using a backpropagation algorithm, and the model parameters can be updated repeatedly until the model performance stops improving or a preset number of training rounds is reached.

[0096] In an embodiment of the present application, a first loss function, a second loss function, and a third loss function are calculated respectively, and the model is trained according to the first loss function, the second loss function, and the third loss function to obtain a target fusion perception model having a target recognition model and a target fusion model. The present application can be optimized in a targeted manner. By combining multiple loss functions, the model can advance in multiple optimization directions simultaneously, thereby accelerating the overall training process.

[0097] Based on the above embodiment, the present application further provides a process for determining a detected three-dimensional bounding box and descriptor information in a training method for a fusion perception model. In the above step 201, based on sample roadside images in at least two directions of each intersection and camera parameters in at least two directions, an initial recognition model is used to obtain multiple detected three-dimensional bounding boxes and multiple descriptor information of the detected three-dimensional bounding boxes, including:

[0098] Based on sample roadside images in at least two directions, camera parameters in at least two directions, and timestamp information of each sample roadside image, an initial recognition model is used for recognition to obtain multiple detected three-dimensional bounding boxes, descriptor information of multiple detected three-dimensional bounding boxes, and timestamp information of the image corresponding to each detected three-dimensional bounding box.

[0099] Among them, each detected three-dimensional bounding box carries the semantic category and heading angle of the corresponding traffic participant object, and the descriptor information of each detected three-dimensional bounding box stores the high-dimensional feature map of the perspective of the corresponding traffic participant object. The semantic category refers to the category label to which the target object belongs, which is used to describe the type of target object. The heading angle refers to the orientation angle of the target object on the horizontal plane, which is usually used to describe the direction or orientation of the target object. For example, when the traffic participant object is a car, the corresponding semantic description is car, and the heading angle is consistent with the direction of travel of the car.

[0100] Optionally, timestamps are used to synchronize roadside scene images captured by different 2D cameras. Since the transmission time of roadside scene images collected by each 2D camera varies, timestamps can ensure that the sample roadside scene images at at least one intersection were collected at the same time. Timestamps can also be used to estimate the trajectory of traffic participants. The trajectory of traffic participants at intermediate moments can be estimated based on the timestamps of the two images.

[0101] Optionally, an initial recognition model is used to perform recognition based on sample roadside images in at least two directions and camera parameters in at least two directions, thereby obtaining a plurality of detected three-dimensional bounding boxes and descriptor information for the plurality of detected three-dimensional bounding boxes. Timestamp information of each sample roadside image can be obtained to verify that the timestamp information of the three-dimensional bounding boxes is consistent with the timestamp information in the corresponding sample roadside image.

[0102] In the present application, when inputting the model, not only the sample roadside image and camera parameters are input, but also the timestamp information corresponding to the sample roadside image is input. While obtaining the detected three-dimensional bounding box and the descriptor information of the detected three-dimensional bounding box, the timestamp information of the image corresponding to the detected three-dimensional bounding box is also obtained, which can ensure data synchronization and consistency. Through the timestamp information, the position and state of the target object at different time points can be tracked, thereby reconstructing its motion trajectory and enhancing the accuracy of trajectory prediction.

[0103] Based on the above embodiments, the present application also provides a process for determining a true three-dimensional bounding box in a training method for a fusion perception model. Figure 4 This is a flow chart of determining a true three-dimensional bounding box in a training method for a fusion perception model provided in this application, such as Figure 4 As shown, based on the above steps 101 to 103, the method further includes:

[0104] Step 401: Obtain a historical 3D point cloud image of each intersection at the same time as the historical roadside image.

[0105] Among them, the historical three-dimensional point cloud image is used to represent the data structure of objects or scenes in the historical three-dimensional space. It is composed of a series of discrete points, each of which contains its coordinates in the three-dimensional space and may also contain other attributes such as color, intensity or reflectivity.

[0106] Optionally, a laser radar is installed at each intersection, and while acquiring historical roadside images of each intersection, historical three-dimensional point cloud images of the same time are collected by the laser radar.

[0107] Step 402: Perform three-dimensional detection on the historical three-dimensional point cloud image to obtain a true three-dimensional bounding box of at least one preset traffic participant object.

[0108] Optionally, three-dimensional detection is performed on the historical three-dimensional point cloud image, the coordinates of at least one preset traffic participant object are determined from the historical three-dimensional point cloud image, and the real three-dimensional bounding box of the at least one preset traffic participant object is determined based on the coordinates of the at least one preset traffic participant object.

[0109] In an embodiment of the present application, by obtaining historical three-dimensional point cloud images of each intersection at the same time as the historical roadside images are obtained, and determining the real three-dimensional bounding box of at least one preset traffic participant object based on the historical three-dimensional point cloud images, the accuracy of the real three-dimensional bounding box can be guaranteed.

[0110] Based on the same inventive concept, an embodiment of the present application also provides a fusion perception method, which performs perception using a target fusion perception model trained by the above-mentioned fusion perception model training method. Figure 5 A flow chart of a fusion perception method provided in an embodiment of the present application is shown as follows: Figure 5 As shown, the method includes:

[0111] Step 501: Acquire roadside images to be detected at a target intersection in at least two directions.

[0112] The two directions can be at least two different directions, such as east, south, west, north, coming, going, etc. The two-dimensional cameras set in two directions may have overlapping areas or no overlapping areas, which is not limited in this embodiment of the present application.

[0113] Optionally, two-dimensional cameras are set at at least two different directions of the target intersection, and the two-dimensional cameras are used to collect roadside images to be detected in at least two directions of the target intersection.

[0114] Step 502: Based on the camera parameters in at least two directions of the target intersection and the roadside images to be detected in at least two directions, a pre-trained target fusion perception model is used to perform perception to obtain a detection result of traffic participants at the target intersection.

[0115] Among them, the target fusion perception model is a model trained using the training method of the fusion perception model.

[0116] Optionally, after acquiring the roadside images to be detected in at least two directions of the target intersection, the roadside images to be detected are cropped according to a preset size so that the sizes of the roadside images to be detected in at least two directions are the same.

[0117] Optionally, the roadside image to be detected, the camera parameters of the corresponding direction of the image to be detected, the number of cameras at the target intersection, the number of channels of the roadside image to be detected, and the height and width of the image to be detected are respectively taken as a set of data, and a pre-trained target fusion perception model is used for perception to obtain the detection results of traffic participants at the target intersection.

[0118] For example, a set of data input to the pre-trained target fusion perception model can be {(N, C, H, W), [camera parameters]} and the image to be detected, where N is the number of cameras, C is the number of channels of the image to be detected, generally 3, HW is the height and width to be detected, and the length of the sensor parameter list is N.

[0119] In an embodiment of the application, roadside images to be detected in at least two directions of the target intersection are obtained, and according to the camera parameters in at least two directions of the target intersection and the roadside images to be detected in at least two directions, a pre-trained target fusion perception model is used for perception to obtain the detection results of traffic participants at the target intersection. The present application only needs to determine the images to be detected in at least two directions of the target intersection, and can then use the pre-trained target fusion perception model for perception to obtain the detection results of traffic participants at the target intersection. There is no need to train the camera of each intersection separately. It is only necessary to determine the roadside images to be detected in at least two directions of the target intersection to identify the traffic participants at the intersection, which reduces the computational complexity and model training cost of the model. The flexibility and scalability of the target fusion perception model are improved.

[0120] Based on the above embodiment, the target fusion perception model includes: a target recognition model and a target fusion model. To this end, the present application also provides a process for determining traffic participants in a fusion perception method. Figure 6A schematic diagram of a process for determining traffic participants in a fusion perception method provided in an embodiment of the present application is shown as follows: Figure 6 As shown, in the above step 502, based on the camera parameters of at least two directions of the target intersection and the roadside images to be detected in at least two directions, a pre-trained target fusion perception model is used to perform perception, and the traffic participant detection results of the target intersection are obtained, including:

[0121] Step 601: Based on the roadside images to be detected in at least two directions of the target intersection and the camera parameters in at least two directions, a target recognition model is used to perform recognition to obtain multiple initial three-dimensional bounding boxes and descriptor information of the multiple initial three-dimensional bounding boxes.

[0122] Optionally, the roadside image to be detected and the camera parameters corresponding to the image to be detected are used as inputs of the target recognition model, and the target recognition model is used for recognition to obtain multiple initial three-dimensional bounding boxes of each roadside image to be detected and descriptor information of the multiple initial three-dimensional bounding boxes.

[0123] Step 602: Based on the descriptor information of the multiple initial three-dimensional bounding boxes, a target fusion model is used to fuse the multiple initial three-dimensional bounding boxes to obtain a traffic participant object detection result.

[0124] Optionally, if the roadside images to be detected from at least two directions of the target intersection have overlapping fields of view, the target fusion model is used to fuse the multiple initial 3D bounding boxes based on their descriptor information. Specifically, similarity is calculated based on the descriptor information of the multiple initial 3D bounding boxes. Initial 3D bounding boxes with a similarity greater than a preset threshold and coordinate differences less than a preset threshold are fused together, and the traffic participant detection result is obtained based on the fused result.

[0125] Optionally, if there is no area of ​​overlapping field of view in the roadside images to be detected in at least two directions of the target intersection, the multiple initial three-dimensional bounding boxes are fused using a target fusion model based on the descriptor information of the multiple initial three-dimensional bounding boxes to obtain the traffic participant object detection result.

[0126] In an embodiment of the present application, a target recognition model is used to perform recognition based on the roadside images to be detected in at least two directions of the target intersection and the camera parameters in at least two directions, thereby obtaining multiple initial three-dimensional bounding boxes and descriptor information of the multiple initial three-dimensional bounding boxes. The present application can accurately identify the three-dimensional bounding boxes of the traffic participants at the target intersection, and can improve the accuracy of target recognition in two directions. Based on the descriptor information of the multiple initial three-dimensional bounding boxes, a target fusion model is used to fuse the multiple initial three-dimensional bounding boxes to obtain the detection results of the traffic participants. The traffic participants can be identified and located more accurately, the repeated identification of the traffic participants can be avoided, and the accuracy of the identification of the traffic participants can be improved.

[0127] For example, Figure 7 A schematic diagram of a roadside image fusion perception to be detected provided in an embodiment of the present application is shown as follows: Figure 7 As shown, the roadside images to be detected at the target intersection in at least two directions and the camera parameters in at least two directions are input into the target recognition model for recognition, obtaining multiple initial 3D bounding boxes and their descriptor information. The initial 3D bounding boxes also carry corresponding semantic categories. The multiple initial 3D bounding boxes and their descriptor information are then input into the target fusion model to obtain traffic participant object detection results. The traffic participant object detection results can be the target 3D detection boxes of the traffic participant objects and their semantic categories.

[0128] Based on the same inventive concept, the embodiments of the present application also provide a training device for a fusion perception model corresponding to the training method for the fusion perception model. Since the principle of solving the problem by the device in the embodiments of the present application is similar to the training method for the fusion perception model in the embodiments of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0129] Figure 8 A structural diagram of a training device for a fusion perception model provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the device includes:

[0130] A first acquisition module 801 is configured to acquire a preset roadside scene dataset, wherein the preset roadside scene dataset includes: sample roadside scene images of at least one intersection, and the sample roadside scene images of each intersection include: historical roadside images in at least two directions of each intersection, camera parameters in at least two directions, and a real three-dimensional bounding box of at least one preset traffic participant at each intersection;

[0131] A first perception module 802 is configured to perform perception using an initial fusion perception model based on historical roadside images from at least two directions of each intersection and camera parameters from at least two directions, to obtain a three-dimensional bounding box for target detection of at least one preset traffic participant;

[0132] The training module 803 is used to train the initial fusion perception model based on the real three-dimensional bounding box of at least one preset traffic participant object and the target detection three-dimensional bounding box of each preset traffic participant object to obtain a target fusion perception model.

[0133] Optionally, the initial fusion perception model includes: an initial recognition model and an initial fusion model; the first perception module 802 is specifically configured to: perform recognition using the initial recognition model based on historical roadside images in at least two directions of each intersection and camera parameters in at least two directions, to obtain multiple detected three-dimensional bounding boxes and descriptor information of the multiple detected three-dimensional bounding boxes;

[0134] According to the descriptor information of multiple detected three-dimensional bounding boxes, an initial fusion model is used to fuse the multiple detected three-dimensional bounding boxes to obtain a target three-dimensional detection box of at least one preset traffic participant object.

[0135] Optionally, the first perception module 802 is specifically used to train the initial recognition model and the initial fusion model based on multiple detected three-dimensional bounding boxes, descriptor information of multiple detected three-dimensional bounding boxes, at least one target three-dimensional detection box of a preset traffic participant object, and at least one real three-dimensional bounding box of a preset traffic participant object to obtain a target fusion perception model having a target recognition model and a target fusion model.

[0136] Optionally, the first perception module 802 is specifically configured to calculate a first loss parameter based on the plurality of detected three-dimensional bounding boxes and the real three-dimensional bounding boxes of the corresponding preset traffic participant objects having the same object identifier;

[0137] Calculating a second loss parameter based on the descriptor information of the plurality of detected three-dimensional bounding boxes and the descriptor information of other detected three-dimensional bounding boxes having the same object identifier;

[0138] Calculating a third loss parameter based on a target three-dimensional detection frame of at least one preset traffic participant object and a true three-dimensional bounding box of each preset traffic participant object;

[0139] Calculating a target loss parameter according to the first loss parameter, the second loss parameter, and the third loss parameter;

[0140] According to the target loss parameter, the initial recognition model and the initial fusion model are trained to obtain a target fusion perception model having a target recognition model and a target fusion model.

[0141] Optionally, the first perception module 802 is specifically used to: use an initial recognition model to perform recognition based on sample roadside images in at least two directions, camera parameters in at least two directions, and timestamp information of each sample roadside image, to obtain multiple detected three-dimensional bounding boxes, descriptor information of multiple detected three-dimensional bounding boxes, and timestamp information of the image corresponding to each detected three-dimensional bounding box; wherein each detected three-dimensional bounding box carries the semantic category and heading angle of the corresponding traffic participant object, and the descriptor information of each detected three-dimensional bounding box stores a high-dimensional feature map of the perspective of the corresponding traffic participant object.

[0142] Optionally, the first acquisition module 801 is further configured to: acquire a historical three-dimensional point cloud image of each intersection at the same time as the historical roadside image;

[0143] Perform three-dimensional detection on the historical three-dimensional point cloud image to obtain a true three-dimensional bounding box of at least one preset traffic participant object.

[0144] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0145] Based on the same inventive concept, the embodiment of the present application also provides a fusion perception device corresponding to the fusion perception method. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned fusion perception method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0146] Figure 9 A schematic diagram of the structure of a fusion sensing device provided in an embodiment of the present application is shown in FIG. Figure 9 As shown, the device includes:

[0147] The second acquisition module 901 is used to acquire roadside images to be detected at a target intersection in at least two directions;

[0148] The second perception module 902 is used to perceive the target intersection based on the camera parameters in at least two directions and the roadside images to be detected in at least two directions, using a pre-trained target fusion perception model to obtain the detection results of traffic participants at the target intersection, wherein the target fusion perception model is a model trained using the training method of any fusion perception model of the first aspect mentioned above.

[0149] Optionally, the target fusion perception model includes: a target recognition model and a target fusion model; the second perception module 902 is specifically configured to: use the target recognition model to perform recognition based on the roadside images to be detected in at least two directions of the target intersection and the camera parameters in at least two directions, to obtain multiple initial three-dimensional bounding boxes and descriptor information of the multiple initial three-dimensional bounding boxes;

[0150] According to the descriptor information of multiple initial 3D bounding boxes, the target fusion model is used to fuse the multiple initial 3D bounding boxes to obtain the traffic participant object detection results.

[0151] The embodiment of the present application also provides a computer device, Figure 10 A schematic diagram of the structure of a computer device provided in an embodiment of the present application is shown in FIG. Figure 10 As shown, the computer device includes: a processor 1001, a memory 1002, and optionally, a bus 1003. The memory 1002 stores machine-readable instructions executable by the processor 1001. When the computer device is running, the processor 1001 communicates with the memory 1002 via the bus 1003. When the machine-readable instructions are executed by the processor 1001, the steps of the training method of the fusion perception model or the steps of the fusion perception method are performed.

[0152] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it executes the steps of the training method of the above-mentioned fusion perception model, or the steps of the fusion perception method.

[0153] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0154] In addition, the functional units in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0155] The above is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the protection scope of the present application.

Claims

1. A training method for a fusion perception model, characterized in that: The method comprises: Acquiring a preset roadside scene dataset, wherein the preset roadside scene dataset includes: sample roadside scene images of at least one intersection, and the sample roadside scene images of each intersection include: historical roadside images in at least two directions of each intersection, camera parameters in the at least two directions, and a real three-dimensional bounding box of at least one preset traffic participant object at each intersection; Based on the historical roadside images of the at least two directions of each intersection and the camera parameters of the at least two directions, an initial fusion perception model is used to perform perception to obtain a target detection three-dimensional bounding box of the at least one preset traffic participant; The initial fusion perception model is trained according to the real three-dimensional bounding box of the at least one preset traffic participant object and the target detection three-dimensional bounding boxes of each preset traffic participant object to obtain a target fusion perception model.

2. The method according to claim 1, characterized in that The initial fusion perception model includes: an initial recognition model and an initial fusion model; the initial fusion perception model is used to perform perception based on the historical roadside images in the at least two directions of each intersection and the camera parameters in the at least two directions to obtain a target detection three-dimensional bounding box of the at least one preset traffic participant, including: Based on the historical roadside images in the at least two directions of each intersection and the camera parameters in the at least two directions, the initial recognition model is used to perform recognition to obtain a plurality of detected three-dimensional bounding boxes and descriptor information of the plurality of detected three-dimensional bounding boxes; According to the descriptor information of the multiple detected three-dimensional bounding boxes, the multiple detected three-dimensional bounding boxes are fused using the initial fusion model to obtain a target three-dimensional detection box of the at least one preset traffic participant object.

3. The method according to claim 2, characterized in that The training of the initial fusion perception model based on the real three-dimensional bounding box of the at least one preset traffic participant object and the target detection three-dimensional bounding boxes of each preset traffic participant object to obtain the target fusion perception model includes: The initial recognition model and the initial fusion model are trained based on the multiple detected three-dimensional bounding boxes, the descriptor information of the multiple detected three-dimensional bounding boxes, the target three-dimensional detection box of the at least one preset traffic participant object, and the real three-dimensional bounding box of the at least one preset traffic participant object to obtain the target fusion perception model having the target recognition model and the target fusion model.

4. The method according to claim 3, characterized in that The method further comprises training the initial recognition model and the initial fusion model based on the multiple detected three-dimensional bounding boxes, the descriptor information of the multiple detected three-dimensional bounding boxes, the target three-dimensional detection box of the at least one preset traffic participant object, and the real three-dimensional bounding boxes of each preset traffic participant object to obtain the target fusion perception model having the target recognition model and the target fusion model, including: Calculating a first loss parameter based on the multiple detected three-dimensional bounding boxes and the real three-dimensional bounding boxes of the corresponding preset traffic participant objects having the same object identifier; Calculating a second loss parameter based on the descriptor information of the multiple detected three-dimensional bounding boxes and the descriptor information of other detected three-dimensional bounding boxes having the same object identifier; Calculating a third loss parameter based on the target three-dimensional detection frame of the at least one preset traffic participant object and the real three-dimensional bounding boxes of the preset traffic participant objects; Calculating a target loss parameter according to the first loss parameter, the second loss parameter, and the third loss parameter; The initial recognition model and the initial fusion model are trained according to the target loss parameter to obtain the target fusion perception model having the target recognition model and the target fusion model.

5. The method according to claim 2, characterized in that The method of using the initial recognition model to perform recognition based on the sample roadside images in the at least two directions of each intersection and the camera parameters in the at least two directions to obtain a plurality of detected three-dimensional bounding boxes and descriptor information of the plurality of detected three-dimensional bounding boxes includes: Based on the sample roadside images in at least two directions, the camera parameters in at least two directions, and the timestamp information of each sample roadside image, the initial recognition model is used for recognition to obtain the multiple detected three-dimensional bounding boxes, the descriptor information of the multiple detected three-dimensional bounding boxes, and the timestamp information of the image corresponding to each detected three-dimensional bounding box; wherein each detected three-dimensional bounding box carries the semantic category and heading angle of the corresponding traffic participant object, and the descriptor information of each detected three-dimensional bounding box stores the perspective high-dimensional feature map of the corresponding traffic participant object.

6. The method according to claim 1, characterized in that Before obtaining the preset roadside scene dataset, the method further includes: Acquire a historical three-dimensional point cloud image of each intersection at the same time as the historical roadside image; Perform three-dimensional detection on the historical three-dimensional point cloud image to obtain a true three-dimensional bounding box of the at least one preset traffic participant object.

7. A fusion perception method, characterized in that: The method comprises: Acquire roadside images to be detected at a target intersection in at least two directions; According to the camera parameters of the at least two directions of the target intersection and the roadside images to be detected in the at least two directions, a pre-trained target fusion perception model is used for perception to obtain the detection results of the traffic participants at the target intersection, wherein the target fusion perception model is a model trained by the training method of the fusion perception model described in any one of claims 1-6 above.

8. The method according to claim 7, characterized in that The target fusion perception model includes: a target recognition model and a target fusion model; The method of obtaining a detection result of a traffic participant at the target intersection by using a pre-trained target fusion perception model to perceive the target intersection based on the camera parameters in the at least two directions and the roadside images to be detected in the at least two directions includes: Based on the roadside images to be detected in the at least two directions of the target intersection and the camera parameters in the at least two directions, the target recognition model is used to perform recognition to obtain a plurality of initial three-dimensional bounding boxes and descriptor information of the plurality of initial three-dimensional bounding boxes; According to the descriptor information of the multiple initial three-dimensional bounding boxes, the target fusion model is used to fuse the multiple initial three-dimensional bounding boxes to obtain the traffic participant object detection result.

9. A computer device, characterized in that: The computer device is used to execute the steps of the training method of the fusion perception model according to any one of claims 1 to 6, or the steps of the fusion perception method according to any one of claims 7 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the training method for the fusion perception model as described in any one of claims 1 to 6, or the steps of the fusion perception method as described in any one of claims 7 to 8.

Images