A three-dimensional target detection method and system based on time-series multi-view information fusion

Abstract

The application discloses a kind of three-dimensional target detection method and system based on timing multi-view information fusion, the method of the present application includes obtaining the image of multiple cameras collected by different view angles of the vehicle and constructing the image frame sequence of current time T Calculate the anchor frame of two-dimensional target, combine the depth of two-dimensional target to camera, and carry out two-dimensional to three-dimensional projection, and carry out random noise to obtain the two-dimensional height reference point after noise addition;The two-dimensional height reference point after noise addition, the fixed reference point of preestablished and the confidence higher front k reference points of last time, combine image features, and the history query of buffer fusion obtains the fusion query of current time, carries out regression and classification to the fusion query of current time to obtain the detection result of three-dimensional target and update the history query of buffer.The present application aims to improve the degree of detail of object detection, improve the 3D target detection accuracy and robustness of bird's eye view.

Description

TECHNICAL FIELD

[0001] The present application relates to target detection technology in the field of automatic driving, robots and intelligent transportation, and in particular to a three-dimensional target detection method and system based on time sequence multi-view information fusion. BACKGROUND

[0002] In the automatic driving environment perception system, 3D target detection as a core module needs to process the visual data collected by multiple cameras in real time and generate an environment representation in the bird's eye view (BEV) perspective, providing a key basis for path planning and decision-making. The traditional multi-view detection scheme processes each camera image independently and then fuses the results, which has problems such as target truncation in the field of view overlap area and insufficient cross-view feature correlation, making it difficult to meet the detection accuracy requirements in complex road conditions. With the application of the Transformer architecture in the field of computer vision, the bird's eye view target detection technology has made breakthrough progress. The PETR series model first converts 2D image features to three-dimensional space through 3D position embedding, enabling 3D target queries to interact with multi-view image features globally, avoiding the cumulative error of depth estimation in traditional methods. The StreamPETR model uses a time sequence modeling mechanism based on object centers, introduces target-guided flow-based time sequence information transmission and processing, and implicitly models the motion state of the target, significantly improving the detection efficiency and temporal consistency in dynamic scenes, while supporting TensorRT inference optimization, providing feasibility for on-board deployment. However, the existing StreamPETR model still has the following limitations in practical application: (1) Limited scene adaptability: In extreme lighting or occlusion scenarios, the model relies on fixed initialized position reference points, lacking scene adaptability, resulting in insufficient robustness for detecting distant high-density small targets such as pedestrians and cones, making it difficult to meet the perception needs of complex urban road conditions. (2) Insufficient utilization of existing information: Research shows that depth estimation and height estimation are equivalent, and depth estimation is an NP problem for 2D cameras, but there is no effective utilization of the height of the detected class by indirectly estimating the depth through the height of the object. (3) No effective fusion of past frame information: The StreamPETR model only uses information from the past 5 frames, but when the blank query enters the attention network, it does not carry effective information, resulting in ineffective fusion of past frame information. SUMMARY

[0003] The technical problem to be solved by the present application: In view of the above problems of the prior art, a three-dimensional target detection method and system based on time sequence multi-view information fusion are provided, which aims to improve the fine degree of object detection and improve the 3D target detection accuracy and robustness of the bird's eye view.

[0004] To solve the above technical problems, the technical scheme adopted by the present application is:

[0005] A three-dimensional target detection method based on time sequence multi-view information fusion, comprising the following steps:

[0006] S101, acquiring multi-view images collected by a camera of the vehicle and constructing an image frame sequence at a current time T;

[0007] S102, extracting image features at the current time T from the image frame sequence at the current time T using a feature extraction network;

[0008] S103, detecting the image features using a target detection network to obtain anchor boxes and height deviations of two-dimensional targets;

[0009] S104, converting the height deviations of the two-dimensional targets into actual heights of the two-dimensional targets, and calculating depths of the two-dimensional targets from the camera according to the actual heights of the two-dimensional targets ;

[0010] S105, projecting the anchor boxes of the two-dimensional targets in combination with the depths of the two-dimensional targets from the camera to obtain a two-dimensional height reference point after adding noise;

[0011] S106, inputting the two-dimensional height reference point after adding noise, a preset fixed reference point, image features at the current time T, the first k reference points with high confidence at the previous time T-1, and an initialized target spliced from a historical query vector cached to a pre-trained three-dimensional detection network to obtain a query vector at the current time T output by the three-dimensional detection network, and performing regression and classification on the query vector at the current time T to obtain a detection result of a three-dimensional target, including confidence of the query vector, anchor boxes of the three-dimensional target, a category of the three-dimensional target, and the first k reference points with high confidence at the current time T.

[0012] Optionally, the view angles of the multi-view images in step S101 include front, left front, left rear, rear, right rear, and right front.

[0013] Optionally, in step S102, when the image features at the current time T are extracted from the image frame sequence at the current time T using the feature extraction network, the extracted image features at the current time T are an image pyramid composed of multiple resolution layer features, and in step S103, the anchor boxes and height deviations of the two-dimensional targets are detected from the image features using the target detection network, which means that: the highest resolution layer feature in the image features is input to the target detection network, and the target detection network detects the highest resolution layer feature The pixel points of the two-dimensional target are regressed to the anchor frame, the height and the confidence of the two-dimensional target point by point, and the anchor frame and the height whose confidence exceeds a preset threshold are selected and output, and the attributes of the anchor frame include the category, the center and the position of the anchor frame.

[0014] Optionally, step S104 comprises:

[0015] S201, the function expression of converting the height deviation of the two-dimensional target into the actual height of the two-dimensional target is:

[0016] ;

[0017] wherein, is the actual height of the i-th two-dimensional target, is the category of the i-th two-dimensional target is the corresponding prior height, is the hyperbolic tangent function, is the height deviation obtained by detecting the i-th two-dimensional target;

[0018] S202, the depth of the two-dimensional target to the camera is calculated according to the actual height of the two-dimensional target :

[0019] ;

[0020] wherein, is the focal length of the camera, is the height of the anchor frame of the two-dimensional target.

[0021] Optionally, step S105 comprises:

[0022] S301, the horizontal deviation angle of the center point of the two-dimensional target to the camera optical axis and the corresponding direction vector are obtained according to the anchor frame center point of the two-dimensional target and the camera intrinsic parameter of the camera, wherein is the coordinate of the anchor frame center point of the two-dimensional target in the camera coordinate system;

[0023] S302, the horizontal deviation angle of the center point of the two-dimensional target to the camera optical axis and the depth of the two-dimensional target to the camera are regarded as polar coordinate representation , and the 3D coordinates of the two-dimensional target in the camera coordinate system are recovered :

[0024] ;

[0025] ​​​S303, convert the 3D coordinates of the two-dimensional target in the camera coordinate system to the three-dimensional coordinates of the point in the world coordinate system Convert to the world coordinate system by using the extrinsic matrix to obtain the three-dimensional coordinates of the point in the world coordinate system

[0026] S304, according to the three-dimensional coordinates of the point in the world coordinate system Calculate the normalized coordinates relative to the specified range around the vehicle Take the obtained two-dimensional height reference point as the three-dimensional coordinates of the point in the world coordinate system:

[0027]

[0028]

[0029]

[0030] wherein, and are the three-dimensional coordinates of the point in the world coordinate system are the minimum and maximum values in the x, y and z axis directions, and are the three-dimensional coordinates of the point in the world coordinate system are the numerical ranges in the x, y and z axis directions;

[0031] S305, add or subtract a random number to the x-axis and y-axis coordinates of the two-dimensional height reference point to realize random noise addition, thereby obtaining a noise-added two-dimensional height reference point.

[0032] Optionally, the initialization target obtained by fusing the cached historical query vectors in step S106 is realized by a global query mixing module GQF, the global query mixing module GQF maintains a global variable and a local variable of different sizes, the global variable and the local variable are respectively used to cache different numbers of historical query vectors with high confidence, and the initialization target obtained by fusing the cached historical query vectors includes:

[0033] S401, extract the global embedding constituted by the historical query vectors cached in the global variable and the local embedding constituted by the historical query vectors cached in the local variable

[0034] S402, splice the global embedding and the local embedding to obtain a fused query by nonlinear mapping ​​​​​​​​​​concatenate the global embedding and the fusion query to obtain an enhanced fusion query through residual weighting ;

[0035] S403, copy and cut the enhanced fusion query to the required length L as the obtained initialization target fused from the cached history query vector.

[0036] Optionally, the global embedding and the local embedding in step S402 are concatenated and nonlinearly mapped to obtain a fusion query , the global embedding and the fusion query are concatenated to obtain an enhanced fusion query through residual weighting The function expression of the enhanced fusion query is as follows:

[0037] ;

[0038] ;

[0039] wherein, is a preliminary fusion query, is a multi-layer perception, is a concatenation operation, is a global embedding obtained by concatenating a preset fixed reference point, the top k reference points with high confidence of the previous time and image features after adding noise, is a memory embedding composed of cached history queries, is a balance coefficient, and the balance coefficient has a value range of .

[0040] The application further provides a three-dimensional target detection system based on time-series multi-view information fusion, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the three-dimensional target detection method based on time-series multi-view information fusion.

[0041] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program or instructions, and the computer program or instructions are programmed or configured to execute the three-dimensional target detection method based on time-series multi-view information fusion through a processor.

[0042] The application further provides a computer program product, comprising a computer program or instructions, and the computer program or instructions are programmed or configured to execute the three-dimensional target detection method based on time-series multi-view information fusion through a processor.

[0043] Compared with the prior art, the present application mainly has the following beneficial effects: the three-dimensional target detection method based on time sequence multi-view information fusion of the present application utilizes the two-dimensional features of multi-view images, designs a target height information prediction network, effectively obtains the shape prior of a three-dimensional target, implicitly maps the multi-view images of a three-dimensional scene to an aerial view, fuses the two-dimensional anchor box information and the shape prior of the target height, and cooperatively reasons out the potential reference points of the three-dimensional target; further introducing the long-range-short-range time sequence correlation constraint of the sequence images reduces the generation space of the invalid anchor boxes near the reference points of the three-dimensional target, effectively improves the precision and robustness of the end-to-end three-dimensional target detection model of the target detection network, and compared with other detection methods in the prior art, the present application can effectively embed the shape prior of a scene target, simultaneously fuse the time sequence information of multi-views at different time, and has better detection effect on the visible targets in the surrounding scene. Figure Three BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 It is a basic flowchart of the method of the embodiment of the present application.

[0045] Figure 2 It is a basic principle diagram of the method of the embodiment of the present application.

[0046] Figure 3 It is a principle diagram of the height estimation based depth in the embodiment of the present application.

[0047] Figure 4 It is a working principle diagram of the global query mixing module in the embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to enable the personnel in the technical field to better understand the technical scheme of the present application, the technical scheme of the present application will be further described in detail below in combination with the drawings of the embodiment of the present application.

[0049] As shown in the drawings, the three-dimensional target detection method based on time sequence multi-view information fusion of the present embodiment comprises the following steps: Figure 1

[0050] S101, acquiring the multi-view images collected by the camera of the vehicle and constructing the image frame sequence at the current time T;

[0051] S102, extracting the image features at the current time T by using a feature extraction network on the image frame sequence at the current time T;

[0052] S103, detecting the anchor boxes and height deviations of two-dimensional targets by using a target detection network on the image features;

[0053] ​​S104, convert the height deviation of the two-dimensional target into the actual height of the two-dimensional target, and calculate the depth of the two-dimensional target to the camera according to the actual height of the two-dimensional target ;

[0054] S105, combine the anchor frame of the two-dimensional target with the depth of the two-dimensional target to the camera perform two-dimensional to three-dimensional projection and random noise addition to obtain a noisy two-dimensional height reference point;

[0055] S106, splice the noisy two-dimensional height reference point (as indicated by ① in Figure 2 , a preset fixed reference point (as indicated by ② in Figure 2 , image features of the current time T (as indicated by ③ in Figure 2 , the top k reference points with high confidence of the previous time T-1 (as indicated by ④ in Figure 2 , and the initialization target fused from the cached historical query vector (as indicated by ⑤ in Figure 2 ), and input them into the pre-trained three-dimensional detection network to obtain the query vector of the current time T output by the three-dimensional detection network, and perform regression and classification on the query vector of the current time T to obtain the detection result of the three-dimensional target, including the confidence of the query vector, the anchor frame of the three-dimensional target, the category of the three-dimensional target, and the top k reference points with high confidence of the current time T. The properties of the anchor frame of the three-dimensional target include the size, position and angle of the anchor frame.

[0056] As shown in Figure 2 , the multiple-view images in step S101 of the embodiment include six views of front, left front, left rear, rear, right rear and right front.

[0057] In step S102, when the image feature extraction network is used to extract the image features of the current time T from the image frame sequence of the current time T, the extracted image features of the current time T are an image pyramid composed of multiple resolution layer features. Specifically, when the image feature extraction network is used to extract the image features of the current time T from the image frame sequence of the current time T in step S102, the images under the multiple views are a series of image frames, and the visual features represent the spatial information of the series of image frames. The expression for extracting the visual features is:

[0058] ;

[0059] In the formula, represents the visual features extracted by the feature extraction network, represents the feature extraction network (such as resnet50), represents the image frame sequence input into the feature extraction network, where The image number is represented by t = 6, which represents the front, left front, left back, back, right back, and right front views, respectively. This indicates the number of image feature layers, typically 4 layers, such as... , , , .

[0060] In step S103 of this embodiment, using a target detection network to detect image features and obtain the anchor frame and height deviation of the two-dimensional target refers to: detecting the highest resolution layer features in the image features. Input the object detection network, and then process the features at the highest resolution layer through the object detection network. The system regresses the anchor boxes, heights, and confidence scores of 2D targets pixel by pixel, and outputs anchor boxes and heights whose confidence scores exceed a preset threshold. The anchor box attributes include the anchor box's category, center, and position. The target detection network can employ any required network model, such as the YOLO model. This embodiment only uses... As network input, the highest resolution layer of image features, and in Within the framework, point-by-point regression is performed to predict the object center, object category, confidence level, and height offset. Pixel-level predictions are used to select the object center location and category predictions with higher confidence levels, and regression is used to obtain the anchor box attributes, including the anchor box's category, center, and location.

[0061] In this embodiment, step S104 includes:

[0062] S201, the functional expression for converting the height deviation of a two-dimensional target into the actual height of the two-dimensional target is:

[0063] ;

[0064] in, For the first The actual height of the two-dimensional target For the first Categories of two-dimensional targets The corresponding prior height, It is the hyperbolic tangent function. For the first The height deviation obtained from the detection of a two-dimensional target;

[0065] S202, Calculate the depth from the two-dimensional target to the camera based on the actual height of the two-dimensional target. :

[0066] ;

[0067] in, The focal length of the camera. The height of the anchor frame for the two-dimensional target.

[0068] Figure 3 This is a schematic diagram illustrating the principle of depth estimation based on height in this embodiment. The camera O can be considered as a pinhole model, with the focal plane (z= f The triangle formed by the image of the target AA' (pixel height) and the camera O, and the triangle formed by the target BB' (actual height) and the camera O, can be modeled as a proportional triangle. This allows us to calculate the pixel height, actual height, and camera focal length. f This allows us to obtain the actual distance from the camera to the target. Therefore, we can calculate the depth from the 2D target to the camera based on the actual height of the 2D target. The function expression.

[0069] In this embodiment, step S105 includes:

[0070] S301, based on the center point of the anchor frame of the two-dimensional target. The horizontal deflection angle of the center point of the two-dimensional target relative to the camera's optical axis is obtained from the camera's intrinsic parameters. and its corresponding direction vector ,in The center point of the anchor frame for the two-dimensional target The coordinates in the camera coordinate system are:

[0071] , ;

[0072] in, This refers to the camera's internal parameters. The image's center point is the reference point; only the horizontal skew angle is relevant (BEV plane, take...). When considering the quantity, we have:

[0073] ;

[0074] in, This is a function used to calculate the angle between the ray from the origin to the point (x,y) in a two-dimensional plane and the positive x-axis.

[0075] S302, the horizontal deflection angle of the center point of the two-dimensional target relative to the camera optical axis. Depth of the 2D target from the camera Consider as polar coordinates Recover the 3D coordinates of the 2D target in the camera coordinate system. :

[0076] ;

[0077] S303, 3D coordinates of the two-dimensional target in the camera coordinate system are calculated The three-dimensional coordinates of the point in the world coordinate system are obtained by converting the external parameter matrix to the world coordinate system ;

[0078] S304, according to the three-dimensional coordinates of the point in the world coordinate system Calculate the normalized coordinates relative to the specified range around the vehicle Take the obtained two-dimensional height reference point as

[0079] , ;

[0080] , ;

[0081] , ;

[0082] wherein, , and are the three-dimensional coordinates of the point in the world coordinate system The minimum and maximum values in the x, y and z axis directions, , and are the three-dimensional coordinates of the point in the world coordinate system The numerical range in the x, y and z axis directions;

[0083] S305, the x-axis and y-axis coordinates of the two-dimensional height reference point are added and subtracted by random numbers to realize random noise addition, so as to obtain the noise-added two-dimensional height reference point.

[0084] Inside the 3D detection network, the anchor box height of 2D detection , the actual height of the target in the anchor box The depth distance from the camera position to the target can be obtained, and after obtaining the internal and external parameters of this camera position, the targets around the vehicle can be converted into normalized reference points in the same coordinate.

[0085] ;

[0086] The effective range of the radar point cloud is , and the scale is

[0087] ;

[0088] Mapping to [0, 1] has:

[0089] , ;

[0090] , ;

[0091] , ;

[0092] wherein, , and are three-dimensional coordinates of the point in the world coordinate system are minimum and maximum values in x, y and z axis directions, , and are three-dimensional coordinates of the point in the world coordinate system are numerical ranges in x, y and z axis directions; and is obtained after taking it as a reference point generated in height. And there is a fixed initialization reference point of the network , which is disturbed by random noise in X and Y directions to make it more close to the actual position of the real target.

[0093] As shown in Figure 4 , the initialization target fused from the cached historical query vectors in step S106 is implemented through a global query fusion module GQF. The global query fusion module GQF maintains a global variable and a local variable of different sizes. The global variable and the local variable are respectively used to cache different numbers of historical query vectors with high confidence. The initialization target fused from the cached historical query vectors includes:

[0094] S401, extracting a global embedding constituted by the historical query vectors cached in the global variable and a local embedding constituted by the historical query vectors cached in the local variable ;

[0095] S402, splicing and nonlinearly mapping the global embedding and the local embedding to obtain a fusion query , splicing the global embedding and the fusion query to obtain an enhanced fusion query through residual weighting ;

[0096] S403, copying and clipping the enhanced fusion query to a required length L to obtain the initialization target fused from the cached historical query vectors.

[0097] In step S402 of the embodiment, the global embedding and local embedding fusion query by splicing and nonlinear mapping , global embedding and fusion query enhanced fusion query by residual weighting The function expression of the enhanced fusion query is:

[0098] ;

[0099] ;

[0100] wherein, is the preliminary fusion query, is a multi-layer perception, is a splicing operation, is a global embedding obtained by splicing the noise-added two-dimensional height reference points, the preset fixed reference points, the top k reference points with high confidence of the previous time, and the image features, is a memory embedding composed of the cached historical queries, is a balance coefficient, and the balance coefficient is in the range of . , , . Wherein, B is batch, N is the number of fixed reference points nums (default 300), d is the dimension (dim), and the default is 256.

[0101] In step S403, the enhanced fusion query is copied and cropped to the required length L as the initialization target obtained by fusing the cached historical query vector. The fusion query is copied and cropped to the required length L, which can be expressed as:

[0102] ;

[0103] ;

[0104] wherein, is the variable repeated r times, represents repeating times in the query dimension, represents the mask length, is the number of repetitions, is the fusion query copied and cropped to the required length L, is the number of fixed reference points. The fusion query is copied and cropped to the required length L and written into the array as the input of the three-dimensional detection network.

[0105] As shown in Figure 2 and Figure 4 , the three-dimensional detection network in this embodiment adopts a propagation transformer (Propagation Transformer), which is an existing model, so its implementation details are not described here. The noisy two-dimensional height reference points (as shown by ① in Figure 4 ), the preset fixed reference points (as shown by ② in Figure 4 ), the image features at the current time T (as shown by ③ in Figure 4 ), the top k reference points with high confidence at the previous time T-1 (as shown by ④ in Figure 4 ), and the initialization target fused from the cached historical query vectors (as shown by ⑤ in Figure 4 ) are spliced and input into the pre-trained three-dimensional detection network. The query vector at the current time T output by the three-dimensional detection network (Propagation Transformer) is regressed and classified through the regression head and the classification head, so that the detection result of the three-dimensional target can be obtained, including the confidence of the query vector, the anchor box of the three-dimensional target, the category of the three-dimensional target, and the top k reference points with high confidence at the current time T. The regression head and the classification head are existing known network structures, so their implementation details are not described here.

[0106] In order to verify the three-dimensional target detection method based on time sequence multi-view information fusion of the embodiment, the nuscenes dataset is used for evaluation in this embodiment. The nuscenes dataset is a large dataset providing full set of automatic vehicle sensor data, including 6 cameras, 1 laser radar, 5 millimeter wave radars, and GPS and inertial measurement unit IMU. The nuscenes dataset has about 1000 scenes with annotated samples (about 20 seconds per scene), of which 850 scenes are used for training and verification, and 150 scenes are used for testing. In this experiment, the AdamW optimizer is used, the initial learning rate is 0.0004, and the batch size is 16. PETRv2, SOLOFusion, BEVFormerv2 and Streampetr are selected as the comparison of the method of this embodiment, and the indicators used include: mean average precision mAP (the larger the better), mean translation error mATE (meters, the smaller the better), mean scale error mASE (1-Intersection over Union, the smaller the better), mean orientation error mAOE (radians, the smaller the better), mean velocity error mAVE (meters / second, the smaller the better), mean attribute error mAAE (1-accuracy, the smaller the better) drop and comprehensive evaluation index NDS (the larger the better), wherein the calculation function expression of the comprehensive evaluation index NDS is:

[0107] ;

[0108] wherein, mAVE is the average precision mean, is a set of detection quality of position, size, orientation, attribute and velocity metrics, is one of the set. The index is calculated using the truth positive (TP) index, half based on detection performance (mAP) and half based on detection performance according to the detection quality of position, size, orientation, attribute and velocity metrics (ATE, ASE, AOE, AVE, AAE). Since mAVE, mAOE and mATE can be greater than 1, each metric can be limited between 0 and 1. The final results are shown in Table 1.

[0109] Table 1: Comparison results of the method of the embodiment and other 3D target detection methods on the nuscenes dataset

[0110]

[0111] As can be seen from Table 1, compared with the existing PETRv2, SOLOFusion, BEVFormerv2 and Streampetr, the average precision mean mAP of the method of the embodiment is respectively 10.7%, 2.9%, 3.2% and 0.6% higher; the average translation error mATE, the average scale error mASE and the average velocity error mAVE are all leading, which proves the superiority of the method of the embodiment.

[0112] In summary, the three-dimensional target detection method based on time sequence multi-view information fusion of the embodiment provides a 3D target detection method under the bird's eye view, which mainly has the following advantages: the method of the embodiment takes high-precision 2D image detection results as spatial anchor points, and constructs a mapping bridge of “pixel-level features-target-level semantics”; after obtaining the 2D detection high-confidence anchor box, the predicted height information and the internal and external parameters of each camera, the target depth can be effectively estimated, and reliable 3D target position initialization can be performed; through the double-path memory structure, the problem of insufficient long-time sequence modeling of the traditional model is solved, and through the MLP fusion mechanism, the complementary enhancement of local and global memory features is realized.

[0113] Furthermore, this embodiment also provides a 3D target detection system based on temporal multi-view information fusion, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the 3D target detection method based on temporal multi-view information fusion. This embodiment also provides a computer-readable storage medium storing a computer program or instructions programmed or configured to execute the 3D target detection method based on temporal multi-view information fusion via a processor. This embodiment also provides a computer program product, including a computer program or instructions programmed or configured to execute the 3D target detection method based on temporal multi-view information fusion via a processor.

[0114] Those skilled in the art will understand that the technical solutions provided by this invention may take the form of a method, system, or computer program product. Therefore, this invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this invention may take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce an implementation of the flowchart... Figure One One or more processes and / or boxes Figure One The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure One One or more processes and / or boxes Figure One The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure One One or more processes and / or boxes Figure One The steps of the function specified in one or more boxes.

[0115] The above merely describes the preferred embodiments of the present application, and the protection scope of the present application is not limited to the above-described embodiments. Any technical solution falling within the concept of the present application shall fall within the protection scope of the present application. It should be noted that, for ordinary skilled persons in the art, some improvements and refinements without departing from the principles of the present application shall also be considered as falling within the protection scope of the present application.

Claims

1. A three-dimensional target detection method based on temporal multi-view information fusion, characterized in that, Includes the following steps: S101, acquire multi-view images captured by the vehicle's camera and construct the image frame sequence at the current time T; S102, Use a feature extraction network to extract image features at the current time T from the image frame sequence at the current time T; S103, using an object detection network to detect image features to obtain the anchor box and height deviation of the two-dimensional object; S104 converts the height deviation of the two-dimensional target into the actual height of the two-dimensional target, and calculates the depth of the two-dimensional target from the camera based on the actual height of the two-dimensional target. ; S105, combine the anchor frame of the two-dimensional target with the depth of the two-dimensional target to the camera. Perform a 2D to 3D projection and add random noise to obtain a noisy 2D height reference point; S106, the noise-added 2D height reference point, the preset fixed reference point, the image features of the current time T, the top k reference points with high confidence from the previous time T-1, and the initial target obtained by fusing from the cached historical query vector are concatenated and input into the pre-trained 3D detection network to obtain the query vector of the current time T output by the 3D detection network. The query vector of the current time T is then regressed and classified to obtain the detection results of the 3D target, including the confidence of the query vector, the anchor box of the 3D target, the category of the 3D target, and the top k reference points with high confidence at the current time T. In step S106, the initialization target obtained by fusing from the cached historical query vectors is implemented through the Global Query Hybridization (GQF) module. The GQF module maintains a global variable and a local variable of different sizes. These global and local variables are used to cache different numbers of historical query vectors with high confidence levels. The initialization target obtained by fusing from the cached historical query vectors includes: S401, Extract the global embedding formed by the cached historical query vectors in the global variables. Local embedding consisting of historical query vectors cached in local variables ; S402, globally embedded and local embedding The fused query is obtained by concatenating and nonlinear mapping. globally embedded and fusion query Enhanced fusion query is obtained through residual weighting. ; S403 will enhance fusion query Copy and trim to the required length L as the initialization target obtained by fusing from the cached historical query vector.

2. The three-dimensional target detection method based on temporal multi-view information fusion according to claim 1, characterized in that, The multi-view image in step S101 includes six viewpoints: front, left front, left back, back, right back, and right front.

3. The three-dimensional target detection method based on temporal multi-view information fusion according to claim 1, characterized in that, In step S102, when the feature extraction network is used to extract the image features of the image frame sequence at the current time T, the extracted image features at the current time T are an image pyramid composed of features at multiple resolution layers. In step S103, the target detection network is used to detect the image features to obtain the anchor box and height deviation of the two-dimensional target, which means: using the highest resolution layer features in the feature pyramid. Input the object detection network, and then process the features at the highest resolution layer through the object detection network. The system regresses the anchor box, height, and confidence level of the two-dimensional target point by point, and selects the anchor box and height with a confidence level exceeding a preset threshold for output. The attributes of the anchor box include the anchor box category, center, and position.

4. The three-dimensional target detection method based on temporal multi-view information fusion according to claim 1, characterized in that, Step S104 includes: S201, the functional expression for converting the height deviation of a two-dimensional target into the actual height of the two-dimensional target is: ； in, For the first The actual height of the two-dimensional target For the first Categories of two-dimensional targets The corresponding prior height, It is the hyperbolic tangent function. For the first The height deviation obtained from the detection of a two-dimensional target; S202, Calculate the depth from the two-dimensional target to the camera based on the actual height of the two-dimensional target. : ； in, The focal length of the camera. The height of the anchor frame for the two-dimensional target.

5. The three-dimensional target detection method based on temporal multi-view information fusion according to claim 1, characterized in that, Step S105 includes: S301, based on the center point of the anchor frame of the two-dimensional target. The horizontal deflection angle of the center point of the two-dimensional target relative to the camera's optical axis is obtained from the camera's intrinsic parameters. and its corresponding direction vector ,in The center point of the anchor frame for the two-dimensional target Coordinates in the camera coordinate system; S302, the horizontal deflection angle of the center point of the two-dimensional target relative to the camera optical axis. Depth of the 2D target from the camera Consider as polar coordinates Recover the 3D coordinates of the 2D target in the camera coordinate system. : ； S303, which sets the 3D coordinates of the 2D target in the camera coordinate system. The three-dimensional coordinates of a point in the world coordinate system are obtained by transforming the extrinsic parameter matrix to the world coordinate system. ; S304, based on the three-dimensional coordinates of a point in the world coordinate system Calculate the normalized coordinates relative to a specified range around the vehicle. Used as the reference point for the obtained two-dimensional height: ， ； ， ； ， ； in, , and These are the three-dimensional coordinates of a point in the world coordinate system. Minimum and maximum values ​​in the x, y, and z axes. , and The three-dimensional coordinates of a point in the world coordinate system The numerical range in the x, y, and z axes; S305 adds or subtracts random numbers to the x-axis and y-axis coordinates of the two-dimensional height reference point to achieve random noise addition, thereby obtaining the noisy two-dimensional height reference point.

6. The three-dimensional target detection method based on temporal multi-view information fusion according to claim 1, characterized in that, In step S402, global embedding will be performed. and local embedding The fused query is obtained by concatenating and nonlinear mapping. globally embedded and fusion query Enhanced fusion query is obtained through residual weighting. The function expression is: ； ； in, It is a multilayer perceptron. For splicing operations, The balance coefficient. The range of values ​​is .

7. A three-dimensional target detection system based on temporal multi-view information fusion, comprising a microprocessor and a memory interconnected, characterized in that, The microprocessor is programmed or configured to execute the three-dimensional target detection method based on temporal multi-view information fusion as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the three-dimensional target detection method based on temporal multi-view information fusion as described in any one of claims 1 to 6.

9. A computer program product, comprising a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the three-dimensional target detection method based on temporal multi-view information fusion as described in any one of claims 1 to 6.

Images