Fusion sensing method, device and equipment based on multiple road test cameras and storage medium

Abstract

The invention provides a fusion sensing method, device and equipment based on multiple road test cameras and a storage medium. The method comprises the following steps: processing a plurality of to-be-processed images based on a pre-constructed encoder to generate an encoder output result, and then carrying out feature convergence on the encoder output result based on a decoder corresponding to the encoder and a preset query vector set to generate a decoder output result, and finally, carrying out perception extraction on an output result of the decoder to obtain a perception result. By designing an encoder-decoder framework which is previously fused as a core feature, multi-camera fusion perception is realized, and target information is extracted from a plurality of to-be-processed images in a targeted manner through a query vector corresponding to a spatial position, so that the information dimension for perceiving and recognizing a target object is enriched, the ill-conditioned inverse transformation of camera projection is avoided, and the image recognition efficiency is improved. The sensing precision is greatly improved, convolutional neural network processing is abandoned, and system computing resource consumption is reduced.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving perception technology, and in particular to a fusion perception method, device, equipment and storage medium based on road test multi-camera. Background Art

[0002] Sensor-based detection and perception is the core information source of vehicle-road collaboration, which determines the application effect of vehicle-road collaboration system's traffic flow statistics, path planning, traffic situation recognition and other functions. Camera-based visual perception is widely used in intelligent driving due to its low cost and high information density.

[0003] At present, the multi-camera perception framework is mainly based on future fusion. After completing the perception detection of a single camera by using the convolutional neural network to output multi-dimensional features, the detected objects of multiple cameras are associated, and the global perception results are output based on the association results.

[0004] However, the object detection in the fusion of the above methods limits the improvement of perception accuracy, reduces the fusion perception accuracy, produces failed association and deduplication, and further reduces the perception performance. Summary of the invention

[0005] The present application provides a fusion perception method, device, equipment and storage medium based on road test multiple cameras, which are used to solve the problems of poor multi-camera perception fusion performance and low accuracy in the prior art.

[0006] In a first aspect, the present application provides a fusion perception method based on drive test multi-camera, comprising:

[0007] Based on a pre-built encoder, a plurality of images to be processed are processed to generate an encoder output result, wherein the encoder is trained by self-supervised learning based on the plurality of images;

[0008] Based on the decoder corresponding to the encoder and a preset query vector set, feature aggregation is performed on the encoder output result to generate a decoder output result;

[0009] The decoder output result is subjected to perception extraction to obtain a perception result, wherein the perception result includes parameters corresponding to at least one traffic object, and the parameters include category, position, heading and identification.

[0010] In combination with the first aspect, in some embodiments, the performing feature aggregation on the encoder output result based on the decoder corresponding to the encoder and a preset query vector set to generate a decoder output result includes:

[0011] Perform decoder autocorrelation feature aggregation on the query vector set to obtain a processed query vector set, wherein the query vector set includes multiple query vectors, each query vector represents a specific position within a camera field of view;

[0012] Based on the processed query vector set, cross-correlation feature aggregation is performed on the encoder output result to generate the decoder output result.

[0013] In combination with the first aspect, in some embodiments, performing decoder cross-correlation feature aggregation on the encoder output result based on the processed query vector set to generate the decoder output result includes:

[0014] For each image to be processed, based on the internal and external parameters of the camera corresponding to the image to be processed, determining a reference point of each query vector in the processed query vector set in the image to be processed;

[0015] According to all reference points and the encoder output result, the image features in each image to be processed are aggregated and extracted by a preset cross-correlation formula to obtain the decoder output result.

[0016] In combination with the first aspect, in some embodiments, determining a reference point of each query vector in the processed query vector set in the image to be processed based on the internal and external parameters of the camera corresponding to the image to be processed includes:

[0017] For each image to be processed, construct a posture projection matrix of the camera according to the internal and external parameters of the camera corresponding to the image to be processed;

[0018] A reference point of each query vector in the image to be processed is calculated according to the pose projection matrix and each query vector.

[0019] In combination with the first aspect, in some embodiments, the processing of multiple images to be processed based on a pre-built encoder to generate an encoder output result includes:

[0020] For each image to be processed, performing block division preprocessing on the image to be processed to obtain a plurality of image block sequences corresponding to the image to be processed;

[0021] For each image block sequence, encoder autocorrelation features of the image block sequence are aggregated using a preset autocorrelation formula to obtain a feature vector corresponding to the image block sequence;

[0022] The feature vectors corresponding to all image block sequences are aggregated to obtain the encoder output result.

[0023] In combination with the first aspect, in some embodiments, performing block division preprocessing on the image to be processed to obtain a plurality of image block sequences corresponding to the image to be processed includes:

[0024] For each image to be processed, the image to be processed is divided into blocks to obtain a plurality of image blocks corresponding to the image to be processed;

[0025] Embedding coding is performed on each image block corresponding to the image to be processed and position coding is superimposed to obtain a plurality of image block sequences corresponding to the image to be processed.

[0026] In combination with the first aspect, in some embodiments, the method further includes:

[0027] Divide the union of the field of view of the camera corresponding to each image to be processed into a grid;

[0028] According to the coordinates of each grid after grid division, determine the query vector corresponding to each grid;

[0029] All query vectors are aggregated to obtain the query vector set.

[0030] In a second aspect, the present application provides a fusion perception device based on a road test multi-camera, comprising:

[0031] An encoding module, used for processing a plurality of images to be processed based on a pre-built encoder to generate an encoder output result, wherein the encoder is obtained by self-supervised learning training based on the plurality of images;

[0032] A decoding module, configured to perform feature aggregation on the encoder output result based on a decoder corresponding to the encoder and a preset query vector set to generate a decoder output result;

[0033] The perception module is used to perform perception extraction on the decoder output result to obtain a perception result, wherein the perception result includes parameters corresponding to at least one traffic object, and the parameters include category, position, heading and identification.

[0034] In conjunction with the second aspect, in some embodiments, the decoding module includes:

[0035] A first aggregation unit is used to perform decoder autocorrelation feature aggregation on the query vector set to obtain a processed query vector set, wherein the query vector set includes multiple query vectors, each query vector represents a specific position within the camera field of view;

[0036] The second aggregation unit is used to perform cross-correlation feature aggregation on the encoder output result based on the processed query vector set to generate the decoder output result.

[0037] In conjunction with the second aspect, in some embodiments, the second convergence unit includes:

[0038] A determination subunit, configured to determine, for each image to be processed, a reference point of each query vector in the processed query vector set in the image to be processed based on internal and external parameters of a camera corresponding to the image to be processed;

[0039] The convergence subunit is used to aggregate and extract the image features in each image to be processed according to all reference points and the encoder output results through a preset cross-correlation formula to obtain the decoder output result.

[0040] In conjunction with the second aspect, in some embodiments, the determining subunit is specifically used to:

[0041] For each image to be processed, construct a posture projection matrix of the camera according to the internal and external parameters of the camera corresponding to the image to be processed;

[0042] A reference point of each query vector in the image to be processed is calculated according to the pose projection matrix and each query vector.

[0043] In conjunction with the second aspect, in some embodiments, the encoding module includes:

[0044] A preprocessing unit, configured to perform block division preprocessing on each image to be processed to obtain a plurality of image block sequences corresponding to the image to be processed;

[0045] A convergence unit is used to converge the encoder autocorrelation features of each image block sequence by a preset autocorrelation formula to obtain a feature vector corresponding to the image block sequence;

[0046] The summarizing unit is used to summarize the feature vectors corresponding to all image block sequences to obtain the encoder output result.

[0047] In conjunction with the second aspect, in some embodiments, the preprocessing unit includes:

[0048] A division subunit is used to divide each image to be processed into blocks to obtain a plurality of image blocks corresponding to the image to be processed;

[0049] The encoding subunit is used to perform embedded encoding and superimposed position encoding on each image block corresponding to the image to be processed, so as to obtain a plurality of image block sequences corresponding to the image to be processed.

[0050] In conjunction with the second aspect, in some embodiments, the apparatus further includes:

[0051] A partitioning module, used for partitioning the union of the field of view of each camera corresponding to each image to be processed into a grid;

[0052] Determination module: the user determines the query vector corresponding to each grid according to the coordinates of each grid after grid division;

[0053] The aggregation module is used to aggregate all query vectors to obtain the query vector set.

[0054] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0055] The memory stores computer-executable instructions;

[0056] The processor executes the computer-executable instructions stored in the memory to implement the fusion perception method based on drive test multiple cameras as described in any one of the first aspects.

[0057] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the fusion perception method based on road test multi-camera as described in any of the aforementioned aspects.

[0058] In a fifth aspect, the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the fusion perception method based on road test multiple cameras as described in any of the aforementioned aspects.

[0059] The fusion perception method, device, equipment and storage medium based on road test multi-camera provided by the present application processes multiple images to be processed based on a pre-built encoder to generate encoder output results, and then performs feature aggregation on the encoder output results based on the decoder corresponding to the encoder and a pre-set query vector set to generate decoder output results, and finally performs perception extraction on the decoder output results to obtain perception results. By designing an encoder-decoder framework with previous fusion as the core feature, multi-camera fusion perception is achieved, and through query vectors corresponding to spatial positions, target information is extracted from multiple images to be processed in a targeted manner, enriching the information dimension of the perceived target object, directly extracting 3D perception positioning information from specific query vectors, avoiding the ill-conditioned inverse transformation of camera projection, effectively improving perception accuracy, abandoning convolutional neural network processing, and reducing system computing resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0061] Figure 1 A flowchart of a first embodiment of a fusion perception method based on multi-camera road test provided in an embodiment of the present application;

[0062] Figure 2 A flowchart of a second embodiment of a fusion perception method based on a multi-camera drive test provided in an embodiment of the present application;

[0063] Figure 3 A flowchart of a third embodiment of a fusion perception method based on a drive test multi-camera provided in an embodiment of the present application;

[0064] Figure 4 A flowchart of a fourth embodiment of a fusion perception method based on drive test multi-cameras provided in an embodiment of the present application;

[0065] Figure 5 A flowchart of a fifth embodiment of a fusion perception method based on drive test multi-cameras provided in an embodiment of the present application;

[0066] Figure 6 A flowchart of a sixth embodiment of a fusion perception method based on drive test multi-cameras provided in an embodiment of the present application;

[0067] Figure 7 A flowchart of Embodiment 7 of the fusion perception method based on drive test multi-camera provided in the embodiment of the present application;

[0068] Figure 8 A schematic diagram of the structure of a first embodiment of a fusion perception device based on a road test multi-camera according to an embodiment of the present application;

[0069] Fig. 9 A schematic diagram of the structure of a second embodiment of a fusion perception device based on a road test multi-camera provided in an embodiment of the present application;

[0070] Fig.10 A schematic diagram of the structure of a third embodiment of a fusion perception device based on a road test multi-camera according to an embodiment of the present application;

[0071] Fig.11 A schematic diagram of the structure of a fourth embodiment of a fusion perception device based on a drive test multi-camera according to an embodiment of the present application;

[0072] Fig.12 A structural diagram of a fifth embodiment of a fusion perception device based on a road test multi-camera provided in an embodiment of the present application;

[0073] Fig.13 A structural diagram of a sixth embodiment of a fusion perception device based on a road test multi-camera provided in an embodiment of the present application;

[0074] Fig.14A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0075] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0076] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0077] Sensor-based detection and perception is the core information source of vehicle-road collaboration, and determines the application effects of functions such as traffic flow statistics, path planning, and traffic situation recognition in the vehicle-road collaboration system. Camera-based visual perception is widely used in intelligent driving due to its low cost and high information density. At present, the multi-camera perception framework is mainly based on post-fusion. After completing the perception detection of a single camera using the convolutional neural network to output multi-dimensional features, the detected objects of multiple cameras are associated, and the global perception results are output based on the association results. However, the object detection in the fusion of the above method limits the improvement of perception accuracy, and the association and deduplication of multiple cameras have large position errors due to the lack of information, which reduces the fusion perception accuracy, resulting in failed association and deduplication, and further reduces the perception performance.

[0078] In response to the above problems, the present application provides a fusion perception method, device, equipment and storage medium based on road test multi-camera, which improves the performance and accuracy of multi-camera fusion perception. Specifically, the current multi-camera perception framework is mainly based on post-fusion. After completing the perception detection of a single camera by using a convolutional neural network to output multi-dimensional features, the detected objects of multiple cameras are associated, and the global perception results are output based on the association results. Object detection in post-fusion does not use the information of all sensors, which limits the improvement of perception accuracy; the association and deduplication of post-fusion multi-cameras first need to obtain the world coordinates through the reverse projection of pixels for the two-dimensional detection information. Due to the lack of information, this is a pathological process with a large position error, which reduces the post-fusion perception accuracy, resulting in failed association and deduplication, and further reduces the perception performance. If the 3D detection target is output by using the Bird-Eye View (BEV) fusion, although the method adopts a multi-camera information aggregation strategy during target detection, thereby avoiding the pathological inverse transformation of pixel coordinates to world coordinates, the method flow relies on the image features extracted by the convolutional neural network, and the encoder-decoder framework using the attention mechanism is used for processing based on the image features, and when extracting information from the camera image, it can only rely on the calibrated internal and external parameters of the camera projection. In view of the above problems, the inventor studied whether it is possible to use the information of multiple cameras at the same time to detect the target, improve the perception detection performance and accuracy, and avoid the performance loss caused by processing links such as deduplication and association. Since the 3D perception information is directly output, there is no pathological processing of the camera inverse transformation, which avoids the perception positioning error introduced by this step, and adopts the Masked Autoencoders (MAE) mechanism, and uses a large number of unlabeled pictures to pre-train the encoder to improve the information aggregation representation ability of the encoder. Based on this, the technical solution of the present application is proposed.

[0079] The fusion perception method based on road test multi-camera provided in the embodiment of the present application can be applied to intelligent transportation systems, autonomous driving technology, security monitoring and other fields. For example, in the field of autonomous driving technology, the method can be used for environmental perception: in an autonomous driving vehicle, the road test multi-camera system can achieve comprehensive perception of the surrounding environment by fusing camera data from different angles and positions. This includes the detection and identification of road signs, pedestrians, vehicles, obstacles and other traffic participants. By fusing data from different cameras, the problem of limited field of view of a single camera can be compensated, and the accuracy and robustness of environmental perception can be improved. Obstacle detection and avoidance: The camera fusion perception system can detect obstacles on the road in real time and evaluate their distance, speed and motion trajectory. This is crucial for autonomous driving vehicles and can help vehicles make obstacle avoidance decisions in a timely manner to ensure driving safety. Path planning and navigation: By fusing multi-camera data, autonomous driving vehicles can build a more accurate environmental model for path planning and navigation. This helps vehicles choose the optimal driving route and cope with complex traffic conditions.

[0080] Optionally, in the autonomous driving vehicle environment perception scenario, the vehicle needs to perceive the surrounding environment in real time to make correct decisions. This scenario includes various road types such as urban roads, highways, and complex intersections. Autonomous driving vehicles capture image information of the surrounding environment through multiple cameras installed around the vehicle body (such as front cameras, side cameras, rear cameras, and surround cameras, etc.). Then, the images collected by multiple cameras are forward-fused and perceived based on the attention encoder-decoder mechanism through the vehicle main controller or external devices, so as to obtain the complete parameters corresponding to the target traffic object, such as category, logo, heading, position, etc. Then, the path can be planned based on the perception results obtained.

[0081] It should be noted that the above-mentioned target traffic objects may be pedestrians, vehicles, traffic signs, etc., and this application does not make any specific limitations.

[0082] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0083] Figure 1 The flowchart of the first embodiment of the fusion perception method based on multi-camera road test provided in the embodiment of the present application is as follows: Figure 1 As shown, the method specifically includes:

[0084] S101: Based on a pre-built encoder, multiple images to be processed are processed to generate encoder output results.

[0085] In this step, in order to simultaneously utilize camera image information from different directions, improve the information content during target detection, enhance perception performance and positioning accuracy, reduce computational load, and improve resource utilization efficiency, a complete encoder-decoder attention architecture is used to process multiple images to be processed corresponding to multiple cameras.

[0086] Specifically, an encoder is obtained by performing self-supervised learning training on an initial encoder constructed based on multiple images, and then multiple images to be processed are input into the encoder for processing to obtain an encoder output result.

[0087] Optionally, for each image to be processed, the image to be processed is preprocessed by block division to obtain multiple image block sequences corresponding to the image to be processed. For each image block sequence, the image block sequence is encoder autocorrelation features are converged by a preset autocorrelation formula to obtain a feature vector corresponding to the image block sequence. The feature vectors corresponding to all image block sequences are aggregated to obtain the encoder output result.

[0088] In one possible implementation, the encoder can use the MAE pre-training strategy and use a large number of unlabeled images for pre-training to improve the encoder's ability to express aggregate information. In the pre-training stage, a portion of the image block area is randomly discarded, for example, the discard ratio is 75%, and the pre-training is performed to reconstruct the discarded blocks, so that a large amount of unlabeled image data can be used.

[0089] S102: Based on the decoder corresponding to the encoder and a preset query vector set, feature aggregation is performed on the encoder output result to generate a decoder output result.

[0090] In this step, after the encoder output result is obtained, the encoder output result is input into a decoder corresponding to the encoder for processing, thereby obtaining a decoder output result.

[0091] Specifically, a query vector set is pre-set, decoder autocorrelation feature aggregation is performed on the query vector set to obtain a processed query vector set, and based on the processed query vector set, cross-correlation feature aggregation is performed on the encoder output result to generate a decoder output result.

[0092] S103: Perform perceptual extraction on the decoder output result to obtain a perceptual result.

[0093] In this step, after the decoder output result is obtained, the subsequent perceptron is used to process and extract the perception result, wherein the perception result includes at least one parameter corresponding to the traffic object, and the parameters include category, location, heading and identification.

[0094] Optionally, after obtaining the decoder output result, it can be further processed and converted through a series of fully connected layers or convolutional layers, or through a target detection algorithm or a target tracking algorithm to obtain a perception result.

[0095] The fusion perception method based on road test multi-camera provided in this embodiment processes multiple images to be processed based on a pre-built encoder to generate encoder output results, and then performs feature aggregation on the encoder output results based on the decoder corresponding to the encoder and a pre-set query vector set to generate decoder output results, and finally performs perception extraction on the decoder output results to obtain perception results. By designing an encoder-decoder framework with fusion as the core feature, multi-camera fusion perception is achieved, and through query vectors corresponding to spatial positions, target information is extracted from multiple images to be processed in a targeted manner, enriching the information dimension of the perceived target object, and directly extracting 3D perception positioning information from specific query vectors, avoiding the ill-conditioned inverse transformation of camera projection, effectively improving perception accuracy, abandoning convolutional neural network processing, and reducing system computing resource consumption.

[0096] Figure 2 The flowchart of the second embodiment of the fusion perception method based on the road test multi-camera provided in the embodiment of the present application is as follows: Figure 2 As shown, based on the above embodiment, step S102 specifically includes:

[0097] S201: performing decoder autocorrelation feature aggregation on a query vector set to obtain a processed query vector set.

[0098] S202: Based on the processed query vector set, perform cross-correlation feature aggregation on the encoder output results to generate a decoder output result.

[0099] The decoder is relative to the encoder in the aforementioned embodiment, so the decoder also needs to perform local information autocorrelation processing. In order to provide perceptual accuracy, a mutual correlation feature aggregation process is set in the decoder, and the decoder performs autocorrelation feature aggregation on the query vector set to obtain a processed query vector set.

[0100] Specifically, self-attention is used to capture the dependencies between different positions within the query vector set.

[0101] After obtaining the processed query vector set, the decoder output result is obtained by performing cross-correlation processing on the encoder output result and the processed query vector set.

[0102] Specifically, based on the internal and external parameters of the camera corresponding to each image to be processed, the reference point of each query vector in the processed query vector set in each image to be processed is determined, and then according to all reference points and the encoder output results, the image features in each image to be processed are aggregated and extracted through a preset cross-correlation formula to obtain the decoder output result.

[0103] Optionally, the decoder corresponds to the encoder structure, and both the self-attention module and the mutual attention module can be connected to the feedforward network for processing to improve the output result. In practical applications, the decoder can also be set up with multi-layer processing, and the processing result of the last layer is determined as the decoder output result.

[0104] The fusion perception method based on multi-camera road test provided in this embodiment performs decoder autocorrelation feature aggregation on the query vector set to obtain a processed query vector set, and based on the processed query vector set, performs cross-correlation feature aggregation on the encoder output result to generate the decoder output result. By designing the query vector corresponding to a specific spatial area, the specific image part information is accurately extracted, and the 3D positioning coordinates are directly output, eliminating the ambiguity of the reverse projection of the pixel to the world coordinate system, and further improving the perception positioning accuracy.

[0105] Figure 3 The flowchart of the third embodiment of the fusion perception method based on the road test multi-camera provided in the embodiment of the present application is as follows: Figure 3 As shown, based on the above embodiment, step S202 specifically includes:

[0106] S301: For each image to be processed, based on the internal and external parameters of the camera corresponding to the image to be processed, determine the reference point of each query vector in the processed query vector set in the image to be processed.

[0107] In this step, in order to eliminate the ambiguity of the reverse projection of pixels to the world coordinate system, the reference point of each query vector in the processed query vector set in the image to be processed can be determined based on the intrinsic and extrinsic parameters of the camera corresponding to the image to be processed.

[0108] Specifically, for each image to be processed, the camera's pose projection matrix is ​​constructed according to the internal and external parameters of the camera corresponding to the image to be processed, and the reference point of each query vector in the image to be processed is calculated according to the pose projection matrix and each query vector.

[0109] S302: Based on all reference points and encoder output results, the image features in each image to be processed are aggregated and extracted by a preset cross-correlation formula to obtain a decoder output result.

[0110] In this step, the pixel coordinates of the query vector obtained after the projection transformation are its parameter points in the image to be processed. In order to aggregate local image feature information, deformable multi-head mutual attention is used to perform image feature aggregation extraction to obtain the decoder output result.

[0111] Specifically, a set of image region translation and weight parameters for extracting information are obtained by linear transformation. The weight parameters are normalized for the rationality of numerical distribution. Due to the block feature description of the image, the image features are discretely distributed in position. When the decoder information is locally converged, bilinear interpolation is used to obtain the offset target position image features.

[0112] Exemplarily, for all query vectors, a 256×40n linear mapping is used to obtain the offset and weight of the deformable multi-head mutual attention. The mutual attention also uses 8 heads to aggregate and extract information from the local 4 encoder image features. Therefore, for any query vector in any image to be processed, 2×4 position bias parameters and 8×4 weight parameters are set.

[0113] The cross-correlation formula is expressed as:

[0114]

[0115] Among them, q j represents the j-th query vector, V represents the encoder output, N sensor Indicates the number of positions corresponding to the query vectors included in different camera fields of view, N head Indicates the number of heads, N value represents the number of pre-set sequence elements, W m Represents the linear transformation weight coefficient, W n represents the linear transformation matrix, A mn represents the mutual attention weight parameter, represents the reference point corresponding to the j-th query vector, Indicates the offset coordinates.

[0116] The fusion perception method based on road test multi-camera provided in this embodiment determines the reference point of each query vector in the processed query vector set in the image to be processed based on the internal and external parameters of the camera corresponding to the image to be processed for each image to be processed, and aggregates and extracts the image features in each image to be processed through a preset cross-correlation formula according to all reference points and the encoder output results to obtain the decoder output result. Through the above method, the reference point of the query vector in the image is determined based on the internal and external parameters of the camera corresponding to the image to be processed, and the image features are aggregated and extracted through a preset cross-correlation formula, which has the advantages of improving positioning accuracy, enhancing feature aggregation effect, improving stability and reliability, and improving processing efficiency for the decoder output result.

[0117] Figure 4 The flowchart of the fourth embodiment of the fusion perception method based on the road test multi-camera provided in the embodiment of the present application is as follows: Figure 4 As shown, based on the above embodiments, step S301 specifically includes:

[0118] S401: For each image to be processed, construct a camera posture projection matrix according to the internal and external parameters of the camera corresponding to the image to be processed.

[0119] In this step, in order to eliminate the ambiguity of the reverse projection of pixels to the world coordinate system, a pose projection matrix is ​​constructed for the camera corresponding to each image to be processed.

[0120] Specifically, for each camera, the learnable tensor P is set according to the internal and external parameters of the camera. i , and then according to P i Construct the pose projection matrix.

[0121] Among them, P i =[p i r i y i x i y i z i ], the pose projection matrix is ​​expressed as:

[0122]

[0123] S402: Calculate the reference point of each query vector in the image to be processed according to the pose projection matrix and each query vector.

[0124] In this step, after the pose projection matrix corresponding to each camera is obtained, each query vector is mapped to each image to be processed based on the matrix to obtain the reference point of each query vector in each image to be processed.

[0125] Specifically, the reference point is determined by the following formula:

[0126]

[0127] Among them, (u ij , v ij ) represents the reference point of the j-th query vector in the i-th camera. (x qj ,y qj , z qj ) represents the j-th query vector.

[0128] It should be noted that the internal and external parameters of each of the above-mentioned cameras are learnable settings, that is, the learnable internal and external parameters of the camera projection obtained through training, which reshapes the construction process of the vehicle-road cooperative system, eliminates the camera calibration link, and reduces the system development cost.

[0129] Optionally, the camera internal parameters can include focal length: The focal length is the distance from the center of the lens to the imaging plane, which determines the size of the camera's viewing angle and the magnification of the image. Pixel size: The pixel size describes the physical size of each pixel on the camera sensor, affecting the resolution and clarity of the image. Optical center: The optical center is the center point of the camera lens and the intersection of the imaging light rays. In image processing, the optical center is usually used as the origin of the image coordinate system. Distortion coefficient: Due to manufacturing and installation errors of the camera lens, the image may produce radial distortion and tangential distortion. The distortion coefficient is used to describe the magnitude and direction of these distortions so that they can be corrected during image processing.

[0130] The camera external parameters can include rotation matrix: The rotation matrix is ​​used to represent the rotation relationship between the camera coordinate system and the world coordinate system. For example, it can be a 3×3 matrix, which can be calculated by the rotation angles (Euler angles or quaternions) around the x, y, and z axes. Translation vector: The translation vector represents the position of the origin of the camera coordinate system in the world coordinate system. For example, it can be a three-dimensional vector, usually represented as [x, y, z].

[0131] In order to limit the distribution area of ​​camera parameters and improve the regression accuracy of learnable camera parameters, 6-dimensional parameters are used to represent the camera projection model, including 3-dimensional rotation parameters and 3-dimensional translation parameters.

[0132] The fusion perception method based on road test multi-camera provided in this embodiment, for each image to be processed, constructs the camera's pose projection matrix according to the internal and external parameters of the camera corresponding to the image to be processed, and calculates the reference point of each query vector in the image to be processed according to the pose projection matrix and each query vector. Through the above method, the camera's pose projection matrix is ​​constructed according to the internal and external parameters of the camera corresponding to the image to be processed, and the reference point of each query vector in the image to be processed is calculated accordingly, which has accurate spatial mapping, improves the accuracy of feature extraction, enhances the robustness of the system, and improves processing efficiency.

[0133] Figure 5 The flowchart of the fifth embodiment of the fusion perception method based on the road test multi-camera provided in the embodiment of the present application is as follows: Figure 5 As shown, based on the above embodiments, step S101 specifically includes:

[0134] S501: For each image to be processed, perform block division preprocessing on the image to be processed to obtain a plurality of image block sequences corresponding to the image to be processed.

[0135] In this step, in order to process the image to be processed more efficiently and to accurately extract specific feature information, each image to be processed may be pre-processed by dividing the image into blocks.

[0136] Specifically, for each image to be processed, the image to be processed is divided into blocks to obtain multiple image blocks corresponding to the image to be processed, and each image block corresponding to the image to be processed is embedded and coded and position-coded to obtain a sequence of multiple image blocks corresponding to the image to be processed.

[0137] S502: for each image block sequence, perform encoder autocorrelation feature aggregation on the image block sequence using a preset autocorrelation formula to obtain a feature vector corresponding to the image block sequence.

[0138] In this step, after obtaining multiple image block sequences corresponding to each image to be processed, in order to capture the global and local feature information in each image block, the encoder's self-attention module is used to aggregate the encoder's autocorrelation features, thereby obtaining a feature vector corresponding to the image block sequence.

[0139] Exemplarily, for each image block sequence, a linear transformation is first used to calculate the offset and weight of the deformed multi-head attention. For example, 8 heads are used, and a 256×40 linear mapping is used. The linear layer output includes 2×4 image block offset coordinates and 8×4 weight parameters. For the encoder deformed multi-head self-attention processing, each image block sequence and the four sequence elements complete attention convergence based on correlation. Each image block sequence corresponds to two offset coordinates in the image to be processed. Since 8-head attention is used, combined with the convergence of feature information of the four sequence elements, each head has 4 weights, a total of 8×4 weight parameters. Here, the 4 weights of each head need to be normalized to optimize the distribution of the number of encoder sequence elements. The autocorrelation formula can be expressed as:

[0140]

[0141] Among them, OutDetreAttn(q i , X) represents the output result of deformed multi-head self-attention, q i represents the i-th element in the image block sequence X, where X represents the image block sequence, and N head Indicates the number of heads, N value represents the number of pre-set sequence elements, W j Represents the linear transformation weight coefficient, W k represents the linear transformation matrix, A jk represents the self-attention weight parameter, Δp k Indicates offset coordinates

[0142] It should be noted that, since the above self-attention processing is a local correlation processing, the above four sequence elements are pre-set and used to perform local feature aggregation on the image block sequence. The above four are just an example. The number of pre-set sequence elements in actual scene processing can be freely set, and the embodiment of this application does not make specific limitations.

[0143] S503: Summarize the feature vectors corresponding to all image block sequences to obtain the encoder output result.

[0144] After obtaining the feature vectors corresponding to all image block sequences, the encoder output results can be summarized.

[0145] Optionally, after obtaining the feature vector corresponding to each image block sequence in the above step S502, in order to increase nonlinear capabilities, feature mapping and integration, and improve model performance, it can also be processed through a feedforward network, wherein both the multi-head self-attention module and the feedforward network use residual connections to increase the convergence speed of model training and avoid gradient disappearance; after the multi-head self-attention module and the feedforward network are residually connected, both layer normalization is performed to concentrate the distribution magnitude of the data. Feedforward network processing includes linear network transformation and nonlinear activation processing, and the specific implementation method is not specifically limited in the embodiments of this application.

[0146] In one possible implementation, the deformation self-attention processing performed in the above steps is a layer of multi-head self-attention module processing. In practical applications, multiple layers of multi-head self-attention module processing can be designed and constructed, that is, the output of the last layer is the encoder output result. It should be noted that each layer of multi-head self-attention module can include deformation head self-attention processing and feedforward network processing.

[0147] The fusion perception method based on road test multi-camera provided in this embodiment performs block division preprocessing on each image to be processed to obtain multiple image block sequences corresponding to the image to be processed, and performs encoder autocorrelation feature aggregation on the image block sequence through a preset autocorrelation formula to obtain a feature vector corresponding to the image block sequence, and summarizes the feature vectors corresponding to all image block sequences to obtain the encoder output result. Through the above method, the computational complexity is reduced, the feature extraction accuracy is improved, the feature representation richness is enhanced, the system flexibility and scalability are improved, and large-scale data set processing is supported.

[0148] Figure 6 This is a flow chart of Embodiment 6 of the fusion perception method based on drive test multi-camera provided in the embodiment of the present application, as shown in FIG. Figure 6 As shown, based on the above embodiments, step S501 specifically includes:

[0149] S601: For each image to be processed, divide the image to be processed into blocks to obtain a plurality of image blocks corresponding to the image to be processed.

[0150] In this step, in order to simultaneously utilize camera image information from different directions, improve the information content during target detection, enhance perception performance and positioning accuracy, better capture local features of the image, and reduce computational complexity, each image to be processed can be divided into multiple blocks for processing.

[0151] Specifically, for each image to be processed, it is divided into l×k blocks that are connected but not intersecting, where l represents the number of intervals divided by the x-axis of the image, and k represents the number of intervals divided by the y-axis of the image.

[0152] Exemplarily, at time t, for n images to be processed from n cameras, each image to be processed is divided into 16×16 blocks.

[0153] S602: Perform embedding coding on each image block corresponding to the image to be processed and superimpose position coding to obtain a plurality of image block sequences corresponding to the image to be processed.

[0154] In this step, in order to improve the expression performance of the encoder, each image block obtained in the above step can be subjected to embedded learnable encoding.

[0155] Specifically, for each image block, a convolutional layer with the same dimension as the image block pixels is used to implement the learnable embedded coding of the image block. The convolutional layer output tensor is flattened into a 1-dimensional vector as the embedded coding of the image block. At the same time, the vector is normalized and the position coding is superimposed on the image block embedded coding to distinguish the different position orders of different image blocks, where the dimension of the position coding vector is the same as the dimension of the embedded coding vector.

[0156] Exemplarily, at time t, for n images to be processed from n cameras, each image to be processed is divided into 16×16 blocks as an example, and a convolutional network with a kernel dimension of 28×28, an input channel of 3, an output channel of 256, a padding of 0, and a step size of 28 is used for processing. This network is actually equivalent to completing the embedded coding of the image block. Unitize the 256 elements of the image block sequence. The encoder position encoding is an embedding, and its embedding parameters are all 256. Each vector of the image block sequence is added to the unitized vector of the corresponding position of the position encoding to complete the position encoding. Since the position encoding parameters are randomly initialized, there is no position information at the beginning. As the training gradually converges, the position encoding parameters characterize the corresponding order of the image blocks.

[0157] Optionally, the above-mentioned regularization can be L2 regularization, weight regularization, etc. The embodiment of the present application does not specifically limit the specific implementation method.

[0158] The fusion perception method based on road test multi-camera provided in this embodiment divides each image to be processed into blocks to obtain multiple image blocks corresponding to the image to be processed, and embeds and superimposes position codes on each image block corresponding to the image to be processed to obtain a sequence of multiple image blocks corresponding to the image to be processed. Through the above method, embedded learnable coding based on image block division, combined with the encoder-decoder architecture, can use masked self-learning technology, use massive unlabeled images, pre-train the encoder, improve the encoder expression performance, and improve the fusion perception accuracy.

[0159] Figure 7 The flowchart of the seventh embodiment of the fusion perception method based on the road test multi-camera provided in the embodiment of the present application is as follows: Figure 7 As shown, based on the above embodiments, the method further includes:

[0160] S701: Divide the union of the field of view of each camera corresponding to each image to be processed into a grid.

[0161] S702: Determine a query vector corresponding to each grid according to the coordinates of each grid after grid division.

[0162] S703: Aggregate all query vectors to obtain a query vector set.

[0163] In order to avoid the pathological inverse transformation of camera projection and perform feature extraction and fusion more accurately, a query vector corresponding to the spatial position can be designed to extract target information from multiple images to be processed in a targeted manner, enriching the information dimension of perceiving and identifying the target object, and directly extracting 3D perception positioning information from a specific query vector.

[0164] Specifically, collect the field of view (FOV) information of the camera corresponding to each image to be processed, and then take the union of all FOV information to obtain a union area that can cover the field of view of all cameras. For example, there are 3 images to be processed, corresponding to 3 cameras, camera A, camera B and camera C, the FOV information of camera A is area 1, the FOV information of camera B is area 2, and the FOV information of camera C is area 3. The obtained union area is {area 1, area 2, area 3}. After obtaining the union area, the union area is gridded, wherein the grid division can be based on pixels, spatial coordinates or other units of measurement, and the embodiment of the present application is not specifically limited. The size and shape of the divided grid can also be determined according to the actual scene. For each grid, the world coordinates of its center point are calculated and recorded, and a unique query vector is assigned to each grid, and then all query vectors are summarized to obtain a query vector set.

[0165] The query vector is a set of learnable embedding vectors corresponding to specific locations in space, which are used to refer to the location coordinates of the corresponding grid.

[0166] For example, for the area covered by n cameras, assuming it is 50m×100m, it is divided into 20,000 grids at intervals of 0.5m both horizontally and vertically (in the specific implementation process, since the target area of ​​interest is generally the road and the nearby sidewalks, it can be divided more specifically and finely to reduce the amount of calculation), so the dimension of the query vector set is 20,000×256, where each query vector corresponds to a specific grid.

[0167] The fusion perception method based on road test multi-camera provided in this embodiment divides the union of the field of view of each camera corresponding to each image to be processed into a grid, determines the query vector corresponding to each grid according to the coordinates of each grid after grid division, and summarizes all query vectors to obtain a query vector set. Through the above method, the query vector design corresponding to a specific spatial area accurately extracts the information of a specific image part, directly outputs the 3D positioning coordinates, eliminates the ambiguity of the reverse projection of pixels to the world coordinate system, and further improves the perception positioning accuracy.

[0168] Figure 8 This is a structural diagram of a first embodiment of a fusion sensing device based on a road test multi-camera according to an embodiment of the present application, as shown in FIG. Figure 8 As shown, the fusion perception device 800 based on the road test multi-camera specifically includes:

[0169] The encoding module 801 is used to process multiple images to be processed based on a pre-built encoder to generate an encoder output result, and the encoder is obtained by self-supervised learning training based on multiple images.

[0170] The decoding module 802 is used to perform feature aggregation on the encoder output result based on the decoder corresponding to the encoder and a preset query vector set to generate a decoder output result.

[0171] The perception module 803 is used to perform perception extraction on the decoder output result to obtain a perception result, which includes parameters corresponding to at least one traffic object, including category, location, heading and identification.

[0172] Fig. 9 This is a structural diagram of a second embodiment of a fusion sensing device based on a road test multi-camera according to an embodiment of the present application, as shown in FIG. Fig. 9 As shown, based on the above embodiment, the decoding module 802 includes:

[0173] The first aggregation unit 901 is used to perform decoder autocorrelation feature aggregation on the query vector set to obtain a processed query vector set, where the query vector set includes multiple query vectors, each query vector representing a specific position within the camera field of view.

[0174] The second aggregation unit 902 is used to perform cross-correlation feature aggregation on the encoder output results based on the processed query vector set to generate a decoder output result.

[0175] Fig.10 This is a structural diagram of Embodiment 3 of the fusion perception device based on road test multi-camera provided in the embodiment of the present application, such as Fig.10 As shown, based on the above embodiment, the second convergence unit 902 specifically includes:

[0176] The determination subunit 1001 is used to determine, for each image to be processed, a reference point of each query vector in the processed query vector set in the image to be processed based on the intrinsic and extrinsic parameters of the camera corresponding to the image to be processed.

[0177] The convergence subunit 1002 is used to aggregate and extract the image features in each image to be processed according to all reference points and the encoder output results through a preset cross-correlation formula to obtain a decoder output result.

[0178] Optionally, the determination subunit 1001 is specifically configured to:

[0179] For each image to be processed, the camera's pose projection matrix is ​​constructed according to the internal and external parameters of the camera corresponding to the image to be processed;

[0180] The reference point of each query vector in the image to be processed is calculated according to the pose projection matrix and each query vector.

[0181] Fig.11 This is a structural diagram of a fourth embodiment of a fusion sensing device based on a road test multi-camera according to an embodiment of the present application, as shown in FIG. Fig.11 As shown, based on the above embodiments, the encoding module 801 specifically includes:

[0182] The preprocessing unit 1101 is used for performing block division preprocessing on each image to be processed to obtain a plurality of image block sequences corresponding to the image to be processed.

[0183] The aggregation unit 1102 is used to aggregate the encoder autocorrelation features of each image block sequence through a preset autocorrelation formula to obtain a feature vector corresponding to the image block sequence.

[0184] The summarizing unit 1103 is used to summarize the feature vectors corresponding to all the image block sequences to obtain the encoder output result.

[0185] Fig.12 This is a structural diagram of a fifth embodiment of a fusion sensing device based on a road test multi-camera according to an embodiment of the present application, as shown in FIG. Fig.12 As shown, based on the above embodiments, the preprocessing unit 1101 specifically includes:

[0186] The division subunit 1201 is used to divide each image to be processed into blocks to obtain a plurality of image blocks corresponding to the image to be processed.

[0187] The encoding subunit 1202 is used to perform embedded encoding and superimpose position encoding on each image block corresponding to the image to be processed, so as to obtain a plurality of image block sequences corresponding to the image to be processed.

[0188] Fig.13 This is a structural diagram of a sixth embodiment of a fusion sensing device based on a road test multi-camera according to an embodiment of the present application, as shown in FIG. Fig.13 As shown, based on the above embodiments, the fusion perception device 800 based on the road test multi-camera further includes:

[0189] The division module 1301 is used to divide the union of the field of view of each camera corresponding to each image to be processed into a grid.

[0190] In determination module 1302, the user determines the query vector corresponding to each grid according to the coordinates of each grid after grid division.

[0191] The aggregation module 1303 is used to aggregate all query vectors to obtain a query vector set.

[0192] The fusion perception device based on drive test multiple cameras provided in the above embodiment is used to execute the fusion perception method based on drive test multiple cameras in any of the above method embodiments. Its implementation principle and technical effect are similar and will not be repeated here.

[0193] Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Fig.14 As shown, the sweeping machine 1400 includes: a processor 1402, and a memory 1401 communicatively connected to the processor 1402;

[0194] The memory 1401 stores computer executable instructions;

[0195] The processor 1402 executes the computer-executable instructions stored in the memory 1401 to implement the fusion perception method based on drive test multiple cameras in any method embodiment.

[0196] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to execute the fusion perception method based on road test multi-camera provided in the above-mentioned various embodiments.

[0197] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0198] Optionally, a readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0199] An embodiment of the present application also provides a computer program product, which includes a computer program. The computer program is stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, it can implement the technical solution provided by any of the above method embodiments.

[0200] In the present application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship; in the formula, the character " / " indicates that the previous and next associated objects are in a "division" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0201] It is to be understood that the various numerical numbers involved in the embodiments of the present application are only for the convenience of description and are not intended to limit the scope of the embodiments of the present application. In the embodiments of the present application, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0202] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0203] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A fusion perception method based on multi-camera road test, characterized in that: include: Based on a pre-built encoder, a plurality of images to be processed are processed to generate an encoder output result, wherein the encoder is trained by self-supervised learning based on the plurality of images; Based on the decoder corresponding to the encoder and a preset query vector set, feature aggregation is performed on the encoder output result to generate a decoder output result; The decoder output result is subjected to perception extraction to obtain a perception result, wherein the perception result includes parameters corresponding to at least one traffic object, and the parameters include category, position, heading and identification.

2. The method according to claim 1, characterized in that The step of performing feature aggregation on the encoder output result based on the decoder corresponding to the encoder and a preset query vector set to generate a decoder output result includes: Perform decoder autocorrelation feature aggregation on the query vector set to obtain a processed query vector set, wherein the query vector set includes multiple query vectors, each query vector represents a specific position within a camera field of view; Based on the processed query vector set, cross-correlation feature aggregation is performed on the encoder output result to generate the decoder output result.

3. The method according to claim 2, characterized in that The step of performing decoder cross-correlation feature aggregation on the encoder output result based on the processed query vector set to generate the decoder output result includes: For each image to be processed, based on the internal and external parameters of the camera corresponding to the image to be processed, determining a reference point of each query vector in the processed query vector set in the image to be processed; According to all reference points and the encoder output result, the image features in each image to be processed are aggregated and extracted by a preset cross-correlation formula to obtain the decoder output result.

4. The method according to claim 3, characterized in that The step of determining a reference point of each query vector in the processed query vector set in the image to be processed based on the internal and external parameters of the camera corresponding to the image to be processed includes: For each image to be processed, construct a posture projection matrix of the camera according to the internal and external parameters of the camera corresponding to the image to be processed; A reference point of each query vector in the image to be processed is calculated according to the pose projection matrix and each query vector.

5. The method according to claim 1, characterized in that The pre-built encoder processes a plurality of images to be processed to generate encoder output results, including: For each image to be processed, performing block division preprocessing on the image to be processed to obtain a plurality of image block sequences corresponding to the image to be processed; For each image block sequence, encoder autocorrelation features of the image block sequence are aggregated using a preset autocorrelation formula to obtain a feature vector corresponding to the image block sequence; The feature vectors corresponding to all image block sequences are aggregated to obtain the encoder output result.

6. The method according to claim 5, characterized in that The preprocessing of dividing the image to be processed into blocks to obtain a plurality of image block sequences corresponding to the image to be processed includes: For each image to be processed, the image to be processed is divided into blocks to obtain a plurality of image blocks corresponding to the image to be processed; Embedding coding is performed on each image block corresponding to the image to be processed and position coding is superimposed to obtain a plurality of image block sequences corresponding to the image to be processed.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Divide the union of the field of view of the camera corresponding to each image to be processed into a grid; According to the coordinates of each grid after grid division, determine the query vector corresponding to each grid; All query vectors are aggregated to obtain the query vector set.

8. A fusion perception device based on road test multiple cameras, characterized in that: include: An encoding module, used for processing a plurality of images to be processed based on a pre-built encoder to generate an encoder output result, wherein the encoder is obtained by self-supervised learning training based on the plurality of images; A decoding module, configured to perform feature aggregation on the encoder output result based on a decoder corresponding to the encoder and a preset query vector set to generate a decoder output result; The perception module is used to perform perception extraction on the decoder output result to obtain a perception result, wherein the perception result includes parameters corresponding to at least one traffic object, and the parameters include category, position, heading and identification.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the fusion perception method based on drive test multiple cameras according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the fusion perception method based on drive test multi-cameras according to any one of claims 1 to 7.

Images