Multi-camera perception and recognition method, device, electronic device, and storage medium

Abstract

The present invention provides a multi-camera perception and recognition method, device, electronic device, and storage medium. Applied in tunnels, the multi-camera field of view covers a 360-degree horizontal range of the vehicle body. The method includes: stitching multiple 2D images captured by the multi-cameras into a surround-view image covering the 360-degree horizontal range of the vehicle body; performing depth prediction on the surround-view image and using a back-projection model to convert the surround-view image into a 3D feature vector of the vehicle body in the 360-degree horizontal range; inputting the 3D feature vector into a multi-task prediction model to predict tunnel road objects; generating a tunnel road object time series based on the tunnel road objects predicted by the multi-task prediction model at the predicted time; inputting the tunnel road object time series into a time series model, and obtaining restored tunnel road objects based on the output of the time series model. The present invention reduces environmental perception costs, reduces the complexity of environmental perception detection, and improves environmental perception detection efficiency in unmanned driving in tunnels.

Description

Technical Field

[0001] The present invention relates to the field of radar point cloud data processing, and in particular to a multi-camera perception and recognition method, device, electronic device, and storage medium for use in tunnels. Background Art

[0002] In subway tunnels and highway underground tunnels, due to insufficient lighting, narrow distances between the left and right sides of the tunnel, and similar colors of the tunnel floor and ceiling, applying a deep learning model to images captured by a monocular camera alone and then fusing the detection results with those of other sensors to process 360-degree environmental perception can easily lead to poor perception results.

[0003] Therefore, how to reduce the cost of environmental perception, reduce the complexity of environmental perception detection, and improve the efficiency of environmental perception detection in unmanned driving in tunnels is a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the Invention

[0004] In order to overcome the defects of the above-mentioned prior art, the present invention provides a multi-camera perception and recognition method, device, electronic device, and storage medium for application in tunnels, so as to reduce the cost of environmental perception, reduce the complexity of environmental perception detection, and improve the efficiency of environmental perception detection during unmanned driving in tunnels.

[0005] According to one aspect of the present invention, a multi-camera perception and recognition method for use in a tunnel is provided, wherein the multi-camera viewing angle covers a 360-degree horizontal range of the vehicle body, including:

[0006] Multiple 2D images collected by multiple cameras are stitched together into a surround image covering the vehicle body in a 360-degree horizontal range;

[0007] Performing depth prediction on the surround view image and converting the surround view image into a 3D feature vector of the vehicle body within a 360-degree horizontal range using a back-projection model;

[0008] Inputting the 3D feature vector into a multi-task prediction model to predict tunnel road objects;

[0009] generating a tunnel road object time series according to the tunnel road objects predicted by the multi-task prediction model at a predicted time;

[0010] The tunnel road object time series is input into a time series model, and a restored tunnel road object is obtained according to an output of the time series model.

[0011] In some embodiments of the present application, stitching multiple 2D images captured by multiple cameras into a surround view image covering 360 degrees of the vehicle body in the horizontal direction includes:

[0012] A multi-camera calibration algorithm is used to stitch multiple 2D images captured by multiple cameras into a surround-view image covering the vehicle body in a 360-degree horizontal range.

[0013] In some embodiments of the present application, performing depth prediction on the surround view picture includes:

[0014] The surround view image is sequentially input into a plurality of encoding layers and a plurality of decoding layers to obtain three weight matrices, where the encoding layer and the decoding layer are a transformer encoding layer and a transformer decoding layer.

[0015] In some embodiments of the present application, using a back-projection model to convert the surround view image into a 3D feature vector of the vehicle body within a 360-degree horizontal range includes:

[0016] Input the three weight matrices output by the decoding layer into the back-projection model;

[0017] Obtaining the 3D feature vector output by the back-projection model as the 3D feature vector of the vehicle body in a 360-degree horizontal range,

[0018] The back-projection model includes multiple first convolutional layers, each of which is used to perform convolution and batch normalization operations.

[0019] In some embodiments of the present application, the multi-task prediction model includes a backbone network for extracting features from a 3D feature vector of the vehicle body within a 360-degree horizontal range, and a branch network connected to the backbone network for respectively predicting multiple tunnel road objects.

[0020] In some embodiments of the present application, generating a tunnel road object time series according to the tunnel road objects predicted by the multi-task prediction model at the predicted time includes:

[0021] Combining the 3D feature vectors of multiple tunnel road objects at the same prediction time into a sequence of frames;

[0022] According to the prediction time, a tunnel road object time series is generated based on multiple sequence frames.

[0023] In some embodiments of the present application, generating a tunnel road object time series according to the tunnel road objects predicted by the multi-task prediction model at the predicted time includes:

[0024] The 3D feature vector of a tunnel road object at the same prediction time is regarded as a sequence of frames;

[0025] According to the prediction time, a plurality of tunnel road object time series are generated for the plurality of tunnel road objects according to the plurality of sequence frames.

[0026] In some embodiments of the present application, the time series model includes multiple second convolutional layers, each second convolutional layer is used to perform convolution and batch normalization on a sequence of frames, the output channel of each second convolutional layer is a 3D coordinate channel, and the output of the time series model is the sum of the output data of each second convolutional layer.

[0027] According to another aspect of the present application, a multi-camera perception and recognition device for use in a tunnel is provided, comprising:

[0028] The stitching module is used to stitch multiple 2D images collected by multiple cameras into a surround view image covering the vehicle body in the horizontal direction of 360 degrees;

[0029] a back-projection module, configured to perform depth prediction on the surround view image and convert the surround view image into a three-dimensional feature vector of the vehicle body within a 360-degree horizontal range using a back-projection model;

[0030] a tunnel road object prediction module, configured to input the three-dimensional feature vector into a multi-task prediction model to predict tunnel road objects;

[0031] A time series generation module, configured to generate a tunnel road object time series based on the tunnel road objects predicted by the multi-task prediction model according to the prediction time;

[0032] The tunnel road object restoration module is used to input the tunnel road object time series into a time series model and obtain the restored tunnel road object according to the output of the time series model.

[0033] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; and a storage medium storing a computer program, wherein the computer program executes the above steps when executed by the processor.

[0034] According to yet another aspect of the present invention, a storage medium is provided, wherein a computer program is stored on the storage medium, and when the computer program is executed by a processor, the steps described above are executed.

[0035] Compared with the prior art, the advantages of the present invention are:

[0036] By stitching together multiple 2D images captured by multiple cameras into a surround view image, depth prediction is performed. Using a back-projection module, a 3D feature vector of the vehicle body in a 360-degree horizontal range is obtained. This 3D feature vector is then input into a multi-task prediction model to predict tunnel road objects. This approach, on the one hand, avoids the need for multiple sensor fusion after model detection, and compared to post-fusion approaches, reduces overall algorithm complexity and time consumption while improving model recognition accuracy and robustness. On the other hand, it only requires multiple monocular cameras with 360-degree coverage, eliminating the need for other sensors such as lidar, reducing costs. Furthermore, stitching together multiple monocular 2D images into a single surround view image requires only a single model for processing, eliminating the need for deep learning models for each camera image, saving system resources. The entire multi-camera perception and recognition process is an end-to-end model. Furthermore, the present application also generates a road object time series according to the tunnel road objects predicted by the multi-task prediction model according to the prediction time, inputs the tunnel road object time series into the time series model, and obtains the restored tunnel road objects according to the output of the time series model, so as to restore the tunnel road objects based on the time series to avoid missed or partially missing detections. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and other features and advantages of the present invention will become more apparent by describing in detail exemplary embodiments thereof with reference to the accompanying drawings.

[0038] Figure 1 A flowchart of a multi-camera perception and recognition method according to an embodiment of the present invention is shown;

[0039] Figure 2 A model architecture diagram of a multi-camera perception and recognition method according to an embodiment of the present invention is shown;

[0040] Figure 3 A schematic diagram of a depth prediction and back-projection model according to an embodiment of the present invention is shown;

[0041] Figure 4 A schematic diagram of a multi-task prediction model according to an embodiment of the present invention is shown;

[0042] Figure 5 A schematic diagram showing a time series model according to an embodiment of the present invention

[0043] Figure 6 A module diagram of a multi-camera perception and recognition device according to an embodiment of the present invention is shown;

[0044] Figure 7 Schematically illustrates a computer-readable storage medium in an exemplary embodiment of the present disclosure;

[0045] Figure 8The figure schematically shows a diagram of an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0047] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0048] In order to address the defects of the prior art, the present invention provides a multi-camera perception and recognition method applied in a tunnel, wherein the viewing angle of the multi-camera covers a 360-degree range in the horizontal direction of the vehicle body.

[0049] See below Figure 1 , Figure 1 A flowchart of a multi-camera perception and recognition method according to an embodiment of the present invention is shown. Figure 1 The steps are as follows:

[0050] Step S110: stitching multiple 2D images captured by multiple cameras into a surround image covering 360 degrees of the vehicle body in the horizontal direction;

[0051] Step S120: performing depth prediction on the surround view image and converting the surround view image into a 3D feature vector of the vehicle body within a 360-degree horizontal range using a back-projection model;

[0052] Step S130: inputting the 3D feature vector into a multi-task prediction model to predict tunnel road objects;

[0053] Step S140: generating a tunnel road object time series according to the tunnel road objects predicted by the multi-task prediction model at the predicted time;

[0054] Step S150: inputting the tunnel road object time series into a time series model, and obtaining a restored tunnel road object according to an output of the time series model.

[0055] Specifically, the model structure involved in the above steps S110 to S140 can be found in Figure 2 , Figure 2 The following is a model architecture diagram of a multi-camera perception and recognition method applied in a tunnel according to an embodiment of the present invention. Figure 2 As shown, multiple monocular cameras can each capture a 2D image, which is then stitched together 210 to obtain a surround view image covering the vehicle body in a 360-degree horizontal range. The 2D images captured by multiple bullet screen cameras form a group of 2D images for stitching 210. Furthermore, the 2D images in the same group are collected at the same time. Timestamps can be aligned so that 2D images collected at the same time are grouped together. Alternatively, each monocular camera can be controlled to capture 2D images at the same sampling time and sampling period, so that each time a 2D image is captured, the captured 2D images are obtained from multiple monocular cameras and stitched together 210.

[0056] The number of 2D images is consistent with the number of monocular cameras. The number of monocular cameras can be set as needed, as long as the viewing angle covers a 360-degree horizontal range of the vehicle body. For example, the number of monocular cameras can be six, respectively set at the four corners of the vehicle body and the midpoints of the two long sides, thereby covering a 360-degree horizontal range of the vehicle body. This application is not limited to this, and other numbers and arrangements of monocular cameras are also within the scope of this application.

[0057] Specifically, due to the narrow tunnel walls and the dark environment in the tunnel, there are fewer image features in each 2D image, making it difficult to directly convert the 2D images taken in the tunnel into 3D. Therefore, in this application, the 2D images captured by multiple monocular cameras are first rotated and translated so that the multiple images can be spliced into a surround view image, thereby not obtaining 3D features from the 2D images alone. After the multiple 2D images are spliced into a surround view image, not only can they be processed uniformly, but feature references between the multiple 2D images can also be achieved to reduce the impact of the tunnel environment on image processing. Among them, the viewing angle of each monocular camera can be determined, so that the direction and range of the 2D images taken by each monocular camera can be known. Therefore, based on the pre-known direction and range, the multiple 2D images can be rotated and translated to complete the splicing of the surround view image. Furthermore, a multi-camera calibration algorithm can be used to splice the multiple 2D images captured by the multiple cameras into a surround view image covering the vehicle body in the horizontal direction of 360 degrees.

[0058] After obtaining the surround view image, since the lane and car images are continuous rather than truncated, they can be input into the back projection model 250 after depth prediction 230 to obtain the 3D feature vector of the car body in the horizontal direction of 360 degrees.

[0059] The obtained 3D feature vector for the vehicle body within a 360-degree horizontal range is input into the multi-task prediction model 240. The multi-task prediction model 240 is used to predict tunnel road objects within the 360-degree range of the vehicle body based on the 3D feature vector. Tunnel road objects may include, but are not limited to, tunnel walls, other vehicles, lane markings, etc. The output of the multi-task prediction model 240 may include whether the 3D feature vector for the vehicle body within a 360-degree horizontal range contains the specified tunnel road object, the type of the tunnel road object, and the 3D location range of the tunnel road object.

[0060] After the multi-task prediction model 240 outputs the predicted tunnel road object (such as tunnel road object AD), it is divided into two groups according to the prediction time (such as T0-T k ), generating a tunnel road object time series based on the tunnel road objects predicted by the multi-task prediction model 240. The generated tunnel road object time series is input into the time series model 260. The time series model 260 is used to refine the predicted tunnel road objects based on the prediction results at multiple times to avoid missing or partially missing tunnel road objects.

[0061] Furthermore, Figure 2 The model architecture shown can be a fully end-to-end model, and end-to-end learning can omit the data annotation required for each independent learning task.

[0062] In the multi-camera perception and recognition method provided by the present invention, multiple 2D images captured by multiple cameras are stitched together into a surround view image, which is then used for depth prediction. A back-projection module is used to obtain a 3D feature vector for the vehicle body within a 360-degree horizontal range. This 3D feature vector is then input into a multi-task prediction model to predict tunnel road objects. This method, on the one hand, avoids the need for multiple sensor fusion after model detection, and compared to post-fusion methods, reduces overall algorithm complexity and time consumption while improving model recognition accuracy and robustness. On the other hand, it only requires multiple monocular cameras with 360-degree coverage, eliminating the need for other sensors such as lidar, thus reducing costs. Furthermore, after stitching multiple monocular 2D images into a single surround view image, only a single model is required for processing, eliminating the need for deep learning models to process each camera image, thus conserving system resources. The entire multi-camera perception and recognition model is an end-to-end model. Furthermore, the present application also generates a tunnel road object time series according to the tunnel road objects predicted by the multi-task prediction model according to the prediction time, inputs the tunnel road object time series into the time series model, and obtains the restored tunnel road object according to the output of the time series model, so as to restore the tunnel road object based on the time series to avoid missed or partially missing detections.

[0063] See below Figure 3 , Figure 3 A schematic diagram of a depth prediction and back projection model according to an embodiment of the present invention is shown. In this application, a codec model (depth prediction 230) is first used to perform depth estimation prediction on the surround view picture, and the back projection model 250 is used to convert it into a 3D point cloud-like feature. The codec model includes multiple coding layers and multiple decoding layers. Among them, the coding layer is a transformer coding layer, and the decoding layer is a transformer decoding layer. The codec model (depth prediction 230) calculates the three weight matrices (query matrix Q, key matrix K, and value matrix V) of the surround view picture input therein. The three weight matrices Q, K, and V can be obtained by matrix multiplication with the three weights based on the image features of the surround view image. The three weight matrices Q, K, and V obtained are input to the back projection model to convert the surround view picture into a 3D feature vector of the vehicle body in a 360-degree range in the horizontal direction. Among them, the back-projection model 250 includes multiple first convolutional layers, each of which can be used to perform convolution and batch normalization operations, thereby back-projecting the three weight matrices into a three-dimensional space to obtain a three-dimensional feature vector of the vehicle body in a 360-degree horizontal range.

[0064] See below Figure 4 , Figure 4 A schematic diagram of a multi-task prediction model according to an embodiment of the present invention is shown. The multi-task prediction model is used to perform multi-task model prediction based on the same 3D feature map extracted from multiple surround view cameras, thereby outputting prediction results for tunnel road objects such as lane lines, tunnel walls, and obstacles.

[0065] Specifically, the multi-task prediction model 240 includes a backbone network 241 for extracting features from the 3D feature vector of the vehicle body within a 360-degree horizontal range, and a branch network AD connected to the backbone network for respectively predicting multiple tunnel road objects (the number is not limited to this).

[0066] After obtaining the 3D feature vector, it can be input into the backbone network 241 for feature extraction, and then input into multiple branch networks. Each branch network is used to predict tunnel road objects such as lane lines, tunnel walls, obstacles, and drivable areas. The backbone network 241 can be used to extract features from the 3D feature vector, diversifying the features and deepening the model, thereby improving model prediction accuracy. The branch networks share the same backbone network 241 to reduce the system resources required by the model. Furthermore, the branch networks can have the same or different structures. Branch networks with the same structure will obtain different model parameters during training, thereby achieving predictions for different tunnel road objects.

[0067] See below Figure 5 , Figure 5A schematic diagram of a time series model according to an embodiment of the present invention is shown. Due to the high acquisition frequency of the unmanned vehicle's monocular camera, the tunnel road objects predicted and output by the multi-task prediction model 270 are strongly correlated in time series. By sequentially inputting the corresponding results of the time series into the time series model, a certain degree of memory is maintained for occluded objects, facilitating their recovery.

[0068] In some embodiments, the 3D feature vectors of multiple tunnel road objects at the same prediction time can be combined into a sequence of frames, and a tunnel road object time series can be generated from these multiple sequence frames at the same prediction time. This creates a holistic time series frame, allowing the recovery of each tunnel road object to be aided by other objects in the 3D feature vector. In other embodiments, the 3D feature vector of a tunnel road object at the same prediction time can be used as a sequence of frames, and multiple tunnel road object time series can be generated for each tunnel road object at the same prediction time based on these multiple sequence frames. This can reduce the amount of sequence frame data processed by the time series model at each time.

[0069] Combine Figure 5 The time series model 290 may include a plurality of second convolutional layers 2911-291N (N is an integer greater than 1), each second convolutional layer is used to perform convolution and batch normalization processing on a sequence of frames, and the output channel of each second convolutional layer is a 3D coordinate channel, which is added via the addition layer 292 as the output of the time series model. Specifically, the plurality of second convolutional layers of the time series model 290 can be calculated in a pipeline form. For example, the second convolutional layer 291N calculates the tunnel road object predicted by T0 in the tunnel road object time series, the second convolutional layer 291N-1 calculates the tunnel road object predicted by T1 in the tunnel road object time series, the second convolutional layer 291N-2 calculates the tunnel road object predicted by T2 in the tunnel road object time series... The second convolutional layer 2911 calculates the tunnel road object predicted by T0 in the tunnel road object time series. k The predicted tunnel road objects are simultaneously calculated and processed by the above convolutional layers and then output to the addition layer 292, which then outputs the restored tunnel road objects A'-D'. The tunnel road object time series is then advanced in this manner. The second convolutional layer 291N calculates the tunnel road object predicted at T1 in the tunnel road object time series, the second convolutional layer 291N-1 calculates the tunnel road object predicted at T2 in the tunnel road object time series, the second convolutional layer 291N-2 calculates the tunnel road object predicted at T3 in the tunnel road object time series, and so on. The second convolutional layer 2911 calculates the tunnel road object predicted at T1 in the tunnel road object time series. k+1The predicted tunnel road objects are simultaneously calculated and processed by the above-mentioned convolutional layers and then output to the addition layer 292 so that the addition layer 292 can output updated restored tunnel road objects A′-D′.

[0070] The above are merely a few specific implementations of the multi-camera perception and recognition method for tunnels according to the present invention. Each implementation can be implemented independently or in combination, and the present invention is not limited thereto. Furthermore, the flowcharts of the present invention are merely illustrative, and the order of execution of the steps is not limited thereto. Step splitting, merging, order swapping, and other synchronous or asynchronous execution methods are all within the scope of the present invention.

[0071] The present invention also provides a multi-camera perception and recognition device, Figure 6 A block diagram of a multi-camera perception and recognition device for use in a tunnel, according to an embodiment of the present invention, is shown. The multi-camera field of view covers a 360-degree horizontal range along the vehicle body. The multi-camera perception and recognition device 200 includes a splicing module 210, a back-projection module 220, a tunnel road object prediction module 230, a time series generation module 240, and a tunnel road object restoration module 250.

[0072] The stitching module 210 is used to stitch multiple 2D images captured by multiple cameras into a surround view image covering 360 degrees of the vehicle body in the horizontal direction;

[0073] The back-projection module 220 is used to perform depth prediction on the surround view image and convert the surround view image into a 3D feature vector of the vehicle body in a 360-degree horizontal range using a back-projection model;

[0074] The tunnel road object prediction module 230 is used to input the 3D feature vector into a multi-task prediction model to predict the tunnel road object;

[0075] The time series generating module 240 is used to generate a tunnel road object time series according to the tunnel road objects predicted by the multi-task prediction model at the predicted time;

[0076] The tunnel road object restoration module 250 is used to input the tunnel road object time series into a time series model, and obtain the restored tunnel road object according to the output of the time series model.

[0077] In the multi-camera perception and recognition device for use in tunnels, the present invention combines multiple 2D images captured by multiple cameras into a surround view image, performs depth prediction, and uses a back-projection module to obtain a 3D feature vector for the vehicle body within a 360-degree horizontal range. This 3D feature vector is then input into a multi-task prediction model to predict tunnel road objects. This eliminates the need for multiple sensor fusion after model detection, reduces overall algorithm complexity and time consumption compared to post-fusion approaches, and improves model recognition accuracy and robustness. Furthermore, it requires only multiple monocular cameras with 360-degree coverage, eliminating the need for other sensors such as lidar, thus reducing costs. Furthermore, after combining multiple monocular 2D images into a single surround view image, only a single model is required for processing, eliminating the need for deep learning models to process each camera image, thus conserving system resources. The entire multi-camera perception and recognition system is an end-to-end model. Furthermore, the present application also generates a tunnel road object time series according to the tunnel road objects predicted by the multi-task prediction model according to the prediction time, inputs the tunnel road object time series into the time series model, and obtains the restored tunnel road object according to the output of the time series model, so as to restore the tunnel road object based on the time series to avoid missed or partially missing detections.

[0078] Figure 6 The multi-camera sensing and recognition device 200 provided by the present invention is merely schematically illustrated. Without violating the inventive concept, the separation, combination, and addition of modules are all within the scope of protection of the present invention. The multi-camera sensing and recognition device 200 provided by the present invention can be implemented using software, hardware, firmware, plug-ins, or any combination thereof, and the present invention is not limited thereto.

[0079] In exemplary embodiments of the present disclosure, a computer-readable storage medium is further provided, on which a computer program is stored. When the program is executed by, for example, a processor, the program can implement the steps of the multi-camera perception and recognition method described in any of the above embodiments. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes program code. When the program product is executed on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the multi-camera perception and recognition method section above.

[0080] refer to Figure 7, a program product 800 for implementing the above method according to an embodiment of the present invention is described. The program product 800 may be a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0081] The program product may be implemented in any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0082] The computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. The data signal propagated may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or component. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0083] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, and the like, as well as conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the tenant computing device, partially on the tenant computing device, as a stand-alone software package, partially on the tenant computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the tenant computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0084] In an exemplary embodiment of the present disclosure, an electronic device is further provided, which may include a processor and a memory for storing executable instructions of the processor. The processor is configured to execute the executable instructions to perform the steps of the multi-camera perception and recognition method described in any of the above embodiments.

[0085] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods, or program products. Therefore, various aspects of the present invention may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0086] Refer to the following Figure 8 An electronic device 600 according to this embodiment of the present invention will be described. Figure 8 The electronic device 600 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0087] like Figure 8 As shown, electronic device 600 is implemented as a general-purpose computing device. Components of electronic device 600 may include, but are not limited to, at least one processing unit 610, at least one storage unit 620, a bus 630 connecting various system components (including storage unit 620 and processing unit 610), a display unit 640, and the like.

[0088] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps of various exemplary embodiments of the present invention described in the multi-camera perception and recognition method section above. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0089] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0090] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.

[0091] Bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0092] The electronic device 600 can also communicate with one or more external devices 700 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable tenants to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). This communication can occur via an input / output (I / O) interface 650. Furthermore, the electronic device 600 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 660. The network adapter 660 can communicate with other modules of the electronic device 600 via the bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the electronic device 600, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0093] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, server, or network device, etc.) to execute the above-mentioned multi-camera perception and recognition method according to the embodiment of the present disclosure.

[0094] Compared with the prior art, the advantages of the present invention are:

[0095] By stitching together multiple 2D images captured by multiple cameras into a surround view image, depth prediction is performed. Using a back-projection module, a 3D feature vector of the vehicle body in a 360-degree horizontal range is obtained. This 3D feature vector is then input into a multi-task prediction model to predict tunnel road objects. This approach, on the one hand, avoids the need for multiple sensor fusion after model detection, and compared to post-fusion approaches, reduces overall algorithm complexity and time consumption while improving model recognition accuracy and robustness. On the other hand, it only requires multiple monocular cameras with 360-degree coverage, eliminating the need for other sensors such as lidar, reducing costs. Furthermore, stitching together multiple monocular 2D images into a single surround view image requires only a single model for processing, eliminating the need for deep learning models for each camera image, saving system resources. The entire multi-camera perception and recognition process is an end-to-end model. Furthermore, the present application also generates a road object time series according to the tunnel road objects predicted by the multi-task prediction model according to the prediction time, inputs the tunnel road object time series into the time series model, and obtains the restored tunnel road objects according to the output of the time series model, so as to restore the tunnel road objects based on the time series to avoid missed or partially missing detections.

[0096] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

Claims

1. A multi-camera perception and recognition method applied in tunnels, characterized in that: The multi-camera viewing angle covers a 360-degree horizontal range of the vehicle body, including: Multiple 2D images collected by multiple cameras are stitched together into a surround image covering the vehicle body in a 360-degree horizontal range; Performing depth prediction on the surround view image and converting the surround view image into a 3D feature vector of the vehicle body within a 360-degree horizontal range using a back-projection model; Inputting the three-dimensional feature vector into a multi-task prediction model to predict tunnel road objects, the multi-task prediction model comprising a backbone network for extracting features from the three-dimensional feature vector of the vehicle body within a 360-degree horizontal range, and branch networks connected to the backbone network for respectively predicting a plurality of tunnel road objects; The 3D feature vector of a tunnel road object at the same prediction time is regarded as a sequence frame; according to the prediction time, a plurality of tunnel road object time series are generated for a plurality of tunnel road objects according to the plurality of sequence frames; The tunnel road object time series is input into a time series model, and a restored tunnel road object is obtained according to an output of the time series model.

2. The multi-camera perception and recognition method according to claim 1, wherein: The process of stitching the multiple 2D images collected by the multiple cameras into a surround view image covering the vehicle body in a 360-degree horizontal range includes: A multi-camera calibration algorithm is used to stitch multiple 2D images captured by multiple cameras into a surround-view image covering the vehicle body in a 360-degree horizontal range.

3. The multi-camera perception and recognition method according to claim 1, wherein: The performing depth prediction on the surround view image includes: The surround view image is sequentially input into a plurality of encoding layers and a plurality of decoding layers to obtain three weight matrices, where the encoding layer and the decoding layer are a transformer encoding layer and a transformer decoding layer.

4. The multi-camera perception and recognition method according to claim 3, wherein: Using a back-projection model, converting the surround view image into a 3D feature vector of the vehicle body in a 360-degree horizontal range includes: Input the three weight matrices output by the decoding layer into the back-projection model; Obtaining the 3D feature vector output by the back-projection model as the 3D feature vector of the vehicle body in a 360-degree horizontal range, The back-projection model includes multiple first convolutional layers, each of which is used to perform convolution and batch normalization operations.

5. The multi-camera perception and recognition method according to claim 1, wherein: in, The time series model includes multiple second convolutional layers, each second convolutional layer is used to perform convolution and batch normalization on a sequence of frames, the output channel of each second convolutional layer is a three-dimensional coordinate channel, and the output of the time series model is the sum of the output data of each second convolutional layer.

6. A multi-camera perception and recognition device used in tunnels, characterized in that: include: The stitching module is used to stitch multiple 2D images collected by multiple cameras into a surround view image covering the vehicle body in the horizontal direction of 360 degrees; a back-projection module, configured to perform depth prediction on the surround view image and convert the surround view image into a three-dimensional feature vector of the vehicle body within a 360-degree horizontal range using a back-projection model; a tunnel road object prediction module, configured to input the 3D feature vector into a multi-task prediction model to predict tunnel road objects, the multi-task prediction model comprising a backbone network for extracting features from the 3D feature vector of the vehicle body within a 360-degree horizontal range, and branch networks connected to the backbone network for respectively predicting a plurality of tunnel road objects; The time series generation module takes the 3D feature vector of a tunnel road object at the same prediction time as a sequence of frames; According to the prediction time, a plurality of tunnel road object time series are generated for the plurality of tunnel road objects according to the plurality of sequence frames; The tunnel road object restoration module is used to input the tunnel road object time series into a time series model and obtain the restored tunnel road object according to the output of the time series model.

7. An electronic device, characterized in that: The electronic device comprises: processor; A storage medium having a computer program stored thereon, wherein the computer program, when executed by the processor, executes the multi-camera perception and recognition method according to any one of claims 1 to 5.

8. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, executes the multi-camera perception and recognition method according to any one of claims 1 to 5.

Images