Multi-camera perception and recognition method, device, electronic device, and storage medium

Abstract

The present invention provides a multi-camera perception and recognition method, device, electronic device, and storage medium. The multi-camera field of view covers a 360-degree horizontal range of a vehicle body. The method includes: inputting multiple 2D images captured by the multi-cameras into a dimensionality conversion model to obtain a 3D feature vector of the vehicle body within a 360-degree horizontal range; inputting the 3D feature vector into a multi-task prediction model to predict road objects; generating a road object time series based on the road objects predicted by the multi-task prediction model at a predicted time; inputting the road object time series into a time series model, and obtaining restored road objects based on the output of the time series model. The present invention reduces environmental perception costs, reduces the complexity of environmental perception detection, and improves environmental perception detection efficiency.

Description

Technical Field

[0001] The present invention relates to the field of unmanned driving data processing, and in particular to a multi-camera perception and recognition method, device, electronic device, and storage medium. Background Art

[0002] In the autonomous driving field, data from LiDAR and monocular cameras are typically fed into deep learning models to generate laser detection results and monocular detection results, respectively. These results are then fused together to perform environmental perception. LiDAR detection data has low data density and lacks texture features like color, resulting in low detection accuracy. Furthermore, monocular cameras lack 360-degree coverage of the environment due to their limited field of view.

[0003] Therefore, how to reduce the cost of environmental perception, reduce the complexity of environmental perception detection, and improve the efficiency of environmental perception detection is a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the Invention

[0004] In order to overcome the defects of the above-mentioned prior art, the present invention provides a multi-camera perception and recognition method, device, electronic device, and storage medium to reduce the cost of environmental perception, reduce the complexity of environmental perception detection, and improve the efficiency of environmental perception detection.

[0005] According to one aspect of the present invention, a multi-camera perception and recognition method is provided, wherein the multi-camera viewing angle covers a 360-degree range in the horizontal direction of the vehicle body, comprising:

[0006] Input multiple 2D images captured by multiple cameras into a dimension conversion model to obtain a 3D feature vector of the vehicle body in a 360-degree horizontal range;

[0007] Inputting the 3D feature vector into a multi-task prediction model to predict road objects;

[0008] generating a road object time series based on the road objects predicted by the multi-task prediction model according to the prediction time;

[0009] The road object time series is input into a time series model, and the restored road object is obtained according to the output of the time series model.

[0010] In some embodiments of the present application, the input of the dimensional conversion model is a 2D image, and the output of the dimensional conversion model is a 3D feature vector of the vehicle body in a 360-degree horizontal range, and the 3D feature vector is the three-dimensional coordinate of each pixel in the 2D image.

[0011] In some embodiments of the present application, the dimensional conversion model includes a multi-layer first convolution layer for processing the input 2D image, the output of the multi-layer first convolution layer is a 3D feature vector of the vehicle body in a 360-degree horizontal range, and the multi-layer first convolution layer forms a residual neural network model.

[0012] In some embodiments of the present application, the multi-task prediction model includes a backbone network for extracting features from a 3D feature vector of the vehicle body within a 360-degree horizontal range, and a branch network connected to the backbone network for respectively predicting multiple road objects.

[0013] In some embodiments of the present application, generating a road object time series according to the road objects predicted by the multi-task prediction model at the predicted time includes:

[0014] Combining the 3D feature vectors of multiple road objects at the same prediction time into a sequence of frames;

[0015] According to the prediction time, a road object time series is generated based on multiple sequence frames.

[0016] In some embodiments of the present application, generating a road object time series according to the road objects predicted by the multi-task prediction model at the predicted time includes:

[0017] The 3D feature vector of a road object at the same prediction time is regarded as a sequence of frames;

[0018] According to the prediction time, a plurality of road object time series are generated for the plurality of road objects according to the plurality of sequence frames.

[0019] In some embodiments of the present application, the time series model includes multiple second convolutional layers, each second convolutional layer is used to perform convolution and batch normalization on a sequence of frames, the output channel of each second convolutional layer is a 3D coordinate channel, and the output of the time series model is the sum of the output data of each second convolutional layer.

[0020] According to another aspect of the present application, a multi-camera perception and recognition device is provided, comprising:

[0021] A dimensionality conversion module is used to input multiple 2D images collected by multiple cameras into a dimensionality conversion model to obtain a 3D feature vector of the vehicle body in a 360-degree horizontal range;

[0022] a road object prediction module, configured to input the 3D feature vector into a multi-task prediction model to predict road objects;

[0023] A time series generation module, configured to generate a road object time series based on the road objects predicted by the multi-task prediction model according to the prediction time;

[0024] The road object restoration module is used to input the road object time series into a time series model and obtain the restored road object according to the output of the time series model.

[0025] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; and a storage medium storing a computer program, wherein the computer program executes the above steps when executed by the processor.

[0026] According to yet another aspect of the present invention, a storage medium is provided, wherein a computer program is stored on the storage medium, and when the computer program is executed by a processor, the steps described above are executed.

[0027] Compared with the prior art, the advantages of the present invention are:

[0028] By inputting multiple 2D images captured by multiple cameras into a dimensionality conversion model, a 3D feature vector of the vehicle body in a 360-degree horizontal range is obtained, and the 3D feature vector is then input into a multi-task prediction model to predict road objects. This avoids the need for multiple sensor fusion after model detection, reduces the overall algorithm complexity and time consumption compared to post-fusion methods, and improves model recognition accuracy and robustness. Furthermore, only multiple monocular cameras with 360-degree coverage are required, eliminating the need for other sensors such as lidar, reducing costs. Furthermore, after converting multiple monocular 2D images into 3D vector space, they become a single feature input map, which only requires a single model for processing, eliminating the need for deep learning models to process each camera image, saving system resources. The entire multi-camera perception and recognition process is an end-to-end model. Furthermore, the present application generates a road object time series based on the road objects predicted by the multi-task prediction model at the predicted time. This road object time series is input into a time series model, and the restored road objects are obtained based on the output of the time series model. Restoration of road objects based on the time series is performed based on the road objects, avoiding missed or partially missing detections. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and other features and advantages of the present invention will become more apparent by describing in detail example embodiments thereof with reference to the accompanying drawings.

[0030] Figure 1 A flowchart of a multi-camera perception and recognition method according to an embodiment of the present invention is shown;

[0031] Figure 2 A model architecture diagram of a multi-camera perception and recognition method according to an embodiment of the present invention is shown;

[0032] Figure 3A schematic diagram showing multiple 2D images according to an embodiment of the present invention is shown;

[0033] Figure 4 A schematic diagram showing acquisition of three-dimensional features from a two-dimensional image according to an embodiment of the present invention is shown;

[0034] Figure 5 A schematic diagram of a dimension conversion model according to an embodiment of the present invention is shown;

[0035] Figure 6 A schematic diagram of a multi-task prediction model according to an embodiment of the present invention is shown;

[0036] Figure 7 A schematic diagram showing a time series model according to an embodiment of the present invention

[0037] Figure 8 A module diagram of a multi-camera perception and recognition device according to an embodiment of the present invention is shown;

[0038] Figure 9 Schematically illustrates a computer-readable storage medium in an exemplary embodiment of the present disclosure;

[0039] Figure 10 The figure schematically shows a diagram of an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0041] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0042] In order to address the defects of the prior art, the present invention provides a multi-camera perception and recognition method, wherein the viewing angle of the multi-camera covers a 360-degree range in the horizontal direction of the vehicle body.

[0043] See below Figure 1 , Figure 1A flowchart of a multi-camera perception and recognition method according to an embodiment of the present invention is shown. Figure 1 The steps are as follows:

[0044] Step S110: Input multiple 2D images captured by multiple cameras into a dimension conversion model to obtain a 3D feature vector of the vehicle body within a 360-degree horizontal range.

[0045] Step S120: Input the 3D feature vector into a multi-task prediction model to predict road objects.

[0046] Step S130: generating a road object time series according to the road objects predicted by the multi-task prediction model at the predicted time.

[0047] Step S140: inputting the road object time series into a time series model, and obtaining a restored road object according to an output of the time series model.

[0048] Specifically, the model structure involved in the above steps S110 to S140 can be found in Figure 2 , Figure 2 FIG. 1 shows a model architecture diagram of a multi-camera perception and recognition method according to an embodiment of the present invention. Figure 2 As shown, multiple monocular cameras can each capture a 2D image to form a set of input features for input into the dimensionality conversion model 220. Furthermore, the 2D images in the same group are collected at the same time. Timestamps can be aligned to group 2D images collected at the same time. Alternatively, each monocular camera can be controlled to capture 2D images at the same sampling time and period, so that each acquisition cycle obtains 2D images from multiple monocular cameras and inputs them into the dimensionality conversion model 220.

[0049] The number of 2D images is consistent with the number of monocular cameras. The number of monocular cameras can be set as needed, as long as the viewing angle covers a 360-degree horizontal range of the vehicle body. For example, the number of monocular cameras can be six, respectively set at the four corners of the vehicle body and the midpoints of the two long sides, thereby covering a 360-degree horizontal range of the vehicle body. This application is not limited to this, and other numbers and arrangements of monocular cameras are also within the scope of this application.

[0050] After multiple 2D images are input into the dimension conversion model 220, a 3D feature vector of the vehicle body in the horizontal 360-degree range can be obtained. Specifically, the dimension conversion model 220 can convert the pixels in the multiple 2D images into a 2D space, and can obtain the 3D coordinates of the pixels of each 2D image as the 3D feature vector of the vehicle body in the horizontal 360-degree range. Specifically, the multiple 2D images can be as follows Figure 3As shown, the vehicle V1 and the pedestrian P1 in multiple 2D images are the same object, so the relative position of each object and the vehicle Vm can be obtained by matching and comparing the same object in multiple 2D images (such as Figure 4 As shown), these objects and corresponding pixel points can be mapped to the 3D control where the vehicle is located.

[0051] The obtained 3D feature vector for the vehicle body within a 360-degree horizontal range is input into the multi-task prediction model 240. The multi-task prediction model 240 is used to predict road objects within the 360-degree range based on the 3D feature vector. Road objects may include, but are not limited to, crosswalks, other vehicles, pedestrians, and road signs. The output of the multi-task prediction model 240 may include whether the 3D feature vector for the vehicle body within a 360-degree horizontal range contains a predetermined road object, the type of road object contained, and the 3D position range of the road object.

[0052] After the multi-task prediction model 240 outputs the predicted road object (such as road object AD), it is k ), generating a road object time series based on the road objects predicted by the multi-task prediction model 240. The generated road object time series is input into the time series model 260. The time series model 260 is used to refine the predicted road objects based on the prediction results at multiple times to avoid omissions or partial loss of road objects.

[0053] Furthermore, Figure 2 The model architecture shown can be a fully end-to-end model, and end-to-end learning can omit the data annotation required for each independent learning task.

[0054] In the multi-camera perception and recognition method provided by the present invention, multiple 2D images captured by multiple cameras are input into a dimensionality conversion model to obtain a 3D feature vector for the vehicle body within a 360-degree horizontal range. This 3D feature vector is then input into a multi-task prediction model to predict road objects. This method, on the one hand, avoids the need for multiple sensor fusion after model detection, and compared to post-fusion methods, reduces the overall algorithm complexity and time consumption, while improving model recognition accuracy and robustness. On the other hand, it only requires multiple monocular cameras with 360-degree coverage, eliminating the need for other sensors such as lidar, thus reducing costs. Furthermore, after converting multiple monocular 2D images into 3D vector space, they become a single feature input map, which only requires a single model for processing, eliminating the need for deep learning models to process each camera image, thus saving system resources. The entire multi-camera perception and recognition model is an end-to-end model. Furthermore, the present application also generates a road object time series according to the road objects predicted by the multi-task prediction model according to the prediction time, inputs the road object time series into the time series model, and obtains the restored road objects according to the output of the time series model, so as to restore the road objects based on the time series to avoid detection omissions or partial missing.

[0055] Specifically, the integration of 2D images from multiple monocular cameras is to project the 2D image features of the single camera into the vehicle coordinate system for fusion. When the images of multiple monocular cameras are encoded in the same 3D vector space, the features of the same object in different monocular images can be represented by the same feature description. This same feature description is converted into a complete object in the 3D vector space (e.g. Figure 3 and Figure 4 Thus, when the 2D images captured by multiple monocular cameras are converted into 3D space, a 360-degree (no blind spot) 3D feature vector can be obtained.

[0056] Specifically, the structure of the dimension conversion model 220 can be seen in Figure 5 , Figure 5 A schematic diagram of a dimensional conversion model according to an embodiment of the present invention is shown. After feature extraction, multiple 2D images are subjected to 2D features to obtain 2D features, which are then input into a residual neural network model 221 formed by multiple layers of first convolutional layers. The output of the residual neural network model 221 is a 3D feature vector of the vehicle body in a 360-degree horizontal range. Each first convolutional layer can also be connected in series with a batch of normalization layers. The residual neural network model 221 has a small size, which can control the number of parameters. In addition, the number of feature maps of the residual neural network model 221 increases layer by layer to ensure the output feature expression capability. At the same time, the residual neural network model 221 uses a large amount of downsampling to improve the efficiency of model propagation.

[0057] See below Figure 6 , Figure 6 A schematic diagram of a multi-task prediction model according to an embodiment of the present invention is shown. The multi-task prediction model is used to perform multi-task model prediction based on the same 3D feature map extracted from multiple surround view cameras, thereby outputting prediction results for road objects such as lane markings, crosswalks, and obstacles.

[0058] Specifically, the multi-task prediction model 240 includes a backbone network 241 for extracting features from the 3D feature vector of the vehicle body within a 360-degree horizontal range, and a branch network AD connected to the backbone network for respectively predicting multiple road objects (the number is not limited to this).

[0059] After obtaining the 3D feature vector, it can be input into the backbone network 241 for feature extraction, and then input into multiple branch networks, each of which is used to predict road objects such as lane lines, obstacles, and drivable areas. The backbone network 241 can be used to extract features from the 3D feature vector, diversifying the features, deepening the model, and improving the model's prediction accuracy. The branch networks share the same backbone network 241 to reduce the system resources required for the model. At the same time, the branch networks can have the same or different structures. Branch networks with the same structure will obtain different model parameters during training, thereby achieving prediction of different road objects.

[0060] See below Figure 7 , Figure 7 A schematic diagram of a time series model according to an embodiment of the present invention is shown. Due to the high acquisition frequency of the unmanned vehicle's monocular camera, the road objects predicted and output by the multi-task prediction model 240 are strongly correlated in time series. By sequentially inputting the corresponding results of the time series into the time series model, a certain memory of occluded objects is maintained, facilitating the recovery of occluded objects.

[0061] In some embodiments, the 3D feature vectors of multiple road objects at the same prediction time can be combined into a sequence of frames, and a road object time series can be generated from these multiple sequence frames at the same prediction time. This creates a holistic time series frame, allowing the restoration of each road object to be aided by other objects in the 3D feature vector. In other embodiments, the 3D feature vector of a road object at the same prediction time can be used as a sequence of frames, and multiple road object time series can be generated for multiple road objects at the same prediction time based on these multiple sequence frames. This can reduce the amount of sequence frame data processed by the time series model at each time.

[0062] Combine Figure 7The time series model 260 may include a plurality of second convolutional layers 2611-261N (N is an integer greater than 1), each second convolutional layer is used to perform convolution and batch normalization processing on a sequence of frames, and the output channel of each second convolutional layer is a 3D coordinate channel, which is added via the addition layer 262 as the output of the time series model. Specifically, the plurality of second convolutional layers of the time series model 260 can be calculated in a pipeline form. For example, the second convolutional layer 261N calculates the road object predicted by T0 in the road object time series, the second convolutional layer 261N-1 calculates the road object predicted by T1 in the road object time series, the second convolutional layer 261N-2 calculates the road object predicted by T2 in the road object time series... The second convolutional layer 2611 calculates the road object predicted by T0 in the road object time series. k The predicted road objects are simultaneously calculated and processed by the above convolutional layers and then output to the addition layer 262, which then outputs the restored road objects A'-D'. The road object time series is then advanced in this manner: the second convolutional layer 261N calculates the road objects predicted at T1 in the road object time series, the second convolutional layer 261N-1 calculates the road objects predicted at T2 in the road object time series, the second convolutional layer 261N-2 calculates the road objects predicted at T3 in the road object time series, and the second convolutional layer 2611 calculates the road objects predicted at T k+1 The predicted road objects are simultaneously calculated and processed by the above convolutional layers and then output to the addition layer 262 for the addition layer 262 to output updated restored road objects A′-D′.

[0063] The above are merely a few specific implementations of the multi-camera perception and recognition method of the present invention. Each implementation can be implemented independently or in combination, and the present invention is not limited to this. Furthermore, the flowchart of the present invention is merely illustrative, and the order of execution between the steps is not limited to this. The splitting, merging, order switching, and other synchronous or asynchronous execution methods of the steps are all within the scope of protection of the present invention.

[0064] The present invention also provides a multi-camera perception and recognition device, Figure 8 A block diagram of a multi-camera perception and recognition device according to an embodiment of the present invention is shown. The multi-camera field of view covers a 360-degree horizontal range of the vehicle body. The multi-camera perception and recognition device 400 includes a dimensionality conversion module 410, a road object prediction module 420, a time series generation module 430, and a road object restoration module 440.

[0065] The dimension conversion module 410 is used to input multiple 2D images collected by multiple cameras into a dimension conversion model to obtain a 3D feature vector of the vehicle body in a 360-degree horizontal range;

[0066] The road object prediction module 420 is used to input the 3D feature vector into a multi-task prediction model to predict road objects;

[0067] The time series generation module 430 is used to generate a road object time series according to the road objects predicted by the multi-task prediction model at the predicted time;

[0068] The road object restoration module 440 is configured to input the road object time series into a time series model, and obtain restored road objects according to the output of the time series model.

[0069] In the multi-camera perception and recognition device provided by the present invention, multiple 2D images captured by multiple cameras are input into a dimensionality conversion model to obtain a 3D feature vector for the vehicle body within a 360-degree horizontal range. This 3D feature vector is then input into a multi-task prediction model to predict road objects. This avoids the need for multiple sensor fusion after model detection, reducing overall algorithm complexity and time consumption compared to post-fusion approaches while improving model recognition accuracy and robustness. Furthermore, only multiple monocular cameras with 360-degree coverage are required, eliminating the need for other sensors such as lidar, thus reducing costs. Furthermore, after converting multiple monocular 2D images into a 3D vector space, a single feature input map is generated, requiring only a single model for processing, eliminating the need for deep learning models to process each camera image, thus conserving system resources. The entire multi-camera perception and recognition system is an end-to-end model. Furthermore, the present application also generates a road object time series according to the road objects predicted by the multi-task prediction model according to the prediction time, inputs the road object time series into the time series model, and obtains the restored road objects according to the output of the time series model, so as to restore the road objects based on the time series to avoid detection omissions or partial missing.

[0070] Figure 8 The multi-camera sensing and recognition device 400 provided by the present invention is merely schematically illustrated. Without violating the inventive concept, the separation, combination, and addition of modules are all within the scope of protection of the present invention. The multi-camera sensing and recognition device 400 provided by the present invention can be implemented using software, hardware, firmware, plug-ins, or any combination thereof, and the present invention is not limited thereto.

[0071] In exemplary embodiments of the present disclosure, a computer-readable storage medium is further provided, on which a computer program is stored. When the program is executed by, for example, a processor, the program can implement the steps of the multi-camera perception and recognition method described in any of the above embodiments. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes program code. When the program product is executed on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the multi-camera perception and recognition method section above.

[0072] refer to Figure 9 , a program product 800 for implementing the above method according to an embodiment of the present invention is described. The program product 800 may be a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0073] The program product may be implemented in any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0074] The computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. The data signal propagated may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or component. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0075] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the tenant computing device, partially on the tenant computing device, as a stand-alone software package, partially on the tenant computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the tenant computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0076] In an exemplary embodiment of the present disclosure, an electronic device is further provided, which may include a processor and a memory for storing executable instructions of the processor. The processor is configured to execute the executable instructions to perform the steps of the multi-camera perception and recognition method described in any of the above embodiments.

[0077] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods, or program products. Therefore, various aspects of the present invention may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0078] Refer to the following Figure 10 An electronic device 600 according to this embodiment of the present invention will be described. Figure 10 The electronic device 600 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0079] like Figure 10 As shown, electronic device 600 is implemented as a general-purpose computing device. Components of electronic device 600 may include, but are not limited to, at least one processing unit 610, at least one storage unit 620, a bus 630 connecting various system components (including storage unit 620 and processing unit 610), a display unit 640, and the like.

[0080] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps of various exemplary embodiments of the present invention described in the multi-camera perception and recognition method section above. For example, the processing unit 610 can perform the following steps: Figure 1Follow the steps shown in .

[0081] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0082] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.

[0083] Bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0084] The electronic device 600 can also communicate with one or more external devices 700 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable tenants to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). This communication can occur via an input / output (I / O) interface 650. Furthermore, the electronic device 600 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 660. The network adapter 660 can communicate with other modules of the electronic device 600 via the bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the electronic device 600, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0085] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, server, or network device, etc.) to execute the above-mentioned multi-camera perception and recognition method according to the embodiment of the present disclosure.

[0086] Compared with the prior art, the advantages of the present invention are:

[0087] By inputting multiple 2D images captured by multiple cameras into a dimensionality conversion model, a 3D feature vector of the vehicle body in a 360-degree horizontal range is obtained, and the 3D feature vector is then input into a multi-task prediction model to predict road objects. This avoids the need for multiple sensor fusion after model detection, reduces the overall algorithm complexity and time consumption compared to post-fusion methods, and improves model recognition accuracy and robustness. Furthermore, only multiple monocular cameras with 360-degree coverage are required, eliminating the need for other sensors such as lidar, reducing costs. Furthermore, after converting multiple monocular 2D images into 3D vector space, they become a single feature input map, which only requires a single model for processing, eliminating the need for deep learning models to process each camera image, saving system resources. The entire multi-camera perception and recognition process is an end-to-end model. Furthermore, the present application generates a road object time series based on the road objects predicted by the multi-task prediction model at the predicted time. This road object time series is input into a time series model, and the restored road objects are obtained based on the output of the time series model. Restoration of road objects based on the time series is performed based on the road objects, avoiding missed or partially missing detections.

[0088] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

Claims

1. A multi-camera perception and recognition method, characterized in that: The multi-camera viewing angle covers a 360-degree horizontal range of the vehicle body, including: Input multiple 2D images captured by multiple cameras into a dimension conversion model to obtain a 3D feature vector of the vehicle body in a 360-degree horizontal range; Inputting the 3D feature vector into a multi-task prediction model to predict road objects; The 3D feature vector of a road object at the same prediction time is regarded as a sequence frame; according to the prediction time, multiple road object time series are generated for multiple road objects based on multiple sequence frames; The road object time series is input into a time series model, and the restored road object is obtained based on the output of the time series model. The time series model includes multiple second convolutional layers, each of which is used to perform convolution and batch normalization on a sequence of frames. The output channel of each second convolutional layer is a three-dimensional coordinate channel, and the output of the time series model is the sum of the output data of each second convolutional layer.

2. The multi-camera perception and recognition method according to claim 1, wherein: The input of the dimensional conversion model is a 2D image, and the output of the dimensional conversion model is a 3D feature vector of the vehicle body in a 360-degree horizontal range. The 3D feature vector is the 3D coordinate of each pixel in the 2D image.

3. The multi-camera perception and recognition method according to claim 2, wherein: The dimensional conversion model includes a multi-layer first convolution layer for processing the input 2D image, the output of the multi-layer first convolution layer is a 3D feature vector of the vehicle body in a 360-degree horizontal range, and the multi-layer first convolution layer forms a residual neural network model.

4. The multi-camera perception and recognition method according to claim 1, wherein: The multi-task prediction model includes a backbone network for extracting features from a 3D feature vector of the vehicle body within a 360-degree horizontal range, and a branch network connected to the backbone network for respectively predicting multiple road objects.

5. A multi-camera perception and recognition device, characterized in that: The multi-camera viewing angle covers a 360-degree horizontal range of the vehicle body, including: A dimensionality conversion module is used to input multiple 2D images collected by multiple cameras into a dimensionality conversion model to obtain a 3D feature vector of the vehicle body in a 360-degree horizontal range; a road object prediction module, configured to input the 3D feature vector into a multi-task prediction model to predict road objects; A time series generation module is used to treat a 3D feature vector of a road object at the same prediction time as a sequence frame; and to generate multiple road object time series for multiple road objects according to the prediction time and multiple sequence frames; A road object restoration module is configured to input the road object time series into a time series model and obtain restored road objects based on the output of the time series model. The time series model includes multiple second convolutional layers, each of which is configured to perform convolution and batch normalization on a sequence of frames. The output channel of each second convolutional layer is a three-dimensional coordinate channel, and the output of the time series model is the sum of the output data of each second convolutional layer.

6. An electronic device, characterized in that: The electronic device comprises: processor; A storage medium having a computer program stored thereon, wherein the computer program, when executed by the processor, executes the multi-camera perception and recognition method according to any one of claims 1 to 4.

7. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, executes the multi-camera perception and recognition method according to any one of claims 1 to 4.

Images