Surround view stitching method based on vehicle vision sensor and related device

Abstract

The application provides a surround view splicing method based on a vehicle vision sensor and related equipment. The method comprises the following steps: acquiring horizontal images collected by a plurality of vision sensors with a horizontal direction as a viewing angle; inputting the acquired plurality of horizontal images into a first feature extraction model and a second feature extraction model in sequence, wherein the second feature extraction model outputs a plurality of overhead feature maps corresponding to the plurality of horizontal images; generating a plurality of overhead images corresponding to the plurality of horizontal images based on the plurality of overhead feature maps; and splicing the plurality of overhead images into a surround view image according to the viewing angle of each vision sensor. The application converts the horizontal images collected by the vision sensor arranged on the vehicle into a surround view overhead image, improves the splicing difficulty, and improves the real-time performance of the overhead image conversion and the surround view image splicing.

Description

Technical Field

[0001] The present invention relates to the field of machine vision and image processing, and in particular to a surround view stitching method based on vehicle vision sensors and related equipment. Background Art

[0002] Currently, in the field of autonomous driving, there is a demand to convert horizontal images captured by cameras into surround-view overhead images in order to obtain multi-dimensional information from planar images, thereby facilitating vehicle positioning and / or map construction during autonomous driving.

[0003] Therefore, how to convert the collected horizontal images into panoramic bird's-eye view images and improve the real-time performance of the bird's-eye view image conversion and the panoramic image stitching is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0004] In order to overcome the defects of the above-mentioned prior art, the present invention provides a surround view stitching method, device, electronic device, and storage medium based on vehicle vision sensors, so as to convert the collected horizontal images into surround view overhead images based on the vision sensors configured on the vehicle, improve the stitching difficulty, and enhance the real-time performance of overhead view image conversion and surround view image stitching.

[0005] According to one aspect of the present invention, a surround view stitching method based on a vehicle vision sensor is provided, comprising:

[0006] Acquire horizontal images collected by multiple visual sensors with horizontal viewing angles;

[0007] Inputting the acquired multiple horizontal images into a first feature extraction model and a second feature extraction model in sequence, wherein the second feature extraction model outputs multiple top-view feature maps corresponding to the multiple horizontal images;

[0008] generating a plurality of overhead images corresponding to a plurality of horizontal images based on the plurality of overhead feature maps;

[0009] The plurality of overhead images are spliced ​​into a surround view image according to the viewing angles of the visual sensors.

[0010] In some embodiments of the present application, sequentially inputting the acquired multiple horizontal images into the first feature extraction model and the second feature extraction model includes:

[0011] Inputting the acquired multiple horizontal images into the first feature extraction model, the first feature extraction model outputting multiple horizontal image features of different sizes corresponding to each of the horizontal images;

[0012] The horizontal image features of the multiple horizontal images of different sizes are input into the second feature extraction model to obtain multiple overhead feature maps corresponding to the multiple horizontal images output by the second feature extraction model.

[0013] In some embodiments of the present application, the first feature extraction model includes a plurality of convolution modules connected in sequence, each of the convolution modules outputs horizontal image features of different dimensions, and the horizontal image features are output by at least part of the convolution modules.

[0014] In some embodiments of the present application, the first feature extraction model is Resnet-50.

[0015] In some embodiments of the present application, the second feature extraction model is used to:

[0016] determining the size of a feature object in the horizontal image feature according to a plurality of horizontal image features of different sizes;

[0017] determining a distance between the feature object and the visual sensor according to a size of the feature object in the horizontal image feature;

[0018] A plurality of horizontal image features of different sizes are converted into a top-view image according to the distance between the feature object and the visual sensor.

[0019] In some embodiments of the present application, the second feature extraction model is a Transformer model.

[0020] In some embodiments of the present application, generating a plurality of overhead images corresponding to a plurality of horizontal images based on the plurality of overhead feature maps includes:

[0021] The plurality of top-view feature maps are respectively input into corresponding upsampling modules to obtain top-view images output by the upsampling modules.

[0022] In some embodiments of the present application, the upsampling module is further configured to perform segmentation of the overhead image to obtain segmentation results of each overhead image.

[0023] In some embodiments of the present application, the acquired multiple horizontal images share model parameters of the first feature extraction model and the second feature extraction model.

[0024] In some embodiments of the present application, stitching the plurality of overhead images into a surround view image according to the viewing angles of the visual sensors includes:

[0025] Determining overlapping areas of adjacent overhead images based on the viewing angles of the visual sensors;

[0026] Based on the confidence levels of the adjacent overhead images, synthesizing and obtaining overlapping images of overlapping areas of the adjacent overhead images;

[0027] The surround view image is obtained by stitching together the non-overlapping images of the overhead images and the overlapping images of the overlapping areas of adjacent overhead images.

[0028] According to another aspect of the present application, a surround view stitching device based on a vehicle visual sensor is provided, comprising:

[0029] A horizontal image acquisition module is used to acquire horizontal images collected by multiple visual sensors with horizontal viewing angles;

[0030] A top-view feature map acquisition module, configured to sequentially input the acquired multiple horizontal images into a first feature extraction model and a second feature extraction model, wherein the second feature extraction model outputs multiple top-view feature maps corresponding to the multiple horizontal images;

[0031] A bird's-eye view image acquisition module, configured to generate a plurality of bird's-eye view images corresponding to a plurality of horizontal images based on a plurality of the bird's-eye view feature maps;

[0032] The surround view stitching module is used to stitch the multiple overhead images into a surround view image according to the viewing angles of the visual sensors.

[0033] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; and a storage medium storing a computer program, wherein the computer program executes the above steps when executed by the processor.

[0034] According to yet another aspect of the present invention, a storage medium is provided, wherein a computer program is stored on the storage medium, and when the computer program is executed by a processor, the steps described above are executed.

[0035] Compared with the prior art, the advantages of the present invention are:

[0036] By sequentially inputting horizontal images collected by multiple visual sensors into a first feature extraction model and a second feature extraction model, a plurality of overhead feature maps corresponding to the plurality of horizontal images are obtained from the output of the second feature extraction model, and a plurality of overhead images corresponding to the plurality of horizontal images are generated based on the plurality of overhead feature maps. According to the viewing angles of the respective visual sensors, the plurality of overhead images are spliced ​​into a surround-view image, thereby converting the collected horizontal images into surround-view overhead images. At the same time, by first converting the horizontal images into overhead images and then performing surround-view splicing, the real-time performance of overhead image conversion and surround-view image splicing is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and other features and advantages of the present invention will become more apparent by describing in detail exemplary embodiments thereof with reference to the accompanying drawings.

[0038] Figure 1 A flowchart of a surround view stitching method based on a vehicle visual sensor according to an embodiment of the present invention is shown;

[0039] Figure 2 The following figure shows the model intent of the surround view stitching method based on the vehicle visual sensor according to an embodiment of the present invention;

[0040] Figure 3 shows a model schematic diagram of a first feature extraction model according to an embodiment of the present invention;

[0041] Figure 4 FIG2 shows a schematic diagram of perspective conversion according to a second feature extraction model according to an embodiment of the present invention;

[0042] Figure 5 A schematic diagram of a surround view image according to a specific embodiment of the present invention is shown;

[0043] Figure 6 A module diagram of a surround view stitching device based on a vehicle visual sensor according to an embodiment of the present invention is shown;

[0044] Figure 7 Schematically illustrates a computer-readable storage medium in an exemplary embodiment of the present disclosure;

[0045] Figure 8 The figure schematically shows a diagram of an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0047] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0048] In order to solve the defects of the prior art, the present invention provides a surround view stitching method based on vehicle visual sensors. Figure 1 , Figure 1 A flowchart of a surround view stitching method based on a vehicle vision sensor according to an embodiment of the present invention is shown. Figure 1 The steps are as follows:

[0049] Step S110: Acquire horizontal images captured by multiple visual sensors with horizontal viewing angles;

[0050] Step S120: inputting the acquired multiple horizontal images into a first feature extraction model and a second feature extraction model in sequence, wherein the second feature extraction model outputs multiple top-view feature maps corresponding to the multiple horizontal images;

[0051] Step S130: generating a plurality of overhead images corresponding to a plurality of horizontal images based on the plurality of overhead feature maps;

[0052] Step S140: stitching the plurality of overhead images into a surround view image according to the viewing angles of the visual sensors.

[0053] In the surround view stitching method based on vehicle vision sensors provided by the present invention, horizontal images collected by multiple vision sensors are sequentially input into a first feature extraction model and a second feature extraction model, and multiple overhead feature maps corresponding to the multiple horizontal images are obtained by the output of the second feature extraction model. Based on the multiple overhead feature maps, multiple overhead images corresponding to the multiple horizontal images are generated. According to the viewing angles of each of the vision sensors, the multiple overhead images are stitched into surround view images, thereby converting the collected horizontal images into surround view overhead images. At the same time, by first converting the horizontal images into overhead images and then performing surround view stitching, the real-time performance of overhead image conversion and surround view image stitching is improved.

[0054] Specifically, the visual sensor may be, for example, a camera module, and the horizontal viewing angle may have a certain amount of deviation, that is, with the horizontal direction as the reference, there may be a deviation angle in the vertical direction. Furthermore, the images collected by multiple visual sensors with horizontal viewing angles may cover a 360-degree range of the vehicle body, thereby facilitating the subsequent surround-view stitching steps. In some specific embodiments, a visual sensor may be set in each of the four directions of the vehicle body. For example, it may be set at the four corners of the vehicle body or at the midpoints of the four faces of the vehicle. This application is not limited to this, and the setting position of each visual sensor is not limited to this. At the same time, the setting position and setting parameters of each visual sensor will be saved on the vehicle side and / or the server side to facilitate determining the viewing angle range of each visual sensor in the subsequent surround-view stitching steps.

[0055] Specifically, the step S120 may include the following steps: inputting the acquired multiple horizontal images into the first feature extraction model, the first feature extraction model outputting multiple horizontal image features of different sizes corresponding to each of the horizontal images; inputting the horizontal image features of different sizes of the multiple horizontal images into the second feature extraction model, and obtaining multiple overhead feature maps corresponding to the multiple horizontal images output by the second feature extraction model. Thus, horizontal image features of different sizes (different dimensions) of each of the horizontal images are extracted by the first feature extraction model. Horizontal image features of different sizes (different dimensions) can provide different feature information to the second feature extraction model (for example, horizontal image features of larger sizes are more conducive to the overhead perspective conversion of farther objects; while horizontal image features of smaller sizes are more conducive to the overhead perspective conversion of closer objects), so that the second feature extraction model can obtain richer image information, which is more conducive to the perspective conversion of the second feature extraction model and improves the robustness of the model.

[0056] Specifically, in the above embodiment, the first feature extraction model includes a plurality of convolution modules connected in sequence, each of which outputs horizontal image features of different dimensions, and the horizontal image features are output by at least some of the convolution modules. In a specific implementation, Figure 3 As shown, the first feature extraction model 202 includes a first convolution module 2021, a second convolution module 2022, a third convolution module 2023, a fourth convolution module 2024, and a fifth convolution module 2025 connected in sequence. Each convolution module can be connected through a batch normalization layer, an activation function layer, a maximum pooling layer, and one or more identification modules (Identity block), but this application is not limited to this. Each convolution module can include a different number of two-dimensional convolution layers, batch normalization layers, activation function layers, etc. as needed, and the convolution parameters of the two-dimensional convolution layers of each convolution module can also be as needed. Figure 3Taking horizontal image 1 (labeled 201) as an example, horizontal image 1 is input to the first convolution module 2021 of the first feature extraction model 202. The first convolution module 2021 performs convolution (and operations such as batch planning one, activation, etc.) on horizontal image 1 to obtain horizontal image features with a first size. The horizontal image features with the first size are input to the second convolution module 2022 of the first feature extraction model 202. The second convolution module 2022 performs convolution (and operations such as batch planning one, activation, etc.) on horizontal image 1 to obtain horizontal image features with a second size. Similarly, the third convolution module 2023 outputs horizontal image features with a third size, the fourth convolution module 2024 outputs horizontal image features with a fourth size, and the fifth convolution module 2025 outputs horizontal image features with a fifth size. Among them, the first size to the fifth size decrease in sequence. In some specific implementations, only the horizontal image features with the second size output by the second convolution module 2022 to the horizontal image features with the fifth size output by the fifth convolution module 2025 can be input into the second feature extraction module 203 to strike a balance between the conversion accuracy and conversion speed of the second feature extraction module 203. The present application can implement more variations. For example, the outputs of all convolution modules of the first feature extraction model can be input into the second feature extraction module; only the output of one convolution module of the first feature extraction model can be input into the second feature extraction module, etc. The present application is not limited to this.

[0057] In some specific implementations, the first feature extraction model can be Resnet-50 (Residual net). Residual networks are used to solve the problem of gradient vanishing during model training and the difficulty of training deep networks. This problem is that the gradient is backpropagated to the previous layer, and repeated multiplication may make the gradient infinitesimal. Therefore, as the number of layers of the network becomes deeper, its performance tends to saturate or even declines rapidly. Therefore, the residual network can make the depth of the model no longer limited by the gradient vanishing, so that the first feature extraction model can obtain horizontal image features of different sizes by increasing the depth.

[0058] Specifically, the second feature extraction model can be used to: determine the size of a feature object in a plurality of horizontal image features of different sizes; determine the distance between the feature object and the visual sensor based on the size of the feature object in the horizontal image features; and convert the plurality of horizontal image features of different sizes into a top-view image based on the distance between the feature object and the visual sensor. Figure 4 , Figure 4A schematic diagram of perspective conversion of a second feature extraction model according to an embodiment of the present invention is shown. A horizontal image feature 301 of size C*H*W (where C is the number of channels, H is the height, and W is the width) is input into the second feature extraction model. The second feature extraction model can fold the horizontal image feature 301 of size C*H*W into a bottleneck feature 302 of size B*W along the height direction by means such as a fully connected layer. Then, the second feature extraction model can expand the bottleneck feature of size B*W along the width direction by means such as a fully connected layer into a polar top-view feature map 303 of size C*Z*W. Then, the second feature extraction model can restore the polar top-view feature map 303 to a top-view feature map 304 of size C*Z*W by means such as Cartesian resampling. Figure 4 It is only a schematic illustration of the process of converting the second feature extraction model into a bird's-eye view feature map based on horizontal image features, and the present application is not limited to this. Furthermore, the second feature extraction model can convert horizontal image features of different sizes of the same horizontal image to obtain bird's-eye view feature maps of different distance ranges, and splice the bird's-eye view feature maps into bird's-eye view feature maps corresponding to the horizontal image. The present application is not limited to this. In some variations, different weights can also be assigned to horizontal image features of different sizes. Horizontal image features with larger sizes have higher weights for the farther parts of the bird's-eye view feature map, and horizontal image features with smaller sizes have higher weights for the closer parts of the bird's-eye view feature map. Thus, based on different weight assignments, bird's-eye view feature maps generated based on horizontal image features of different sizes of the same horizontal image are spliced ​​into bird's-eye view feature maps corresponding to the horizontal image. The present application can implement more variations, which will not be elaborated here.

[0059] In some specific implementations of the present application, the second feature extraction model is a Transformer model, but the present application is not limited to this.

[0060] See below Figure 2 , Figure 2 The figure shows the model concept of the surround view stitching method according to an embodiment of the present invention.

[0061] The horizontal images 1-n (labeled 201) are sequentially input to the first feature extraction module 202 and the second feature extraction model 203. The multiple top-view feature maps corresponding to the horizontal images 1-n output by the second feature extraction model 203 are respectively input to multiple upsampling modules 1-n (labeled 204). The top-view images 1-n (labeled 205) output by each upsampling module 1-n are respectively subjected to surround stitching 206 to obtain a surround image 207. The obtained surround image can be as follows Figure 5 shown.

[0062] like Figure 2 As shown, since the conversion process from the horizontal perspective to the top-view perspective of different cameras is similar, the horizontal images 1-n (labeled 201) share the first feature extraction module 202 and the second feature extraction model 203, so there is no need to provide a corresponding first feature extraction module 202 and second feature extraction model 203 for each horizontal image 1-n. In some variations, the horizontal images 1-n (labeled 201) can also share the model parameters of the first feature extraction module 202 and the second feature extraction model 203, so that the acquisition of the top-view feature map of each horizontal image 1-n (labeled 201) can be performed in parallel. This embodiment does not require the training of multiple models and can improve the efficiency of acquiring the top-view feature map.

[0063] Furthermore, the first feature extraction module 202 and the second feature extraction module 203 can be trained using open source datasets such as ImageNet and Nuscenes, so that the first feature extraction module 202 and the second feature extraction module 203 have higher robustness.

[0064] Specifically, since the visual sensors that collect different horizontal images 1-n are installed in different orientations, the upsampling step of the overhead feature map is performed by multiple upsampling modules 1-n that do not share model parameters, so that each horizontal image 1-n can be converted into an accurate overhead image 1-n (labeled 205). In some specific implementations, each of the upsampling modules 1-n is also used to perform segmentation of the overhead image to obtain a segmentation result of each overhead image. Specifically, the segmentation of the overhead image is used to distinguish the pixel areas corresponding to different objects in the overhead image. In some preferred examples, the segmentation result can also include identification labels for different objects to mark each pixel area as a corresponding object.

[0065] Specifically, when performing surround stitching 206, the following steps may also be included: determining the overlapping area of ​​adjacent overhead images based on the viewing angle of each of the visual sensors; synthesizing the overlapping images of the overlapping areas of adjacent overhead images based on the confidence of the adjacent overhead images; stitching the surround image based on the non-overlapping images of each overhead image and the overlapping images of the overlapping areas of adjacent overhead images. Furthermore, when performing surround stitching 206, the external parameter relationship between the visual sensors may be first determined based on the viewing angle and installation position of each of the visual sensors, and each overhead image may be converted to the same coordinate system. In some specific implementations, a calibration plate may be used to determine the combination method of the overlapping areas between the overhead images of the visual sensors. However, for some cases where calibration is difficult and the sensitivity to surround segmentation details is low, the confidence level may be used to assist in the combination of the overlapping areas. For example, the overlapping image I of the overlapping areacross It can be obtained according to the following formula:

[0066] I cross =(1-P mask )*I k +P mask *I k-1 ,

[0067] Among them, I k is the image in the overlapping area of ​​the adjacent overhead images k and k-1 in the overhead image k, I k-1 The image in the overlapping area of ​​the adjacent overhead image k and k-1 in the overhead image k-1 is the adjacent overhead image, P mask is the confidence level of the top-view image k-1. In some embodiments, the confidence level can be output along with the top-view feature map when the second feature extraction model outputs the top-view feature map. This eliminates the need for an additional confidence level calculation step and improves the overall efficiency of the model. This is not intended to be limiting; other methods for obtaining confidence levels are also within the scope of this application.

[0068] In some specific applications of this application, the obtained bird's-eye view surround image can be used to filter the laser point cloud. Specifically, dynamic obstacles can be identified based on the surround image at consecutive moments, thereby filtering the laser point cloud corresponding to these dynamic obstacles. In a specific implementation, after segmenting the bird's-eye view image to obtain segmentation results and feature labels, a preliminary screening for dynamic obstacles can be performed based on the feature labels. For example, segmented features with feature labels such as pedestrians and vehicles are necessarily dynamic obstacles and can be pre-labeled, eliminating the need for subsequent dynamic obstacle identification and filtering within the laser point cloud. This reduces the computational effort required to identify dynamic obstacles, improves laser surround view stitching efficiency, and reduces system load. Furthermore, in some variations, static feature labels, such as plants and buildings, can be maintained. Since these static features are inherently immobile, preliminary screening can be performed after determining the feature labels of the segmentation results. This eliminates the need to identify dynamic obstacles for segmentation results whose feature labels match the static feature labels. The laser point cloud corresponding to these segmented features will be retained without filtering. This reduces the computational effort required to identify dynamic obstacles, improves laser point cloud filtering efficiency, and reduces system load. In some implementations, after segmenting the overhead image to obtain segmentation results and feature labels, the feature labels of the segmentation results can be assigned to the filtered and retained laser point cloud. This eliminates the need for object recognition and feature labeling of the laser point cloud, reduces the computational effort required to assign feature labels to the laser point cloud, improves laser point cloud object recognition efficiency, and reduces system load.

[0069] The above are merely a few specific implementations of the vehicle vision sensor-based surround view stitching method of the present invention. Each implementation can be implemented independently or in combination, and the present invention is not limited thereto. Furthermore, the flowchart of the present invention is merely illustrative, and the order of execution of the steps is not limited thereto. Steps can be split, combined, swapped, or executed synchronously or asynchronously in any other manner, all within the scope of the present invention.

[0070] The present invention also provides a surround view splicing device based on a vehicle visual sensor, Figure 6 The module diagram of a surround view stitching device based on a vehicle vision sensor according to an embodiment of the present invention is shown. The surround view stitching device based on a vehicle vision sensor 400 includes a horizontal image acquisition module 410, a top view feature map acquisition module 420, a top view image acquisition module 430, and a surround view stitching module 440.

[0071] The horizontal image acquisition module 410 is used to acquire horizontal images collected by a plurality of visual sensors with a horizontal viewing angle;

[0072] The top-view feature map acquisition module 420 is used to sequentially input the acquired multiple horizontal images into a first feature extraction model and a second feature extraction model, and the second feature extraction model outputs multiple top-view feature maps corresponding to the multiple horizontal images;

[0073] The overhead image acquisition module 430 is configured to generate a plurality of overhead images corresponding to a plurality of horizontal images based on the plurality of overhead feature maps;

[0074] The surround view stitching module 440 is configured to stitch the plurality of overhead view images into a surround view image according to the viewing angles of the visual sensors.

[0075] In the surround view stitching device based on vehicle vision sensors provided by the present invention, horizontal images collected by multiple vision sensors are sequentially input into a first feature extraction model and a second feature extraction model, and multiple overhead feature maps corresponding to the multiple horizontal images are obtained by the output of the second feature extraction model. Based on the multiple overhead feature maps, multiple overhead images corresponding to the multiple horizontal images are generated. According to the viewing angles of each of the vision sensors, the multiple overhead images are stitched into surround view images, thereby converting the collected horizontal images into surround view overhead images. At the same time, by first converting the horizontal images into overhead images and then performing surround view stitching, the real-time performance of overhead image conversion and surround view image stitching is improved.

[0076] Figure 6The vehicle vision sensor-based surround view stitching device 400 provided by the present invention is merely schematically illustrated. Without violating the inventive concept, the separation, combination, and addition of modules are all within the scope of protection of the present invention. The vehicle vision sensor-based surround view stitching device 400 provided by the present invention can be implemented using software, hardware, firmware, plug-ins, or any combination thereof, and the present invention is not limited thereto.

[0077] In exemplary embodiments of the present disclosure, a computer-readable storage medium is also provided, storing a computer program. When executed by, for example, a processor, the program can implement the steps of the surround view stitching method based on a vehicle vision sensor described in any of the aforementioned embodiments. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, comprising program code. When executed on a terminal device, the program code causes the terminal device to execute the steps of the various exemplary embodiments of the present invention described in the aforementioned surround view stitching method based on a vehicle vision sensor section of this specification.

[0078] refer to Figure 7 , a program product 800 for implementing the above method according to an embodiment of the present invention is described. The program product 800 may be a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0079] The program product may be implemented in any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0080] The computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. The data signal propagated may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or component. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0081] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the tenant computing device, partially on the tenant computing device, as a stand-alone software package, partially on the tenant computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the tenant computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0082] In an exemplary embodiment of the present disclosure, an electronic device is further provided, which may include a processor and a memory for storing executable instructions of the processor. The processor is configured to execute the executable instructions to perform the steps of the surround view stitching method based on a vehicle vision sensor described in any of the above embodiments.

[0083] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods, or program products. Therefore, various aspects of the present invention may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0084] Refer to the following Figure 8 An electronic device 600 according to this embodiment of the present invention will be described. Figure 8 The electronic device 600 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0085] like Figure 8 As shown, electronic device 600 is implemented as a general-purpose computing device. Components of electronic device 600 may include, but are not limited to, at least one processing unit 610, at least one storage unit 620, a bus 630 connecting various system components (including storage unit 620 and processing unit 610), a display unit 640, and the like.

[0086] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps of various exemplary embodiments of the present invention described in the above-mentioned surround view stitching method based on vehicle visual sensors. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0087] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0088] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.

[0089] Bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0090] The electronic device 600 can also communicate with one or more external devices 700 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable tenants to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). This communication can occur via an input / output (I / O) interface 650. Furthermore, the electronic device 600 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 660. The network adapter 660 can communicate with other modules of the electronic device 600 via the bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the electronic device 600, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0091] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, server, or network device, etc.) to execute the above-mentioned surround view stitching method based on vehicle vision sensors according to the embodiments of the present disclosure.

[0092] Compared with the prior art, the advantages of the present invention are:

[0093] By sequentially inputting horizontal images collected by multiple visual sensors into a first feature extraction model and a second feature extraction model, a plurality of overhead feature maps corresponding to the plurality of horizontal images are obtained from the output of the second feature extraction model, and a plurality of overhead images corresponding to the plurality of horizontal images are generated based on the plurality of overhead feature maps. According to the viewing angles of the respective visual sensors, the plurality of overhead images are spliced ​​into a surround-view image, thereby converting the collected horizontal images into surround-view overhead images. At the same time, by first converting the horizontal images into overhead images and then performing surround-view splicing, the real-time performance of overhead image conversion and surround-view image splicing is improved.

[0094] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

Claims

1. A surround view stitching method based on vehicle vision sensor, characterized in that: include: Acquire multiple horizontal images captured by a visual sensor with a horizontal viewing angle; input the multiple horizontal images acquired into a first feature extraction model, the first feature extraction model outputting multiple horizontal image features of different sizes corresponding to each of the horizontal images; input the multiple horizontal image features of different sizes of the horizontal images into a second feature extraction model, and obtain multiple overhead feature maps corresponding to the multiple horizontal images output by the second feature extraction model; the second feature extraction model is used to: determine the size of the feature object in the horizontal image feature based on the multiple horizontal image features of different sizes; determine the distance between the feature object and the visual sensor based on the size of the feature object in the horizontal image feature; assign weights to the horizontal image features based on the distance between the feature object and the visual sensor, horizontal image features with larger sizes have higher weights for the farther parts of the overhead feature map, and horizontal image features with smaller sizes have higher weights for the closer parts of the overhead feature map, and splice the overhead feature maps generated based on horizontal image features of different sizes of the same horizontal image into an overhead feature map corresponding to the horizontal image; generating a plurality of overhead images corresponding to a plurality of horizontal images based on the plurality of overhead feature maps; The plurality of overhead images are spliced ​​into a surround view image according to the viewing angles of the visual sensors.

2. The surround view stitching method based on vehicle visual sensor according to claim 1, characterized in that: The first feature extraction model includes a plurality of convolution modules connected in sequence, each of the convolution modules outputs horizontal image features of different dimensions, and the horizontal image features are output by at least part of the convolution modules.

3. The surround view stitching method based on vehicle visual sensor according to claim 2, characterized in that: The first feature extraction model is Resnet-50.

4. The surround view stitching method based on vehicle visual sensor according to claim 1, characterized in that: The second feature extraction model is a Transformer model.

5. The surround view stitching method based on vehicle visual sensor according to claim 1, characterized in that: The step of generating a plurality of overhead images corresponding to a plurality of horizontal images based on the plurality of overhead feature maps comprises: The plurality of top-view feature maps are respectively input into corresponding upsampling modules to obtain top-view images output by the upsampling modules.

6. The surround view stitching method based on vehicle visual sensor according to claim 5, characterized in that: The upsampling module is further configured to perform segmentation of the overhead image to obtain a segmentation result of each overhead image.

7. The surround view stitching method based on vehicle visual sensor according to claim 1, characterized in that: The acquired multiple horizontal images share model parameters of the first feature extraction model and the second feature extraction model.

8. The surround view stitching method based on vehicle visual sensor according to claim 1, characterized in that: The step of stitching the plurality of overhead images into a surround view image according to the viewing angles of the visual sensors includes: Determining overlapping areas of adjacent overhead images based on the viewing angles of the visual sensors; Based on the confidence levels of the adjacent overhead images, synthesizing and obtaining overlapping images of overlapping areas of the adjacent overhead images; The surround view image is obtained by stitching together the non-overlapping images of the overhead images and the overlapping images of the overlapping areas of adjacent overhead images.

9. A surround view stitching device based on vehicle vision sensor, characterized in that: include: A horizontal image acquisition module is used to acquire horizontal images collected by multiple visual sensors with horizontal viewing angles; A bird's-eye view feature map acquisition module is configured to input the acquired multiple horizontal images into a first feature extraction model, the first feature extraction model outputting multiple horizontal image features of different sizes corresponding to each of the horizontal images; inputting the horizontal image features of the multiple horizontal images of different sizes into a second feature extraction model to obtain multiple bird's-eye view feature maps corresponding to the multiple horizontal images output by the second feature extraction model; the second feature extraction model is configured to: determine the size of the feature object in the horizontal image features based on the multiple horizontal image features of different sizes; determine the distance between the feature object and the visual sensor based on the size of the feature object in the horizontal image features; assign weights to the horizontal image features based on the distance between the feature object and the visual sensor, with larger horizontal image features having higher weights for farther portions of the bird's-eye view feature map, and smaller horizontal image features having higher weights for closer portions of the bird's-eye view feature map; and splicing the bird's-eye view feature maps generated based on horizontal image features of different sizes from the same horizontal image into a bird's-eye view feature map corresponding to the horizontal image; A bird's-eye view image acquisition module, configured to generate a plurality of bird's-eye view images corresponding to a plurality of horizontal images based on a plurality of the bird's-eye view feature maps; The surround view stitching module is used to stitch the multiple overhead images into a surround view image according to the viewing angles of the visual sensors.

10. An electronic device, characterized in that: The electronic device comprises: processor; A storage medium having a computer program stored thereon, wherein the computer program, when executed by the processor, executes the surround view stitching method based on a vehicle vision sensor according to any one of claims 1 to 8.

11. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the surround view stitching method based on the vehicle vision sensor according to any one of claims 1 to 8 is executed.

Images