Multi-camera image processing method and system, electronic equipment and medium

Abstract

The invention provides a multi-camera image processing method and system, electronic equipment and a medium, and belongs to the technical field of aerial view, and the method comprises the steps: obtaining fisheye images shot by a plurality of fisheye cameras; converting the image features of each fisheye image into corresponding aerial view features through a feature transmission matrix; and performing aerial view feature fusion through a distortion perception learning pooling layer. According to the method, the advantage of a large field angle of the fisheye camera is utilized, efficient feature conversion is achieved in combination with matrix multiplication, feature fusion is optimized through a distortion perception pooling layer, and the problem of radial image distortion of the fisheye camera is effectively solved; according to the invention, through feature compression and matrix decomposition technologies, the calculation amount and memory occupation of multi-camera BEV feature fusion are greatly reduced; the shielding reasoning module is introduced, and the shielding problem in the BEV space can be effectively solved.

Description

Technical Field

[0001] This invention relates to the field of bird's-eye view technology, and in particular to a multi-camera image processing method and system, electronic device, and medium. Background Technology

[0002] In the fields of autonomous driving and robotics, bird's-eye view (BEV) technology has become one of the core means of environmental perception. BEV can unify data from multiple sensors (such as cameras, LiDAR, etc.) into a global perspective, providing an intuitive environmental representation for autonomous driving systems, facilitating subsequent path planning, obstacle detection, and decision control. However, existing BEV technologies still face many challenges in areas such as multi-camera fusion, fisheye camera processing, and occlusion inference. Summary of the Invention

[0003] This invention aims to solve at least one of the above-mentioned problems in the prior art and proposes an efficient fisheye image to bird's-eye view method. This method is based on multi-camera BEV feature extraction and semantic segmentation using fisheye cameras. Through efficient feature extraction, fusion and occlusion inference, it improves the accuracy and efficiency of BEV technology and provides more reliable environmental perception capabilities for autonomous driving systems.

[0004] In a first aspect, embodiments of the present invention provide a multi-camera image processing method, comprising: Acquire fisheye images captured by multiple fisheye cameras; The image features of each fisheye image are converted into corresponding bird's-eye view features using a feature transfer matrix; Bird's-eye view feature fusion is performed by learning pooling layers through distortion perception.

[0005] In a preferred embodiment, the step of converting the image features of each fisheye image into corresponding bird's-eye view features through a feature transfer matrix includes: Let the image features of a fisheye camera be... ,in It refers to the number of cameras. and These are the height and width of the image features. This is the number of feature channels, and the feature transfer matrix is... The bird's-eye view features are ,in and It is the height and width of the bird's-eye view grid. It is the number of bins at depth; The image features of a fisheye image are converted into bird's-eye view features using the following formula: , In formula (1), It is an intermediate feature obtained by combining image features F with depth distribution D through an outer product operation.

[0006] In a preferred embodiment, the step of fusing bird's-eye view features through a distortion-aware learning pooling layer includes: For each bird's-eye view grid, the feature weights from different cameras are calculated using a weighting function; The final bird's-eye view features are generated using the following formula for learning pooling layers with distortion perception: , In formula (2), It is the bird's-eye view feature of the k-th camera. These are the corresponding weights.

[0007] In a preferred embodiment, before the step of converting the image features of each fisheye image into corresponding bird's-eye view features using a feature transfer matrix, the method further includes a step of compressing the image features of each fisheye image using a PE module. Let the image features of a fisheye image be... ,in It refers to the number of cameras. and These are the height and width of the image features, and C is the number of feature channels; The PE module is used to compress image features through max pooling and convolution operations to obtain compressed image features. and depth distribution The calculation formula for the PE module is as follows: , In formula (3), MaxPool represents the max pooling operation and Conv represents the convolution operation.

[0008] In a preferred embodiment, the step of converting the image features of each fisheye image into corresponding bird's-eye view features through a feature transfer matrix includes: Use the RR module to transfer the feature matrix Decomposed into Ring matrices and Ray matrix The Ring matrix encodes distance information, and the Ray matrix encodes direction information. The image features of a fisheye image are converted into bird's-eye view features using the following formula: , In formula (4), This represents the Hadamard product.

[0009] In a preferred embodiment, the method further includes a step of processing occlusion issues in the bird's-eye view space through an occlusion inference module: It has There are 1 camera, and the image features of each camera are: For each camera, the occlusion inference module is used to extract image features. Projecting onto the bird's-eye view space yields bird's-eye view features. ; The occlusion inference module uses the following formula to fuse the bird's-eye view features from multiple cameras to obtain the fused bird's-eye view features. : , In formula (5), It is the weight of the k-th camera; The occlusion inference module calculates the occlusion probability p(o) of each bird's-eye view grid by statistically analyzing its visibility under different camera views. The formula is as follows: , In formula (6), It is an indicator function, indicating whether the k-th camera has observed the bird's-eye view grid. It is the normalized threshold; Using occlusion probability The features of the bird's-eye view are weighted to obtain the final semantic segmentation result of the bird's-eye view. The specific formula is as follows: , In formula (7), Represents the Hadamard product. This represents the visibility probability of the grid.

[0010] In a preferred embodiment, the camera weights are dynamically adjusted using the following formula. : , In formula (8), It's about adjusting parameters. It is the occlusion probability of the k-th camera.

[0011] In a second aspect, embodiments of the present invention provide a multi-camera image processing system, the system being configured to implement any of the methods described in the first aspect, the system comprising: a first acquisition unit, configured to acquire fisheye images captured by multiple fisheye cameras; and a feature extraction and fusion unit, configured to convert the image features of each fisheye image into corresponding bird's-eye features through a feature transfer matrix, and to perform bird's-eye feature fusion through a distortion-aware learning pooling layer.

[0012] Thirdly, embodiments of the present invention provide an electronic device, including: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement any of the methods described in the first aspect.

[0013] Fourthly, embodiments of the present invention provide a computer-readable medium storing a computer program that, when executed by a processor, implements the steps of any of the methods described in the first aspect.

[0014] Beneficial effects of this invention: This invention utilizes the large field of view advantage of fisheye cameras, combines matrix multiplication to achieve efficient feature transformation, and performs feature fusion through distortion-aware pooling layers. This effectively solves the radial distortion problem of fisheye camera images, significantly improves the accuracy and robustness of feature extraction, and ensures accurate capture of environmental information even in complex scenes. This invention significantly reduces the computational load and memory consumption of multi-camera BEV feature fusion through feature compression and matrix factorization techniques, making the feature fusion process more efficient and suitable for autonomous driving scenarios with high real-time requirements. Based on efficient feature transformation, this invention introduces an occlusion inference module, which can effectively handle the occlusion problem in BEV space. Combined with a multi-task head design, the model can perform semantic classification and occlusion prediction simultaneously, significantly improving the semantic segmentation accuracy in complex scenarios. This invention not only improves the accuracy and efficiency of BEV feature extraction and semantic segmentation, but also provides more reliable environmental perception capabilities for autonomous driving systems. The technical solution of this invention is not only applicable to the field of autonomous driving, but can also be widely applied to scenarios that require environmental perception, such as robot navigation and intelligent transportation systems, providing a new solution for the application of multi-camera systems. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall process of a multi-camera image processing method provided in an embodiment of the present invention.

[0016] Figure 2 A flowchart illustrating a multi-camera image processing method provided in this embodiment of the invention. Figure 1 .

[0017] Figure 3 A flowchart illustrating a multi-camera image processing method provided in this embodiment of the invention. Figure 2 .

[0018] Figure 4A flowchart illustrating a multi-camera image processing method provided in this embodiment of the invention. Figure 3 .

[0019] Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0022] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0023] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0024] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0025] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0026] In this invention, some technical terms have the following meanings: BEV: Bird's Eye View, usually refers to a bird's-eye view generated by converting fisheye or multi-camera images, used for autonomous driving perception and scene understanding.

[0027] LSS: Lift-Splat-Shoot is a deep learning model that projects two-dimensional image features into three-dimensional space to generate BEV maps.

[0028] FTM: Feature Transfer Matrix, used to extract features from fisheye images and efficiently generate BEV views.

[0029] FOV: Field of view, the range of the scene that the camera can capture.

[0030] IPM: Inverse Perspective Mapping, a method for converting an image from a perspective view to a bird's-eye view (BEV) view.

[0031] PE technology: Prime Extraction is a technique in the MatrixVT method used to optimize BEV feature extraction. Its core is to achieve efficient multi-camera feature conversion to bird's-eye view (BEV) through feature compression and matrix operations.

[0032] RR technique: Ring & Ray Decomposition is a technique in the MatrixVT method used to optimize the computational efficiency of feature transformation from multiple cameras to bird's-eye view (BEV). Its core is to reduce computational complexity by splitting the dimensions of the feature transformation matrix (FTM).

[0033] Fisheye cameras are widely used in autonomous driving due to their large field of view, but their images suffer from severe radial distortion, making traditional feature extraction methods difficult to apply directly. Existing methods typically eliminate distortion through image correction (such as cylinder correction), but this process results in the loss of some field of view and introduces interpolation artifacts. Furthermore, the nonlinear optical characteristics of fisheye cameras make feature extraction and fusion more complex, and existing methods often perform poorly when dealing with fisheye cameras.

[0034] In related technologies, multi-camera BEV feature extraction methods mainly rely on geometric transformations and deep learning. Geometric transformation methods (such as inverse perspective mapping, IPM) project image features into the BEV space using camera parameters, but they assume a flat ground surface, making them difficult to handle complex scenes. Deep learning methods (such as Lift-Splat-Shoot, LSS) "lift" image features into 3D space by predicting pixel depth distribution, and then "project" them into the BEV space. However, these methods have limitations when dealing with fisheye cameras. While the large field of view of fisheye cameras can capture more environmental information, their images suffer from severe radial distortion, leading to inaccurate feature extraction when directly applying existing methods.

[0035] The challenge of multi-camera BEV feature fusion lies in how to efficiently integrate feature information from different cameras. Existing methods typically employ symmetric pooling (such as mean pooling and max pooling) to fuse features, but this approach ignores the characteristics of different cameras (such as field of view and resolution), resulting in poor fusion performance. Furthermore, the high dimensionality of multi-camera features significantly increases computational and memory consumption, limiting the real-time performance of the algorithm. While some studies have attempted to reduce computational cost through feature compression (such as channel pruning and matrix factorization), these methods often sacrifice feature integrity, affecting the accuracy of the BEV representation.

[0036] The goal of BEV semantic segmentation is to classify each grid cell in the BEV space into a specific semantic category (such as vehicle, road, pedestrian, etc.). Existing methods are typically based on deep learning models (such as U-Net and BEVFormer), but in complex scenes, occlusion problems severely affect the accuracy of segmentation. When an object is occluded by other objects in the BEV space, existing methods struggle to accurately infer the semantic information of the occluded region, resulting in incomplete segmentation results. While some studies have attempted to alleviate this problem through occlusion inference modules, these modules often rely on complex post-processing or additional sensor data (such as LiDAR), increasing the complexity and cost of the system.

[0037] One related technology proposes an obstacle detection method and device based on a fisheye camera, mainly addressing the problem of inaccurate obstacle detection caused by image distortion in fisheye cameras during autonomous driving. The core steps of this technical solution include: acquiring fisheye images with overlapping perspectives, converting them into a top-down view, calculating the true height of obstacles through sub-block division and height assumptions, and performing clustering and labeling. This technical solution solves the distortion problem of fisheye cameras through top-down view transformation, is applicable to the detection of any type of target, and reduces missed detections and false detections. However, this technical solution has the following drawbacks: 1. Although it uses multi-camera information, it does not involve feature compression and efficient fusion, resulting in high computational complexity; 2. It does not involve BEV (bird's-eye view) feature extraction and fusion, failing to fully utilize the large field of view advantage of fisheye cameras for global environmental perception; 3. It does not involve occlusion reasoning, failing to effectively handle occlusion problems in complex scenes, potentially leading to missed or false detections in target identification.

[0038] The second related technology proposes a target object recognition method suitable for fisheye cameras, mainly addressing the problem of inaccurate target object localization caused by image distortion in roadside scenes using fisheye cameras. The core steps of this technical solution include: acquiring images captured by the fisheye camera, performing distortion correction, identifying targets and correcting the pixel coordinates of the detection box center point, and finally calculating the geodetic coordinates of the target object using a projection transformation matrix. This technical solution reduces the impact of image edge distortion on localization by correcting pixel coordinates, thus improving the accuracy of target object recognition. However, this technical solution has the following drawbacks: 1. It relies entirely on a single viewpoint and lacks the fusion of multi-camera information; 2. It does not involve BEV (bird's-eye view) feature extraction and fusion, failing to fully utilize the large field of view advantage of fisheye cameras for global environmental perception; 3. It does not involve occlusion reasoning, failing to effectively handle occlusion problems in complex scenes, potentially leading to missed or false detections in target object recognition.

[0039] In summary, the BEV technology involved in the relevant technologies still has many problems in fisheye camera processing, multi-camera fusion, and occlusion inference.

[0040] This invention proposes a multi-camera image processing method based on multi-camera BEV (Bird's-EyeView) feature extraction and semantic segmentation using fisheye cameras, aiming to solve the problems of difficult fisheye camera processing, low efficiency of multi-camera fusion, and inaccurate occlusion inference in related technologies.

[0041] like Figure 1As shown, the overall process of a multi-camera image processing method provided by this embodiment of the invention is as follows: acquiring fisheye images captured by multiple fisheye cameras; utilizing the large field of view of fisheye cameras, combining matrix multiplication to achieve efficient feature transformation, and performing feature fusion through a distortion-aware pooling layer; performing feature compression through a PE module and matrix decomposition through an RR module to reduce computational load and memory usage while preserving feature integrity; and introducing an occlusion inference module on the basis of efficient feature transformation to improve the accuracy of semantic segmentation in complex scenes.

[0042] Figure 2 This is a flowchart illustrating a multi-camera image processing method provided in an embodiment of the present invention; as shown below. Figure 2 As shown, the method includes: Acquire fisheye images captured by multiple fisheye cameras; The image features of each fisheye image are converted into corresponding bird's-eye view features using a feature transfer matrix; Bird's-eye view feature fusion is performed by learning pooling layers through distortion perception.

[0043] The Feature Transfer Matrix (FTM) can be viewed as a sparse matrix that describes the mapping relationship from image features of a fisheye image to the BEV grid of a bird's-eye view.

[0044] In some embodiments, the step of converting the image features of each fisheye image into corresponding bird's-eye view features through a feature transfer matrix includes: Let the image features of a fisheye camera be... ,in It refers to the number of cameras. and These are the height and width of the image features. This represents the number of feature channels, and the Feature Transfer Matrix (FTM) is... The bird's-eye view features are ,in and It is the height and width of the BEV grid in the bird's-eye view. It is the number of bins at depth; The image features of a fisheye image are converted into bird's-eye view features using the following formula: , In formula (1), It is an intermediate feature obtained by combining image features F with depth distribution D through an outer product operation.

[0045] In the BEV feature fusion process of fisheye cameras, traditional symmetric pooling methods (such as mean pooling and max pooling) cannot effectively handle the large field of view and radial distortion of fisheye cameras. To address this, this invention proposes a distortion-aware learning pooling layer strategy, which optimizes BEV feature fusion by introducing a learnable weight function. By introducing this strategy, the distortion problem of fisheye cameras can be better handled, improving the accuracy of BEV feature fusion.

[0046] In some embodiments, the step of fusing bird's-eye view features through a distortion-aware learning pooling layer includes: For each bird's-eye view BEV grid, feature weights from different cameras are calculated using a weighting function that is learned and can adaptively adjust feature contributions based on the camera's field of view and distortion characteristics. The final bird's-eye view BEV features are generated using the following formula for learning pooling layers based on distortion perception: , In formula (2), It is the BEV feature of the k-th camera. These are the corresponding weights.

[0047] The Prime Extraction module compresses the image features from the fisheye camera to reduce feature dimensionality; the Ring & Ray Decomposition module decomposes the Feature Transporting Matrix (FTM) into two matrices to reduce its sparsity and improve the efficiency of BEV feature fusion.

[0048] In some embodiments, such as Figure 3 As shown, before the step of converting the image features of each fisheye image into corresponding bird's-eye view features through the feature transfer matrix, the method further includes a step of compressing the image features of each fisheye image using the PE module: Let the image features of a fisheye image be , where It refers to the number of cameras. and These are the height and width of the image features, and C is the number of feature channels; The PE module is used to compress image features through max pooling and convolution operations to obtain compressed image features. and depth distribution The calculation formula for the PE module is as follows: , In formula (3), MaxPool represents the max pooling operation and Conv represents the convolution operation.

[0049] The Prime Extraction module compresses the high-dimensionality of image features, thereby reducing the sparsity and computational cost of the feature transfer matrix. (Compressed features) and depth distribution It retains key information while significantly reducing computational complexity.

[0050] In some embodiments, the step of converting the image features of each fisheye image into corresponding bird's-eye view features through a feature transfer matrix includes: Use the RR module to transfer the feature matrix Decomposed into Ring matrices and Ray matrix The Ring matrix encodes distance information, and the Ray matrix encodes direction information. The image features of a fisheye image are converted into bird's-eye view features using the following formula: , In formula (4), This represents the Hadamard product.

[0051] The sparsity of the feature transfer matrix is ​​significantly reduced through Ring & Ray Decomposition technology, and the computational complexity is reduced from... FLOPs reduced to FLOPs, memory usage also increased from Reduce to .

[0052] Utilizing an efficient feature transformation mechanism, image features from multiple cameras are converted into BEV features. Then, an occlusion inference module is introduced to address occlusion issues in the BEV space, ensuring accurate inference and segmentation of semantic information in occluded regions. The core idea of ​​the occlusion inference module is to use feature fusion from multiple cameras and apply the fused bird's-eye view BEV features... Calculate the occlusion probability of each BEV mesh cell. The fused BEV features are weighted to obtain the final BEV semantic segmentation result.

[0053] In some embodiments, such as Figure 4 As shown, it also includes steps for handling occlusion issues in the bird's-eye view space through an occlusion inference module: It has There are 1 camera, and the image features of each camera are: For each camera, the occlusion inference module is used to extract image features. Projecting the bird's-eye view onto the BEV space yields the bird's-eye view BEV features. ; The occlusion inference module uses the following formula to fuse the bird's-eye view BEV features from multiple cameras to obtain the fused bird's-eye view BEV features. : , In formula (5), It is the weight of the k-th camera, which is usually adjusted according to the camera's viewing angle and occlusion. The occlusion inference module is used to calculate the occlusion probability p(o) of each BEV grid in the bird's-eye view by statistically analyzing its visibility under different camera views. The formula is as follows: , In formula (6), It is an indicator function, indicating whether the k-th camera has observed the bird's-eye view BEV grid. It is a normalized threshold, calculated using the occlusion probability. The occlusion reasoning module can infer the semantic information of the occluded area; Using occlusion probability The BEV features of the bird's-eye view are weighted to obtain the final semantic segmentation result of the bird's-eye view BEV. The specific formula is as follows: , In formula (7), Represents the Hadamard product. The visibility probability of the grid is represented by this value. In this way, the occlusion inference module can effectively handle the occlusion problem in the BEV space and improve the accuracy of BEV semantic segmentation.

[0054] To further improve the accuracy of occlusion inference, this invention optimizes the multi-camera feature fusion mechanism by introducing an adaptive weighting mechanism, which dynamically adjusts the camera weights based on the viewpoint and occlusion status of each camera. .

[0055] In some embodiments, the camera weights are dynamically adjusted using the following formula. : , In formula (8), It's about adjusting parameters. This represents the occlusion probability of the k-th camera. An adaptive weighting mechanism can better fuse features from multiple cameras, improving the accuracy of occlusion inference.

[0056] Based on the same inventive concept, embodiments of the present invention also provide a multi-camera image processing system, the system being configured to implement any of the methods described in the above embodiments, the system comprising: The first acquisition unit is used to acquire fisheye images captured by multiple fisheye cameras; The feature extraction and fusion unit is used to convert the image features of each fisheye image into corresponding bird's-eye view features through the feature transfer matrix, and to perform bird's-eye view feature fusion through a distortion-aware learning pooling layer.

[0057] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Figure 5 As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement any of the methods described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0058] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0059] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0060] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0061] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable medium. This computer-readable medium stores a computer program, wherein, when executed by a processor, the program implements the steps of any of the methods described in the above embodiments. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0062] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0063] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0064] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0065] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0066] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0067] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0068] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0069] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0070] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0071] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A multi-camera image processing method, characterized in that, include: Acquire fisheye images captured by multiple fisheye cameras; The image features of each fisheye image are converted into corresponding bird's-eye view features using a feature transfer matrix; Bird's-eye view feature fusion is performed by learning pooling layers through distortion perception.

2. The method according to claim 1, characterized in that, The step of converting the image features of each fisheye image into corresponding bird's-eye view features through the feature transfer matrix includes: Let the image features of a fisheye camera be... ,in It refers to the number of cameras. and These are the height and width of the image features. This is the number of feature channels, and the feature transfer matrix is... The bird's-eye view features are ,in and It is the height and width of the bird's-eye view grid. It is the number of bins at depth; The image features of a fisheye image are converted into bird's-eye view features using the following formula: ， In formula (1), It is an intermediate feature obtained by combining image features F with depth distribution D through an outer product operation.

3. The method according to claim 2, characterized in that, The steps for bird's-eye view feature fusion through distortion-perceived learning pooling layers include: For each bird's-eye view grid, the feature weights from different cameras are calculated using a weighting function; The final bird's-eye view features are generated using the following formula for learning pooling layers with distortion perception: ， In formula (2), It is the bird's-eye view feature of the k-th camera. These are the corresponding weights.

4. The method according to claim 1, characterized in that, Before the step of converting the image features of each fisheye image into corresponding bird's-eye view features through the feature transfer matrix, the method further includes a step of compressing the image features of each fisheye image using the PE module: Let the image features of a fisheye image be... ,in It refers to the number of cameras. and These are the height and width of the image features, and C is the number of feature channels; The PE module is used to compress image features through max pooling and convolution operations to obtain compressed image features. and depth distribution The calculation formula for the PE module is as follows: ， In formula (3), MaxPool represents the max pooling operation and Conv represents the convolution operation.

5. The method according to claim 4, characterized in that, The step of converting the image features of each fisheye image into corresponding bird's-eye view features through the feature transfer matrix includes: Use the RR module to transfer the feature matrix Decomposed into Ring matrices and Ray matrix The Ring matrix encodes distance information, and the Ray matrix encodes direction information. The image features of a fisheye image are converted into bird's-eye view features using the following formula: ， In formula (4), This represents the Hadamard product.

6. The method according to any one of claims 1 to 5, characterized in that, It also includes steps for handling occlusion issues in the bird's-eye view space through an occlusion inference module: It has There are 1 camera, and the image features of each camera are: For each camera, the occlusion inference module is used to extract image features. Projecting onto the bird's-eye view space yields bird's-eye view features. ; The occlusion inference module uses the following formula to fuse the bird's-eye view features from multiple cameras to obtain the fused bird's-eye view features. : ， In formula (5), It is the weight of the k-th camera; The occlusion inference module calculates the occlusion probability p(o) of each bird's-eye view grid by statistically analyzing its visibility under different camera views. The formula is as follows: ， In formula (6), It is an indicator function, indicating whether the k-th camera has observed the bird's-eye view grid. It is the normalized threshold; Using occlusion probability The features of the bird's-eye view are weighted to obtain the final semantic segmentation result of the bird's-eye view. The specific formula is as follows: ， In formula (7), Represents the Hadamard product. This represents the visibility probability of the grid.

7. The method according to claim 6, characterized in that, The camera weights are dynamically adjusted using the following formula. : ， In formula (8), It's about adjusting parameters. It is the occlusion probability of the k-th camera.

8. A multi-camera image processing system, characterized in that, The system is configured to implement the method as described in any one of claims 1 to 7, the system comprising: The first acquisition unit is used to acquire fisheye images captured by multiple fisheye cameras; The feature extraction and fusion unit is used to convert the image features of each fisheye image into corresponding bird's-eye view features through the feature transfer matrix, and to perform bird's-eye view feature fusion through a distortion-aware learning pooling layer.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images