Image processing method, computer program product, storage medium and electronic device

Abstract

The application relates to the technical field of image processing, and provides an image processing method, a computer program product, a storage medium and an electronic device. The image processing method comprises the following steps: acquiring an image to be processed, and extracting an image feature of the image to be processed by using a backbone network; acquiring a plurality of initial query features, each initial query feature corresponding to a sub-region in a BEV space; inputting the plurality of initial query features into a decoding network, and performing interactive calculation with the image feature input into the decoding network to obtain a plurality of final query features; obtaining a plurality of target segmentation results in the plurality of sub-regions by using a segmentation network based on the plurality of final query features; and obtaining an overall target segmentation result in the BEV space according to the target segmentation results in the plurality of sub-regions. The method significantly improves the accuracy of target segmentation in the BEV space.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to an image processing method, a computer program product, a storage medium, and an electronic device. Background Technology

[0002] In autonomous driving, it is often necessary to segment lane lines in the bird's-eye view (BEV) space based on images captured by onboard cameras in order to perform subsequent driving planning and other operations.

[0003] In existing technologies, image features are typically extracted from images captured by onboard cameras, and depth information is estimated. The estimated depth information is then used to transform the image features into a BEV (Battery, Electric Vehicle) space, and finally, the features in the BEV space are used for lane segmentation. However, the depth information estimated by this method already has a large error, and transforming the image features into the BEV space further amplifies the error, resulting in low accuracy in lane segmentation. Summary of the Invention

[0004] The purpose of this application is to provide an image processing method, a computer program product, a storage medium, and an electronic device to improve the above-mentioned technical problems.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] In a first aspect, embodiments of this application provide an image processing method, comprising: acquiring an image to be processed and extracting image features of the image to be processed using a backbone network; acquiring multiple initial query features; wherein each initial query feature corresponds to a sub-region in a BEV space, the BEV space being a planar region corresponding to the shooting range of the image to be processed from a bird's-eye view; inputting the multiple initial query features into a decoding network and performing interactive calculations with the image features also input into the decoding network to obtain multiple corresponding final query features; based on the multiple final query features, using a segmentation network to obtain target segmentation results within the corresponding multiple sub-regions; and obtaining an overall target segmentation result in the BEV space based on the target segmentation results within the multiple sub-regions.

[0007] The targets in the above method may include, but are not limited to, various map elements, such as lane lines, drivable areas, vehicles, pedestrians, or one or more other elements. The beneficial effects of this method include:

[0008] Firstly, since the initial query features correspond to sub-regions in the BEV space, the final query features corresponding to the sub-regions in the BEV space can be directly obtained after interactive calculation with the image features. This process does not require converting the image features to the BEV space, thus avoiding the errors caused by the conversion and significantly improving the accuracy of target segmentation in the BEV space.

[0009] Secondly, since the final query features correspond to sub-regions in the BEV space, the target segmentation result in the corresponding sub-region can be directly obtained based on each final query feature. Then, by combining the target segmentation results in each sub-region, the overall target segmentation result in the BEV space can be easily obtained. Because the relative positional relationships between the sub-regions are clear, it is also very clear how to combine the target segmentation results in each sub-region, which is conducive to obtaining a more accurate target segmentation result.

[0010] In one implementation of the first aspect, obtaining multiple initial query features includes: obtaining multiple sets of coordinates corresponding to the multiple initial query features; wherein each set of coordinates includes at least an x-coordinate and a y-coordinate, and the x-coordinate and y-coordinate corresponding to each initial query feature are located in a sub-region in the BEV space corresponding to the initial query feature; encoding the multiple sets of coordinates using a query feature encoding network to obtain the corresponding multiple initial query features; wherein the dimension of an initial query feature is greater than the dimension of a set of coordinates.

[0011] In the above implementation, since the initial query features are obtained by encoding the coordinates of a sub-region located in the BEV space, it is equivalent to defining the initial query features within the sub-region of the BEV space. Therefore, after interacting with image features, the final query features corresponding to the sub-region in the BEV space can be directly obtained without converting the image features to the BEV space, thus avoiding the errors caused by conversion and significantly improving the accuracy of target segmentation in the BEV space. Furthermore, since the final query features correspond to the sub-regions in the BEV space, the target segmentation result in the corresponding sub-region can be directly obtained based on each final query feature. Then, by combining the target segmentation results of each sub-region, it is easy to obtain target segmentation results over a larger range.

[0012] Furthermore, the above implementation does not directly use the coordinates within the sub-region as the initial query feature. Instead, it utilizes a query feature encoding network to encode the coordinates within the sub-region as the initial query feature. This expands the dimension of the coordinates (e.g., from two or three dimensions to 256 dimensions), giving the resulting initial query feature a larger optimization space. This facilitates better optimization of the initial query feature in the decoding network (the optimized result is the final query feature), thereby improving the accuracy of target segmentation. In contrast, if the coordinates within the sub-region are directly used as the initial query feature, due to the low dimension of the coordinates, changes in the value of any dimension will have a significant impact on the final optimization result, making fine-grained numerical optimization difficult.

[0013] In one implementation of the first aspect, each set of coordinates also includes a z-coordinate obtained through learning.

[0014] Although image features do not explicitly contain depth information, depth estimation can be performed based on them. Therefore, image features can be considered to implicitly contain depth information, meaning they are three-dimensional features. Since the BEV space is actually a planar region, if the initial query features are encoded solely based on the x and y coordinates in the BEV space, they might only be two-dimensional features. Therefore, to enable better interaction between the initial query features and image features, the above implementation adds a z-coordinate representing height information, thus making the initial query features three-dimensional as well.

[0015] Furthermore, since there is actually no height information in the BEV space, the above implementation also sets the z-coordinate as a learnable parameter, so that a reasonable z-coordinate value can be learned during the training process of the network.

[0016] In one implementation of the first aspect, the query feature encoding network includes a position encoder and a feature encoder. Encoding the multiple sets of coordinates using the query feature encoding network to obtain the corresponding multiple initial query features includes: encoding the multiple sets of coordinates using the position encoder to obtain corresponding multiple position encoded features; wherein the dimension of one position encoded feature is greater than the dimension of a set of coordinates; and encoding the multiple position encoded features using the feature encoder to obtain the corresponding multiple initial query features; wherein the feature encoder is a multilayer perceptron.

[0017] In the above implementation, the position encoder is mainly used to encode coordinates to expand the dimension of the coordinates. However, the position encoded features obtained by the position encoder are essentially still features of a single point (the point corresponding to the coordinates) and cannot represent the features of a sub-region. The feature encoder, on the other hand, adopts a multilayer perceptron structure (i.e., a fully connected neural network), so it can extend the features of a single point to a larger spatial range, so that the obtained initial query features can better represent the corresponding sub-region.

[0018] In one implementation of the first aspect, obtaining target segmentation results within corresponding sub-regions using a segmentation network based on the multiple final query features includes: inputting the multiple final query features into the segmentation network to obtain multiple sets of Discrete Cosine Transform (DCT) coefficients; and performing an Inverse Discrete Cosine Transform (IDCT) on the multiple sets of DCT coefficients to obtain target segmentation results within corresponding sub-regions.

[0019] In the above implementation, the segmentation network does not directly output the target segmentation result in the spatial domain, but instead outputs the DCT coefficients in the frequency domain, and then uses IDCT to obtain the target segmentation result in the spatial domain. DCT is commonly used for image compression; after compression, only a small number of DCT coefficients need to be saved to largely restore the original image. Therefore, for the scheme of this application, the segmentation network only needs to calculate a small number of DCT coefficients to obtain a larger-sized target segmentation result, which not only facilitates the application of the target segmentation result (which has certain size requirements), but also reduces the computational load and storage resource consumption in the target segmentation process.

[0020] In one implementation of the first aspect, the decoding network includes at least one decoder connected in sequence. Each decoder decodes multiple query features input to it and outputs multiple decoded query features. The multiple query features input to the first decoder are the multiple initial query features, and the multiple decoded query features output by the last decoder are the multiple final query features. Each decoder includes a self-attention layer and a cross-attention layer. When decoding, the self-attention layer performs interactive calculations on the multiple query features input to it, and the cross-attention layer performs interactive calculations on the multiple query features output by the self-attention layer and the image features.

[0021] In the above implementation, if the decoding network includes multiple decoders, the decoding process can be regarded as an iterative optimization process of the initial query features. Each time the query features pass through a decoder, attention operations (self-attention layer) are performed inside the query features, and attention operations (cross-attention layer) are also performed between the query features and the image features. That is, an optimization is performed according to the attention mechanism. The final query features output by the last decoder are the result of multiple rounds of optimization. They can better represent the features of the corresponding sub-region in the BEV space. Therefore, subsequent target segmentation based on the final query features can also obtain high-precision segmentation results.

[0022] In one implementation of the first aspect, the image to be processed is a multi-frame image, wherein each frame corresponds to a viewpoint and is captured by a camera mounted on a mobile carrier, and the overall shooting range of the multi-frame image covers the four sides of the mobile carrier.

[0023] Taking the case where the mobile carrier is a vehicle as an example, in the above implementation method, since the multi-view images cover the four sides of the vehicle (360° range), the area centered on the vehicle can be comprehensively segmented in the BEV space, thereby perceiving various targets around the vehicle, which is beneficial for performing tasks such as vehicle driving planning.

[0024] Secondly, embodiments of this application provide an image processing apparatus, comprising: an image feature acquisition module, configured to acquire an image to be processed and extract image features of the image to be processed using a backbone network; a query feature acquisition module, configured to acquire multiple initial query features; wherein each initial query feature corresponds to a sub-region in a BEV space, the BEV space being a planar region corresponding to the shooting range of the image to be processed from a bird's-eye view; a decoding module, configured to input the multiple initial query features into a decoding network and perform interactive calculations with the image features also input into the decoding network to obtain multiple final query features; a target segmentation module, configured to obtain target segmentation results within the corresponding multiple sub-regions using a segmentation network based on the multiple final query features; and a result aggregation module, configured to obtain an overall target segmentation result in the BEV space based on the target segmentation results within the multiple sub-regions.

[0025] Thirdly, embodiments of this application provide a computer program product, including computer program instructions, which, when read and executed by a processor, perform the method provided in the first aspect or any possible implementation thereof.

[0026] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the method provided in the first aspect or any possible implementation thereof.

[0027] Fifthly, embodiments of this application provide an electronic device, including: a memory and a processor, wherein the memory stores computer program instructions, and the computer program instructions are read and executed by the processor to perform the method provided in the first aspect or any possible implementation of the first aspect. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 The steps of the image processing method provided in the embodiments of this application are illustrated.

[0030] Figure 2 It shows the setup of the multiple cameras and the location of the BEV space;

[0031] Figure 3 A method applicable to Figure 1 The model structure of the Chinese method;

[0032] Figure 4 It shows Figure 3 One possible structure for a decoding network;

[0033] Figure 5 The functional modules included in the image processing apparatus provided in the embodiments of this application are shown;

[0034] Figure 6 This illustration shows one possible structure for the electronic device provided in the embodiments of this application. Detailed Implementation

[0035] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, it has been applied in numerous fields, such as security, urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, smart healthcare, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beautification, makeup, medical aesthetics, and intelligent temperature measurement. The image processing method in this application embodiment also utilizes relevant artificial intelligence technologies.

[0036] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. It should be noted that similar reference numerals and letters in the following drawings indicate similar items; therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0037] The terms “comprising,” “including,” or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0038] Figure 1 The steps of an image processing method provided in an embodiment of this application are illustrated. This method can be, but is not limited to, [the steps described above]. Figure 6The electronic device in the process performs the operation; for possible structures of this electronic device, please refer to the following section. Figure 6 The explanation. Figure 3 A method applicable to Figure 1 The model structure of the Chinese method is introduced in the following sections. Figure 1 When using the methods described, appropriate combinations will be made. Figure 3 The content will be elaborated upon. (Refer to...) Figure 1 Image processing methods include:

[0039] Step S110: Obtain the image to be processed and extract the image features of the image to be processed using the backbone network.

[0040] The image to be processed is an image captured by a camera. The image to be processed may contain targets that need to be segmented, such as lane lines, drivable areas, vehicles, pedestrians, or one or more map elements. The following text mainly takes the segmentation of lane lines, drivable areas, and vehicles as examples.

[0041] It should be noted that, firstly, the image to be processed may not contain the target to be segmented, but for the sake of clarity, the following text will mainly focus on the case where the image to be processed contains the target.

[0042] Secondly, the solution in this application does not segment the image to be processed itself, but rather segments the BEV space. Here, BEV space refers to the planar area corresponding to the shooting range of the image to be processed from a bird's-eye view. The shooting range of the image to be processed can also be understood as the field of view of the camera capturing the image. The concept of BEV space will be explained with examples later.

[0043] The camera that captures the image to be processed can be one or more cameras. If there is only one camera, there is only one shooting angle, and the image to be processed is a single-view image (one frame). If there are multiple cameras, there are multiple shooting angles (one angle for each camera), and the image to be processed is a multi-view image (one frame for each angle, for a total of multiple frames).

[0044] The following is combined Figure 2 Here's an example with multiple cameras. Figure 2On the left, six cameras (of course, the number can be changed to other numbers as needed) are shown in black circles, numbered 1 to 6. These six cameras are mounted on a mobile carrier to capture images around the carrier. Cameras 1 to 3 are mounted at the front of the carrier, and cameras 4 to 6 are mounted at the rear. Each camera has a different shooting angle. The two dotted lines extending from each black circle represent the field of view of the corresponding camera, or the shooting range of the image to be processed captured by that camera. The six frames captured by these six cameras at the same time can be used as the images to be processed in step S110. If the six cameras work continuously, images to be processed can be continuously generated, thereby continuously executing the image processing method provided in this embodiment.

[0045] The term "mobile carrier" as used above refers to the object used to support the camera, in contrast to the camera itself. "Mobile" means that the carrier can move. For example, a mobile carrier could be a vehicle, a robot, a pedestrian wearing a helmet (with the camera mounted on the helmet), and so on. Obviously, a camera can also be mounted on a stationary carrier (e.g., a wall, traffic light, utility pole), but the following examples will primarily focus on mobile carriers.

[0046] Continue to refer to Figure 2 The six cameras seamlessly cover the perimeter of the mobile vehicle. At this point, the BEV space can be defined as a square area centered on the mobile vehicle from a bird's-eye view perspective, for example, a square area with sides of 51.2m centered on the mobile vehicle. Figure 2 As shown on the right.

[0047] It should be noted that, firstly, strictly speaking, the field of view of the six cameras does not cover the mobile vehicle itself (and a few blind spots around it). However, when the mobile vehicle itself is not too large relative to the BEV space, it can be approximated as a point. Therefore, there is no void in the center of the BEV space.

[0048] Secondly, the field of view of the cameras is not infinitely extended. Strictly speaking, the superimposed fields of view of the six cameras form a circular area rather than a square area from a bird's-eye view. However, for the convenience of subsequent segmentation calculations, defining the BEV space as a square area is not inappropriate, since the area within which the target is segmented depends on business needs. Of course, defining the BEV space as other shapes, such as circles or rectangles, is also possible.

[0049] If the number or arrangement of cameras is... Figure 2 The definition of the BEV space may change depending on the specific example shown. For instance, if... Figure 2If only camera 1 is retained, then the BEV space can be defined as a triangular area located in front of the mobile vehicle from a bird's-eye view, and so on.

[0050] exist Figure 2 The reason for setting up multiple cameras and capturing images from multiple perspectives is twofold. Firstly, it's driven by business needs. Taking a vehicle as an example, because multi-view images cover a larger area, the corresponding BEV space can be defined more broadly, allowing for target segmentation over a wider range. For instance... Figure 2 The six cameras in the BEV (Battery Electric Vehicle) system cover the entire area around the vehicle, allowing the BEV space to be defined as a square area centered on the vehicle from a bird's-eye view. By performing comprehensive target segmentation within this area, various targets around the vehicle can be perceived, which is beneficial for executing tasks such as autonomous driving planning. In contrast, a single camera can only capture images from a single perspective, thus only perceiving targets within a small area, such as targets directly in front of the vehicle, and therefore cannot effectively perform driving planning.

[0051] On the other hand, multi-view images are equivalent to having more samples (although steps S110-S150 are steps in the model inference stage, if multi-view images are used in the inference stage, they must also be used in the training stage), thus improving the performance of the model (including the backbone network, decoding network, segmentation network, query feature encoding network, etc. mentioned later) and obtaining more accurate target segmentation results. Furthermore, adjacent views can have a certain degree of overlap ( Figure 2 This illustrates the case of overlap, which in the image means that images from different perspectives contain some of the same content. The existence of this same content is also beneficial to the optimization of the model.

[0052] It should be noted that although the image to be processed is captured by a camera, step S110 does not necessarily involve directly obtaining the image from the camera. For example, it may also involve downloading the image from the network, etc.

[0053] The backbone network can be a pre-trained neural network with no specific structure restrictions; for example, it can use ResNet, VGG, SENet, etc. After obtaining the image to be processed, it can be input into the backbone network. After processing by the backbone network, the image features of the image to be processed are output, such as... Figure 3 As shown in the left branch. If the image to be processed consists of multiple frames, these frames can be stitched together before being input into the backbone network. Optionally, the image to be processed can also be scaled, normalized, or otherwise processed before being input into the backbone network.

[0054] Step S120: Obtain multiple initial query features.

[0055] When querying a database, a query condition is usually specified, and then the query condition is submitted to the database management system. The database management system matches the data in the database with the query condition, obtains the corresponding query results, and returns them.

[0056] To better understand the meaning of query features in this application, we can draw an analogy with a database example. The initial query features correspond to the query conditions in the example above, the final query features (see step S130) correspond to the query results in the example above, and the operation performed by the decoding network (see step S130) corresponds to the matching operation performed by the database management system in the example above. However, while "query" in a database can be understood as a search, the meaning of "query" in this application is more abstract. It can be understood as the process of providing necessary information (initial query features) to obtain the desired result (final query features that can be used for target segmentation).

[0057] Each initial query feature in step S120 corresponds to a sub-region in the BEV space, or in other words, each initial query feature is a feature within a sub-region of the BEV space. The sub-region in the BEV space can be a small region obtained by dividing the BEV space according to a certain rule, for example, referring to... Figure 2 The BEV space can be evenly divided into 16 square sub-regions. Therefore, the initial query features obtained in step S120 will have 16 features, each corresponding to a sub-region. It should be understood that, depending on the division rules, the number of sub-regions may not be 16, their shape may not be square, and the size of each sub-region may not be the same.

[0058] In one implementation, the location information of the sub-region (referring to the location information of the sub-region in the BEV space) can be encoded using a query feature encoding network to obtain the initial query features corresponding to the sub-region. For example, the location information here can be implemented as the coordinates of a point (e.g., the center point) in the sub-region, such as... Figure 3 As shown in the right-hand branch, since the coordinates of this point are located within a sub-region, the initial query features generated by the encoding are naturally defined within the sub-region, or in other words, correspond to the sub-region. In this implementation, "query" can be understood as the process of providing the location information of a sub-region to obtain features within that sub-region that can be used for target segmentation. How to encode the location information of the sub-region will be explained in detail later; it will not be elaborated on here.

[0059] In another implementation, the initial query features can be pre-calculated and stored, allowing them to be directly read during step S120. For example, since the sub-region division method is predetermined, the location information of the sub-regions is also predetermined. Furthermore, since the encoding of the location information of the sub-regions is also predetermined (e.g., the parameters of the query feature encoding network are fixed during model training), the initial query features corresponding to each sub-region can be pre-calculated, rather than being calculated only during step S120. In particular, if the same initial query features are used for different images to be processed, this implementation significantly improves the efficiency of target segmentation because it avoids calculating the initial query features each time.

[0060] Step S130: Input multiple initial query features into the decoding network and perform interactive calculations with the image features that are also input into the decoding network to obtain multiple corresponding final query features.

[0061] The decoding network can be a pre-trained neural network whose inputs are multiple initial queries and image features, and whose outputs are multiple final query features, such as... Figure 3 As shown in the middle section. The final query features correspond one-to-one with the initial query features, thus each final query feature also corresponds to a sub-region in the BEV space. Optionally, some preprocessing can be performed on the initial query features before inputting them into the decoding network; similar preprocessing is applied to image features. The specific structure of the decoding network is not limited; examples of possible structures will be provided later, and will not be elaborated upon here.

[0062] The so-called "decoding" can be understood as the process of parsing the encoded features (initial query features) into the required results (final query features), which is actually the "query" process mentioned in step S120, or it can be considered as the process of optimizing the query features (optimizing the initial query features into the final query features).

[0063] During decoding, the decoding network performs interactive computation on the initial query features and image features. "Interactive computation" means that these two types of feature data are fused in some way. For example, if the initial query features are obtained by encoding the location information of a sub-region, they do not contain image features. Obviously, such features cannot be directly used for target segmentation because they lack target information. However, after interactive computation, the final query features contain both the location information of the corresponding sub-region and the image features within that sub-region. This effectively represents the characteristics of the BEV space within that sub-region, and thus, based on the final query features, effective target segmentation can be performed within the sub-region of the BEV space.

[0064] Step S140: Based on multiple final query features, use a segmentation network to obtain target segmentation results within the corresponding multiple sub-regions.

[0065] The segmentation network can be a pre-trained neural network, and its specific structure is not limited. Step S140 can be implemented in several different ways:

[0066] In one implementation, the segmentation network takes multiple final query features as input and outputs the target segmentation results within multiple corresponding sub-regions (each final query feature is input into the segmentation network and outputs the target segmentation result within a corresponding sub-region). For example, if there are 16 final query features, the output will be the target segmentation results within the corresponding 16 sub-regions.

[0067] The target segmentation result can be in the form of a binary mask. For example, a 64×64 (width×height) mask can represent a sub-region in the BEV space. If the value of a pixel in the mask is 1, it indicates that a target exists at that pixel; if the value of a pixel in the mask is 0, it indicates that a target does not exist at that pixel. If there are multiple targets to segment, the segmentation network can output a mask for each target. For example, it can output a mask for lane lines, a mask for drivable areas, and a mask for vehicles (excluding the vehicle where the camera is located). (See reference...) Figure 3 At the bottom, each small square represents the target segmentation result within a sub-region. Note... Figure 3 To facilitate observation, the masks corresponding to the three targets are superimposed on one image, and the mask corresponding to each target is not shown separately.

[0068] It's understandable that a segmentation network may not directly output a binarized mask. For example, it may be necessary to perform a binarization operation on the output of the segmentation network to obtain a binarized mask. Of course, it is also possible to consider the binarization operation as a layer in the segmentation network.

[0069] In another implementation, the input to the segmentation network is multiple final query features, but the output is not the target segmentation result within a sub-region, but some intermediate result. Based on these intermediate results, the target segmentation results within the corresponding multiple sub-regions can be further obtained.

[0070] For example, refer to Figure 3 The segmentation network outputs multiple sets of DCT coefficients (each final query feature is input into the segmentation network and outputs a corresponding set of DCT coefficients). Then, IDCT is performed on these multiple sets of DCT coefficients to obtain the target segmentation results within multiple sub-regions corresponding to the multiple final query features. This example will be explained in more detail later; it will not be elaborated on here.

[0071] Step S150: Based on the target segmentation results within multiple sub-regions, obtain the overall target segmentation result in the BEV space.

[0072] Since the locations of each sub-region in the BEV space are known, after obtaining the target segmentation results within each sub-region in step S140, these target segmentation results are combined according to the locations of each sub-region in the BEV space to obtain the overall target segmentation result in the BEV space. For example, referring to... Figure 3 By combining the target segmentation results from 16 sub-regions, a larger overall target segmentation result can be obtained.

[0073] In some implementations, if the overall target segmentation result is too small to meet business requirements, it can be upsampled to obtain a larger target segmentation result.

[0074] Here is a brief summary Figure 1 The method described above has the following beneficial effects:

[0075] Firstly, since the initial query features correspond to sub-regions in the BEV space, the final query features corresponding to the sub-regions in the BEV space can be directly obtained after interactive calculation with the image features. This process does not require converting the image features to the BEV space, thus avoiding the errors caused by the conversion and significantly improving the accuracy of target segmentation in the BEV space.

[0076] Secondly, since the final query features correspond to sub-regions in the BEV space, the target segmentation result in the corresponding sub-region can be directly obtained based on each final query feature. Then, by combining the target segmentation results in each sub-region, the overall target segmentation result in the BEV space can be easily obtained. Because the relative positional relationships between the sub-regions are clear, it is also very clear how to combine the target segmentation results in each sub-region, which is conducive to obtaining a more accurate target segmentation result.

[0077] Based on the above embodiments, the following describes how to encode the corresponding initial query features in step S120 according to the location information of the sub-regions in the BEV space:

[0078] In one implementation, step S120 may further include:

[0079] Step A: Obtain multiple sets of coordinates corresponding to multiple initial query features.

[0080] Each initial query feature corresponds to a set of coordinates, with each set including at least x and y coordinates, corresponding to a two-dimensional point (x, y). The two-dimensional point (x, y) corresponding to each initial query feature is located within a sub-region in the BEV space corresponding to that initial query feature. For example, (x, y) can, but is not limited to, be the center point of the sub-region. Clearly, (x, y) represents, to some extent, the location information of its sub-region.

[0081] Reference Figure 3 In the upper left corner of the left branch, there are 16 initial query features corresponding to 16 two-dimensional points, which are represented by black circles. You can first divide the BEV space into 16 square sub-regions evenly, and then take the center point of each sub-region to obtain the coordinates of these 16 two-dimensional points. Alternatively, you can directly sample the two-dimensional points in the BEV space evenly to obtain the coordinates of these 16 two-dimensional points, and so on.

[0082] Optionally, each coordinate set may also include a z-coordinate. The z-coordinate is set as a learnable parameter, and its value can be determined during model training. When the z-coordinate is included, each coordinate set can be considered as a three-dimensional point (x, y, z), where z represents the height. (See reference...) Figure 3 In the upper right corner of the left branch, 16 white circles represent 16 z-coordinates. When concatenated with 16 two-dimensional points (x, y) (plus signs in the diagram), 16 three-dimensional points (x, y, z) are obtained. When the model starts training, the z-coordinates can be randomly selected or take a default value. After the model is trained, the z-coordinates also learn a set of reasonable values.

[0083] The reasons for increasing the z-coordinate are analyzed as follows:

[0084] Although image features do not explicitly contain depth information, depth estimation can be performed based on them. Therefore, image features can be considered to implicitly contain depth information, meaning they are three-dimensional features. Since the BEV space is actually a planar region, if the initial query features are encoded solely based on the x and y coordinates in the BEV space (step B), the initial query features may only be two-dimensional features. Therefore, to enable better interaction between the initial query features and image features, a z-coordinate representing height information can be added, transforming the initial query features into three-dimensional features as well.

[0085] Furthermore, since there is actually no height information in the BEV space, it is not suitable to set the z-coordinate as a priori value. Instead, it is more appropriate to learn a reasonable z-coordinate value during the model training process.

[0086] It should be noted that it is also possible not to set the z-coordinate. For example, the height information implied by the z-coordinate can also be learned by the query feature encoding network in step B. However, if the network is left to learn the z-coordinate implicitly, the uncertainty of its learning results will increase, and it will be difficult to know for sure whether the network has learned the z-coordinate. Explicitly specifying the z-coordinate as a learnable parameter will make the learning results of the network more explicit and will help improve the final target segmentation results.

[0087] Furthermore, since the x and y coordinates can be calculated once the division of the BEV space and sub-regions is determined, and the z coordinate (if any) can be determined after the model is trained, in one implementation, multiple sets of coordinates corresponding to multiple initial query features can be calculated before step A is executed, and the calculated sets of coordinates can be read directly in step A, instead of calculating these coordinates on the spot.

[0088] Previously, when explaining step S120, we also mentioned an implementation method for pre-calculating initial query features. Compared with the implementation method introduced earlier, although step B cannot be omitted when pre-calculating coordinates, since the coordinate dimension is very low (generally only 2 to 3 dimensions), saving the pre-calculated coordinates requires less storage space than saving the pre-calculated initial query features, which is beneficial for saving storage resources.

[0089] Step B: Encode multiple sets of coordinates using a query feature encoding network to obtain multiple corresponding initial query features.

[0090] The query feature encoding network can be a pre-trained neural network. Its input is multiple sets of coordinates obtained in step A, and its output is multiple initial query features corresponding to the multiple sets of coordinates (each set of coordinates is input into the query feature encoding network and outputs one initial query feature). For example, if 16 sets of coordinates are input, then 16 corresponding initial query features will be output.

[0091] The dimension of an initial query feature obtained after encoding is greater than the dimension of a set of coordinates (all initial query features have the same dimension, and all sets of coordinates also have the same dimension). That is, the query feature encoding network can expand the dimension of the coordinates through encoding. For example, the initial query feature can be 256-dimensional, while a set of coordinates is 2 to 3-dimensional (depending on whether there is a z-coordinate).

[0092] It should be understood that if the initial query features are pre-calculated, and only the pre-calculated initial query features are read in step S120, then it is not necessary to set up the query feature encoding network (but the network still needs to be set up during the model training stage).

[0093] The following is a brief analysis of the implementation methods corresponding to steps A to B. In this implementation, since the initial query features are obtained by encoding the coordinates (at least the x and y coordinates are located within the BEV space) of a sub-region in the BEV space, it is equivalent to defining the initial query features within a sub-region in the BEV space. Therefore, after interacting with the image features, the final query features corresponding to the sub-region in the BEV space can be obtained directly without converting the image features to the BEV space, thus avoiding the errors caused by conversion and significantly improving the accuracy of target segmentation in the BEV space. Furthermore, since the final query features correspond to the sub-regions in the BEV space, the target segmentation result in the corresponding sub-region can be obtained directly based on each final query feature. Then, by combining the target segmentation results in each sub-region, it is easy to obtain target segmentation results over a larger range.

[0094] Furthermore, this implementation does not directly use the coordinates within the sub-region as the initial query feature. Instead, it utilizes a query feature encoding network to encode the coordinates within the sub-region as the initial query feature. During the encoding process, the dimensionality of the coordinates is expanded, giving the resulting initial query feature a larger optimization space (e.g., expanding from 2-3 optimizable dimensions to 256 optimizable dimensions). This facilitates better optimization of the initial query feature in the decoding network (as mentioned earlier, the decoding process can also be viewed as an optimization process for the query feature), thereby improving the accuracy of target segmentation. In contrast, if the coordinates within the sub-region are directly used as the initial query feature, due to the low dimensionality of the coordinates, changes in the value of any dimension will have a significant impact on the final optimization result, making fine-grained numerical optimization difficult.

[0095] Continue to refer to Figure 3 In an alternative approach, the query feature encoding network may further include a position encoder and a feature encoder, in which case step B may further include:

[0096] Step B1: Use a position encoder to encode multiple sets of coordinates to obtain multiple corresponding position encoded features.

[0097] In this case, the dimension of a location encoding feature is greater than the dimension of a set of coordinates (the dimensions of each location encoding feature are the same, and the dimensions of each set of coordinates are also the same). That is, step B1 is mainly used to expand the dimension of the coordinates to increase the optimizable space of the features. For example, the coordinates can be expanded from 2 to 3 dimensions to 128 dimensions.

[0098] A position encoder can be implemented as a combination of one or more trigonometric functions (e.g., sine or cosine functions), which is similar to the trigonometric series expansion of a function (expanding 2-3 dimensional coordinates to 128 dimensions).

[0099] However, it's important to note that the encoding algorithm used by a position encoder can be a fixed algorithm, meaning it doesn't have learnable parameters. Therefore, the transformation from coordinates to position encoded features can be considered merely a conversion of coordinate representation; the resulting position encoded features are still those of a single point (a two-dimensional or three-dimensional point represented by coordinates). (Refer to...) Figure 3 In the middle of the left branch, each black dot represents a location encoding feature, which can be regarded as the feature at the center point of the sub-region (small square).

[0100] Step B2: Encode multiple location-coded features using a feature encoder to obtain multiple corresponding initial query features.

[0101] The feature encoder can adopt a multi-layer perceptron (MLP) structure, which is a fully connected neural network. Its input is multiple positional encoding features obtained in step B1, and its output is multiple initial query features corresponding to the multiple positional encoding features (each positional encoding feature is input into the feature encoder and outputs one initial query feature). For example, if 16 positional encoding features are input, then 16 corresponding initial query features are output.

[0102] Feature encoders can be used to further expand the dimensionality of features, for example, from 128 dimensions to 256 dimensions. More importantly, with the help of multilayer perceptron computation, feature encoders can achieve a mapping from single-point features to region features: as described in step B1, location-encoded features are only single-point features and cannot represent the features of the entire sub-region. Therefore, it is inappropriate to directly use location-encoded features and image features for interactive computation. The attention of the final query features obtained in this way will be focused on a single point in the sub-region, which cannot achieve good target segmentation. However, multilayer perceptrons can enable the various dimensions of location-encoded features to perform mutual computation, which is equivalent to blurring the coordinates corresponding to the location-encoded features, expanding them from single-point features to a larger spatial range, thereby enabling the initial query features to better represent the corresponding sub-region.

[0103] In an alternative approach, the query feature encoding network can also directly employ a multilayer perceptron (MLP), which directly encodes the coordinates as initial query features. However, the problem with this implementation is the significant dimensionality difference between the coordinates and the initial query features. For example, the features might be 2-3 dimensions, while the initial query features are 256 dimensions. This can make it difficult for the MLP to fit the relationship between the input and output. In steps B1-B2, the task of expanding the coordinate dimensions is mainly accomplished by the position encoder. Since the position encoder does not require a neural network, there is no problem with fitting. Furthermore, the dimensionality of the position encoded features output by the position encoder is already quite close to that of the initial query features. For example, the position encoded features are 128 dimensions, while the initial query features are 256 dimensions. In this case, the MLP (feature encoder) has no difficulty fitting the coordinates.

[0104] Based on the above embodiments, the following describes the possible structures of the decoding network in step S130:

[0105] Reference Figure 4 The decoding network may further include at least one decoder connected in sequence. Each decoder decodes multiple query features input to it and outputs multiple corresponding decoded query features. The multiple query features input to the first decoder are multiple initial query features, while the multiple decoded query features output by the last decoder are multiple final query features.

[0106] Consider a decoding network comprising multiple decoders (a single decoder is simply a simplified version of multiple decoders). The decoding process can be viewed as an iterative optimization of the initial query features. Each pass through a decoder represents one round of optimization, and the final query feature output by the last decoder is the result of these multiple optimizations. This feature effectively represents the features within the corresponding sub-region of the BEV space, thus enabling high-precision object segmentation based on the final query feature. During the query feature optimization process, the decoder interactively calculates the query features with image features.

[0107] Furthermore, an attention mechanism can be implemented in the decoding network to enhance the expressive power of query features and improve target segmentation results. For example, referring to... Figure 4 Each decoder contains a self attention layer and a cross attention layer, and it is possible that the decoder may also contain other layers.

[0108] In this process, each decoder uses a self-attention layer to perform interactive calculations on multiple query features input to the decoder, while a cross-attention layer is used to perform interactive calculations on multiple query features output from the self-attention layer and image features (of course, if the self-attention layer and the cross-attention layer are not directly connected, i.e., there are other layers in between, then the cross-attention layer is used to perform interactive calculations on multiple query features input to the layer and image features).

[0109] The aforementioned "interactive computation" for attentional network layers refers to the mutual computation between queries, keys, and values, or attention operations. The general calculation process is as follows: calculate the weight coefficients representing attention based on the query and key, and then perform a weighted summation of the values ​​based on the weight coefficients. For the meanings of the query, key, and value, as well as the specific process of attention operations, please refer to existing technologies, which will not be elaborated here.

[0110] For the self-attention layer in the decoder, the Query is (or originates from) the query feature input to the decoder, while the Key and Value are calculated based on the Query. Therefore, the function of the self-attention layer can be summarized as optimizing the query feature using an attention mechanism. For the cross-attention layer in the decoder, the Query is (or originates from) the query feature output by the self-attention layer, while the image features serve as the Key and Value (or the Key and Value originate from the image features). Therefore, the function of the cross-attention layer can also be summarized as optimizing the query feature using image features through an attention mechanism.

[0111] Based on the above embodiments, the following describes the implementation method of obtaining the target segmentation result based on DCT coefficients in step S140. In this implementation method, step S140 may further include:

[0112] Step a: Input multiple final query features into the segmentation network to obtain multiple sets of corresponding DCT coefficients.

[0113] Step b: Perform IDCT on multiple sets of DCT coefficients to obtain target segmentation results within the corresponding multiple sub-regions.

[0114] To understand the schemes in steps a to b, let's first briefly introduce DCT. DCT is commonly used in the field of image compression, and its typical approach is as follows (this is only a basic principle and may differ slightly from actual compression standards):

[0115] The image to be compressed is divided into blocks, for example, 8×8 blocks. A Direct Transformation (DCT) is performed on each block, resulting in another 8×8 block. Each pixel value within the block represents a DCT coefficient. DCT is a transform that converts image signals from the spatial domain to the frequency domain, and it has an energy concentration effect. This is visually represented by the fact that in the 8×8 DCT coefficient block, the values ​​in the upper left corner are larger (mainly representing low-frequency information in the image), while the values ​​in the remaining parts are smaller (mainly representing high-frequency information or noise in the image), even approaching 0.

[0116] Performing IDCT on an 8×8 DCT coefficient block allows for lossless reconstruction of the 8×8 image block. However, for lossy image compression, reconstructing the original image is not required. Therefore, the smaller values ​​in the 8×8 DCT coefficient block can be set to zero, i.e., only a few DCT coefficients in the upper left corner are retained, such as only the 10 DCT coefficients in the upper left corner. In this way, the original 64-value image block is compressed into 10 values, and subsequent image block transmission and storage only use these 10 values.

[0117] During image block decompression, simply arrange these 10 DCT coefficients in the top left corner of the 8×8 block, pad the remaining DCT coefficients with 0, and then perform IDCT. Of course, because the smaller DCT coefficients are set to zero during compression, the original image block cannot be recovered losslessly; a small amount of image detail will be lost. However, the main content of the image block is still preserved, as it is primarily represented by these 10 DCT coefficients in the top left corner.

[0118] Returning to the scheme of this application, the target segmentation result within a sub-region can be regarded as an image block, such as a 64×64 block. This image block can be compressed using a small number of DCT coefficients (e.g., 64 or 128). By performing IDCT on these DCT coefficients, the target segmentation result can be approximately restored. Steps a to b can be regarded as the reverse application of this compression process. In step a, the segmentation network is first used to predict the DCT coefficients corresponding to the target segmentation result within the sub-region. In step b, the DCT coefficients are then used to restore the target segmentation result within the sub-region. Although this restoration involves a small loss of accuracy, it is sufficient in many application scenarios.

[0119] The advantages of the schemes in steps a to b are briefly analyzed below:

[0120] If the target segmentation result within the sub-region is directly calculated using the final query features, the computational load on the segmentation network becomes excessive when the target segmentation result is large, such as 64×64 or 128×128. To avoid this problem, smaller target segmentation results, such as 16×16 or 32×32, must be calculated first, and then upsampling is performed as needed. However, small-sized target segmentation results have lower segmentation accuracy (some small targets will be ignored), and upsampling cannot improve the segmentation accuracy.

[0121] If the DCT coefficients are calculated first, and then the IDCT is used to calculate the target segmentation result within the sub-region, the computational load of the segmentation network is not large because the number of DCT coefficients is not large (for example, 128 is significantly smaller than 64×64 or 128×128). That is, only a small amount of computational and storage resources are needed to obtain a large target segmentation result, and there is no need to upsample the target segmentation result.

[0122] Furthermore, the DCT coefficients calculated in step a can be in vector form, but before performing IDCT, these DCT coefficients need to be shaped into DCT coefficient blocks. One shaping method is to first pad the DCT coefficients with zeros (for example, if the size of the target segmentation result of the sub-region is 64×64, then 64×64-128 zeros need to be padded after 128 DCT coefficients), and then arrange the zero-padded DCT coefficients into coefficient blocks according to the ZigZag method. For the ZigZag method, please refer to the existing technology, which will not be described in detail here.

[0123] Figure 5 The functional modules included in the image processing apparatus 200 provided in an embodiment of this application are shown. (Refer to...) Figure 5 The image processing apparatus 200 includes:

[0124] Image feature acquisition module 210 is used to acquire the image to be processed and extract the image features of the image to be processed using the backbone network;

[0125] The query feature acquisition module 220 is used to acquire multiple initial query features; wherein each initial query feature corresponds to a sub-region in the BEV space, and the BEV space is the planar region corresponding to the shooting range of the image to be processed from a bird's-eye view.

[0126] Decoding module 230 is used to input the multiple initial query features into the decoding network and perform interactive calculations with the image features that are also input into the decoding network to obtain the corresponding multiple final query features;

[0127] The target segmentation module 240 is used to obtain target segmentation results within multiple sub-regions based on the multiple final query features using a segmentation network;

[0128] The result summarization module 250 is used to obtain the overall target segmentation result in the BEV space based on the target segmentation results in the multiple sub-regions.

[0129] In one implementation of the image processing device 200, the query feature acquisition module 220 acquires multiple initial query features, including: acquiring multiple sets of coordinates corresponding to the multiple initial query features; wherein each set of coordinates includes at least an x-coordinate and a y-coordinate, and the x-coordinate and y-coordinate corresponding to each initial query feature are located in a sub-region in the BEV space corresponding to the initial query feature; encoding the multiple sets of coordinates using a query feature encoding network to obtain the corresponding multiple initial query features; wherein the dimension of an initial query feature is greater than the dimension of a set of coordinates.

[0130] In one implementation of the image processing apparatus 200, each set of coordinates also includes a z-coordinate obtained through learning.

[0131] In one implementation of the image processing device 200, the query feature encoding network includes a position encoder and a feature encoder. The query feature acquisition module 220 uses the query feature encoding network to encode the multiple sets of coordinates to obtain the corresponding multiple initial query features, including: using the position encoder to encode the multiple sets of coordinates to obtain the corresponding multiple position encoded features; wherein, the dimension of one position encoded feature is greater than the dimension of a set of coordinates; using the feature encoder to encode the multiple position encoded features to obtain the corresponding multiple initial query features; wherein, the feature encoder is a multilayer perceptron.

[0132] In one implementation of the image processing device 200, the target segmentation module 240 obtains target segmentation results within multiple corresponding sub-regions using a segmentation network based on the multiple final query features, including: inputting the multiple final query features into the segmentation network to obtain multiple sets of DCT coefficients; and performing IDCT on the multiple sets of DCT coefficients to obtain target segmentation results within the multiple corresponding sub-regions.

[0133] In one implementation of the image processing device 200, the decoding network includes at least one decoder connected in sequence. Each decoder decodes multiple query features input to it and outputs multiple decoded query features. The multiple query features input to the first decoder are the multiple initial query features, and the multiple decoded query features output by the last decoder are the multiple final query features. Each decoder includes a self-attention layer and a cross-attention layer. When decoding, the self-attention layer performs interactive calculations on the multiple query features input to it, and the cross-attention layer performs interactive calculations on the multiple query features output by the self-attention layer and the image features.

[0134] In one implementation of the image processing device 200, the image to be processed is a multi-frame image, wherein each frame corresponds to a viewpoint and is captured by a camera mounted on a mobile carrier, and the overall shooting range of the multi-frame image covers the four sides of the mobile carrier.

[0135] The image processing apparatus 200 provided in this application embodiment can be used to execute the image processing method provided in this application embodiment. The implementation principle and technical effects of the apparatus have been described in the foregoing method embodiment. For the sake of brevity, any parts not mentioned in the apparatus embodiment can be referred to the corresponding content in the method embodiment.

[0136] Figure 6 The structure of the electronic device 300 provided in an embodiment of this application is shown. (Refer to...) Figure 6 The electronic device 300 includes a processor 310, a memory 320, and a camera 330. These components are interconnected and communicate with each other via a communication bus 340 and / or other forms of connection mechanism (not shown).

[0137] The processor 310 includes one or more (only one is shown in the figure), which can be an integrated circuit chip with signal processing capabilities. The processor 310 can be a general-purpose processor, including a Central Processing Unit (CPU), a Microcontroller Unit (MCU), a Network Processor (NP), or other conventional processors; it can also be a special-purpose processor, including a Graphics Processing Unit (GPU), a Neural-network Processing Unit (NPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Furthermore, when there are multiple processors 310, some can be general-purpose processors and others can be special-purpose processors.

[0138] The memory 320 includes one or more (only one is shown in the figure), which may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0139] Processor 310 and other possible components can access memory 320, read and / or write data therein. In particular, one or more computer program instructions can be stored in memory 320, and processor 310 can read and execute these computer program instructions to implement the image processing method provided in the embodiments of this application.

[0140] The camera 330 includes one or more (only one is shown in the figure), which can be a device for acquiring images (including video), for example, acquiring the image to be processed in step S110, and storing the image to be processed in memory 320.

[0141] Understandable. Figure 6 The structure shown is for illustrative purposes only; the electronic device 300 may also include components that are more advanced than those shown. Figure 6 The more or fewer components shown, or having the same Figure 6 Different configurations are shown. For example, electronic device 300 may also include a communication module for communicating with other devices. As another example, electronic device 300 may also not include camera 330, and the image to be processed may be saved to memory 320 via other means (e.g., network download).

[0142] Figure 6 The components shown can be implemented using hardware, software, or a combination thereof. Electronic device 300 may be a physical device, such as an in-vehicle device, wearable device, PC, or server, or a virtual device, such as a virtual machine or container. Furthermore, electronic device 300 is not limited to a single device; it can also be a combination of multiple devices or a cluster of numerous devices.

[0143] The installation scenario for electronic device 300 is not limited. For example, electronic device 300 can be installed on a mobile carrier (installation method can be found in [reference]). Figure 2 During the movement of the mobile carrier, the camera 330 of the electronic device 300 captures the image to be processed, and the processor 310 reads and runs the computer program instructions in the memory 320 to implement the image processing method provided in this embodiment. Alternatively, the electronic device 300 may not be mounted on any carrier; for example, the image to be processed may be stored in the memory 320 through a channel other than direct acquisition. The processor 310 can still read and run the computer program instructions in the memory 320 to implement the image processing method provided in this embodiment.

[0144] This application also provides a computer-readable storage medium storing computer program instructions. These computer program instructions are read and executed by a processor to perform the image processing method provided in this application. For example, the computer-readable storage medium can be implemented as follows: Figure 6 The memory 320 in the electronic device 300.

[0145] This application also provides a computer program product, which includes computer program instructions. These computer program instructions are read and executed by a processor to perform the image processing method provided in this application.

[0146] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An image processing method, characterized by, include: The image to be processed is acquired, and the image features of the image to be processed are extracted using the backbone network; Multiple initial query features are obtained; wherein each initial query feature corresponds to a sub-region in the bird's-eye view BEV space, and the BEV space is the planar region corresponding to the shooting range of the image to be processed in the bird's-eye view. The multiple initial query features are input into the decoding network and interactively calculated with the image features that are also input into the decoding network to obtain the corresponding multiple final query features; Based on the multiple final query features, a segmentation network is used to obtain target segmentation results within the corresponding multiple sub-regions; Based on the target segmentation results within the multiple sub-regions, the overall target segmentation result in the BEV space is obtained; The process of obtaining multiple initial query features includes: Obtain multiple sets of coordinates corresponding to the multiple initial query features; wherein each set of coordinates includes at least x coordinates and y coordinates, and the x coordinates and y coordinates corresponding to each initial query feature are located in the sub-region corresponding to the initial query feature in the BEV space; The multiple sets of coordinates are encoded using a query feature encoding network to obtain the corresponding multiple initial query features; wherein the dimension of an initial query feature is greater than the dimension of a set of coordinates. The query feature encoding network includes a location encoder and a feature encoder. The location encoder uses a fixed algorithm to encode the multiple sets of coordinates to expand the coordinate dimension and does not have learnable parameters. The feature encoder is a multilayer perceptron used to map the location encoded features from single-point features to regional features of corresponding sub-regions.

2. The image processing method of claim 1, wherein, Each set of coordinates also includes the z-coordinate obtained through learning.

3. The image processing method of claim 2, wherein, The step of encoding the multiple sets of coordinates using a query feature encoding network to obtain the corresponding multiple initial query features includes: The position encoder is used to encode the multiple sets of coordinates to obtain multiple corresponding position encoding features; wherein, the dimension of a position encoding feature is greater than the dimension of a set of coordinates. The feature encoder is used to encode the plurality of location-coded features to obtain the corresponding plurality of initial query features; wherein, the feature encoder is a multilayer perceptron.

4. The image processing method according to any one of claims 1-3, characterized in that, The step of obtaining target segmentation results within multiple sub-regions using a segmentation network based on the multiple final query features includes: The multiple final query features are input into the segmentation network to obtain the corresponding multiple sets of Discrete Cosine Transform (DCT) coefficients. The inverse offline cosine transform (IDCT) is performed on the multiple sets of DCT coefficients to obtain the target segmentation results in the corresponding multiple sub-regions.

5. The image processing method according to any one of claims 1-4, characterized in that, The decoding network includes at least one decoder connected in sequence. Each decoder is used to decode multiple query features input to the decoder and output multiple decoded query features. The multiple query features input to the first decoder are the multiple initial query features, and the multiple decoded query features output by the last decoder are the multiple final query features. Each decoder includes a self-attention layer and a cross-attention layer. When decoding, the self-attention layer is used to perform interactive calculations on multiple query features input to the decoder, and the cross-attention layer is used to perform interactive calculations on multiple query features output by the self-attention layer and the image features.

6. The image processing method according to any one of claims 1-3, characterized in that, The image to be processed is a multi-frame image, where each frame corresponds to a viewpoint and is captured by a camera mounted on a mobile carrier. The overall shooting range of the multi-frame image covers the four sides of the mobile carrier.

7. A computer program product, characterized in that, It includes computer program instructions, which, when read and executed by a processor, perform the method as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when read and executed by a processor, perform the method as described in any one of claims 1-6.

9. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores computer program instructions, which are read and executed by the processor to perform the method according to any one of claims 1-6.

Images