A lightweight and lossless method for fine-grained panoramic environment perception

Abstract

The present invention discloses a distortion-free, lightweight, fine-grained panoramic environment perception method, which can effectively solve the distortion problem in panoramic images and the ambiguity problem of object importance ranking. By accurately mining the importance clues in panoramic scenes, the method significantly improves the computer's ability to automatically perceive the panoramic environment. In addition, the present invention adopts a lightweight network structure design to enable it to reach real-time processing speed, optimize the use of computing resources, and fully meet the real-time requirements of downstream tasks such as autonomous driving and video navigation. This method that combines high-precision perception with high-efficiency processing will greatly promote the widespread application of panoramic image technology in real-time application scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of salient object ranking methods, and in particular relates to a distortion-free lightweight panoramic environment fine-grained perception method. Background Art

[0002] 360° panoramic images can provide a full range of vision of 360 degrees horizontally and 180 degrees vertically, and have been widely used in fields such as autonomous driving, video navigation, and virtual reality. This panoramic field of view expands the breadth of environmental perception, but it also increases the complexity of environmental perception, because operators or automatic systems need to respond quickly in a vast and changing environment. In order to accurately process such highly complex panoramic images and achieve efficient decision support, fine-grained perception of the panoramic environment is required, among which object salience / importance ranking is a powerful tool. By accurately evaluating the importance of each object in the environment, we can better understand and predict potential dynamic changes in the environment, thereby providing safer and more efficient response strategies.

[0003] Existing salient object ranking methods have made some progress in understanding and modeling important objects, but these methods usually involve complex network structures, resulting in slow reasoning speed. For example, mainstream ranking networks involve complex multi-scale feature fusion and non-maximum suppression (NMS) post-processing operations, which are particularly time-consuming when processing large-scale data sets. In order to speed up the model operation, model lightweight technology has been widely used in classification, detection and image segmentation fields, mainly through model compression and lightweight model design. Model compression involves compressing and optimizing the pre-trained model to reduce the scale of model parameters and reduce computational complexity. However, in order to ensure that the performance of the compressed model is not affected, it is usually necessary to rely on a large amount of pre-training data and sufficient computing resources, thereby increasing the training cost. In contrast, lightweight model design does not rely on pre-training, and directly designs a concise and efficient neural network model from scratch. The present invention follows this model lightweight design.

[0004] Panoramic images are usually stored in the equirectangular projection (ERP) format, and the projection characteristics of this format cause significant distortion in the upper and lower levels of the image. Existing object importance ranking methods often do not work well in dealing with this distortion caused by lens characteristics, resulting in reduced accuracy of object importance ranking, which in turn affects subsequent accurate decision-making. In addition, there are many objects in panoramic images, which makes many existing methods ambiguous in judging the importance of objects, and it is difficult to distinguish the importance of different objects with the same semantics. Most importantly, traditional salient object ranking methods mainly identify salient objects in the environment. This low-information prediction result greatly restricts the computer's ability to perceive and understand the scene.

[0005] In summary, there are three technical challenges in fine-grained perception of panoramic environments: First, image distortion. Panoramic image stitching and lens curvature can easily cause severe image distortion. This distortion distorts the shape and size of objects in the image, posing a challenge to the accuracy of scene analysis and object importance prediction, because standard image processing algorithms often assume that images have normal proportions and perspectives. Second, the accuracy of object importance ranking. In complex panoramic environments, there are many objects, and they are distorted and distorted in visual presentation, resulting in ambiguity in the existing methods in determining the importance of different objects with the same semantics, making it difficult to effectively distinguish them. Third, model lightweight and efficiency issues. The challenge of lightweight design is how to significantly reduce its computational complexity and memory usage while maintaining model performance.

[0006] This paper starts with the problem of fine-grained object importance ranking in panoramic scenes, and explores fine-grained perception and analysis of panoramic environments. The fine-grained object importance ranking is defined as all identifiable important objects in the segmented image, and they are ranked according to their significance / importance. Summary of the invention

[0007] In order to solve the technical problems existing in the background technology, the present invention aims to provide a lightweight, distortion-free, fine-grained panoramic environment perception method, which can achieve fine-grained, high-precision and efficient comprehensive perception and analysis of the panoramic environment. The present invention solves the geometric distortion problem in the panoramic image by introducing a distortion adaptive encoder; in view of the ambiguity problem of the object importance ranking, a self-paced iteration strategy is proposed, and each iteration only determines the importance ranking of one object until the importance ranking of all objects is completed; the overall network structure adopts a lightweight encoder and decoder structure design, so that the model can achieve real-time processing speed.

[0008] In order to solve the technical problem, the technical solution of the present invention is:

[0009] A light-weight, non-distorted, panoramic environment fine-grained perception method, the method comprising:

[0010] S1: Obtain a dataset of fine-grained object importance ranking of panoramic scene images and build a distortion-free lightweight initial model for fine-grained object importance ranking;

[0011] S2: using the panoramic scene image fine-grained object importance ranking dataset to train the initial model to obtain a fine-grained object importance ranking prediction model that is more suitable for panoramic images;

[0012] S3: preprocess each ERP panorama of the panoramic scene image fine-grained object importance ranking dataset to obtain a standardized ERP panoramic image;

[0013] S4: inputting the standardized ERP panoramic image into a distortion adaptive encoder to obtain distortion-free ERP panoramic image features;

[0014] S5: inputting the undistorted ERP panorama features into a lightweight pixel decoder to obtain multi-scale global features;

[0015] S6: Multi-scale features are input into a dual-path Transformer decoder to obtain refined image features and object queries. The lightweight three-branch prediction head is used to obtain category prediction, pixel-level binary mask prediction, and object importance ranking prediction. The prediction results together constitute the final fine-grained scene importance perception result.

[0016] The present invention introduces an importance ranking method with a lightweight encoder and decoder network structure, which can achieve a real-time inference speed of 15 frames per second, so that the model of the present invention can not only be conveniently deployed on mobile hardware, but also enhance the availability and adaptability of the model in downstream application scenarios.

[0017] The distortion adaptive encoder proposed in this invention is specifically used to deal with the geometric distortion problem of ERP panoramic images. Its innovation lies in the use of window grid segmentation and intra-window rotation consistency measurement technology, which can be used as a universal plug-in and embedded in the existing 2D salient object ranking network to enable it to adapt to the processing requirements of panoramic scenes.

[0018] The importance ranking branch of the present invention adopts a self-paced iterative update strategy, in which only the most important objects are inferred in each iteration, effectively avoiding the ambiguity of the relative importance ranking. Each iteration shares the network parameters, which not only reduces the parameter scale of the model, but also ensures that the number of important objects is not limited by the number of objects, thereby enabling more comprehensive important object reasoning.

[0019] The present invention adopts a phased task design strategy, first using a lightweight model to distinguish important foregrounds from unimportant backgrounds, and then performing a more complex importance ranking of panoramic objects, ensuring the efficiency and accuracy of the model at each stage.

[0020] Further, the step S1 comprises:

[0021] S101: Acquire a real ERP panoramic scene image data set;

[0022] S102: Manually annotating the panoramic scene image data set to obtain binary segmentation masks of all identifiable important objects and real human eye gaze point data of the entire image;

[0023] S103: using the binary segmentation mask of the important object and the ERP panoramic saliency map generated by real human eye gaze point data, the importance of the important object in the ERP image is evaluated by calculating the maximum value of the panoramic saliency map within the object mask, and an initial instance-level object importance ranking data set is obtained;

[0024] S104: Manually review and supplement annotations are performed on the initial data set to ensure the accuracy and fine-grainedness of the data set, obtain a final fine-grained object importance ranking data set, and divide it into a training set and a test set.

[0025] Furthermore, the distortion-free lightweight fine-grained object importance ranking initial model adopts the FastInst model as the basic network architecture.

[0026] Further, the step S2 trains the initial model, specifically including:

[0027] S201: using the binary segmentation mask of the important object and the object category label to train the pixel-level binary mask branch and the classification branch respectively;

[0028] S202: After the training is completed, the trained model weights are fixed, and the fine-grained object importance ranking labels are introduced. The pixel-level binary mask branch, the classification branch, and the newly added importance ranking branch are comprehensively fine-tuned so that the model can handle the tasks of these three branches at the same time, thereby obtaining a fine-grained object importance ranking prediction model that is more suitable for panoramic images.

[0029] Furthermore, the image preprocessing adopts image demeaning processing, and standardization of the panoramic image is achieved by subtracting the pixel mean of the training set image from each ERP panoramic image.

[0030] Further, the step S4 specifically includes:

[0031] S401: Mapping the standardized ERP panoramic image into a plurality of distortion-free sub-window patches, wherein the distortion-free mapping rule is expressed by the following formula group:

[0032] Subwins sph =Grid avg (f ERP→sph (ERP)

[0033] Subwins nodis =f sph→plane (Dila(Subwins sph )),

[0034] Among them, ERP represents the image in the equirectangular projection format. Since the ERP projection maps the longitude and latitude lines into vertical lines with constant spacing, it causes different degrees of distortion at different angles.ERP→sph Represents the projection mapping function of ERP to the sphere, Grid avg Indicates an average grid division of the sphere, Subwins sph represents the set of spherical sub-windows after segmentation, Dila represents the expansion operation, which increases the area of ​​each spherical sub-window by 1.5 times to obtain overlapping sub-windows, and f sph→plane Represents the projection mapping function from sphere to plane, Subwins nodis represents the distortion-free subwindow patch;

[0035] S402: Feed the multiple undistorted sub-window patches into a linear projection layer, and then obtain undistorted ERP panorama features through multiple SwinTransformer blocks, wherein the SwinTransformer block adds a layer of RotaryAttentionModule RAM after the original shifted window-based SW-MSA module to solve the problem of discontinuity of objects at both ends in the ERP format;

[0036] The improved SwinTransformer block is specifically implemented as follows:

[0037] x h =WMSA(LN(x h-1 ))+x h-1 ,

[0038] x h+1 =SWMSA(LN(x h ))+x h ,

[0039] x h+2 =RAM(LN(x h+1 ))+x h+1 ,

[0040] Among them, WMSA represents multi-head self-attention based on regular window, SWMSA represents multi-head self-attention based on sliding window, RAM represents rotational attention module, and LN represents normalization layer.

[0041] Furthermore, the lightweight pixel decoder is improved on the basis of the initial pixel decoder. The improved lightweight pixel decoder makes key structural adjustments based on the original pixel decoder structure, and adjusts the level of the output multi-scale global features from the fourth convolutional layer C 4 Change to the fifth network layer C 5 , and add an auxiliary classification head after the five convolutional layers to produce class probability predictions for each pixel;

[0042] The improved lightweight pixel decoder has the following structure:

[0043]

[0044] Among them, F 5 represents the high-level features output by the fifth SwinTransformer block, PPM represents the pyramid pooling module, CU represents the convolution plus sampling operation, represents context aggregation features, Conv represents convolution operation, Represents matrix addition, and Concat represents superposition operation.

[0045] Further, the step S6 specifically includes:

[0046] S601: The fused multi-scale feature C 5 , combined with K instances to activate the guided object query Q g and N auxiliary learnable object queries Q a Input into the dual-path Transformer decoder to obtain a finer-grained pixel feature F pxi and object query QE;

[0047] S602: performing a multi-layer full connection operation on the object query QE to obtain a final category prediction;

[0048] S603: By analyzing the pixel-level feature F pxi And object query QE multiplied to get the pixel-level binary mask prediction of the important object;

[0049] S604: Perform a cross-attention operation on the object query QE to update the object query. The updating process is shown in the following formula:

[0050]

[0051] Among them, CrossAttn represents cross attention, N represents the number of objects contained in a single image, and qe i and qe j The instance-level features belonging to any object query QE, V his Indicates the access history code. Represents the updated object query;

[0052] S605: Query the updated object Perform two residual gated graph convolution operations to learn the relationship between objects. The graph convolution operation is expressed using the following formula:

[0053]

[0054] Among them, RGCN represents the residual gated graph convolution operation, represents the updated object query, ω represents the learnable parameters of graph convolution, and V' represents the updated node features;

[0055] S606: Normalize the node feature V' through the Sigmoid function to obtain the most important object index Rank of the current iteration i ;

[0056] S607: Execute the self-paced iteration strategy and repeat the above steps S604, S605 and S606 until the importance rankings of all objects are obtained.

[0057] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any one of the above-mentioned light-weight, distortion-free, panoramic environment fine-grained perception methods is implemented.

[0058] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements any one of the above-mentioned methods for lightweight, fine-grained perception of a panoramic environment without distortion.

[0059] Compared with the prior art, the advantages of the present invention are:

[0060] Existing salient object ranking methods usually involve complex network structures, resulting in slow inference speed. The present invention effectively reduces the amount of model computation by designing lightweight encoder and decoder network structures, thereby significantly improving model training and inference speed. In addition, the distortion adaptive encoder of the model can effectively alleviate the geometric distortion problem of ERP projection.

[0061] Among the existing model lightweight technologies, model distillation methods are widely used, but this method is overly dependent on large models and pre-training of large-scale data sets, and is prone to losing key information during the knowledge transfer process. The present invention proposes a new method that combines lightweight model design with parameter sharing strategy. This method reduces the dependence on large pre-trained models, effectively reduces the number of model parameters, and improves computational efficiency.

[0062] Existing salient object ranking methods have limitations in the number of salient objects detected, usually up to 8 objects, or the model outputs a fixed number of salient objects (usually 5) during inference. This inherent limitation of the design reduces the effectiveness of the model in fully characterizing the perception ability of the real human eye. The present invention adopts a self-paced iterative update strategy, inferring only the most salient objects in each iteration until the importance ranking of all objects is obtained. This method not only allows the model to flexibly handle any number of important objects and avoid the ambiguity of ranking, but also more accurately simulates the attention mechanism of human vision.

[0063] The current mainstream salient object ranking method adopts an end-to-end training method, which easily leads to the model weight biased towards a single task branch, affecting the overall optimization and performance. The present invention adopts a phased task training strategy to avoid the complexity of simultaneous multi-task optimization by gradually optimizing different task modules. This method simplifies the training process while ensuring the efficiency and accuracy of the model at each stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 , a flow chart of the light-weight, non-distorted, panoramic environment fine-grained perception method;

[0065] Figure 2 , a schematic diagram of the overall network structure of a light-weight, distortion-free, panoramic environment fine-grained perception method;

[0066] Figure 3 ,Schematic diagram of the self-paced iteration strategy for ranking object importance;

[0067] Figure 4 , schematic diagram of the branch network structure details for importance ranking prediction. DETAILED DESCRIPTION

[0068] The specific implementation mode of the present invention is described below in conjunction with embodiments:

[0069] It should be noted that the structures, proportions, sizes, etc. shown in this specification are only used to match the contents disclosed in the specification so that people familiar with this technology can understand and read them, and are not used to limit the conditions under which the present invention can be implemented. Any structural modification, change in proportional relationship or adjustment of size should still fall within the scope of the technical content disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0070] At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" cited in this specification are only for the convenience of description and are not used to limit the scope of implementation of the present invention. Changes or adjustments to their relative relationships should be regarded as the scope of implementation of the present invention without substantially changing the technical content.

[0071] Embodiment 1:

[0072] Figure 1 A flow chart of a light-weight, non-distorted panoramic environment fine-grained perception method according to the present disclosure is provided, which includes the following steps:

[0073] S1: Obtain a dataset of fine-grained object importance rankings in panoramic scene images and divide it into training and test sets for model learning and verification.

[0074] To reduce the time cost of manual labeling, S1 mainly includes the following steps:

[0075] S11: Acquire a real ERP panoramic scene image data set.

[0076] In a specific embodiment, an existing salient object detection dataset ODI-SOD is used as an initial data source, where the dataset contains 6,263 high-resolution ERP panoramic images with pixel-level annotations.

[0077] S12: Performing detailed manual annotation on the panoramic scene image data set to obtain binary segmentation masks of all identifiable important objects and real human eye gaze point data of the entire image.

[0078] In a specific embodiment, the ODI-SOD dataset contains important objects that have not been instance segmented, so these important objects need to be segmented into separate important instances through manual inspection. The real human eye gaze point data of the ERP image is collected through a head-mounted display device and Gaussian blur processing is performed on it to generate an ERP panoramic saliency map.

[0079] S13: Using the binary segmentation mask of the important object and the ERP panoramic saliency map generated by real human eye gaze point data, the importance of the important object in the ERP image is evaluated by calculating the maximum value of the panoramic saliency map within the object mask, and an initial instance-level object importance ranking dataset is obtained.

[0080] In a specific embodiment, the pixel-level binary mask of the important object and the ERP panoramic saliency map are used to distinguish the importance of different objects by calculating the maximum saliency value in the object mask, and the importance ranking of the object is obtained according to the saliency value to obtain the initial instance-level object importance ranking data set. Although there are several distortions in the objects in the ERP format, because only the maximum value point in the object is selected, it is independent of the object size, so the distortion does not affect the calculation of the importance ranking.

[0081] S14: Manually review the initial dataset and perform necessary supplementary annotations to ensure the accuracy and fine-grainedness of the dataset, thereby forming a final fine-grained object importance ranking dataset, which is then divided into a training set and a test set in a ratio of 7:3 to support subsequent model training and verification.

[0082] In a specific implementation, the manual review includes checking whether there are important objects or the ranking of important objects is missing. 6,000 pairs of data are obtained through the above supplementary annotation, 4,200 pairs of images are selected as training sets, and the remaining 1,800 pairs of images are selected as test sets.

[0083] S2: constructing an initial object importance ranking model, training the initial object importance ranking model based on the panoramic image fine-grained object importance ranking dataset, and determining the trained object importance ranking model as a fine-grained object importance ranking prediction model.

[0084] In a specific embodiment, referring to the attached Figure 2 The initial object importance ranking model uses the FastInst model as the backbone network, uses a distortion adaptive encoder to replace the original Resnet50 feature extraction network, and adds an importance ranking branch to obtain a fine-grained object importance ranking model adapted to panoramic images.

[0085] The initial object importance ranking model is trained using the panoramic image object importance ranking dataset, and the specific training steps include:

[0086] S21: The first stage of training includes using the binary segmentation mask of the important object and the object category label to train the pixel-level binary mask branch and the classification branch respectively.

[0087] S22: After the first stage of training is completed, the trained model weights are fixed, and the fine-grained object importance ranking labels are introduced. The pixel-level binary mask branch, classification branch, and newly added importance ranking branch are comprehensively fine-tuned so that the model can handle the tasks of these three branches at the same time. Finally, the fine-tuned model is the fine-grained object importance ranking prediction model.

[0088] In a specific embodiment, the training loss is calculated as follows:

[0089] L total = l ia-q +l mask +l cla +l rank ,

[0090] Among them, l ia-q represents the instance activation guided object query loss, using cross entropy loss, lmask represents the important object segmentation mask loss, using binary cross entropy loss and diceloss, l cla and l rank They represent classification loss and ranking loss respectively, both using binary cross entropy loss.

[0091] S3: The image preprocessing adopts image mean removal processing, which specifically achieves standardization of the panoramic image by subtracting the pixel mean of the training set image from each ERP panoramic image.

[0092] S4: The preprocessed ERP panoramic image is input into the distortion adaptive encoder of the object importance ranking prediction model, and features are extracted at multi-scale levels from low to high to obtain the distortion-free multi-scale image feature F i ,i∈{3,4,5}. The distortion adaptive encoder comprises:

[0093] S41: Mapping the standardized distorted ERP panoramic image into a plurality of distortion-free sub-window patches, wherein the distortion-free mapping rule is represented by the following formula group:

[0094] Subwins sph =Grid avg (f ERP→sph (ERP)

[0095] Subwins nodis =f sph→plane (Dila(Subwins sph )),

[0096] Among them, ERP represents the image in the equirectangular projection format. Since the ERP projection maps the longitude and latitude lines into vertical lines with constant spacing, it causes different degrees of distortion at different angles. ERP→sph Represents the projection mapping function of ERP to the sphere, Grid avg Indicates an average grid division of the sphere, Subwins sph represents the set of spherical sub-windows after segmentation, Dila represents the expansion operation, which increases the area of ​​each sub-window on the sphere to obtain overlapping sub-windows, and f sph→plane Represents the projection mapping function from sphere to plane, Subwins nodis Represents a distortion-free subwindow patch.

[0097] In a specific embodiment, each distorted ERP panoramic image is evenly divided into 64 undistorted sub-window patches, and the area of ​​each sub-window on the sphere is increased to 1.5 times in the expansion operation.

[0098] S42: Feed the multiple undistorted sub-window patches into the linear projection layer, and then obtain the undistorted panoramic image multi-scale features through multiple SwinTransformer blocks, wherein the SwinTransformer block adds a layer of rotational attention module (RAM) behind the original shift window-based SW-MSA module to solve the problem of discontinuity of objects at both ends in the ERP format.

[0099] In a specific implementation, the improved SwinTransformer block is implemented as follows:

[0100] x h =WMSA(LN(x h-1 ))+x h-1 ,

[0101] x h+1 =SWMSA(LN(x h ))+x h ,

[0102] x h+2 =RAM(LN(x h+1 ))+x h+1 ,

[0103] Among them, WMSA represents multi-head self-attention based on regular window, SWMSA represents multi-head self-attention based on sliding window, RAM represents rotational attention module, and LN represents normalization layer.

[0104] S5: The lightweight pixel decoder is improved on the basis of the initial pixel decoder, and the improved lightweight pixel decoder has the following structure:

[0105] Based on the original pixel decoder structure, a key structural adjustment is made to increase the level of the output multi-scale global features from the fourth convolutional layer C 4 Change to the fifth network layer C 5 , and an auxiliary classification head is added after the five convolutional layers to produce class probability predictions for each pixel.

[0106] In a specific implementation, the improved lightweight pixel decoder has the following structure:

[0107]

[0108] Among them, F 5 represents the high-level features output by the fifth SwinTransformer block, PPM represents the pyramid pooling module, CU represents the convolution plus sampling operation, represents context aggregation features, Conv represents convolution operation, Represents matrix addition, and Concat represents superposition operation.

[0109] S6: Select C 5 The global features of the level are combined with K instance activations to guide the object query Q g and N auxiliary learnable object queries Q a The data is input into the dual-path Transformer decoder to obtain more fine-grained pixel features and object queries, and then three lightweight branches are used to obtain category prediction, pixel-level binary mask prediction, and importance ranking prediction. Finally, the pixel-level binary mask prediction and importance ranking prediction are converted to grayscale to obtain the final instance-level fine-grained scene importance perception result.

[0110] In a specific embodiment, the S6 further comprises:

[0111] S61: The fused multi-scale feature C 5 , combined with K instances to activate the guided object query Q g and N auxiliary learnable object queries Q a Input into the dual-path Transformer decoder to obtain a finer-grained pixel feature F pxi and object query QE.

[0112] S62: Perform multi-layer full connection operations on the object query QE to obtain a final category prediction.

[0113] In a specific embodiment, the object query QE performs 3 layers of fully connected operations to obtain the final category prediction.

[0114] S63: By analyzing the pixel-level feature F pxi And the object query QE is multiplied to obtain the pixel-level binary mask prediction of the important object.

[0115] S64: Perform a cross-attention operation on the object query to update the object query.

[0116] In a specific embodiment, referring to the attached Figure 4 , the updating process is shown in the following formula:

[0117]

[0118] Among them, CrossAttn represents cross attention, N represents the number of objects contained in a single image, and qe i and qe j The instance-level features belonging to any object query QE, V his Indicates the access history code. Represents the updated object query;

[0119] S65: Perform two residual gated graph convolution operations on the updated object query to learn the relationship between objects.

[0120] In a specific embodiment, referring to the attached Figure 4 , the graph convolution operation is expressed as follows:

[0121]

[0122] Among them, RGCN represents the residual gated graph convolution operation, ω represents the learnable parameters of the graph convolution, and V' represents the updated node features.

[0123] S66: Normalize the node feature V' using the Sigmoid function to obtain the most important object index Rank of the current iteration i .

[0124] S67: Execute the self-paced iteration strategy, refer to the attached Figure 3 , repeat the above steps S64, S65 and S66 several times until the importance ranking of all objects is obtained.

[0125] In a specific embodiment, according to the above specific implementation methods, the comparison results of the evaluation results of a lightweight panoramic environment fine-grained perception method without distortion of the present invention and some existing mainstream models can be obtained as shown in Appendix 1. It can be seen from the performance comparison table that the method can obtain better evaluation results on the new data set proposed by the present invention and can achieve a faster reasoning speed.

[0126] Table 1 compares the object importance ranking results with the existing mainstream model importance ranking results.

[0127]

[0128] Embodiment 2:

[0129] This embodiment provides a terminal device, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of a distortion-free lightweight panoramic environment fine-grained perception method, including the following steps:

[0130] S1: Obtain a dataset of fine-grained object importance ranking of panoramic scene images and build a distortion-free lightweight initial model for fine-grained object importance ranking;

[0131] S2: using the panoramic scene image fine-grained object importance ranking dataset to train the initial model to obtain a fine-grained object importance ranking prediction model that is more suitable for panoramic images;

[0132] S3: preprocess each ERP panorama of the panoramic scene image fine-grained object importance ranking dataset to obtain a standardized ERP panoramic image;

[0133] S4: inputting the standardized ERP panoramic image into a distortion adaptive encoder to obtain distortion-free ERP panoramic image features;

[0134] S5: inputting the undistorted ERP panorama features into a lightweight pixel decoder to obtain multi-scale global features;

[0135] S6: Multi-scale features are input into a dual-path Transformer decoder to obtain refined image features and object queries. The lightweight three-branch prediction head is used to obtain category prediction, pixel-level binary mask prediction, and object importance ranking prediction. The prediction results together constitute the final fine-grained scene importance perception result.

[0136] Embodiment 3:

[0137] This embodiment provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory.

[0138] The processor may load and execute one or more instructions stored in a computer-readable storage medium to implement the corresponding steps of the above-mentioned embodiment related to a method for fine-grained perception of a lightweight panoramic environment without distortion; the processor may load and execute the following steps of one or more instructions in the computer-readable storage medium:

[0139] S1: Obtain a dataset of fine-grained object importance ranking of panoramic scene images and build a distortion-free lightweight initial model for fine-grained object importance ranking;

[0140] S2: using the panoramic scene image fine-grained object importance ranking dataset to train the initial model to obtain a fine-grained object importance ranking prediction model that is more suitable for panoramic images;

[0141] S3: preprocess each ERP panorama of the panoramic scene image fine-grained object importance ranking dataset to obtain a standardized ERP panoramic image;

[0142] S4: inputting the standardized ERP panoramic image into a distortion adaptive encoder to obtain distortion-free ERP panoramic image features;

[0143] S5: inputting the undistorted ERP panorama features into a lightweight pixel decoder to obtain multi-scale global features;

[0144] S6: Multi-scale features are input into a dual-path Transformer decoder to obtain refined image features and object queries. The lightweight three-branch prediction head is used to obtain category prediction, pixel-level binary mask prediction, and object importance ranking prediction. The prediction results together constitute the final fine-grained scene importance perception result.

[0145] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0147] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0149] The preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

[0150] Many other changes and modifications may be made without departing from the concept and scope of the present invention.It should be understood that the present invention is not limited to the specific embodiments, and the scope of the present invention is defined by the appended claims.

Claims

1. A lightweight, distortion-free, panoramic environment fine-grained perception method, characterized in that: The method comprises: S1: Obtain a dataset of fine-grained object importance ranking of panoramic scene images and build a distortion-free lightweight initial model for fine-grained object importance ranking; S2: using the panoramic scene image fine-grained object importance ranking dataset to train the initial model to obtain a fine-grained object importance ranking prediction model that is more suitable for panoramic images; S3: preprocess each ERP panorama of the panoramic scene image fine-grained object importance ranking dataset to obtain a standardized ERP panoramic image; S4: inputting the standardized ERP panoramic image into a distortion adaptive encoder to obtain distortion-free ERP panoramic image features; S5: inputting the undistorted ERP panorama features into a lightweight pixel decoder to obtain multi-scale global features; S6: Input the multi-scale features into the dual-path Transformer decoder to obtain refined image features and object queries. The lightweight three-branch prediction head obtains category prediction, pixel-level binary mask prediction, and object importance ranking prediction respectively. The prediction results together constitute the final fine-grained scene importance perception result. The lightweight pixel decoder is improved on the basis of the initial pixel decoder. The improved lightweight pixel decoder makes key structural adjustments based on the original pixel decoder structure, changes the level of the output multi-scale global features from the fourth convolutional layer C4 to the fifth network layer C5, and adds an auxiliary classification head after the five convolutional layers to generate class probability predictions for each pixel; The improved lightweight pixel decoder has the following structure: Among them, F5 represents the high-level features output by the fifth SwinTransformer block, PPM represents the pyramid pooling module, and CU represents the convolution plus sampling operation. represents context aggregation features, Conv represents convolution operation, Represents matrix addition, Concat represents superposition operation; The step S6 specifically includes: S601: Combine the fused multi-scale feature C5 with K instances to activate the guided object query Q g And N auxiliary learnable object queries Q a Input into the dual-path Transformer decoder to obtain a finer-grained pixel feature F pxi and object query QE; S602: performing a multi-layer full connection operation on the object query QE to obtain a final category prediction; S603: By analyzing the pixel-level feature F pxi And object query QE multiplied to get the pixel-level binary mask prediction of the important object; S604: Perform a cross-attention operation on the object query QE to update the object query. The updating process is shown in the following formula: Among them, CrossAttn represents cross attention, N represents the number of objects contained in a single image, and qe i and qe j The instance-level features belonging to any object query QE, V his Indicates the access history code. Represents the updated object query; S605: Query the updated object Perform two residual gated graph convolution operations to learn the relationship between objects. The graph convolution operation is expressed using the following formula: Among them, RGCN represents the residual gated graph convolution operation, represents the updated object query, ω represents the learnable parameters of graph convolution, and V′ represents the updated node features; S606: Normalize the node feature V′ using the Sigmoid function to obtain the most important object index Rank of the current iteration i ; S607: Execute the self-paced iteration strategy and repeat the above steps S604, S605 and S606 until the importance rankings of all objects are obtained.

2. The method for lightweight panoramic environment fine-grained perception without distortion according to claim 1, characterized in that: The step S1 comprises: S101: Acquire a real ERP panoramic scene image data set; S102: Manually annotating the panoramic scene image data set to obtain binary segmentation masks of all identifiable important objects and real human eye gaze point data of the entire image; S103: using the binary segmentation mask of the important object and the ERP panoramic saliency map generated by real human eye gaze point data, the importance of the important object in the ERP image is evaluated by calculating the maximum value of the panoramic saliency map within the object mask, and an initial instance-level object importance ranking data set is obtained; S104: Manually review and supplement annotations are performed on the initial data set to ensure the accuracy and fine-grainedness of the data set, obtain a final fine-grained object importance ranking data set, and divide it into a training set and a test set.

3. The method for lightweight, non-distorted, panoramic environment fine-grained perception according to claim 1, characterized in that: The distortion-free lightweight fine-grained object importance ranking initial model adopts the FastInst model as the basic network architecture.

4. The method for lightweight, non-distorted, panoramic environment fine-grained perception according to claim 1, characterized in that: The step S2 trains the initial model, specifically including: S201: using the binary segmentation mask of the important object and the object category label to train the pixel-level binary mask branch and the classification branch respectively; S202: After the training is completed, the trained model weights are fixed, and the fine-grained object importance ranking labels are introduced. The pixel-level binary mask branch, the classification branch, and the newly added importance ranking branch are comprehensively fine-tuned so that the model can handle the tasks of these three branches at the same time, thereby obtaining a fine-grained object importance ranking prediction model that is more suitable for panoramic images.

5. The method for lightweight, non-distorted, panoramic environment fine-grained perception according to claim 1, characterized in that: The image preprocessing adopts image mean removal processing, and standardization of the panoramic image is achieved by subtracting the pixel mean of the training set image from each ERP panoramic image.

6. The method for lightweight, non-distorted, panoramic environment fine-grained perception according to claim 1, characterized in that: The step S4 specifically includes: S401: Mapping the standardized ERP panoramic image into a plurality of distortion-free sub-window patches, wherein the distortion-free mapping rule is expressed by the following formula group: Subwins sph =Grid avg (f ERP→sph (ERP)), Subwins nodis =f sph→plane (Dila(Subwins sph )), Among them, ERP represents the image in the equirectangular projection format. Since the ERP projection maps the longitude and latitude lines into vertical lines with constant spacing, it causes different degrees of distortion at different angles. ERP→sph Represents the projection mapping function of ERP to the sphere, Grid avg Indicates an average grid division of the sphere, Subwins sph represents the set of spherical sub-windows after segmentation, Dila represents the expansion operation, which increases the area of ​​each spherical sub-window by 1.5 times to obtain overlapping sub-windows, and f sph→plane Represents the projection mapping function from sphere to plane, Subwins nodis represents the distortion-free subwindow patch; S402: Feed the multiple undistorted sub-window patches into a linear projection layer, and then obtain undistorted ERP panorama features through multiple SwinTransformer blocks, wherein the SwinTransformer block adds a layer of RotaryAttentionModule RAM after the original shifted window-based SW-MSA module to solve the problem of discontinuity of objects at both ends in the ERP format; The improved SwinTransformer block is specifically implemented as follows: x h =WMSA(LN(x h-1 ))+x h-1 , x h+1 =SWMSA(LN(x h ))+x h , x h+2 =RAM(LN(x h+1 ))+x h+1 , Among them, WMSA represents multi-head self-attention based on regular window, SWMSA represents multi-head self-attention based on sliding window, RAM represents rotational attention module, and LN represents normalization layer.

7. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a distortion-free lightweight panoramic environment fine-grained perception method according to any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements a distortion-free, lightweight, panoramic environment fine-grained perception method according to any one of claims 1 to 6.

Images