A light field camera semantic segmentation method, system and electronic device
By extracting shallow features from the sub-aperture image of the light field camera and using a double-headed self-attention network and cross-attention mechanism, the high memory cost and information redundancy in semantic segmentation of the light field camera image is solved, achieving high efficiency semantic segmentation and optimization of computing resources.
Patent Information
- Application Number
- CN202411008638.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-07-26
AI Technical Summary
The light field camera faces high memory costs and redundant data processing problems during image semantic segmentation, and cannot effectively utilize angle information, resulting in high computing pressure and redundant information. The existing technology ignores the implicit angular relationship of the light field image, resulting in boundaries. Blur or mix.
By extracting shallow features from sub-aperture images, a double-headed self-attention network is used to extract multi-scale features, dynamic weighting and cross-attention mechanisms to complement and correct heterogeneous information interactions, avoid information redundancy and calculation pressure, and generate stable semantic prediction results.
It significantly reduces the pressure on the network to the equipment, improves the effective compression and extraction of light field image information, avoids information redundancy and calculation pressure, and greatly improves the efficiency and accuracy of semantic segmentation.
Smart Images

Figure QLYQS_1 
Figure QLYQS_2 
Figure SMS_161
Abstract
Description
Technical Field
[0001] The present application belongs to the field of light field image processing technology, and specifically relates to a light field camera semantic segmentation method, system and electronic equipment. Background Art
[0002] Unlike traditional cameras, light field cameras utilize their unique full-aperture optical systems to capture more comprehensive light information about a scene, including its angle and intensity. This allows light field cameras to capture not only traditional two-dimensional images but also four-dimensional light field data, providing richer spatial and angular information than ordinary cameras.
[0003] In order to realize the application of light field cameras in the field of intelligent agents (such as robotics and autonomous driving), it is also necessary to perform semantic segmentation on the light field images captured by the light field camera. The core of semantic segmentation is to assign a category label to each pixel in the light field image, thereby helping the system accurately identify and understand various objects in the surrounding environment.
[0004] However, light field images have the following problems in image semantic segmentation:
[0005] (1) High memory costs are required to process redundant sub-aperture information, as well as additional work associated with complex data preprocessing. For example, assigning a feature extractor to each light field image may place an excessive burden on the device. In particular, using a light field camera with a 640×480 spatial resolution and a 9×9 angular resolution will produce a 4320×5760 macro-pixel image, which will increase the data flow of the network module by 81 times, and compressing the light field image into a depth map will further increase the complexity of the model;
[0006] (2) It is impossible to effectively utilize the angular information contained in the light field image. In related technologies, the angular information contained in the light field image is utilized by stacking the images into an array of video sequences containing only one-dimensional angular information. However, this method ignores the implicit angular relationship of the light field image, and directly merging images or features will result in blurred or mixed boundaries. This phenomenon becomes particularly serious as the number of sub-aperture images increases.
[0007] Therefore, it is necessary to provide a light field camera semantic segmentation method, system and electronic device to solve the problems raised in the above background technology. Summary of the Invention
[0008] The present application provides a light field camera semantic segmentation method, system and electronic device, which directly extract shallow features from sub-aperture images, save manual operation costs and improve system operation speed. The fusion of shallow features and the characteristic spatial angle and spatial feature separation guided by the light field can achieve effective compression and extraction of light field image information at a low computational cost, avoiding the problems of information redundancy and high equipment computing pressure caused by parallel processing of multiple groups of images at the same time; the spatial features and angular features obtained by the shallow feature embedding fusion will be embedded into high-dimensional information representing multi-scale global features extracted by two lightweight dual-headed self-attention networks; the heterogeneous information in the high-dimensional angular features and spatial features is interactively supplemented and corrected in a dynamic weighted manner to prevent the mismatch of heterogeneous information caused by direct fusion; the cross-attention mechanism is used to efficiently fuse the corrected heterogeneous information to avoid information aliasing caused by direct feature superposition; the fused features are sent to the decoder and post-processed to generate a stable, designated personal privacy-protected image, which can effectively solve at least one technical problem involved in the background technology.
[0009] In order to solve the above technical problems, the technical solution of this application is:
[0010] This application provides a light field camera semantic segmentation method, comprising the following steps:
[0011] Step S1, performing feature precoding on an unfocused original image captured by a light field camera to generate shallow spatial information embedding of a central image and a sub-aperture image corresponding to the central image;
[0012] Step S2: A mapping network is used to calculate the feature weights of a single sub-aperture image. The shallow spatial information embedding of the sub-aperture image is then multiplied pixel by pixel by the feature weight using the Pytorch broadcast mechanism to obtain a pixel-by-pixel weighted shallow angular information embedding. The shallow angular information embedding of the sub-aperture image is then superimposed element-by-element with the shallow spatial information embedding of the central image to generate an angular information embedding rich in angular information. The shallow spatial information embedding of the central image is kept unchanged as the spatial information embedding rich in texture information.
[0013] Step S3: introducing a dual-path multi-scale feature extraction network to extract multi-scale high-dimensional information embeddings representing global features from the spatial information embedding and the angular information embedding, respectively; establishing global feature associations within the spatial information embedding and the angular information embedding, and obtaining high-dimensional spatial information embedding and high-dimensional angular information embedding;
[0014] In step S4, the heterogeneous information in the high-dimensional angular information embedding and the high-dimensional spatial information embedding is interactively supplemented and corrected by dynamic weighting and rearrangement, the corrected heterogeneous information is fused, and sent to the feature encoder to generate a semantic prediction result.
[0015] Preferably, the feature precoding process in any dimension is expressed as:
[0016] ;
[0017] ;
[0018] Where, represents the depthwise separable convolution used on sub-aperture images; represents the depth-wise separable convolution used on the center image; Represents the shallow spatial information embedding of sub-aperture images; Represents the shallow spatial information embedding of the central image; Represents dimension.
[0019] Preferably, during the feature precoding process, depthwise separable convolution with different parameters is used between the central image and the sub-aperture images; and depthwise separable convolution with the same parameters is used between different sub-aperture images.
[0020] Preferably, step S2 specifically includes the following steps:
[0021] Step S21, calculate the pixel-by-pixel Euclidean distance between the shallow spatial information embedding of each group of sub-aperture images and the shallow spatial information embedding of the central image. The calculation process is expressed as:
[0022] ;
[0023] Where, Represents the spatial coordinates of the pixel point Next, the pixel Euclidean distance between the shallow spatial information embedding of the sub-aperture image and the shallow spatial information embedding of the central image; Indicates pixel Euclidean distance calculation;
[0024] Step S22: Use the mapping method to map the Euclidean distance corresponding to each pixel point to In the range of , each pixel is used as the weight corresponding to the central image, and the weight is assigned to the shallow spatial information embedding of the sub-aperture image. The shallow spatial information embedding of the weighted sub-aperture image is recombined with the shallow spatial information embedding of the central image to obtain the angle information embedding. The recombining process is expressed as:
[0025] ;
[0026] in, Indicates the angular information embedding that aggregates all sub-aperture angular information in this dimension.
[0027] Preferably, the dual-path multi-scale feature extraction network uses two self-attention networks that do not share weights. , Establish global connections within the angular information embedding and spatial information embedding, which are rich in different semantic information: and spatial information embedding First, it is transformed into two dimensions through deformation operation. The token is then transformed into the self-attention network required by the linear mapping ; Then, the key values of the self-attention network are mapped with the help of a multi-layer perceptron, and the original dimension is of Use interpolation to reduce the size to .
[0028] Preferably, the extraction process of high-dimensional angle information embedding is expressed as:
[0029] ;
[0030] ;
[0031] ;
[0032] ;
[0033] Where, Represents the feature deformation operation in Python code. The tensor transformation is ,in Represents the decoupling operation, splitting the features along the channel dimension, and the splitting result is Three tensors of the same dimension; Indicates the use of linear mapping to transform features from dimension C Mapping to dimensions 3C ; Indicates that the original After interpolation operation The compressed key value afterwards; represents the attention mechanism; Represents high-dimensional angular information embedding;
[0034] The extraction process of high-dimensional spatial information embedding is expressed as:
[0035] ;
[0036] ;
[0037] ;
[0038] ;
[0039] Where, Represents high-dimensional spatial information embedding.
[0040] Preferably, step S4 specifically includes the following steps:
[0041] For high-dimensional angle information embedding and high-dimensional spatial information embedding , splice it horizontally and vertically respectively, and then encode the spliced features into a series of feature tokens through feature deformation operation, and combine the feature tokens with the three mapping matrices Multiply them together to get the characteristic token ; The feature token Send it to the multi-head attention network to calculate the similarity score, and convert the The token is multiplied with the similarity score of each attention sub-head to obtain the sub-feature token of each attention sub-head, and the sub-feature tokens of multiple attention sub-heads are concatenated to obtain the complete final feature token; then linear mapping and multi-layer perceptron and residual link are used to fill in the secondary information lost in the high-dimensional operation of the final feature token, and the dimension is mapped back to the dimension of the input to complete the correction process; finally, through decoupling operation, the recovered spatial information embedding and angular information embedding are obtained respectively.
[0042] Preferably, the process of fusing the corrected heterogeneous information includes the following steps: embedding the recovered spatial information and the angular information in dimensions for splicing, establishing local correlation of dimensional features with the help of point convolution and depth separation convolution with the same number of dimensions, enriching its nonlinear expression ability with the help of ReLU function, and finally performing dimensionality reduction calculation on the aggregated features with the help of point convolution, and obtaining the prediction results at this scale with the help of jump connection.
[0043] The present application also provides a light field camera semantic segmentation system for implementing the above-mentioned light field camera semantic segmentation method, comprising:
[0044] A feature precoding network is used to perform feature precoding on an unfocused original image captured by a light field camera to generate shallow spatial information embedding of a central image and a sub-aperture image corresponding to the central image;
[0045] The shallow information aggregation module uses a mapping network to calculate the feature weights of a single sub-aperture image. The shallow spatial information embedding of the sub-aperture image is then multiplied pixel by pixel by the feature weight using the Pytorch broadcast mechanism to obtain a pixel-by-pixel weighted shallow angular information embedding. The shallow angular information embedding of the sub-aperture image is then superimposed on the shallow spatial information embedding of the central image in a high-dimensional element-by-element manner to generate an angular information embedding rich in angular information. The shallow spatial information embedding of the central image is kept unchanged as the spatial information embedding rich in texture information.
[0046] A dual-path multi-scale feature extraction network extracts multi-scale high-dimensional information embeddings representing global features from spatial information embeddings and angular information embeddings, respectively, and establishes global feature associations within the spatial information embeddings and angular information embeddings to obtain high-dimensional spatial information embeddings and high-dimensional angular information embeddings;
[0047] A multi-scale feature correction and fusion module uses dynamic weighting and rearrangement to interactively supplement and correct the heterogeneous information in the high-dimensional angular information embedding and high-dimensional spatial information embedding, and fuses the corrected heterogeneous information;
[0048] The feature encoder is used to receive the fused heterogeneous information and generate semantic prediction results.
[0049] The present application also provides an electronic device, including a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are read by the processor and execute the above-mentioned light field camera semantic segmentation method when executed.
[0050] The beneficial effects of this application are:
[0051] (1) It overcomes the traditional network's dependence on image quality, avoids the adverse effects of the light field camera's unfocused original image on the perception results, and uses lightweight feature precoding to significantly reduce the huge pressure on the network to use the device;
[0052] (2) Selectively screen shallow sub-aperture spatial information embeddings and integrate multi-scale shallow spatial information embeddings to generate spatial information embeddings and angular information embeddings rich in texture information;
[0053] (3) It is possible to extract multi-scale high-dimensional information embeddings representing global features from shallow multi-scale spatial information embeddings;
[0054] (4) Dynamic weighting is used to interactively supplement and correct heterogeneous information in high-dimensional angular information embedding and spatial information embedding to prevent heterogeneous information mismatch caused by direct fusion;
[0055] (5) Using the cross-attention mechanism to efficiently fuse the corrected heterogeneous information to avoid information aliasing caused by direct feature superposition;
[0056] (6) Compared with traditional light field semantic segmentation networks, it has faster inference speed while maintaining high accuracy, and has the advantages of low computational cost and high operating efficiency. DETAILED DESCRIPTION
[0057] The following is a clear and complete description of the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application. Example 1
[0058] This embodiment provides a semantic segmentation method for a light field camera, comprising the following steps:
[0059] Step S1 : performing feature precoding on an unfocused original image captured by a light field camera to generate shallow spatial information embedding of a central image and a sub-aperture image corresponding to the central image.
[0060] The unfocused original image captured by the light field camera includes the center image and subaperture images , where Represents a three-dimensional real matrix; H represents the image height; W represents the image width; C represents the number of image channels of the image, and the initial number of RGB image channels is 3; i Indicates the number of the sub-aperture image.
[0061] Center image and the central image The corresponding set of sub-aperture images Use depth-wise separable convolution for feature precoding to generate a central image and subaperture images The multi-scale shallow spatial information embedding of . The feature precoding process in any dimension is expressed as:
[0062] ;
[0063] ;
[0064] Where, represents the depthwise separable convolution used on sub-aperture images; represents the depth-wise separable convolution used on the center image; Represents the shallow spatial information embedding of sub-aperture images; Represents the shallow spatial information embedding of the central image; Represents dimension.
[0065] In the feature precoding process, the shallow spatial information output at the previous scale is embedded as the input of the next scale. In this embodiment, feature precoding includes four scales, namely: The feature precoding process in 64 dimensions takes the original RGB image as input; the feature precoding process in 128 dimensions takes the output in 64 dimensions as input; the feature precoding process in 312 dimensions takes the output in 128 dimensions as input; and the feature precoding process in 524 dimensions takes the output in 312 dimensions as input.
[0066] After feature precoding at four scales, shallow spatial information embedding at four scales is obtained. The shallow spatial information embedding at each scale includes the shallow spatial information embedding of the sub-aperture image and the shallow spatial information embedding of the central image. The shallow spatial information embedding of the sub-aperture image at four scales is expressed as: and ; The shallow spatial information embedding of the central image at four scales is expressed as: .
[0067] During the feature precoding process, in order to ensure the uniqueness of the spatial information of the central image, depthwise separable convolution with different parameters is used between the central image and the sub-aperture images; depthwise separable convolution with the same parameters is used between different sub-aperture images.
[0068] The feature precoding process is implemented through a deep separable convolutional network. It does not rely on the preprocessing of light field sub-aperture images, but directly processes the unfocused original images captured by the light field camera. This can overcome the traditional network's dependence on image quality and avoid the equipment pressure brought by direct encoding.
[0069] In step S2, a mapping network is used to calculate the feature weights of a single sub-aperture image. The shallow spatial information embedding of the sub-aperture image is then multiplied pixel by pixel by the feature weight using the Pytorch broadcast mechanism to obtain a pixel-by-pixel weighted shallow angular information embedding. The shallow angular information embedding of the sub-aperture image is then superimposed on the shallow spatial information embedding of the central image in high-dimensional information element-by-element fashion to generate an angular information embedding rich in angular information. The shallow spatial information embedding of the central image is kept unchanged as the spatial information embedding rich in texture information.
[0070] Step S2 specifically includes the following steps:
[0071] Step S21, calculate the pixel-by-pixel Euclidean distance between the shallow spatial information embedding of each group of sub-aperture images and the shallow spatial information embedding of the central image. The calculation process is expressed as:
[0072] ;
[0073] Where, Represents the spatial coordinates of the pixel point Next, the pixel Euclidean distance between the shallow spatial information embedding of the sub-aperture image and the shallow spatial information embedding of the central image; Indicates pixel Euclidean distance calculation;
[0074] Step S22: Use the mapping method to map the Euclidean distance corresponding to each pixel point to In the range of , each pixel is used as the weight corresponding to the central image, and the weight is assigned to the shallow spatial information embedding of the sub-aperture image. The shallow spatial information embedding of the weighted sub-aperture image is recombined with the shallow spatial information embedding of the central image to obtain the angle information embedding. The recombining process is expressed as:
[0075] ;
[0076] Where, Indicates the angular information embedding that aggregates all sub-aperture angular information in this dimension.
[0077] In step S3, a dual-path multi-scale feature extraction network is introduced to extract multi-scale high-dimensional information embeddings representing global features from the spatial information embedding and the angular information embedding respectively, and a global feature association is established within the spatial information embedding and the angular information embedding to obtain high-dimensional spatial information embedding and high-dimensional angular information embedding.
[0078] The dual-path multi-scale feature extraction network uses two self-attention networks that do not share weights , Global connections are established within the angular information embedding and spatial information embedding, which are rich in different semantic information. and spatial information embedding First, it is transformed into two dimensions through deformation operation. The token is then transformed into the self-attention network required by the linear mapping In addition, in order to further reduce the computational complexity of the network, the key values of the self-attention network are mapped with the help of a multi-layer perceptron, and the original dimension is of Use interpolation to reduce the size to Through two different sets of self-attention networks, we finally obtain high-dimensional information embedding that represents global features.
[0079] The extraction process of high-dimensional angle information embedding is expressed as:
[0080] ;
[0081] ;
[0082] ;
[0083] ;
[0084] Where, Represents the feature deformation operation in Python code. The tensor transformation is ,in Represents the decoupling operation, splitting the features along the channel dimension, and the splitting result is Three tensors of the same dimension; Indicates the use of linear mapping to transform features from dimension C Mapping to dimensions 3C ; Indicates that the original After interpolation operation The compressed key value afterwards; represents the attention mechanism; Represents high-dimensional angular information embedding;
[0085] The extraction process of high-dimensional spatial information embedding is expressed as:
[0086] ;
[0087] ;
[0088] ;
[0089] ;
[0090] Where, Represents high-dimensional spatial information embedding.
[0091] With the help of the calculation process of the attention mechanism, global connections are established within the angular information embedding and spatial information embedding rich in different semantic information, and finally high-dimensional angular information embedding and high-dimensional spatial information embedding representing global features are obtained.
[0092] In step S4, the heterogeneous information in the high-dimensional angular information embedding and the high-dimensional spatial information embedding is interactively supplemented and corrected by dynamic weighting and rearrangement, the corrected heterogeneous information is fused, and sent to the feature encoder to generate a semantic prediction result.
[0093] Light field cameras use a single-shot, multiple-image strategy, so it's difficult to align images of different sub-apertures using the same information. Specifically, light field cameras exhibit a certain degree of pixel offset in both the horizontal and vertical directions. The farther away from the center point the sensor is, the greater the offset in the image information captured, necessitating corrective alignment of the image information.
[0094] For high-dimensional angle information embedding and high-dimensional spatial information embedding , splice it horizontally and vertically respectively, and then encode the spliced features into a series of feature tokens through feature deformation operation, and combine the feature tokens with the three mapping matrices Multiply them together to get the characteristic token ; The feature token Send it to the multi-head attention network to calculate the similarity score, and convert the The token is multiplied with the similarity score of each attention sub-head to obtain the sub-feature token of each attention sub-head, and the sub-feature tokens of multiple attention sub-heads are concatenated to obtain the complete final feature token; then linear mapping and multi-layer perceptron and residual link are used to fill in the secondary information lost in the high-dimensional operation of the final feature token, and the dimension is mapped back to the dimension of the input to complete the correction process; finally, through decoupling operation, the recovered spatial information embedding and angular information embedding are obtained respectively.
[0095] Take the horizontal correction in 64 dimensions as an example:
[0096] First, for the features as well as , splice the two horizontally to obtain the spliced features ,in Refers to the height and width of the information embedded in the current module;
[0097] Then, Encoded as a sequence of feature tokens ,in Indicates the number of feature tokens;
[0098] Feature Token By mapping the matrix , , Multiply them to get the corresponding tokens ; then They are all split along the dimension layer and sent to eight attention heads to calculate the similarity score. The formula is as follows:
[0099] ;
[0100] Where, represents the similarity score; Representative The number of features per attention sub-head; Indicates Function to calculate the probability score in the attention mechanism;
[0101] Based on the calculated similarity score, the sub-feature tokens of each attention sub-head are Correction is performed, and the correction process is expressed as:
[0102]
[0103] Where, Indicates the Attention-grabbing Token;
[0104] Then the final feature token with the complete attention mechanism is Expressed as:
[0105] ;
[0106] Where, Indicates that features are superimposed at the dimension level.
[0107] On this basis, linear mapping and multi-layer perceptron are used with residual links to fill in the secondary information lost during the high-dimensional operation and map the final dimension back to the input dimension of 64. The horizontal correction process is completed, and the formula is expressed as follows:
[0108] ;
[0109] Where, represents a linear mapping operation, represents a multilayer perceptron, Represents the stitching features after correction.
[0110] Finally, through the decoupling operation, the recovered spatial information embedding is obtained and angle information embedding .
[0111] The process of fusing the corrected heterogeneous information includes the following steps: embedding the restored spatial information and the angular information in the dimension for splicing, using the same number of point convolutions and depth separation convolutions as the dimension to establish the local correlation of the dimensional features, and using The function enriches its nonlinear expression ability, and finally uses point convolution to reduce the dimension of the aggregated features, and cooperates with jump connections to obtain the prediction results at this scale.
[0112] Taking 64 dimensions as an example, the fusion process includes the following steps:
[0113] S51: Embed the recovered spatial information and angle information embedding Splicing is performed on 64 dimensions to obtain a total of 128 dimensions of features , the splicing process is expressed as:
[0114] ;
[0115] Where, Indicates that features are superimposed at the dimension level;
[0116] S52: Use point convolution and depth separation convolution with the same number of dimensions to establish local correlation of dimensional features and obtain features ;
[0117] ;
[0118] Where, Represents point-wise convolution.
[0119] S53: With the help of The function enriches its nonlinear expression ability, and finally uses point convolution to reduce the dimension of the aggregated features, and cooperates with jump connections to obtain the prediction results at this scale. ;
[0120] ;
[0121] Where, Indicates the use of Function activation.
[0122] Specifically, in this embodiment, the light field image is uniformly The image is enhanced by random inversion and random scaling, with a probability of 0.5 and a scaling factor of .
[0123] Each channel of the input light field image is processed in a regularized manner, and the regularization coefficient is The encoding and decoding networks were trained jointly for 300 training epochs with a batch size of 3. The initial learning rate was set to 0.0010 and then dropped to 0.0006 after 30 training epochs.
[0124] A comparative experiment was conducted to compare the method proposed in this application with six existing methods. Performance was evaluated using the mIoU, Acc, and mAcc metrics. The results are shown in Table 1. mIoU represents the ratio of the intersection-over-union of the predicted and true regions for each category, averaged across all categories; Acc represents the proportion of correctly classified pixels to the total number of pixels; and mAcc represents the accuracy calculated for each category and averaged across all categories. These metrics are widely used in various image semantic segmentation tasks.
[0125] Table 1 Comparison of the accuracy of this application and various methods
[0126]
[0127] As can be seen from Table 1, the method provided by this application has achieved the best performance in all indicators and has brought about a significant improvement in accuracy. The reason is that this application can strengthen the training of the model to provide powerful visual features and masks.
[0128] Example 2
[0129] This embodiment provides a light field camera semantic segmentation system, including:
[0130] A feature precoding network is used to perform feature precoding on an unfocused original image captured by a light field camera to generate shallow spatial information embedding of a central image and a sub-aperture image corresponding to the central image;
[0131] The shallow information aggregation module uses a mapping network to calculate the feature weights of a single sub-aperture image. The shallow spatial information embedding of the sub-aperture image is then multiplied pixel by pixel by the feature weight using the Pytorch broadcast mechanism to obtain a pixel-by-pixel weighted shallow angular information embedding. The shallow angular information embedding of the sub-aperture image is then superimposed on the shallow spatial information embedding of the central image in a high-dimensional element-by-element manner to generate an angular information embedding rich in angular information. The shallow spatial information embedding of the central image is kept unchanged as the spatial information embedding rich in texture information.
[0132] A dual-path multi-scale feature extraction network extracts multi-scale high-dimensional information embeddings representing global features from spatial information embeddings and angular information embeddings, respectively, and establishes global feature associations within the spatial information embeddings and angular information embeddings to obtain high-dimensional spatial information embeddings and high-dimensional angular information embeddings;
[0133] A multi-scale feature correction and fusion module uses dynamic weighting and rearrangement to interactively supplement and correct the heterogeneous information in the high-dimensional angular information embedding and high-dimensional spatial information embedding, and fuses the corrected heterogeneous information;
[0134] The feature encoder is used to receive the fused heterogeneous information and generate semantic prediction results.
[0135] Example 3
[0136] This embodiment provides an electronic device, including a processor and a memory. The memory stores computer program instructions, and the computer program instructions are read by the processor and execute the above-mentioned light field camera semantic segmentation method when executed.
[0137] More specific examples (a non-exhaustive list) of the memory include the following: an electrical connection with one or more wires (electronic device), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the memory may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in the computer memory.
[0138] The above describes the embodiments of the present application, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.
Claims
1. A light field image semantic segmentation method, characterized in that: The steps include: Step S1, performing feature precoding on an unfocused original image captured by a light field camera to generate shallow spatial information embedding of a central image and a sub-aperture image corresponding to the central image; Step S2, using a mapping network to calculate the feature weight of a single sub-aperture image, and then using the Pytorch broadcast mechanism to multiply the shallow spatial information embedding of the sub-aperture image by the feature weight pixel by pixel to obtain a shallow angle information embedding weighted by pixel, and then superimposing the shallow angle information embedding of the sub-aperture image and the shallow spatial information embedding of the central image on an element-by-element basis to generate an angle information embedding rich in angle information; keeping the shallow spatial information embedding of the central image unchanged as the spatial information embedding rich in texture information; Step S3, introducing a dual-path multi-scale feature extraction network, extracting multi-scale high-dimensional information embeddings representing global features from the spatial information embedding and the angular information embedding respectively, and establishing feature global associations within the spatial information embedding and the angular information embedding, to obtain high-dimensional spatial information embedding and high-dimensional angular information embedding; Step S4, interactively supplementing and correcting the heterogeneous information in the high-dimensional angular information embedding and the high-dimensional spatial information embedding by means of dynamic weighting and rearrangement, fusing the corrected heterogeneous information, and sending it to the feature encoder to generate a semantic prediction result.
2. The method for semantic segmentation of light field images according to claim 1, characterized in that: The process of feature precoding in any dimension is expressed as: ; ; In the formula, Represents the sub-aperture image Depthwise separable convolution used; Represents the center image Depthwise separable convolution used; Represents the shallow spatial information embedding of sub-aperture images; Represents the shallow spatial information embedding of the central image; Represents the dimension, i Indicates the number of the sub-aperture image.
3. The method for semantic segmentation of light field images according to claim 2, characterized in that: During the feature precoding process, depthwise separable convolutions with different parameters are used between the central image and the sub-aperture images; and depthwise separable convolutions with the same parameters are used between different sub-aperture images.
4. The method for semantic segmentation of light field images according to claim 2, characterized in that: Step S2 specifically includes the following steps: Step S21, calculate the pixel-by-pixel Euclidean distance between the shallow spatial information embedding of each group of sub-aperture images and the shallow spatial information embedding of the central image. The calculation process is expressed as: ; In the formula, Represents the spatial coordinates of the pixel point Next, the pixel Euclidean distance between the shallow spatial information embedding of the sub-aperture image and the shallow spatial information embedding of the central image; Indicates pixel Euclidean distance calculation; Step S22, using a mapping method, maps the Euclidean distance corresponding to each pixel point to In the range, the weight of each pixel corresponding to the central image is used, and the weight is assigned to the shallow spatial information embedding of the sub-aperture image; the shallow spatial information embedding of the weighted sub-aperture image is recombined with the shallow spatial information embedding of the central image to obtain the angle information embedding. The recombining process is expressed as: ; in, Indicates the angular information embedding that aggregates all sub-aperture angular information in this dimension; Indicates that the Euclidean distance corresponding to each pixel is mapped to A value in the range.
5. The method for semantic segmentation of light field images according to claim 4, characterized in that: The dual-path multi-scale feature extraction network uses two self-attention networks that do not share weights. , Establish global connections between the angular information embedding and the spatial information embedding, which are rich in different semantic information: and spatial information embedding First, it is transformed into two dimensions through deformation operation. The token is then transformed into the self-attention network using a linear mapping. ; Then, the key values of the self-attention network are mapped with the help of a multi-layer perceptron, and the original dimension is of Use interpolation to reduce the size to .
6. The method for semantic segmentation of light field images according to claim 5, characterized in that: The extraction process of high-dimensional angle information embedding is expressed as: ; ; ; ; In the formula, Represents the feature deformation operation in Python code. The tensor transformation is ,in It represents the decoupling operation, splitting the features along the channel dimension, and the splitting result is Three tensors of the same dimension; Indicates the use of linear mapping to transform features from dimension C Mapping to dimensions 3C ; Indicates that the original After interpolation The compressed key value afterwards; represents the attention mechanism; Represents high-dimensional angular information embedding; The extraction process of high-dimensional spatial information embedding is expressed as: ; ; ; ; In the formula, Represents high-dimensional spatial information embedding.
7. The method for semantic segmentation of light field images according to claim 6, characterized in that: Step S4 specifically includes the following steps: For high-dimensional angle information embedding And high-dimensional spatial information embedding , splicing them horizontally and vertically respectively, and then encoding the spliced features into a series of feature tokens through feature deformation operation, and the feature tokens are combined with the three mapping matrices Multiply them together to get the characteristic token ; The feature token Send it to the multi-head attention network to calculate the similarity score, and convert the The token is multiplied with the similarity score of each attention sub-head to obtain the sub-feature token of each attention sub-head, and the sub-feature tokens of multiple attention sub-heads are concatenated to obtain the complete final feature token; Then, linear mapping, multi-layer perceptron and residual link are used to complete the secondary information lost in the high-dimensional operation of the final feature token, and the dimension is mapped back to the dimension of the input to complete the correction process; finally, through the decoupling operation, the restored spatial information embedding and angular information embedding are obtained respectively.
8. The method for semantic segmentation of light field images according to claim 7, characterized in that: The process of fusing the corrected heterogeneous information includes the following steps: embedding the restored spatial information and the angular information in the dimension for splicing, establishing the local correlation of the dimensional level features with the help of point convolution and depth separation convolution with the same number of dimensions, enriching its nonlinear expression ability with the help of ReLU function, and finally using point convolution to reduce the dimension of the aggregated features, and combining with jump connections to obtain the prediction results at this scale.
9. A light field image semantic segmentation system for implementing the light field image semantic segmentation method according to any one of claims 1 to 8, characterized in that: include: A feature precoding network is used to perform feature precoding on an unfocused original image acquired by a light field camera, and to generate shallow spatial information embedding of a central image and a sub-aperture image corresponding to the central image; The shallow information aggregation module uses a mapping network to calculate the feature weight of a single sub-aperture image, and then uses the Pytorch broadcast mechanism to multiply the shallow spatial information embedding of the sub-aperture image by the feature weight pixel by pixel to obtain a shallow angle information embedding that is weighted pixel by pixel. The shallow angle information embedding of the sub-aperture image is then superimposed on the shallow spatial information embedding of the central image in high-dimensional information element by element to generate an angle information embedding rich in angle information. The shallow spatial information embedding of the central image is kept unchanged as a spatial information embedding rich in texture information. A dual-path multi-scale feature extraction network extracts multi-scale high-dimensional information embeddings representing global features from spatial information embeddings and angular information embeddings respectively, and establishes feature global associations within the spatial information embeddings and angular information embeddings to obtain high-dimensional spatial information embeddings and high-dimensional angular information embeddings; A multi-scale feature correction and fusion module uses dynamic weighting and rearrangement to interactively supplement and correct the heterogeneous information in the high-dimensional angle information embedding and the high-dimensional spatial information embedding, and fuses the corrected heterogeneous information; The feature encoder is used to receive the fused heterogeneous information and generate semantic prediction results.
10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are read by the processor and execute the light field image semantic segmentation method according to any one of claims 1 to 8 when executed.
