High-precision integrated imaging 3D salient target detection method with texture features

Abstract

The invention provides a high-precision integrated imaging 3D salient target detection method with texture features, and the method comprises the steps: obtaining four-dimensional light field data, carrying out the high-precision salient target detection of a micro-image array, carrying out the texture feature compensation of a salient micro-image array, and carrying out the three-dimensional reconstruction of a high-precision salient micro-image array with texture features. According to the method, collected four-dimensional light field data is used for generating a micro-image array, the micro-image array is used as input of high-precision salient target detection, cross-regional global salient features are fully mined through recombination folding operation and cyclic coding and decoding, a high-precision salient micro-image array is generated, and then a layered texture feature compensation method is used for realizing high-precision salient target detection. According to the method, the texture features of the region salient micro-image array and the edge salient micro-image array are respectively restored and generated, and finally three-dimensional reconstruction is performed on the high-precision salient micro-image array with the texture features by means of the lens array, so that a subsequent computer vision task is effectively simplified, and fine and colorful three-dimensional display is provided for a salient target.

Description

1. Technical Field

[0001] The present invention relates to the field of salient object detection in computer vision tasks, and more specifically, to a high-precision integrated imaging 3D salient object detection method with texture features. 2. Background Technology

[0002] Integrated imaging is a naked-eye three-dimensional stereo imaging technology with full parallax and full-color display. It uses lens arrays to achieve efficient storage and reconstruction of 3D image data, thereby increasing the information carrying capacity and display dimension. With the development of science and technology, the rich depth information and real texture details contained in integrated imaging 3D images provide strong support for medical image visualization analysis and autonomous driving environment perception. However, how to quickly extract key and effective 3D information from these massive data is a key issue in integrated imaging image analysis and reconstruction. The 3D salient object detection method can identify the most eye-catching objects from 3D images, detect and segment visually unique image areas, and greatly simplify subsequent computer vision tasks.

[0003] In the related research of salient object detection, traditional salient object detection cannot reconstruct three-dimensional objects due to the lack of depth information, which in turn hinders the understanding of spatial context. Combining salient object detection with integrated imaging has initially explored a path to solve this limitation, aiming to reconstruct 3D salient objects. However, this method still faces major challenges such as limited resolution of individual micro-images and insufficient integration of global information, which leads to fatal problems such as loss of salient object details, false detection, and missed detection. In addition, the output of existing 3D salient object detection methods is usually limited to black and white binary images, which fails to fully display the rich and colorful texture features on the surface of 3D salient objects, thereby limiting the ability to provide users with an immersive 3D experience. III. Summary of the invention

[0004] The present invention proposes a high-precision integrated imaging 3D salient target detection method with texture features, the core of which is to effectively integrate four-dimensional light field data to enhance the recognition accuracy of 3D salient targets and restore their real texture features. The method first samples the four-dimensional light field and uses the obtained micro-image array as a network input. Subsequently, it is segmented to obtain a micro-image sequence, the micro-image sequence is reorganized and folded, and n-times cyclic encoding is performed to achieve different ways of reorganizing the four-dimensional light field information of micro-images in different regions, fully extract the salient features of the micro-image sequence, and then the results are input to the decoding module in parallel to achieve multi-scale feature fusion and image resolution restoration, and obtain a high-precision salient micro-image array. After that, the salient micro-image array is subjected to classification texture feature compensation processing, the regional salient micro-image array is subjected to image thresholding processing, and the boundary salient micro-image array is processed by a deep convolutional generative adversarial network to generate the real color texture features of the boundary of the salient object, thereby outputting a high-precision salient micro-image array with texture features, and finally reconstructing the 3D image of the salient object with the help of an integrated imaging display device.

[0005] The method includes four processes: acquisition of four-dimensional light field data, high-precision salient target detection of micro-image array, texture feature compensation of salient micro-image array, and three-dimensional reconstruction of salient micro-image array with texture features. The specific process is shown in the attached Figure 1 shown.

[0006] In the process of acquiring the four-dimensional light field data, a real camera array can be established to collect the real four-dimensional light field information, or a virtual camera array can be established in computer graphics software to obtain virtual four-dimensional light field information, and then a micro-image array is generated through a pixel mapping algorithm. The micro-image array contains multi-angle information of the four-dimensional light field and contains rich geometric information of the scene. Comprehensive analysis of the micro-image array can fully and effectively mine multi-angle three-dimensional saliency relationships in different spatial ranges.

[0007] The high-precision salient target detection process of the micro-image array is shown in the attached Figure 2 As shown. The M×N micro-image array is divided according to the size of the micro-image to form a micro-image sequence. After reconstruction, the micro-image sequence is obtained as a three-dimensional tensor, which is then divided into k×k adjacent micro-images to form a small rectangular area (patch). In the folding operation, a convolution kernel of the same size as the patch is used, with a step size of s and a padding of p. The result is convolved with the corresponding elements in the patch, and after passing through the flattening layer, a micro-image sequence with increased length is obtained. It is embedded into the encoder together with the sinusoidal position code as input, and n attention heads are set. After the multi-head attention layer, multiple groups of query matrices Q, key matrices K and value matrices V of micro-image sequences at each level are obtained. The normalized attention weight A of the i-th head is obtained through dot product calculation, scaling factor and activation function. i ,Right now:

[0008]

[0009] According to A i For the corresponding attention head V i The context vector of the corresponding position in the micro-image sequence is obtained by weighted summation, which is fully connected and then enters the multi-layer perceptron module, as shown in the attached figure. Figure 3 As shown in the figure, the context vector is further extracted from the high-level micro-image features in the hidden layer after nonlinear transformation, and the output of the encoder is obtained after residual connection and layer normalization based on feature dimension. In the decoding module, the decoder and the reverse recombination folding operation are used for collaborative upsampling. In the masked multi-head attention layer, the dimensions of the input and output features of the micro-image sequence are set, and independent attention weights are calculated for all positions of the micro-image in the standard set. In the encoder-decoder attention sublayer, the image feature dimension of this layer is set to be the same as the input and output feature dimensions, so as to facilitate the calculation of the correlation between cross-region micro-images by scaling the dot product and the residual connection calculation of two consecutive micro-image sequence layers. In the multi-layer perceptron and fully connected layers, the input features of the previous layer are upsampled. In order to retain the detail information, this layer fuses the low-level features of the micro-image sequences in the encoders of different scales. With the help of reverse recombination folding and the cyclic superposition of the decoder, the image size is restored to the original size of the micro-image array. m cyclic decoding modules and the same number of scale factors r are set to gradually restore the feature micro-image array to the original resolution, and obtain a high-precision salient micro-image array.

[0010] The texture feature compensation process of the significant micro-image array is as shown in the attached Figure 4 As shown in the figure, the black and white significant micro-image array is divided into a regional significant micro-image array and an edge significant micro-image array by a Gaussian filter, where each micro-image array contains M×N pixels. The regional significant micro-image array is subjected to image thresholding, where the white area restores the texture features of the corresponding area of ​​the original image f(x, y), and the black area is retained. The obtained image g(x, y) is expressed as:

[0011]

[0012] The texture features of the boundary salient micro-image array are generated using a deep convolutional generative adversarial network. The generator consists of transposed convolutional layers and is defined to generate color images of three channels: red, green, and blue. The random noise z and the grayscale boundary map with a resolution of M×N are used as the input of the generator. The random noise z is converted into a higher-dimensional spatial representation and concatenated with the grayscale boundary map to obtain the original feature map of size a×b×c. It is upsampled through the transposed convolutional layer. Batch normalization and rectified linear unit activation functions are used in each transposed convolutional layer. The last layer of the generator superimposes the hyperbolic tangent function and outputs a color image of size M×N×3. The color image output by the generator enters the discriminator, and is downsampled through a series of convolutional layers and rectified linear unit activation functions to capture features of different scales. After normalization, the S-type function is used in the last layer to map the output to the interval (0,1) to obtain the total loss value and update the parameters of the generator, and finally generate a boundary salient micro-image array with texture features. The image is fused with the regional significant micro-image array with texture features by using image addition to obtain a significant micro-image array with texture features.

[0013] In the three-dimensional reconstruction process of the significant micro-image array with texture features, the micro-image array obtained in the third step is reconstructed with the four-dimensional light field angle and spatial information by means of a lens array to obtain a three-dimensional significant object with texture features.

[0014] The present invention proposes a high-precision integrated imaging 3D salient target detection method with texture features. According to the characteristics of micro-image arrays containing rich light field spatial information and angle information, a method for reorganizing and folding micro-image sequences is designed, and combined with the Transformer encoder and decoder, the salient clues of cross-regional micro-images in the micro-image array are fully mined to achieve high-precision integrated imaging 3D salient target detection. At the same time, a hierarchical texture feature compensation method for salient micro-image arrays is proposed, and the texture features of regional and boundary salient micro-image arrays are restored by using image thresholding and deep convolution generative adversarial networks, and the real details and color information of the surface of salient three-dimensional objects are restored. Therefore, the three-dimensional salient objects obtained after reconstruction with the help of lens arrays can not only effectively simplify subsequent computer vision tasks, but also provide fine and colorful three-dimensional display of salient targets. IV. Description of the drawings

[0015] Attached Figure 1 This is a flow chart of a high-precision integrated imaging 3D salient target detection method with texture features proposed in the present invention.

[0016] Attached Figure 2 Flowchart for high-precision salient object detection on micro-image arrays.

[0017] Attached Figure 3 Schematic diagram of the structure of a multi-layer perceptron for high-precision salient object detection.

[0018] Attached Figure 4 Flow chart of the hierarchical texture feature compensation method for significant micro-image arrays.

[0019] It should be understood that the above drawings are only schematic and are not drawn to scale. V. Specific implementation methods

[0020] A typical embodiment of a high-precision integrated imaging 3D salient target detection method with texture features proposed by the present invention is described in detail below, and the present invention is further described in detail. It is necessary to point out here that the following embodiments are only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Personnel skilled in the art in this field may make some non-essential improvements and adjustments to the present invention based on the above content of the present invention, which still fall within the scope of protection of the present invention.

[0021] The present invention proposes a high-precision integrated imaging 3D salient target detection method with texture features, which specifically includes four processes: acquisition of four-dimensional light field data, high-precision salient target detection of micro-image array, texture feature compensation of salient micro-image array, and three-dimensional reconstruction of salient micro-image array with texture features. The specific process is shown in the attached figure. Figure 1 shown.

[0022] The acquisition process of the four-dimensional light field data is to establish a virtual camera array in the computer graphics software to obtain the virtual four-dimensional light field information, and then generate a micro-image array through a pixel mapping algorithm. The micro-image array contains multi-angle information of the four-dimensional light field and contains rich geometric information of the scene. Comprehensive analysis of the micro-image array can fully and effectively mine multi-angle three-dimensional saliency relationships in different spatial ranges.

[0023] In the high-precision salient target detection process of the micro-image array, the micro-image array of size 2000×2000 is divided according to the size of the micro-image to form a micro-image sequence. After the micro-image sequence is reconstructed, a three-dimensional tensor is obtained, which is then divided into 7×7 adjacent micro-images to form a small rectangular area (patch). In the folding operation, a convolution kernel of the same size as the patch is used, with a step size of 4 and a padding of 2. The result is convolved with the corresponding elements in the patch, and after passing through the flattening layer, a micro-image sequence with increased length is obtained, which is embedded into the encoder together with the sinusoidal position code as input, and 6 attention heads are set. After the multi-head attention layer, multiple groups of query matrices Q, key matrices K and value matrices V of micro-image sequences at each level are obtained, and the normalized attention weight A of the i-th head is obtained through dot product calculation, scaling factor and activation function i ,Right now:

[0024] According to Ai The value V of the corresponding attention head i The weighted sum is performed to obtain the context vector of the corresponding position in the micro-image sequence, which is fully connected and then fed into the multi-layer perceptron module, as shown in the attached figure. Figure 3 As shown in the figure, the context vector is further extracted from the high-level micro-image features in the hidden layer after nonlinear transformation, and the output of the encoder is obtained after residual connection and layer normalization based on feature dimension. In the decoding module, the decoder and the reverse recombination folding operation are used for collaborative upsampling. In the masked multi-head attention layer, the dimensions of the input and output features of the micro-image sequence are set to 768, and independent attention weights are calculated for all positions of the micro-image in the standard set. In the encoder-decoder attention sublayer, the image feature dimension of this layer is set to be the same as the input and output feature dimensions, so as to facilitate the calculation of the correlation between cross-region micro-images by scaling the dot product and the residual connection calculation of two consecutive micro-image sequence layers. In the multi-layer perceptron and fully connected layers, the input features of the previous layer are upsampled. In order to retain the detail information, this layer fuses the low-level features of the micro-image sequences in the encoders of different scales. The image size is restored to the original size of the micro-image array by using the reverse recombination folding and the cyclic superposition of the decoder. Three cyclic decoding modules and the same number of scale factors r are set, which are 8, 4, and 1 respectively, so that the output feature maps are upsampled to the original micro-image array size. 1. Gradually restore the characteristic micro-image array to its original resolution to obtain a high-precision significant micro-image array.

[0025] The texture feature compensation process of the significant micro-image array is as shown in the attached Figure 4 As shown in FIG. 1 , the black and white significant micro-image array is divided into a regional significant micro-image array and an edge significant micro-image array by a Gaussian filter, where each micro-image array contains 2000×2000 pixels. The regional significant micro-image array is subjected to image thresholding, where the white area restores the texture features of the corresponding area of ​​the original micro-image array f(x, y), and the black area is retained. The obtained image g(x, y) is expressed as:

[0026]

[0027] The texture features of the boundary salient micro-image array are generated using a deep convolutional generative adversarial network. The generator consists of a transposed convolutional layer and is defined to generate a color image with three channels: red, green, and blue. The random noise z and a grayscale boundary map with a resolution of 2000×2000 are used as the input of the generator. The random noise z is converted into a higher-dimensional spatial representation and concatenated with the grayscale boundary map to obtain an original feature map of size 125×125×256. It is upsampled through a transposed convolutional layer, and batch normalization and rectified linear unit activation functions are used in each transposed convolutional layer. The last layer of the generator superimposes the hyperbolic tangent function and outputs a color image of size 2000×2000×3. The color image output by the generator enters the discriminator, and is downsampled through a series of convolutional layers and rectified linear unit activation functions to capture features of different scales. After normalization, the S-type function is used in the last layer to map the output to the (0,1) interval to obtain the total loss value and update the parameters of the generator, and finally generate a boundary salient micro-image array with texture features. The image is fused with the regional significant micro-image array with texture features by using image addition to obtain a significant micro-image array with texture features.

[0028] In the process of reconstructing a three-dimensional object with a significant micro-image array having texture features, the micro-image array obtained in the third step is used to reconstruct four-dimensional light field angle and spatial information with the help of a lens array to obtain a three-dimensional significant object with texture features.

[0029] The present invention proposes a high-precision integrated imaging 3D salient target detection method with texture features. According to the characteristics of micro-image arrays containing rich light field spatial information and angle information, a method for reorganizing and folding micro-image sequences is designed, and combined with the Transformer encoder and decoder, the salient clues of cross-regional micro-images in the micro-image array are fully mined to achieve high-precision integrated imaging 3D salient target detection. At the same time, a hierarchical texture feature compensation method for salient micro-image arrays is proposed, and the texture features of regional and boundary salient micro-image arrays are restored by using image thresholding and deep convolution generative adversarial networks, and the real details and color information of the surface of salient three-dimensional objects are restored. Therefore, the three-dimensional salient objects obtained after reconstruction with the help of lens arrays can not only effectively simplify subsequent computer vision tasks, but also provide fine and colorful three-dimensional display of salient targets.

Claims

1. A high-precision integrated imaging 3D salient object detection method with texture features, characterized in that: The method comprises four processes: acquisition of four-dimensional light field data, high-precision significant target detection of a micro-image array, texture feature compensation of a significant micro-image array, and three-dimensional reconstruction of a high-precision significant micro-image array with texture features. In the process of acquiring four-dimensional light field data, a real or virtual camera array is established to collect light field information at multiple angles, and a micro-image array is generated by a pixel mapping algorithm. In the high-precision salient target detection process of the micro-image array, the micro-image array is segmented to obtain a micro-image sequence, which is encoded and decoded n times in a cycle to obtain a high-precision salient micro-image array; In the process of compensating the texture features of the salient micro-image array, the texture features of the regional salient micro-image array and the boundary salient micro-image array are restored in layers to obtain a high-precision salient micro-image array with texture features. In the process of three-dimensional reconstruction of the salient micro-image array with texture features, a lens array is used to obtain a colorful 3D salient object, which fully displays the rich texture details on its surface, and realizes high-precision integrated imaging 3D salient target detection and reconstruction with texture features.

2. The high-precision integrated imaging 3D salient target detection method with texture features according to claim 1, characterized in that: A real camera array can be established to collect real four-dimensional light field information, or a virtual camera array can be established in computer graphics software to obtain virtual four-dimensional light field information, and a micro-image array can be obtained using a pixel mapping algorithm. The micro-image array contains multi-angle information of the four-dimensional light field, and performing salient target detection on it can fully capture the salient dependencies of micro-images in a larger range.

3. The high-precision integrated imaging 3D salient target detection method with texture features according to claim 1, characterized in that: The micro-image sequence is obtained by dividing the micro-image array according to the micro-image size, which contains multi-angle information of the four-dimensional light field and rich scene geometry information. After reorganization, a three-dimensional tensor is obtained. Multiple adjacent micro-images in the three-dimensional tensor are taken to form a small rectangular area (patch). In the folding operation, the patch is convolved with a convolution kernel of the same size. After the convolution result passes through the flattening layer, a cross-regional micro-image sequence with increased length is obtained, which facilitates the mining of micro-image saliency relationships in a larger spatial range during the encoding process.

4. The high-precision integrated imaging 3D salient target detection method with texture features according to claim 1, characterized in that: The cross-region micro-image sequence and the sinusoidal position code are embedded into the encoder as input. In the multi-head attention layer, n attention heads are set to obtain multiple sets of query matrices Q, key matrices K and value matrices V of micro-image sequences at each level, and then the attention weight A of the i-th head micro-image sequence is calculated. i , and compare it with the value matrix V of the corresponding attention head i After weighted summation, the context vector of the micro-image at the corresponding position is obtained. This vector can extract the high-level semantic features of the micro-image in the hidden layer after full connection and nonlinear transformation, and use it as an important basis for saliency prediction. It is output after residual connection and layer normalization.

5. The high-precision integrated imaging 3D salient target detection method with texture features according to claim 1, characterized in that: In the decoding module, independent attention weights are assigned to the micro-images at all positions of the standard salient micro-image array; in the encoder-decoder attention layer, the feature dimensions are set to be the same as the input features and the output features to achieve the correlation between cross-region micro-images in the micro-image sequence and the calculation of two consecutive micro-image sequence layers; in the multi-layer perceptron, upsampling of image features and multi-scale fusion of low-level features of micro-image sequences at each level of the encoder are simultaneously achieved; By using the reverse recombination and folding operation and the cyclic superposition of the decoder, the image size is restored to the same size as the original micro-image array, and finally a high-precision significant micro-image array is obtained.

6. The high-precision integrated imaging 3D salient target detection method with texture features according to claim 1, characterized in that: The significant micro-image array is divided into a regional significant micro-image array and a boundary significant micro-image array to compensate for texture features in a hierarchical manner. Both significant micro-image arrays contain M×N pixels. The regional significant micro-image array is subjected to image thresholding. The white area restores the texture features of the original micro-image array f(x, y), and the black area is retained. The obtained regional significant micro-image array g(x, y) with texture features satisfies:

7. The high-precision integrated imaging 3D salient target detection method with texture features according to claim 1, characterized in that: The texture features of the boundaries of salient objects are generated using a deep convolutional generative adversarial network. The generator consists of a transposed convolution layer and can generate a three-channel color image of red, green and blue. The generator converts the random noise z and splices it with the grayscale boundary saliency map with a resolution of M×N to obtain an original feature map of size a×b×c. Through iterative transposed convolution layers and using batch normalization and rectified linear unit activation functions in each layer, a hyperbolic tangent function is superimposed on the last layer to output a color image of size M×N×3. In the discriminator, the color image is passed through a series of convolution layers and rectified linear unit activation functions to capture its features at different scales, normalized and using the S-type function, the output is mapped to the interval (0,1) to obtain the total loss value, and the parameters of the generator are continuously updated to generate a boundary salient micro-image array with texture features. The image is fused with the regional significant micro-image array with texture features by using image addition to obtain a significant micro-image array with texture features.

8. The high-precision integrated imaging 3D salient target detection method with texture features according to claim 1, characterized in that: The generated high-precision salient micro-image array with texture features is used to reconstruct a true-color 3D salient object using a lens array.

Images