A 3D object saliency detection method using integrated imaging in multi-target scenes

Abstract

The present invention discloses a method for saliency detection of 3D objects using integrated imaging in a multi-target scene. The method comprises four parts: microelement super-resolution reconstruction, saliency detection, pixel position division, and three-dimensional reconstruction. The microelement super-resolution reconstruction part segments the microimage array based on the microelement concept and performs super-resolution reconstruction; the saliency detection part predicts salient areas based on the edge perception principle, and uses a residual module to optimize image details to improve the accuracy of saliency detection; the pixel position division part uses an anchor point detection method to segment different objects, obtains a salient microimage array of a single target object, and amplifies it; the three-dimensional reconstruction part reconstructs a stereoscopic salient image of a single target object through an integrated imaging display device. The method can not only obtain a stereoscopic salient image of a single target object in a multi-target scene, avoiding attention distraction and visual confusion caused by multiple objects, but also improves the accuracy of saliency detection.

Description

1. Technical Field

[0001] The present invention relates to the field of image processing, and more specifically, to a method for detecting saliency of 3D objects using integrated imaging in a multi-target scene. 2. Background Technology

[0002] Integrated imaging saliency detection can detect and locate the most salient regions in an image or video based on multi-view, multi-channel, and in-depth image information. It has important applications in visual tracking, image compression, and target recognition. Since most real-world scenes contain multiple salient targets and complex information, saliency detection of a single target object in multi-target scenarios is more applicable. This is particularly true in fields such as medical image processing, military reconnaissance, and surveillance. It can eliminate interference from ineffective salient regions and focus on exploring the three-dimensional features of the target object. This not only helps doctors accurately locate a few deformed cancer cells among numerous cells for diagnosis and surgical planning, but also helps avoid distraction and visual confusion caused by multiple similar objects in military reconnaissance and surveillance, facilitating the identification and tracking of key targets.

[0003] Currently, 3D saliency detection methods for integrated imaging mainly include refocusing saliency detection, video saliency detection, and co-saliency detection. To optimize refocusing saliency detection, Liu et al. proposed a learnable weighted descriptor method, utilizing a hierarchical multimodal fusion strategy to fully exploit co-saliency semantics and eliminate the blurring effect of focal slice features. Furthermore, Zhang et al. employed a two-stage saliency detection based on refocusing to refine target details for different focal slices. In video saliency detection, Li et al. utilized a fully convolutional neural network to achieve 3D saliency detection for integrated imaging by processing sequences of local micro-image elements. In co-saliency detection, Kou et al. collaboratively maximized the spatial and angular correlation of the four-dimensional light field, improving the accuracy and continuity of 3D saliency maps. However, existing methods are poorly applicable to complex scenes containing multiple objects, suffering from issues such as missing salient regions, incomplete objects, and inaccurate saliency maps. Furthermore, in multi-object scenes, the detection results of these methods often contain multiple salient regions, which can easily cause visual confusion and hinder the study of the position or shape information of individual objects. 3. Summary of the Invention

[0004] The present invention proposes a method for 3D object saliency detection based on integrated imaging in multi-target scenes, which consists of four parts: micro-element super-resolution reconstruction, saliency detection, pixel position division, and 3D reconstruction. The convolutional neural network model is trained by training the obtained micro-image array sample set, as shown in the attached figure. Figure 1As shown in the figure. First, in the micro-element super-resolution reconstruction, the micro-image array is segmented based on the micro-element idea to obtain the micro-image element set and perform super-resolution reconstruction. Secondly, in the saliency detection, the salient area is predicted according to the edge perception principle, and the residual module is used to optimize the image details. After combination, a salient micro-image array is obtained. Then, in the pixel position division, the anchor point detection method is used to obtain the boundary information of a single target object, realize the segmentation of different objects, obtain the salient micro-image array of a single target object and amplify it. Finally, the integrated imaging display device is used to reconstruct the stereo salient image of a single target object. The proposed method applies pixel position division to integrated imaging saliency detection for the first time to obtain a stereo salient image of a single target object in a multi-target scene, avoiding the attention distraction and visual confusion caused by multiple similar objects, and improving the accuracy and applicability of saliency detection.

[0005] The specific implementation steps of the present invention are as follows.

[0006] Step 1: The micro-element super-resolution reconstruction adopts the micro-element idea to perform fine segmentation on the micro-image array to obtain a micro-image element set, which is used as the input of the enhanced super-resolution generative adversarial network to supplement the image information and obtain a high-resolution micro-image element set.

[0007] Furthermore, the enhanced super-resolution generative adversarial network incorporates a high-order degradation model. The network's generator is uniquely designed, employing a residual-within-residual dense block as its core component, facilitating deep feature extraction and accurate restoration of image detail. Meanwhile, the discriminator utilizes an efficient encoder-decoder architecture, aiming to enhance the network's ability to discriminate high-resolution features and global consistency of image content.

[0008] Furthermore, the high-order degradation model can simulate complex degradation problems, solve the degradation caused by the imaging system and network transmission process, and make the super-resolution reconstruction result closer to the real situation. The model includes blurring, resizing, denoising, and image compression, as shown in the attached figure. Figure 2 As shown in the figure, the model loops through the four degradation processes described above twice, randomly selecting an algorithm for each process to ensure high diversity and unpredictability in the degradation process. Blurring algorithms include isotropic or anisotropic Gaussian filtering and two-dimensional sinc filtering; resizing includes bicubic interpolation, bilinear interpolation, and adaptive average pooling; noise distribution includes Gaussian noise, Poisson noise, color noise, and grayscale noise; and image compression includes image resizing and two-dimensional sinc filtering to remove redundant information.

[0009] Furthermore, the generator is characterized by dense residual links, with residual edges connected at both ends. The output of each convolutional layer serves as the input of all the following nodes.

[0010] Furthermore, the discriminator is divided into three parts: downsampling, upsampling, and output layers. The downsampling layer consists of three convolutional layers with spectral normalization, with the number of channels doubled in each layer. The upsampling layer uses traditional interpolation and also consists of three convolutional layers with spectral normalization, with the number of channels halved in each layer. The output layer consists of two convolutional layers with spectral normalization and one convolutional output layer.

[0011] Step 2: The saliency detection utilizes a supervised encoder-decoder architecture to simultaneously capture the contextual information and low-level detail features of micro-image elements. This architecture predicts salient regions based on edge perception principles and enhances prediction accuracy through residual refinement. Finally, the salient micro-image element sets are combined and arranged to obtain a complete salient micro-image array. This is then compared with the true values ​​to obtain the value of the loss function, which is used as feedback to guide the network optimization and learning process, ensuring high-quality output of the salient micro-image array, as shown in the attached figure. Figure 3 shown.

[0012] Furthermore, the encoder consists of an input layer with 64 convolutional filters and six convolutional modules. The first four convolutional modules adopt the main stage design concept of the IResNet framework, each of which contains three basic structures: a starting residual block, an intermediate residual block, and an ending residual block. The last two convolutional modules perform convolution after maximum pooling to obtain the same receptive field as the input micro-image element. The decoder is symmetrical with the encoder.

[0013] Furthermore, the process of predicting salient areas based on the edge perception principle is to utilize the characteristic that the grayscale changes at the edge in the image are most drastic, differentiate or calculate the second-order differential of each pixel point in the image to determine the edge pixel points, and then calculate the intensity or response value of the detected edge. Thereafter, the saliency score of each pixel point is calculated based on the edge intensity or response value and converted into a saliency image.

[0014] Furthermore, the residual refinement module is mainly composed of a simple encoder-decoder structure, and each convolutional layer has n filters, which are used to learn the difference between the salient region prediction map and the true value, thereby improving the image area and boundary.

[0015] Step 3: The pixel position is divided, and the re-parameterized VGG network is used in the backbone network to extract image features. The target position or bounding box coordinates are predicted by the regression sub-network in the effective detection head. Finally, the micro-image array is scanned. According to the discrimination results of each micro-image element, the image is segmented to obtain the significant micro-image array of a single target object, and it is amplified, as shown in the attached figure. Figure 4 At the same time, the true value of the significant micro-image array is manually labeled, segmented, and amplified to obtain the true value of the significant micro-image array of a single target object, which is compared with the test result to obtain the value of the loss function.

[0016] Furthermore, the anchor points are equivalent to presetting possible object bounding boxes on the micro-image array to be predicted, that is, presetting the object region that the feature data may represent. Each region is typically composed of two attributes: size and scale: the region area and the width-to-height ratio of the region rectangle, which can also be the width and height data of the anchor point. Each anchor point is set based on the pixels of the feature image, and multiple anchor points of different scales can be preset for the same feature. Anchor point detection refers to scanning and detecting based on the predicted anchor points of different sizes to obtain the object's bounding box.

[0017] Furthermore, the magnification of the significant micro-image array of a single target object utilizes traditional image magnification methods to supplement information and fully utilize the resolution of the screen display.

[0018] Step 4: Perform three-dimensional reconstruction through the integrated imaging display device to obtain a three-dimensional salient image of a single target object. IV. Description of the Figures

[0019] Figure 1 Schematic diagram of the structure of the integrated imaging 3D object saliency detection method in a multi-target scenario of the present invention;

[0020] Figure 2 Schematic diagram of a high-order degradation model for the micro-element super-resolution reconstruction part of the present invention;

[0021] Figure 3 Schematic diagram of the saliency detection network of the present invention;

[0022] Figure 4 Array of significant micro-images of a single target object generated for the present invention. V. Specific Implementation Methods

[0023] The following describes in detail a typical embodiment of the present invention's integrated imaging 3D object saliency detection method in a multi-target scenario, further specifically describing the present invention. It is important to note that the following embodiments are intended only to further illustrate the present invention and are not to be construed as limiting the scope of protection of the present invention. Non-essential improvements and adjustments made by persons skilled in the art based on the above-described disclosure remain within the scope of protection of the present invention.

[0024] The present invention proposes a method for 3D object saliency detection based on integrated imaging in multi-target scenes, which consists of four parts: micro-element super-resolution reconstruction, saliency detection, pixel position division, and 3D reconstruction. The convolutional neural network model is trained by training the obtained micro-image array sample set, as shown in the attached figure. Figure 1As shown, the method applies pixel position division to integrated imaging saliency detection for the first time to obtain a stereo salient image of a single target object in a multi-target scene, avoiding attention distraction and visual confusion caused by multiple similar objects, and improving the accuracy and applicability of saliency detection.

[0025] In the microelement super-resolution reconstruction, the present invention segments a microimage array based on the microelement concept to obtain microimage elements and then performs super-resolution reconstruction. Secondly, in saliency detection, salient regions are predicted based on edge perception principles, and image details are optimized using a residual module to combine and obtain a salient microimage array. Finally, in pixel location segmentation, anchor point detection is used to obtain boundary information of a single target object, enabling segmentation of different objects. This results in a salient microimage array of the single target object and amplification of the image. Finally, a 3D salient image of the single target object is reconstructed using an integrated imaging display device.

[0026] The specific implementation steps of the present invention are as follows:

[0027] Step 1: The micro-element super-resolution reconstruction adopts the micro-element idea to perform fine segmentation on the micro-image array to obtain a micro-image element set, which is used as the input of the enhanced super-resolution generative adversarial network to supplement the image information and obtain a high-resolution micro-image element set.

[0028] Furthermore, the enhanced super-resolution generative adversarial network incorporates a high-order degradation model. The network's generator is uniquely designed, employing a residual-within-residual dense block as its core component, facilitating deep feature extraction and accurate restoration of image detail. Meanwhile, the discriminator utilizes an efficient encoder-decoder architecture, aiming to enhance the network's ability to discriminate high-resolution features and global consistency of image content.

[0029] Furthermore, the high-order degradation model can simulate complex degradation problems, solve the degradation caused by the imaging system and network transmission process, and make the super-resolution reconstruction result closer to the real situation. The model includes blurring, resizing, denoising, and image compression, as shown in the attached figure. Figure 2 As shown in the figure, the model loops through the four degradation processes described above twice, randomly selecting an algorithm for each process to ensure high diversity and unpredictability in the degradation process. Blurring algorithms include isotropic or anisotropic Gaussian filtering and two-dimensional sinc filtering; resizing includes bicubic interpolation, bilinear interpolation, and adaptive average pooling; noise distribution includes Gaussian noise, Poisson noise, color noise, and grayscale noise; and image compression includes image resizing and two-dimensional sinc filtering to remove redundant information.

[0030] Furthermore, the generator is characterized by dense residual links, where the ends of the residual edges are connected in a concat manner. The output of each convolutional layer serves as the input of all the following nodes.

[0031] Furthermore, the discriminator is divided into three parts: downsampling, upsampling, and output layer. The downsampling layer includes three convolutional layers with spectral normalization, with the number of channels doubled in each layer; the upsampling layer uses the nearest neighbor interpolation method and also includes three convolutional layers with spectral normalization, with the number of channels halved in each layer; the output layer includes two convolutional layers with spectral normalization and one convolutional output layer.

[0032] Step 2: The saliency detection utilizes a supervised encoder-decoder architecture to simultaneously capture the contextual information and low-level detail features of micro-image elements. This architecture predicts salient regions based on edge perception principles and enhances prediction accuracy through residual refinement. Finally, the salient micro-image element sets are combined and arranged to obtain a complete salient micro-image array. This is then compared with the true values ​​to obtain the value of the loss function, which is used as feedback to guide the network optimization and learning process, ensuring high-quality output of the salient micro-image array, as shown in the attached figure. Figure 3 shown.

[0033] Furthermore, the encoder consists of an input layer with 64 3×3 convolutional filters with a stride of 1, and six convolutional modules. The first four convolutional modules adopt the main stage design concept of the IResNet framework, each of which contains three basic structures: a starting residual block, a middle residual block, and an ending residual block. The last two convolutional modules perform convolution after maximum pooling to obtain the same receptive field as the input micro-image element. The decoder is symmetrical with the encoder.

[0034] Furthermore, the process of predicting salient areas based on the edge perception principle is to utilize the characteristic that the grayscale changes at the edge in the image are most drastic, differentiate or calculate the second-order differential of each pixel point in the image to determine the edge pixel points, and then calculate the intensity or response value of the detected edge. Thereafter, the saliency score of each pixel point is calculated based on the edge intensity or response value and converted into a saliency image.

[0035] Furthermore, the residual refinement module is mainly composed of a simple encoder-decoder structure, and each convolutional layer has 64 filters of size 3×3, which are used to learn the difference between the salient region prediction map and the true value, thereby improving the image area and boundaries.

[0036] Step 3: The pixel position is divided, and the re-parameterized VGG network is used in the backbone network to extract image features. The target position or bounding box coordinates are predicted by the regression sub-network in the effective detection head. Finally, the micro-image array is scanned. According to the discrimination results of each micro-image element, the image is segmented to obtain the significant micro-image array of a single target object, and it is amplified, as shown in the attached figure. Figure 4 At the same time, the true value of the significant micro-image array is manually annotated and segmented to obtain the true value of the significant micro-image array of a single target object, which is compared with the test result to obtain the specific value of the loss function.

[0037] Furthermore, the anchor points are equivalent to presetting possible object bounding boxes on the micro-image array to be predicted, that is, presetting the object region that the feature data may represent. Each region is typically composed of two attributes: size and scale: the region area and the width-to-height ratio of the region rectangle, which can also be the width and height data of the anchor point. Each anchor point is set based on the pixels of the feature image, and multiple anchor points of different scales can be preset for the same feature. Anchor point detection refers to scanning and detecting based on the predicted anchor points of different sizes to obtain the object's bounding box.

[0038] Furthermore, the magnification of the significant micro-image array of a single target object utilizes conventional image magnification methods to supplement information and fully utilize the resolution of the screen display.

[0039] Step 4: Perform three-dimensional reconstruction through the integrated imaging display device to obtain a three-dimensional salient image of a single target object.

Claims

1. A method for 3D object saliency detection using integrated imaging in a multi-target scene, characterized in that: The method consists of four parts: microelement super-resolution reconstruction, saliency detection, pixel position segmentation, and 3D reconstruction. A convolutional neural network model is trained by training a micro-image array sample set obtained through training. The microelement super-resolution reconstruction part segments the micro-image array based on the microelement concept to obtain a micro-image element set and then performs super-resolution reconstruction. The saliency detection part predicts salient areas based on edge perception principles, optimizes image details using a residual module, and then combines them to obtain a salient micro-image array. The pixel position division part uses the anchor point detection method to obtain the boundary information of a single target object, realize the segmentation of different objects, obtain the salient micro-image array of the single target object and amplify it; the 3D reconstruction part uses the integrated imaging display device to reconstruct the 3D salient image of the single target object; The saliency detection method utilizes a supervised encoder-decoder architecture to simultaneously capture contextual information and low-level detail features of micro-image elements. This architecture predicts salient regions based on edge perception and enhances prediction accuracy through residual refinement. Finally, the salient micro-image element sets are combined and arranged to obtain a complete salient micro-image array. This is then compared with the true values ​​to obtain the value of a loss function, which is used as feedback to guide the network optimization and learning process, ensuring high-quality output of the salient micro-image array. The encoder consists of an input layer with 64 convolutional filters and six convolutional modules. The first four convolutional modules adopt the main stage design concept of the IResNet framework. Each main stage contains three basic structures: a starting residual block, an intermediate residual block, and an ending residual block. The last two convolutional modules perform convolution after maximum pooling to obtain the same receptive field as the input micro-image element. The decoder is symmetrical with the encoder. The process of predicting salient areas based on the edge perception principle is to utilize the characteristic that the grayscale changes at the edge of the image are the most drastic, differentiate or calculate the second-order differential of each pixel in the image to determine the edge pixels, then calculate the strength or response value of the detected edge, and then calculate the saliency score of each pixel based on the edge strength or response value and convert it into a saliency image; The pixel position division is performed by using a re-parameterized VGG network in the backbone network to extract image features, and a regression sub-network is used in the effective detection head to predict the target position or bounding box coordinates. Finally, the micro-image array is scanned, and based on the discrimination results of each micro-image element, the image is segmented to obtain a salient micro-image array of a single target object, and the salient micro-image array is amplified. Simultaneously, the true value of the salient micro-image array is manually labeled, segmented, and amplified to obtain the true value of the salient micro-image array of the single target object, which is compared with the test result to obtain the specific value of the loss function. The anchor point is equivalent to presetting a possible object bounding box on the micro-image array to be predicted, that is, presetting the object area that the feature data may represent. Each area is usually composed of two attributes: size and scale, namely the area of ​​the area and the width-to-height ratio of the area rectangle, which can also be the width-to-height data of the anchor point. Each anchor point is set based on the pixel points of the feature map, and the same feature can simultaneously preset multiple anchor points of different scales. The anchor point detection refers to scanning and detecting based on the predicted anchor points of different sizes to obtain the bounding box of the object.

2. The method for 3D object saliency detection using integrated imaging in a multi-target scenario according to claim 1, wherein: In the micro-element super-resolution reconstruction, the micro-element idea is adopted to perform fine segmentation on the micro-image array to obtain micro-image element sets, which are then used as the input of the enhanced super-resolution generative adversarial network to supplement the image information and obtain high-resolution micro-image element sets; The enhanced super-resolution generative adversarial network incorporates a high-order degradation model. The network's generator is uniquely designed, using a residual-within-residual dense block as its core component, which facilitates deep feature extraction and accurate restoration of image details. Meanwhile, the discriminator utilizes an efficient encoder-decoder architecture to improve the network's ability to discriminate high-resolution features and global consistency of image content. The high-order degradation model can simulate complex degradation problems, solve the degradation of images caused by the imaging system and network transmission process, and make the super-resolution reconstruction results closer to the actual situation; The model includes blurring, resizing, denoising, and image compression. The model loops through the four degradation processes twice, randomly selecting an algorithm for each process to ensure high diversity and unpredictability in the degradation process. The blurring algorithm includes isotropic or anisotropic Gaussian filtering and two-dimensional sine filtering; resizing includes bicubic interpolation, bilinear interpolation, and adaptive average pooling; noise distribution includes Gaussian noise, Poisson noise, color noise, and grayscale noise; and image compression includes image resizing and two-dimensional sine filtering to remove redundant information. The generator is characterized by dense residual links, with residual edges connected at both ends; the output of each convolutional layer serves as the input of all the nodes below; The discriminator is divided into three parts: downsampling, upsampling and output layer; downsampling includes three convolutional layers with spectral normalization, and the channels of each layer are doubled; upsampling uses traditional interpolation method and also includes three convolutional layers with spectral normalization, and the channels of each layer are halved; the output layer includes two convolutional layers with spectral normalization and one convolutional output layer.

3. The method for 3D object saliency detection using integrated imaging in a multi-target scenario according to claim 1, wherein: The residual refinement module in saliency detection is mainly composed of a simple encoder-decoder structure. Each convolutional layer has n filters, which are used to learn the difference between the salient region prediction map and the true value, thereby improving the image area and boundaries.

4. The method for 3D object saliency detection using integrated imaging in a multi-target scenario according to claim 1, wherein: The magnification of the significant micro-image array of a single target object utilizes traditional image magnification methods to supplement information and fully utilize the resolution of the screen display.

5. The method for 3D object saliency detection using integrated imaging in a multi-target scenario according to claim 1, wherein: The three-dimensional reconstruction is performed through the integrated imaging display device to obtain a stereoscopic salient image of a single target object.

Images