Integrated imaging 3D object saliency detection method in multi-target scene

Abstract

The invention discloses an integrated imaging 3D object saliency detection method in a multi-target scene. The method comprises four parts of infinitesimal super-resolution reconstruction, saliency detection, pixel position division and three-dimensional reconstruction. The infinitesimal super-resolution reconstruction part is used for segmenting a micro-image array based on an infinitesimal thought and carrying out super-resolution reconstruction; the saliency detection part predicts a salient region according to an edge perception principle, optimizes image details by using a residual module, and improves the saliency detection accuracy; according to the pixel position division part, segmentation of different objects is achieved through an anchor point detection method, and a significant micro-image array of a single target object is obtained and amplified; the three-dimensional reconstruction part reconstructs a three-dimensional salient image of a single target object through an integrated imaging display device. According to the method, the three-dimensional saliency image of the single target object in the multi-target scene can be obtained, distraction and visual confusion caused by multiple objects are avoided, and the accuracy of saliency detection is improved.

Description

1. Technical Field

[0001] The present invention relates to the field of image processing, and more specifically, to a 3D object saliency detection method using integrated imaging in a multi-target scene. 2. Background Technology

[0002] Integrated imaging saliency detection can detect and locate the most salient areas in images or videos based on multi-view, multi-channel, and in-depth image information, and has very important applications in visual tracking, image compression, and target recognition. Since most actual scenes contain multiple salient targets and the information is complex, saliency detection of a single target object in a multi-target scene is more applicable, especially in the fields of medical image processing, military reconnaissance, and surveillance. It can eliminate the interference of invalid salient areas and focus on exploring the three-dimensional features of the target object, which is helpful for doctors to accurately locate a few deformed cancer cells among many cells for diagnosis and surgical planning. In military reconnaissance and surveillance, it can also avoid distraction and visual confusion caused by multiple similar objects, making it easier to identify and track key targets.

[0003] At present, the integrated imaging 3D saliency detection methods mainly include refocusing saliency detection, video saliency detection and co-saliency detection. To optimize the refocusing saliency detection method, Liu et al. proposed a learnable weight descriptor method, using a hierarchical multimodal fusion strategy to fully explore the co-saliency semantics and eliminate the blurring effect of focal slice features. Furthermore, Zhang et al. used two-stage saliency detection for different focal slices based on refocusing to refine the target details. In video saliency detection, Li et al. used a fully convolutional neural network to achieve integrated imaging 3D saliency detection by processing local micro-image element sequences. In the co-saliency detection method, Kou et al. collaboratively maximized the spatial and angular correlation of the four-dimensional light field to improve the accuracy and continuity of the 3D saliency map. However, the existing methods are poorly applicable in complex scenes containing multiple targets, and there are problems such as partial loss of saliency areas, incomplete objects, and inaccurate saliency maps. Moreover, in multi-target scenes, the detection results of the above methods mostly contain multiple saliency areas, which are easy to cause visual confusion and are not conducive to studying the position or shape information of a single target object. III. Summary of the invention

[0004] The present invention proposes a 3D object saliency detection method for integrated imaging in a multi-target scene, which consists of four parts: micro-element super-resolution reconstruction, saliency detection, pixel position division and 3D reconstruction. The designed convolutional neural network model is trained by training the obtained micro-image array sample set, as shown in the attached figure. Figure 1As shown. First, in the micro-element super-resolution reconstruction, the micro-image array is segmented based on the micro-element idea to obtain the micro-image element set and perform super-resolution reconstruction. Secondly, in the saliency detection, the salient area is predicted according to the edge perception principle, and the image details are optimized using the residual module, and the salient micro-image array is obtained after combination. Then, in the pixel position division, the anchor point detection method is used to obtain the boundary information of a single target object, realize the segmentation of different objects, obtain the salient micro-image array of a single target object and amplify it. Finally, the integrated imaging display device is used to reconstruct the stereo salient image of a single target object. The proposed method applies pixel position division to integrated imaging saliency detection for the first time to obtain a stereo salient image of a single target object in a multi-target scene, avoids the attention distraction and visual confusion caused by multiple similar objects, and improves the accuracy and applicability of saliency detection.

[0005] The specific implementation steps of the present invention are as follows.

[0006] Step 1: The micro-element super-resolution reconstruction adopts the micro-element idea to perform fine segmentation on the micro-image array to obtain a micro-image element set, and uses it as the input of the enhanced super-resolution generative adversarial network to supplement the image information and obtain a high-resolution micro-image element set.

[0007] Furthermore, the enhanced super-resolution generative adversarial network incorporates a high-order degradation model. The generator of the network is uniquely designed, using residual-in-residual dense blocks as core components, which is conducive to deep feature extraction and accurate restoration of image details. At the same time, the discriminator uses an efficient encoder-decoder architecture to improve the network's ability to identify high-resolution features and the global consistency of image content.

[0008] Furthermore, the high-order degradation model can simulate complex degradation problems, solve the degradation of images caused by the imaging system and network transmission process, and make the super-resolution reconstruction results closer to the actual situation. The model includes blurring, resizing, denoising, and image compression, as shown in the attached Figure 2 As shown. The model cycles through the above four degradation processes twice, and randomly selects an algorithm for each process to ensure the high diversity and unpredictability of the degradation process. Among them, the blur algorithm includes isotropic or anisotropic Gaussian filtering and two-dimensional sine filtering; resizing includes bicubic interpolation, bilinear interpolation and adaptive average pooling; noise distribution includes Gaussian noise cloth, Poisson noise, color noise, and grayscale noise; image compression includes controlling image size and two-dimensional sine filtering to remove redundant information.

[0009] Furthermore, the generator is characterized by dense residual links, and the residual edges are connected at both ends. The output of each convolutional layer will serve as the input of all the following nodes.

[0010] Furthermore, the discriminator is divided into three parts: downsampling, upsampling and output layer. Downsampling includes three convolutional layers with spectral normalization, and the channels of each layer are doubled; upsampling uses traditional interpolation method and also includes three convolutional layers with spectral normalization, and the channels of each layer are halved; the output layer includes two convolutional layers with spectral normalization and one convolutional output layer.

[0011] Step 2: The saliency detection uses a supervised encoder-decoder structure to capture the contextual information and low-level detail features of micro-image elements at the same time. This structure predicts salient areas based on the edge perception principle and enhances the prediction accuracy through residual refinement. Finally, the salient micro-image element sets are combined and arranged to obtain a complete salient micro-image array, and compared with the true value to obtain the value of the loss function, which is used as feedback to guide the optimization and learning process of the network to ensure the high-quality output of the salient micro-image array, as shown in the attached figure. Figure 3 shown.

[0012] Furthermore, the encoder consists of an input layer and six convolution modules, the input layer has 64 convolution filters; the first four convolution modules adopt the main stage design concept of the IResNet framework, each main stage includes three basic structures of the starting residual block, the middle residual block and the ending residual block; the last two convolution modules obtain the same receptive field as the input micro-image element after convolution after maximum pooling. The decoder is symmetrical with the encoder.

[0013] Furthermore, the process of predicting the salient area based on the edge perception principle is to utilize the characteristic that the grayscale change of the edge in the image is the most drastic, differentiate or calculate the second-order differential of each pixel point of the image to determine the edge pixel point, and then calculate the strength or response value of the detected edge, and then calculate the significance score of each pixel point based on the edge strength or response value and convert it into a significance image.

[0014] Furthermore, the residual refinement module is mainly a simple encoder-decoder structure, and each convolutional layer has n filters for learning the difference between the salient region prediction map and the true value, thereby improving the image region and boundary.

[0015] Step 3: The pixel position is divided, and the re-parameterized VGG network is used in the backbone network to extract image features. The target position or bounding box coordinates are predicted by the regression sub-network in the effective detection head. Finally, the micro-image array is scanned. According to the discrimination result of each micro-image element, the image is segmented to obtain the significant micro-image array of a single target object, and it is enlarged, as shown in the attached figure. Figure 4 At the same time, the true value of the significant micro-image array is manually annotated, segmented and enlarged to obtain the true value of the significant micro-image array of a single target object, which is compared with the test result to obtain the value of the loss function.

[0016] Furthermore, the anchor point is equivalent to presetting a possible object bounding box on the micro-image array to be predicted, that is, presetting the object area that the feature data may represent. Each area is usually composed of two attributes: size and ratio, that is, the area of ​​the area and the width-to-height ratio of the area rectangle, which can also be the width-to-height data of the anchor point. Each anchor point is set based on the pixel points of the feature map, and the same feature can simultaneously preset multiple anchor points of different scales. The anchor point detection refers to scanning and detecting according to the predicted anchor points of different sizes to obtain the bounding box of the object.

[0017] Furthermore, the enlargement of the significant micro-image array of a single target object utilizes conventional image enlargement methods to supplement information and fully utilize the resolution of the screen display.

[0018] Step 4: Perform three-dimensional reconstruction through an integrated imaging display device to obtain a stereoscopic salient image of a single target object. IV. Description of the drawings

[0019] Figure 1 It is a schematic diagram of the structure of the integrated imaging 3D object saliency detection method in a multi-target scene of the present invention;

[0020] Figure 2 A schematic diagram of a high-order degradation model of a micro-element super-resolution reconstruction part of the present invention;

[0021] Figure 3 It is a schematic diagram of the saliency detection network of the present invention;

[0022] Figure 4 An array of salient micro-images of a single target object generated for the present invention. V. Specific implementation methods

[0023] A typical embodiment of the integrated imaging 3D object saliency detection method in a multi-target scene of the present invention is described in detail below, and the present invention is further described in detail. It is necessary to point out here that the following embodiments are only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Personnel skilled in the art in this field may make some non-essential improvements and adjustments to the present invention based on the above content of the present invention, which still fall within the scope of protection of the present invention.

[0024] The present invention proposes a 3D object saliency detection method for integrated imaging in a multi-target scene, which consists of four parts: micro-element super-resolution reconstruction, saliency detection, pixel position division and 3D reconstruction. The designed convolutional neural network model is trained by training the obtained micro-image array sample set, as shown in the attached figure. Figure 1As shown, the method applies pixel position division to integrated imaging saliency detection for the first time to obtain a stereoscopic salient image of a single target object in a multi-target scene, avoiding attention distraction and visual confusion caused by multiple similar objects, and improving the accuracy and applicability of saliency detection.

[0025] In the micro-element super-resolution reconstruction, the present invention divides the micro-image array based on the micro-element idea to obtain micro-image elements and performs super-resolution reconstruction. Secondly, in the saliency detection, the salient area is predicted according to the edge perception principle, and the image details are optimized through the residual module to obtain the salient micro-image array. Then, in the pixel position division, the anchor point detection method is used to obtain the boundary information of a single target object to achieve the segmentation of different objects, and the salient micro-image array of the single target object is obtained and amplified. Finally, the integrated imaging display device is used to reconstruct the stereoscopic salient image of the single target object.

[0026] The specific implementation steps of the present invention are as follows:

[0027] Step 1: The micro-element super-resolution reconstruction adopts the micro-element idea to perform fine segmentation on the micro-image array to obtain a micro-image element set, and uses it as the input of the enhanced super-resolution generative adversarial network to supplement the image information and obtain a high-resolution micro-image element set.

[0028] Furthermore, the enhanced super-resolution generative adversarial network incorporates a high-order degradation model. The generator of the network is uniquely designed, using residual-in-residual dense blocks as core components, which is conducive to deep feature extraction and accurate restoration of image details. At the same time, the discriminator uses an efficient encoder-decoder architecture to improve the network's ability to identify high-resolution features and the global consistency of image content.

[0029] Furthermore, the high-order degradation model can simulate complex degradation problems, solve the degradation of images caused by the imaging system and network transmission process, and make the super-resolution reconstruction results closer to the actual situation. The model includes blurring, resizing, denoising, and image compression, as shown in the attached Figure 2 As shown. The model cycles through the above four degradation processes twice, and randomly selects an algorithm for each process to ensure the high diversity and unpredictability of the degradation process. Among them, the blur algorithm includes isotropic or anisotropic Gaussian filtering and two-dimensional sine filtering; resizing includes bicubic interpolation Bicubic function, bilinear interpolation and adaptive average pooling; noise distribution includes Gaussian noise cloth, Poisson noise color noise, and grayscale noise; image compression includes controlling image size and two-dimensional sine filtering to remove redundant information.

[0030] Furthermore, the generator is characterized by dense residual links, and the two ends of the residual edges are connected in a concat manner. The output of each convolutional layer will serve as the input of all the following nodes.

[0031] Furthermore, the discriminator is divided into three parts: downsampling, upsampling and output layer. Downsampling includes three convolution layers with spectral normalization, and the channels of each layer are doubled; upsampling uses the neighboring interpolation method and also includes three convolution layers with spectral normalization, and the channels of each layer are halved; the output layer includes two convolution layers with spectral normalization and one convolution output layer.

[0032] Step 2: The saliency detection uses a supervised encoder-decoder structure to capture the contextual information and low-level detail features of micro-image elements at the same time. This structure predicts salient areas based on the edge perception principle and enhances the prediction accuracy through residual refinement. Finally, the salient micro-image element sets are combined and arranged to obtain a complete salient micro-image array, and compared with the true value to obtain the value of the loss function, which is used as feedback to guide the optimization and learning process of the network to ensure the high-quality output of the salient micro-image array, as shown in the attached figure. Figure 3 shown.

[0033] Furthermore, the encoder consists of an input layer and six convolution modules, the input layer has 64 convolution filters of size 3×3 and stride 1; the first four convolution modules adopt the main stage design concept of the IResNet framework, each main stage includes three basic structures of the starting residual block, the middle residual block and the ending residual block; the last two convolution modules obtain the same receptive field as the input micro-image element after convolution after maximum pooling. The decoder is symmetrical with the encoder.

[0034] Furthermore, the process of predicting the salient area based on the edge perception principle is to utilize the characteristic that the grayscale change of the edge in the image is the most drastic, differentiate or calculate the second-order differential of each pixel point of the image to determine the edge pixel point, and then calculate the strength or response value of the detected edge, and then calculate the significance score of each pixel point based on the edge strength or response value and convert it into a significance image.

[0035] Furthermore, the residual refinement module is mainly a simple encoder-decoder structure, and each convolutional layer has 64 filters of size 3×3, which are used to learn the difference between the salient region prediction map and the true value, thereby improving the image region and boundary.

[0036] Step 3: The pixel position is divided, and the re-parameterized VGG network is used in the backbone network to extract image features. The target position or bounding box coordinates are predicted by the regression sub-network in the effective detection head. Finally, the micro-image array is scanned. According to the discrimination result of each micro-image element, the image is segmented to obtain the significant micro-image array of a single target object, and it is enlarged, as shown in the attached figure. Figure 4 At the same time, the true value of the significant micro-image array is manually annotated and segmented to obtain the true value of the significant micro-image array of a single target object, which is compared with the test result to obtain the specific value of the loss function.

[0037] Furthermore, the anchor point is equivalent to presetting a possible object bounding box on the micro-image array to be predicted, that is, presetting the object area that the feature data may represent. Each area is usually composed of two attributes: size and ratio, that is, the area of ​​the area and the width-to-height ratio of the area rectangle, which can also be the width-to-height data of the anchor point. Each anchor point is set based on the pixel points of the feature map, and the same feature can simultaneously preset multiple anchor points of different scales. The anchor point detection refers to scanning and detecting according to the predicted anchor points of different sizes to obtain the bounding box of the object.

[0038] Furthermore, the enlargement of the significant micro-image array of a single target object utilizes conventional image enlargement methods to supplement information and fully utilize the resolution of the screen display.

[0039] Step 4: Perform three-dimensional reconstruction through an integrated imaging display device to obtain a stereoscopic salient image of a single target object.

Claims

1. A method for 3D object saliency detection using integrated imaging in a multi-target scene, characterized in that: The method consists of four parts: micro-element super-resolution reconstruction, saliency detection, pixel position division and three-dimensional reconstruction. The designed convolutional neural network model is trained by training and sorting the obtained micro-image array sample set; the micro-element super-resolution reconstruction part divides the micro-image array based on the micro-element idea to obtain the micro-image element set and performs super-resolution reconstruction; the saliency detection part predicts the salient area according to the edge perception principle, optimizes the image details by using the residual module, and obtains the salient micro-image array by combination; The pixel position division part uses the anchor point detection method to obtain the boundary information of a single target object, realize the segmentation of different objects, obtain the significant micro-image array of a single target object and amplify it; the three-dimensional reconstruction part uses the integrated imaging display device to reconstruct the three-dimensional significant image of a single target object; The method applies pixel position division to integrated imaging saliency detection for the first time, obtains a salient micro-image array of a single target object in a multi-target scene, avoids attention distraction and visual confusion caused by multiple similar objects, and improves the accuracy and applicability of saliency detection.

2. The method for 3D object saliency detection by integrated imaging in a multi-target scene according to claim 1, characterized in that: In the micro-element super-resolution reconstruction, the micro-element idea is adopted to perform fine segmentation on the micro-image array to obtain a micro-image element set, which is used as the input of the enhanced super-resolution generative adversarial network to supplement the image information and obtain a high-resolution micro-image element set; The enhanced super-resolution generative adversarial network incorporates a high-order degradation model. The generator of the network is uniquely designed, using residual-in-residual dense blocks as core components, which is conducive to deep feature extraction and accurate restoration of image details. At the same time, the discriminator uses an efficient encoder-decoder architecture to improve the network's ability to identify high-resolution features and the global consistency of image content. The high-order degradation model can simulate complex degradation problems, solve the degradation of images caused by the imaging system and network transmission process, and make the results of super-resolution reconstruction closer to the actual situation; The model includes blurring, resizing, denoising, and image compression, as shown in Figure 2; the model cycles through the above four degradation processes twice, and randomly selects an algorithm for each process to ensure the high diversity and unpredictability of the degradation process; the blurring algorithm includes isotropic or anisotropic Gaussian filtering and two-dimensional sine filtering; resizing includes bicubic interpolation, bilinear interpolation, and adaptive average pooling; noise distribution includes Gaussian noise cloth, Poisson noise, color noise, and grayscale noise; image compression includes controlling image size and two-dimensional sine filtering to remove redundant information; The generator is characterized by dense residual links, with both ends of the residual edges connected; the output of each convolutional layer will serve as the input of all the following nodes; The discriminator is divided into three parts: downsampling, upsampling and output layer; the downsampling includes three convolutional layers with spectral normalization, and the channels of each layer are doubled; the upsampling uses the traditional interpolation method and also includes three convolutional layers with spectral normalization, and the channels of each layer are halved; the output layer includes two layers of convolution with spectral normalization and one convolutional output layer.

3. The method for 3D object saliency detection by integrated imaging in a multi-target scene according to claim 1, characterized in that: The saliency detection uses a supervised encoder-decoder structure to capture the contextual information and low-level detail features of micro-image elements at the same time; the structure predicts the salient area based on the edge perception principle and enhances the prediction accuracy through residual refinement; finally, the salient micro-image element sets are combined and arranged to obtain a complete salient micro-image array, and compared with the true value to obtain the value of the loss function, which is used as feedback to guide the optimization and learning process of the network to ensure the high-quality output of the salient micro-image array, as shown in Figure 3; The encoder consists of an input layer and six convolution modules. The input layer has 64 convolution filters. The first four convolution modules adopt the main stage design concept of the IResNet framework. Each main stage includes three basic structures: a starting residual block, an intermediate residual block, and an ending residual block. The last two convolution modules obtain the same receptive field as the input micro-image element after convolution after maximum pooling. The decoder is symmetrical with the encoder. The process of predicting the salient area based on the edge perception principle is to utilize the characteristic that the grayscale change of the edge in the image is the most drastic, differentiate or calculate the second-order differential of each pixel point of the image to determine the edge pixel point, and then calculate the strength or response value of the detected edge, and then calculate the significance score of each pixel point based on the edge strength or response value and convert it into a significance image.

4. The method for 3D object saliency detection by integrated imaging in a multi-target scene according to claim 1, characterized in that: The residual refinement module described in saliency detection is mainly a simple encoder-decoder structure with n filters in each convolutional layer, which is used to learn the difference between the salient region prediction map and the true value, thereby improving the image region and boundary.

5. The method for 3D object saliency detection by integrated imaging in a multi-target scene according to claim 1, characterized in that: The pixel position division uses a re-parameterized VGG network to extract image features in the backbone network part, and predicts the target position or bounding box coordinates through a regression sub-network in the effective detection head part. Finally, the micro-image array is scanned, and the image is segmented according to the discrimination result of each micro-image element to obtain a significant micro-image array of a single target object, and the significant micro-image array is enlarged, as shown in Figure 4; at the same time, the real value of the significant micro-image array is manually marked, segmented and enlarged to obtain the real value of the significant micro-image array of a single target object, and compared with the test result to obtain the specific value of the loss function; the anchor point is equivalent to presetting a possible object bounding box on the micro-image array to be predicted, that is, presetting the object area that the feature data may represent, each area is usually composed of two attributes of size and ratio, that is, the area of ​​the region and the width and height ratio of the region rectangle, which can also be the width and height data of the anchor point; each anchor point is set on the basis of the pixel point of the feature map, and the same feature can simultaneously preset multiple anchor points of different scales; the anchor point detection refers to scanning and detecting according to the predicted anchor points of different sizes to obtain the bounding box of the object.

6. The method for 3D object saliency detection by integrated imaging in a multi-target scene according to claim 1, characterized in that: The enlargement of the significant micro-image array of a single target object utilizes conventional image enlargement methods to supplement information and fully utilize the resolution of the screen display.

7. The method for 3D object saliency detection by integrated imaging in a multi-target scene according to claim 1, characterized in that: The three-dimensional reconstruction is performed through an integrated imaging display device to obtain a three-dimensional salient image of a single target object.

Images