Photoacoustic imaging method and device, electronic equipment and computer storage medium

By integrating the hollow pyramid pooling module and the generation adversarial network of multi-scale information network in the image significance prediction network, the problem of poor image feature extraction simplification and generalization capabilities in traditional methods is solved, and efficient and accurate significance is achieved Sexual image generation.

CN118941818AInactive Publication Date: 2024-11-12WUHAN XIN MICROELECTRONICS TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410981207.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-11-12
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The traditional image significance prediction network cannot fully capture the rich semantic information and structural features in the image, resulting in inaccurate and fine detection results, and poor generalization capabilities, resulting in inefficient efficiency in generating significant images.

Method used

Using an image significance prediction model based on the significance generative adversarial network, the multi-scale feature information of the image is extracted by fusing the hollow pyramid pooling module in the multi-scale information network, and using the idea of ​​generating adversarial to distinguish the significance map, the training is obtained. The model is highly generalized.

Benefits of technology

It improves the accuracy and generalization ability of image significance prediction, enhances the generation efficiency of significant images, and maintains efficient generation of significant images in different data sets and application scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941818A_ABST
    Figure CN118941818A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of image processing, and discloses a photoacoustic imaging method and device, electronic equipment and a computer storage medium, and the method comprises the steps: obtaining a to-be-processed visual image; the visual image to be processed is acquired based on an optical acquisition device or / and an acoustic acquisition device; performing image preprocessing on the to-be-processed visual image to obtain a preprocessed image; inputting the preprocessed image into an image saliency prediction model to obtain a saliency image of the to-be-processed visual image output by the image saliency prediction model; wherein the image saliency prediction model is obtained by training an initial model based on a sample visual image and a corresponding image saliency result label; the initial model is formed by fusing a cavity pyramid pooling module in a multi-scale information network in a generator of the saliency generative adversarial network. According to the method, the accuracy of image saliency prediction can be improved, the generalization ability of image saliency prediction can be enhanced, and then the generation efficiency of the saliency image can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a photoacoustic imaging method, device, electronic equipment and computer storage medium. Background Art

[0002] When extracting image features based on traditional image saliency prediction networks, they are unable to fully capture the rich semantic information and structural features in the image, resulting in inaccurate and intricate detection results. In addition, commonly used image saliency prediction networks have poor generalization capabilities for different data sets or application scenarios.

[0003] Therefore, the generation of saliency images based on current image saliency prediction methods is inefficient due to insufficient accuracy and poor generalization ability. Summary of the invention

[0004] The present invention provides a photoacoustic imaging method, device, electronic device and computer storage medium, which are used to solve the problem of inefficiency in generating saliency images based on the current image saliency prediction method due to insufficient accuracy and poor generalization ability, and to improve the generation efficiency of image saliency.

[0005] A photoacoustic imaging method, comprising:

[0006] Acquire a visual image to be processed; the visual image to be processed is acquired based on an optical collector and / or an acoustic collector;

[0007] Performing image preprocessing on the visual image to be processed to obtain a preprocessed image;

[0008] The preprocessed image is input into an image saliency prediction model to obtain a saliency image of the visual image to be processed output by the image saliency prediction model; wherein the image saliency prediction model is obtained by training an initial model based on a sample visual image and a corresponding image saliency result label; and the initial model is formed by integrating a dilated pyramid pooling module in a multi-scale information network into a generator of a saliency generative adversarial network.

[0009] According to the photoacoustic imaging method provided by the present invention, the image saliency prediction model is determined based on the following method:

[0010] Obtain sample visual images and corresponding image saliency result labels;

[0011] Performing image preprocessing on the sample visual image to obtain a sample preprocessed image;

[0012] Inputting the sample preprocessed image into the generator of the initial model to obtain a predicted saliency image output by the generator;

[0013] Inputting the predicted saliency image and its corresponding image saliency result label into the discriminator of the initial model to obtain the adversarial loss of the initial model output by the discriminator;

[0014] If it is determined that the adversarial loss has not converged or has not reached a preset loss threshold, updating parameters of the generator and the discriminator respectively;

[0015] If it is determined that the adversarial loss converges or reaches a preset loss threshold, the initial model is determined as the image saliency prediction model.

[0016] According to the photoacoustic imaging method provided by the present invention, the generator of the initial model at least includes an encoder, a dilated pyramid pooling module and a decoder.

[0017] According to the photoacoustic imaging method provided by the present invention, the image preprocessing is performed on the visual image to be processed to obtain the preprocessed image, including:

[0018] Converting the color space of the visual image to be processed to obtain a hue channel, a saturation channel and a lightness channel;

[0019] Performing histogram equalization on the brightness channel to obtain a new brightness channel;

[0020] Combining the new lightness channel, the hue channel and the saturation channel to obtain a balanced image;

[0021] Fill the edges of the equalized image to obtain a filled image;

[0022] The pixel value of each channel of each pixel point in the filled image is normalized to obtain a preprocessed image.

[0023] According to the photoacoustic imaging method provided by the present invention, the step of performing histogram equalization on the brightness channel to obtain a new brightness channel includes:

[0024] Determining the probability of occurrence of each brightness value in the brightness channel;

[0025] According to each probability value, a histogram mapping of each brightness value is performed to obtain corresponding new brightness values;

[0026] A new luminance channel is determined based on each new luminance value.

[0027] According to the photoacoustic imaging method provided by the present invention, after obtaining the saliency image of the visual image to be processed output by the image saliency prediction model, the method further comprises:

[0028] Visual image registration is performed on the visual image to be processed according to the saliency image; the visual image registration includes any one of element size adjustment, visual image alignment, image space correction after element size adjustment, image space correction after visual image alignment, or image space correction after visual image alignment and element size adjustment.

[0029] According to the photoacoustic imaging method provided by the present invention, after obtaining the saliency image of the visual image to be processed output by the image saliency prediction model, the method further comprises:

[0030] Obtaining a binary image of the saliency image;

[0031] Determining a saliency evaluation index of an eye fixation point of the saliency image according to the saliency image and the binarized image;

[0032] Determining an object saliency evaluation index of the saliency image according to the saliency image and a real saliency image of the visual image to be processed;

[0033] An evaluation value of the saliency image is determined according to the object saliency evaluation index and the eye fixation point saliency evaluation index.

[0034] The present invention also provides a photoacoustic imaging device, comprising:

[0035] An acquisition module, used for acquiring a visual image to be processed; the visual image to be processed is acquired based on an optical collector and / or an acoustic collector;

[0036] A preprocessing module, used for performing image preprocessing on the visual image to be processed to obtain a preprocessed image;

[0037] An input module is used to input the preprocessed image into an image saliency prediction model to obtain a saliency image of the visual image to be processed output by the image saliency prediction model; wherein the image saliency prediction model is obtained by training an initial model based on a sample visual image and a corresponding image saliency result label; and the initial model is formed by integrating a dilated pyramid pooling module in a multi-scale information network into a generator of a saliency generative adversarial network.

[0038] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the photoacoustic imaging method as described in any one of the above items when executing the program.

[0039] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program implements any of the above-mentioned photoacoustic imaging methods when executed by a processor.

[0040] The present invention also provides a computer product having a computer program stored thereon, wherein the computer program implements any of the above-mentioned photoacoustic imaging methods when executed by a processor.

[0041] The photoacoustic imaging method provided in real time by the present invention obtains an image saliency prediction model by pre-training an initial model based on a sample visual image and a corresponding image saliency result label. Therefore, after obtaining a visual image to be processed and performing image preprocessing on the visual image to be processed, the obtained preprocessed image is input into the image saliency prediction model, and a saliency image of the visual image to be processed output by the image saliency prediction model can be obtained. Since the initial model is formed after fusing a dilute pyramid pooling module in a multi-scale information network in a generator of a saliency generative adversarial network, the dilute pyramid pooling module can extract and fuse multi-scale feature information of the input image to achieve fine-grained division of the predicted saliency image, thereby solving the problem of oversimplification of image feature extraction and improving the accuracy of image saliency prediction. At the same time, through the idea of ​​generative adversarial generation in the saliency generative adversarial network, the process of generating a saliency map from an input image is used as a generator part, and is input into a discriminator together with a saliency image result label to distinguish whether the saliency map is predicted or real, so that the trained image saliency prediction model has strong generalization. Therefore, the present invention can improve the accuracy of image saliency prediction and enhance the generalization ability of image saliency prediction, thereby improving the generation efficiency of saliency images. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the process of the photoacoustic imaging method provided by the present invention.

[0043] Figure 2 It is a schematic diagram of the network architecture of the generator in the initial model provided by the present invention.

[0044] Figure 3 It is a network structure diagram of the training process of the image saliency prediction model provided by the present invention.

[0045] Figure 4 is a schematic structural diagram of a photoacoustic imaging device provided by the present invention;

[0046] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0048] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0049] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present invention.

[0050] An embodiment of the present invention provides a photoacoustic imaging method. Figure 1 is a schematic diagram of the process of the photoacoustic imaging method provided by the present invention, such as Figure 1 As shown, the method includes:

[0051] Step 100: Obtain a visual image to be processed.

[0052] Step 200: perform image preprocessing on the visual image to be processed to obtain a preprocessed image.

[0053] Step 300: Input the preprocessed image into the image saliency prediction model to obtain a saliency image of the visual image to be processed output by the image saliency prediction model.

[0054] In this embodiment, the visual image to be processed is acquired based on an optical collector or / and an acoustic collector, that is, the visual image to be processed can be an optical visual image, an acoustic visual image, or an optical visual image and an acoustic visual image. The image saliency prediction model is obtained by training the initial model based on the sample visual image and the corresponding image saliency result label; the initial model is formed by integrating the dilated pyramid pooling module in the multi-scale information network in the generator of the saliency generative adversarial network.

[0055] It should be noted that the executor of the photoacoustic imaging method provided in the embodiment of the present invention may be a server, a computer device, etc., such as a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc.

[0056] The server or computer device of the present invention may be provided with or connected with a photoacoustic imaging device. In the present invention, the photoacoustic imaging device may be controlled to execute the photoacoustic imaging method of the present invention by controlling the photoacoustic imaging device. The present invention may integrate the dilute pyramid pooling module in the multi-scale information network (MSI-Net) in advance in the generator of the saliency generative adversarial network (Saliency Generative Adversial Network, SalGAN), thereby forming a model that can be used for image saliency prediction as an initial model based on the saliency generative adversarial network after integrating the dilute pyramid pooling module.

[0057] It should be noted that the generator of the saliency generative adversarial network of the present invention can be obtained by improving the backbone network of VGG16, specifically, the downsampling operation in the two maximum pooling (MaxPooling) is omitted in the backbone network of VGG16, and the atrous spatial pyramid pooling (ASPP) module is added to obtain multi-scale features of the image. Among them, VGG16 is a convolutional neural network architecture, which includes 16 convolutional layers and fully connected layers.

[0058] Figure 2 is a schematic diagram of the network architecture of the generator in the initial model provided by the present invention, such as Figure 2As shown, the generator G may include an encoder, an ASPP module and a decoder. After the input image is input to the generator, the generator may output a predicted saliency map.

[0059] Furthermore, the present invention can obtain visual images for information promotion as sample visual images, for example, poster images can be obtained as sample visual images. And, image saliency result labels are respectively annotated for each sample visual image to obtain image saliency result labels for the corresponding sample visual images.

[0060] Furthermore, image preprocessing may be performed on each sample visual image in the sample data set to obtain a corresponding sample preprocessed image.

[0061] Furthermore, a data set for model training can be established based on each sample preprocessed image and the corresponding image saliency result label. The present invention can further divide the data set into a training set (train_image_dataset) and a validation set (validation_image_dataset) according to a certain ratio (e.g., 8:2).

[0062] Furthermore, the initial model can be trained according to the data set, and after the training is completed, an image saliency prediction model can be obtained that can predict image saliency according to the input preprocessed visual image and output a saliency image.

[0063] Based on this, the present invention can obtain a visual image that needs to be predicted for saliency as a visual image to be processed, for example, obtain any poster image that needs to be predicted for saliency as a visual image to be processed.

[0064] Furthermore, the visual image to be processed may be subjected to image preprocessing, wherein the image preprocessing may include operations such as histogram equalization, edge filling, and normalization that are performed successively.

[0065] Furthermore, after completing operations such as histogram equalization, edge filling and normalization, the preprocessed image can be input into the image saliency prediction model, and the image saliency prediction model performs saliency prediction based on the input image. After the image saliency prediction model completes the prediction, the saliency image of the visual image to be processed output by the image saliency prediction model is obtained.

[0066] In the embodiment of the present invention, an image saliency prediction model is obtained by pre-training an initial model based on a sample visual image and a corresponding image saliency result label. Therefore, after obtaining a visual image to be processed and performing image preprocessing on the visual image to be processed, the obtained preprocessed image is input into the image saliency prediction model, and a saliency image of the visual image to be processed output by the image saliency prediction model can be obtained. Since the initial model is formed after fusing a dilute pyramid pooling module in a multi-scale information network in a generator of a saliency generative adversarial network, the dilute pyramid pooling module can extract and fuse multi-scale feature information of the input image to achieve fine-grained division of the predicted saliency image, thereby solving the problem of oversimplification of image feature extraction and improving the accuracy of image saliency prediction. At the same time, through the idea of ​​generative adversarial generation in a saliency generative adversarial network, the process of generating a saliency map from an input image is used as a generator part, and is input into a discriminator together with a saliency image result label to distinguish whether the saliency map is predicted or real, so that the trained image saliency prediction model has strong generalization. Therefore, the present invention can improve the accuracy of image saliency prediction and enhance the generalization ability of image saliency prediction, thereby improving the generation efficiency of saliency images.

[0067] Because the model has strong generalization properties, it can be extended to different data sets and different application scenarios without changing the loss function.

[0068] In one embodiment, the visual image to be processed is subjected to image preprocessing to obtain a preprocessed image, including: converting the color space of the visual image to be processed to obtain a hue channel, a saturation channel, and a lightness channel; performing histogram equalization on the lightness channel to obtain a new lightness channel; combining the new lightness channel, hue channel, and saturation channel to obtain a balanced image; performing edge filling on the balanced image to obtain a filled image; and normalizing the pixel values ​​of each channel of each pixel point in the filled image to obtain the preprocessed image.

[0069] Furthermore, the luminance channel is subjected to histogram equalization to obtain a new luminance channel, including: determining the probability of occurrence of each luminance value in the luminance channel; performing histogram mapping of each luminance value according to each probability value to obtain corresponding new luminance values; and determining a new luminance channel according to each new luminance value.

[0070] When the present invention converts the color space of the visual image to be processed, the color visual image to be processed can be converted into the HSV (hue, saturation, value) color space, thereby obtaining the hue channel, saturation channel and value channel. Among them, the HSV color space is a way to describe color, where H represents hue (0°-360°), S represents saturation (0-1), and V represents value (0-1). For value V: V=0 represents black, V=1 represents white, and the values ​​in between represent different values ​​of value.

[0071] Furthermore, the brightness channel may be subjected to histogram equalization to obtain a new brightness channel. Specifically, the brightness histogram of the visual image to be processed may be calculated based on the information of the brightness channel, thereby determining the probability of occurrence of each brightness value in the brightness channel.

[0072] More specifically, the probability of each brightness value appearing in the brightness channel can be determined according to the following calculation formula:

[0073]

[0074] Among them, M and N represent the number of pixels in length and width of the visual image to be processed, r i represents the brightness value of the i-th level, n i Represents brightness as r i The number of pixels, p r (r i ) represents the brightness value r i Probability of occurrence.

[0075] Furthermore, the corresponding brightness value s after histogram equalization can be calculated by the histogram mapping function T(r): i , the specific calculation formula is:

[0076]

[0077] Among them, s i Represents r i The new brightness value obtained by the mapping function, M·N represents the total number of pixels in the image, n j Represents the j-th level brightness value r j The number of pixels. i

[0078] After rounding off, the new brightness value histogram formula is calculated as follows:

[0079]

[0080] Among them, n' i Represents the brightness value s of the i-th level after histogram equalization i The number of pixels.

[0081] Thus, a plurality of new brightness values ​​are obtained, namely, brightness values ​​after histogram equalization.

[0082] Furthermore, the original brightness value of the brightness channel in the visual image to be processed is replaced by the brightness value after histogram equalization, so as to obtain the brightness channel of the visual image to be processed after equalization, that is, a new brightness channel.

[0083] Furthermore, the present invention can recombines the equalized brightness channel with the H (hue) channel and the S (saturation) channel of the original color image to obtain an equalized image. Furthermore, the present invention can fill the edges of the equalized image with black pixels so that the length and width of the image are the same to obtain a filled image. This operation can add a black border to the equalized image.

[0084] Furthermore, the pixel values ​​of each pixel point and each channel in the padded image can be normalized. This operation can ensure that the image size meets the input requirements of the network model of SalGAN fused with MSI-Net without changing the image features.

[0085] The embodiment of the present invention can better maintain color stability, avoid color shift and distortion, make the image more vivid and visually impactful, improve the display effect of the image, and ensure that the processed image meets the input requirements of the model, so that the image features can be highlighted, which is conducive to the deep learning model to capture image information and improve the visual experience, thereby helping to improve the generation efficiency of saliency images.

[0086] In one embodiment, the image saliency prediction model is determined based on the following method: obtaining a sample visual image and a corresponding image saliency result label; performing image preprocessing on the sample visual image to obtain a sample preprocessed image; inputting the sample preprocessed image into the generator of the initial model to obtain a predicted saliency image output by the generator; inputting the predicted saliency image and its corresponding image saliency result label into the discriminator of the initial model to obtain the adversarial loss of the initial model output by the discriminator; if it is determined that the adversarial loss has not converged or has not reached a preset loss threshold, updating the parameters of the generator and the discriminator respectively; if it is determined that the adversarial loss has converged or reached a preset loss threshold, determining the initial model as the image saliency prediction model.

[0087] Specifically, the present invention further can obtain visual images used for information promotion as sample visual images, for example, poster images can be obtained as sample visual images. And, image saliency result labels are respectively annotated for each sample visual image to obtain image saliency result labels for the corresponding sample visual images.

[0088] Furthermore, image preprocessing may be performed on each sample visual image in the sample data set to obtain a corresponding sample preprocessed image.

[0089] Furthermore, a data set for model training can be established based on each sample preprocessed image and the corresponding image saliency result label. The present invention can further divide the data set into a training set (train_image_dataset) and a validation set (validation_image_dataset) according to a certain ratio (e.g., 8:2).

[0090] Furthermore, the sample preprocessed image obtained by image preprocessing the training set (train_image_dataset) in the dataset according to the above-mentioned image preprocessing method is input into the generator G (generator) of the initial model. The generator G (generator) performs convolution (Conv) and pooling (Pooling) operations on the input image through the encoder (Encoder), reduces the size while increasing the dimension, extracts the hidden feature information in the image, and obtains multi-scale feature information. Furthermore, the multi-scale feature information can be fused through the ASPP module, and upsampled (Upsampling) through the decoder (Decoder) to obtain the predicted saliency image, and the predicted saliency map dataset (total_saliencyMap_predict) is formed by multiple predicted saliency images.

[0091] It should be noted that the present invention calculates the mean square error (MSE) of each image by the corresponding pixel points between the predicted saliency map dataset and the image saliency result label in the dataset (also referred to as the true saliency map dataset (total_saliencyMap_dataset)), and the preliminary content loss can be obtained as follows:

[0092]

[0093] Among them, N is the total number of pixels, S j is the pixel value corresponding to the normalized pixel of the true saliency map, The pixel value corresponding to the normalized pixel in the predicted saliency map.

[0094] At the same time, S jIt can also be interpreted as an estimate of the probability that a particular pixel is noticed by the observer. However, the observer may notice more than one pixel, so each prediction value needs to be considered independent of other prediction values. By adding an element-wise sigmoid to the final output of the generator G, the pixel-level prediction can be regarded as the probability of an independent binary random variable. In this case, binary cross entropy is used as the content cost of the generated result. That is, the average value of the single binary cross entropy (BCE) over all pixels:

[0095]

[0096] The predicted saliency map dataset (total_saliencyMap_predict) and the true saliency map dataset (total_saliencyMap_dataset) are input into the discriminator D (Discriminator) to obtain the adversarial cost of the network model.

[0097] Furthermore, the loss function calculated by combining the error of the discriminator D and the binary cross entropy improves the stability and convergence speed of adversarial training compared to the loss function of the traditional GAN ​​network. Therefore, the final loss function of the network in adversarial training, i.e., the initial model, can be expressed as:

[0098]

[0099] Where L is the binary cross entropy loss function, is the probability that the discriminator is deceived, 1 represents the sample is true, and 0 represents the sample is false; α represents the preset weight coefficient, which can be set and adjusted according to actual needs.

[0100] use Rather than It can provide a stronger gradient to ensure that the loss value of the discriminator D being deceived at the beginning of training is relatively stable. The loss of the discriminator is:

[0101]

[0102] Therefore, the present invention can input the predicted saliency image and its corresponding image saliency result label into the discriminator of the initial model to obtain the adversarial loss of the initial model output by the discriminator.

[0103] Furthermore, it can be determined whether the adversarial loss converges or reaches a preset loss threshold. If so, it can be determined that the model training is completed and an image saliency prediction model is obtained.

[0104] If it is determined that the adversarial loss has not converged or has not reached the preset loss threshold, the loss function gradient can be determined based on the adversarial loss, and the parameters of the generator G and the discriminator D in the model training can be further updated according to the back propagation of the loss function gradient, and the loss function can be continuously minimized so that the generator G and the discriminator D can compete with each other and gradually improve the performance.

[0105] Furthermore, when each round of training is completed, the present invention can verify its loss value (i.e., adversarial loss) by inputting a validation set (validation_image_dataset) to prevent the model from falling into a local optimum. When the loss values ​​of the generator G and the discriminator D converge, the model training is completed and an image saliency prediction model is obtained.

[0106] Figure 3 is a network structure diagram of the training process of the image saliency prediction model provided by the present invention, such as Figure 3 As shown, in one embodiment, after the input image is input to the generator G of the initial model, the generator can output a predicted saliency map, and the predicted saliency map is input to the discriminator D. The discriminator D can output an adversarial cost, determine the gradient of the loss function according to the adversarial loss, and update the parameters of the generator G and the discriminator D in the model training according to the back propagation of the gradient of the loss function, and continuously minimize the loss function, so that the generator G and the discriminator D compete with each other and gradually improve the performance until the model training is completed. In the training process, the binary cross entropy loss (BCE cost) can also be calculated according to the predicted saliency map and the real saliency map.

[0107] The present invention utilizes the MSI-Net network to preliminarily output a predicted saliency result map; and continuously optimizes the output of the MSI-Net network through the idea of ​​generative adversarial generation of the SalGAN network structure. The encoder and decoder structures of the MSI-Net network can extract salient features of the visual image, and the ASPP module can decompose and fuse feature information of different scales in the visual image. The SalGAN network structure can reduce the model's dependence on the loss function when training different data sets, and improve the generalization of the model; and through the generative adversarial method, the convergence speed of the model during training is increased, and the accuracy of the output saliency prediction map is improved, thereby improving the generation efficiency of saliency images.

[0108] In addition, the MSI-Net adopts an encoder-decoder network structure. The image input to MSI-Net is first input to the encoder to extract the semantic feature information of the image pixel level. On this basis, after being processed by MSI-Net, the encoded image information is decoded and reconstructed. By extracting the semantic information of each pixel in the image, pixel-level saliency prediction can be achieved.

[0109] In one embodiment, after obtaining the saliency image of the visual image to be processed output by the image saliency prediction model, the method further includes: obtaining a binary image of the saliency image; determining an eye fixation point saliency evaluation index of the saliency image according to the saliency image and the binary image; determining an object saliency evaluation index of the saliency image according to the saliency image and the real saliency image of the visual image to be processed; and determining an evaluation value of the saliency image according to the object saliency evaluation index and the eye fixation point saliency evaluation index.

[0110] After obtaining the saliency image of the visual image to be processed output by the image saliency prediction model, the present invention can also evaluate the saliency prediction result. Specifically, the generated saliency image can be analyzed and evaluated by two scales: object saliency detection and eye fixation saliency.

[0111] More specifically, a binary image of the saliency image may be obtained first.

[0112] Furthermore, the Normalized Scanpath Saliency (NSS) was selected as the evaluation index of the gaze point saliency. The larger the NSS, the better the model performance.

[0113] Specifically, the eye fixation point saliency evaluation index of the saliency image can be determined based on the saliency image and the binary image in combination with the following calculation formula:

[0114]

[0115] Among them, P is the saliency image, Q is the saliency image after binarization, i is the subscript of the pixel, and Q i is the pixel value of the i-th pixel in the binary image, and N is the total number of pixel values ​​in the image.

[0116]

[0117] Among them, P i is the saliency value of the i-th pixel in the saliency image, μ is the average saliency value of a randomly selected reference saliency map (usually 0), and σ is the standard deviation of the saliency value corresponding to the subject's eye movement data.

[0118] The present invention can select F β The score is used as an evaluation index of object saliency, F β The larger the model, the better the performance.

[0119] Therefore, the object saliency evaluation index of the saliency image can be determined based on the saliency image and the real saliency image of the visual image to be processed, combined with the following calculation formula:

[0120]

[0121] Among them, β 2 Generally, it is set to 0.3 to ensure that the weight of the precision rate is higher than the recall rate, because when the model marks all the output images as target areas, the recall rate is 100%, but the precision rate is very low. The calculation formula of the precision rate and the recall rate is:

[0122]

[0123] Among them, TP (True Positive) refers to the number of positive examples correctly predicted by the model, FP (False Positive) refers to the number of negative examples incorrectly predicted as positive examples by the model, and FN (False Negative) refers to the number of positive examples incorrectly predicted as negative examples by the model.

[0124] Furthermore, the present invention can combine the eye gaze point significance evaluation index NSS with the object significance evaluation index F β The scores are normalized respectively.

[0125] Furthermore, according to the type of input image (for example, the matching weights of the two evaluation indicators of the corresponding type can be set in advance according to whether the interface image is rich in layers), the matching weights of the two evaluation indicators are determined, and then the evaluation value of the saliency image output by the model is obtained by weighted combination of the normalized evaluation indicators.

[0126] The present invention is based on the eye gaze point evaluation index NSS and the object significance evaluation index F β The scores are normalized and weighted to evaluate the accuracy of the significance prediction map, thereby evaluating the practicality of the model. By adjusting the weight parameters of the two evaluation indicators according to the actual application scenario, the prediction accuracy of the model in various scenarios can be accurately evaluated.

[0127] In one embodiment, after obtaining the saliency image of the visual image to be processed output by the image saliency prediction model, it also includes: performing visual image registration on the visual image to be processed according to the saliency image; the visual image registration includes any one of element size adjustment, visual image alignment, image space correction after element size adjustment, image space correction after visual image alignment, or visual image alignment and image space correction after element size adjustment.

[0128] After obtaining the saliency image of the visual image to be processed output by the image saliency prediction model, the embodiment of the present invention can also perform visual image registration on the visual image to be processed according to the saliency image.

[0129] Specifically, the present invention may only perform element size adjustment on the visual image to be processed, may only perform visual image alignment on the visual image to be processed, may further perform image space correction after performing element size adjustment on the visual image to be processed, may further perform image space correction after performing visual image alignment on the visual image to be processed, may also perform element size adjustment and visual image alignment on the visual image to be processed, and further perform image space correction after completing the element size adjustment and visual image alignment.

[0130] Specifically, when adjusting the size of elements, the eye fixation area in the visual image to be processed can be determined by analyzing the saliency prediction result map. This area is crucial to the overall visual effect and information communication, and needs to be retained and emphasized during the resizing process. According to the guidance of the saliency image, the elements in the visual image to be processed are cropped and scaled to ensure that important content areas are retained and highlighted while maintaining the overall visual balance. According to the guidance of the saliency image, the size and position of the eye fixation area in the visual image to be processed are further adjusted to make it more prominent and eye-catching, thereby improving the user experience.

[0131] When aligning visual images, elements of different perspectives or levels in the visual image to be processed can be aligned to ensure that they are consistent in space, so that users can more intuitively compare and understand the differences or connections between these elements. Specifically, baseline alignment methods or image feature matching technology can be used. Among them, the application of baseline alignment methods can further calibrate elements with different stacking relationships and different scales, and eliminate transformation differences such as displacement and rotation. Image feature matching technology is used to achieve a more natural alignment effect by finding similar features of similar elements in the visual image.

[0132] When performing image space correction, for element tilt caused by element size adjustment, the element in the visual image to be processed is transformed to keep the element horizontal or vertical in space. For spatial distortion caused by element stretching and cropping, the distortion correction tool is used to repair it. For areas with severe perspective deformation, the perspective transformation tool is used to adjust the perspective effect of the area to make it smoother and more realistic. While performing image space correction, the blank areas or irregular edges generated after correction are removed through image cropping, and redundant elements are eliminated to achieve visual image alignment.

[0133] The embodiment of the present invention achieves highlighting of the user's area of ​​interest in the visual image, elimination of redundant parts, and improvement of the overall visual sense of the image through three steps of element size adjustment, visual image alignment, and image space correction. It can enhance the visual impact and memorability of the image and improve the visual image delivery effect.

[0134] Based on any of the above embodiments, Figure 4 It is a schematic structural diagram of the three-dimensional reconstruction device provided by the present invention.

[0135] like Figure 4 As shown, the device comprises:

[0136] The acquisition module 401 is used to acquire a visual image to be processed; the visual image to be processed is acquired based on an optical collector and / or an acoustic collector;

[0137] A preprocessing module 402 is used to perform image preprocessing on the visual image to be processed to obtain a preprocessed image;

[0138] The input module 403 is used to input the preprocessed image into the image saliency prediction model to obtain the saliency image of the visual image to be processed output by the image saliency prediction model; wherein the image saliency prediction model is obtained by training an initial model based on a sample visual image and a corresponding image saliency result label; the initial model is formed by integrating the dilated pyramid pooling module in the multi-scale information network in the generator of the saliency generative adversarial network.

[0139] The photoacoustic imaging device provided by the embodiment of the present invention obtains an image saliency prediction model by pre-training an initial model based on a sample visual image and a corresponding image saliency result label. Therefore, after obtaining a visual image to be processed and performing image preprocessing on the visual image to be processed, the obtained preprocessed image is input into the image saliency prediction model, and a saliency image of the visual image to be processed output by the image saliency prediction model can be obtained. Since the initial model is formed after fusing a dilute pyramid pooling module in a multi-scale information network in a generator of a saliency generative adversarial network, the dilute pyramid pooling module can extract and fuse multi-scale feature information of the input image to achieve fine-grained division of the predicted saliency image, thereby solving the problem of oversimplification of image feature extraction and improving the accuracy of image saliency prediction. At the same time, through the idea of ​​generative adversarial generation in the saliency generative adversarial network, the process of generating a saliency map from the input image is used as a generator part, and is input into a discriminator together with a saliency image result label to distinguish whether the saliency map is predicted or real, so that the trained image saliency prediction model has strong generalization. Therefore, the present invention can improve the accuracy of image saliency prediction and enhance the generalization ability of image saliency prediction, thereby improving the generation efficiency of saliency images.

[0140] See also Figure 5 , Figure 5 Schematic diagram of an embodiment of the electronic device provided by the present invention. Figure 5 As shown, an embodiment of the present invention provides an electronic device 500, including a memory 510, a processor 520, and a computer program 511 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 511, the following steps are implemented:

[0141] Acquire a visual image to be processed; the visual image to be processed is acquired based on an optical collector and / or an acoustic collector;

[0142] Performing image preprocessing on the visual image to be processed to obtain a preprocessed image;

[0143] The preprocessed image is input into an image saliency prediction model to obtain a saliency image of the visual image to be processed output by the image saliency prediction model; wherein the image saliency prediction model is obtained by training an initial model based on a sample visual image and a corresponding image saliency result label; and the initial model is formed by integrating a dilated pyramid pooling module in a multi-scale information network into a generator of a saliency generative adversarial network.

[0144] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0145] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0147] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0149] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0150] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A photoacoustic imaging method, characterized in that: include: Acquire a visual image to be processed; The visual image to be processed is acquired based on an optical collector and / or an acoustic collector; Performing image preprocessing on the visual image to be processed to obtain a preprocessed image; The preprocessed image is input into an image saliency prediction model to obtain a saliency image of the visual image to be processed output by the image saliency prediction model; wherein the image saliency prediction model is obtained by training an initial model based on a sample visual image and a corresponding image saliency result label; and the initial model is formed by integrating a dilated pyramid pooling module in a multi-scale information network into a generator of a saliency generative adversarial network.

2. The photoacoustic imaging method according to claim 1, characterized in that: The image saliency prediction model is determined based on the following method: Obtain sample visual images and corresponding image saliency result labels; Performing image preprocessing on the sample visual image to obtain a sample preprocessed image; Inputting the sample preprocessed image into the generator of the initial model to obtain a predicted saliency image output by the generator; Inputting the predicted saliency image and its corresponding image saliency result label into the discriminator of the initial model to obtain the adversarial loss of the initial model output by the discriminator; If it is determined that the adversarial loss has not converged or has not reached a preset loss threshold, updating parameters of the generator and the discriminator respectively; If it is determined that the adversarial loss converges or reaches a preset loss threshold, the initial model is determined as the image saliency prediction model.

3. The photoacoustic imaging method according to claim 2, characterized in that: The generator of the initial model includes at least an encoder, a dilated pyramid pooling module and a decoder.

4. The photoacoustic imaging method according to claim 1, characterized in that: The performing image preprocessing on the visual image to be processed to obtain a preprocessed image includes: Converting the color space of the visual image to be processed to obtain a hue channel, a saturation channel and a lightness channel; Performing histogram equalization on the brightness channel to obtain a new brightness channel; Combining the new lightness channel, the hue channel and the saturation channel to obtain a balanced image; Fill the edges of the equalized image to obtain a filled image; The pixel value of each channel of each pixel point in the filled image is normalized to obtain a preprocessed image.

5. The photoacoustic imaging method according to claim 4, characterized in that: The performing histogram equalization on the brightness channel to obtain a new brightness channel includes: Determining the probability of occurrence of each brightness value in the brightness channel; According to each probability value, a histogram mapping of each brightness value is performed to obtain corresponding new brightness values; A new luminance channel is determined based on each new luminance value.

6. The photoacoustic imaging method according to any one of claims 1 to 5, characterized in that: After obtaining the saliency image of the visual image to be processed output by the image saliency prediction model, the method further includes: Visual image registration is performed on the visual image to be processed according to the saliency image; the visual image registration includes any one of element size adjustment, visual image alignment, image space correction after element size adjustment, image space correction after visual image alignment, or image space correction after visual image alignment and element size adjustment.

7. The photoacoustic imaging method according to any one of claims 1 to 5, characterized in that: After obtaining the saliency image of the visual image to be processed output by the image saliency prediction model, the method further includes: Obtaining a binary image of the saliency image; Determining a saliency evaluation index of an eye fixation point of the saliency image according to the saliency image and the binarized image; Determining an object saliency evaluation index of the saliency image according to the saliency image and a real saliency image of the visual image to be processed; An evaluation value of the saliency image is determined according to the object saliency evaluation index and the eye fixation point saliency evaluation index.

8. A photoacoustic imaging device, characterized in that: include: An acquisition module, used for acquiring a visual image to be processed; The visual image to be processed is acquired based on an optical collector and / or an acoustic collector; A preprocessing module, used for performing image preprocessing on the visual image to be processed to obtain a preprocessed image; An input module is used to input the preprocessed image into an image saliency prediction model to obtain a saliency image of the visual image to be processed output by the image saliency prediction model; wherein the image saliency prediction model is obtained by training an initial model based on a sample visual image and a corresponding image saliency result label; and the initial model is formed by integrating a dilated pyramid pooling module in a multi-scale information network into a generator of a saliency generative adversarial network.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the photoacoustic imaging method according to any one of claims 1 to 7 when executing the program.

10. A non-transitory computer-readable storage medium, characterized in that: The storage medium stores a computer software program, and when the computer software program is executed by the processor, the photoacoustic imaging method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Non-uniform illumination color image enhancement method

    CN110047051A

  • Light field saliency target detection method based on generative adversarial convolutional neural network

    CN111369522A

  • Method and device for tracking object in video, electronic equipment and storage medium

    CN115100240A

  • Human voice accompaniment separation method, network training method, equipment and storage medium

    CN116778946A