A method, system and medium for out-of-focus image generation and gaze target identification based on deep learning

Abstract

The application discloses a kind of methods, systems and media based on defocus image generation and gaze target identification of deep learning, it is related to biomedical engineering and medical imaging service field, including: using pre-trained monocular metric depth estimation model, the absolute depth map reflecting scene real physical distance is predicted to RGB image;Utilize pre-trained visual saliency detection model to carry out feature analysis to RGB image, generate pixel-level saliency map;Based on absolute depth map and saliency map, the defocus amount distribution of whole image is calculated, and defocus image is obtained;Determine high saliency area coordinate set based on saliency map, and draw boundary box on RGB image with this, obtain labeled image;The object name or semantic description of gaze target is identified and output by visual language model by inputting labeled image and specific instruction into visual language model.The application is realized to human eye visual perception process from physical perception to semantic cognitive understanding full-link simulation using pure software mode.

Description

Technical Field

[0001] This application relates to the fields of biomedical engineering and medical imaging services, and in particular to a method, system and medium for generating defocused images and recognizing gaze targets based on deep learning. Background Technology

[0002] In fields such as virtual reality display, myopia prevention mechanism research, and human-computer interaction, accurately simulating the human eye's visual perception process in natural scenes is of great significance. This process includes not only the physical perception of the scene's three-dimensional structure (depth and defocus) but also the semantic cognition of the visual focus. The human visual system has a "foveal fixation" characteristic, meaning that when observing a scene, the human eye only forms a clear image of the fixated point, while non-fixated areas are blurred (defocused). This mechanism not only helps the brain quickly obtain key information from complex scenes through differences in sharpness but also reflects the observer's perception of three-dimensional spatial depth.

[0003] To simulate the aforementioned visual perception process, current technical solutions largely rely on a combination of hardware devices. Typically, researchers use eye trackers to capture the observer's gaze point in real time, while simultaneously using depth cameras to acquire depth information of the scene. This allows them to calculate the defocus amount and obtain a defocused image, thus simulating the defocusing effect of the human eye. However, these solutions have significant limitations in practical applications: firstly, their over-reliance on hardware devices such as eye trackers and depth cameras results in high costs and poor portability; secondly, the lack of semantic understanding prevents accurate semantic understanding of the gaze target, limiting their application scope. Summary of the Invention

[0004] The purpose of this application is to provide a method, system, and medium for generating defocused images and recognizing gazed targets based on deep learning. It adopts a pure software approach, using depth estimation and saliency detection to generate defocused images that conform to the physiological characteristics of the human eye. At the same time, it combines a visual language model to accurately recognize gazed targets, thereby realizing the full-link simulation of the human eye's visual perception process from physical perception to semantic cognition and understanding.

[0005] To achieve the above objectives, this application provides the following solution.

[0006] In a first aspect, this application provides a method for generating defocused images and recognizing gazed targets based on deep learning. The method includes: acquiring a single RGB image to be processed; the RGB image being a scene image representing human observation; inputting the RGB image into a pre-trained monocular depth estimation model, and using the pre-trained monocular depth estimation model to predict an absolute depth map reflecting the true physical distance of the scene; inputting the RGB image into a pre-trained visual saliency detection model, and using the pre-trained visual saliency detection model to perform feature analysis on the RGB image to generate a pixel-level saliency map; both the monocular depth estimation model and the visual saliency detection model are deep learning models; calculating the defocus distribution of the entire image based on the absolute depth map and the saliency map according to the principle of geometrical optics imaging to obtain a defocused image; determining a set of coordinates of highly saliency regions based on the saliency map, and using the set of coordinates of highly saliency regions to draw bounding boxes on the RGB image to obtain an annotated image; inputting the annotated image and specific instructions into a visual language model, and using the visual language model to identify and output the object name or semantic description of the gazed target.

[0007] In a second aspect, this application also provides a computer system, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the deep learning-based defocus image generation and gaze target recognition method described in the first aspect.

[0008] Thirdly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the deep learning-based defocus image generation and gaze target recognition method described in the first aspect.

[0009] Based on the specific embodiments provided in this application, the following technical effects are disclosed.

[0010] First, this application no longer relies on eye trackers to capture fixation points and depth cameras to obtain scene depth to simulate human vision. Instead, it employs a purely software-based approach. First, a pre-trained monocular metric depth estimation model is used to infer an absolute depth map reflecting the true physical distance from a single RGB image, providing a precise spatial geometric reference for subsequent defocus calculations. Then, a pre-trained visual saliency detection model is used to simulate the human visual cortex's response mechanism to RGB image features and high-level semantics, generating a pixel-level saliency map. Finally, through the aforementioned depth estimation and saliency detection, a defocused image conforming to the physiological characteristics of the human eye is generated, effectively restoring physical visual perception. Second, this application also uses the saliency map to determine a set of highly saliency region coordinates and labels the image accordingly. Then, the labeled image and specific instructions are used as prompts input to the Vision Language Model (VLM) to guide semantic understanding and accurately identify the physical information of the fixated target. Therefore, this application, using a purely software approach, achieves a full-link simulation of the human visual perception process from physical perception to semantic cognitive understanding. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of the deep learning-based defocus image generation and gaze target recognition method in the embodiments of this application.

[0013] Figure 2 This is a diagram showing the internal structure of the computer system in an embodiment of this application. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] This application mainly focuses on making specific improvements to the following technical aspects.

[0016] (1) Monocular depth estimation technique: Traditional monocular depth estimation is often limited by the scale ambiguity problem of "relative depth" (i.e., it can only determine the front-to-back relationship of objects, but cannot know the specific distance). This application adopts a monocular depth estimation model trained on large-scale data. This technique can directly infer the absolute depth map reflecting the scale of the real physical world from a single RGB image through feature alignment and scale recovery algorithms. This provides an accurate spatial geometric benchmark for subsequent calculation of defocus based on optical imaging principles, and overcomes the hardware limitations of traditional methods that rely on binocular cameras or depth sensors.

[0017] (2) Saliency Detection Technology: In order to simulate human eye fixation behavior without an eye tracker, this application introduces saliency detection technology. This technology uses a deep neural network to simulate the response mechanism of the human visual cortex to image features (such as color, brightness, and edges) and high-level semantics (such as faces and text), generating a pixel-level saliency (probability distribution) map. This technology solves the problem of fixation point prediction, enabling the system to automatically identify the region in the image that is most attractive to the observer as the fixation region.

[0018] (3) Semantic understanding technology of visual language model based on saliency mask cues: Traditional image recognition classifies the entire image, which cannot reflect the selective attention process of "gaze". This application innovatively uses saliency maps to draw bounding boxes and inputs the processed image as a visual cue into the VLM. This technology forces the multimodal large model to focus on the simulated human eye gaze area, thereby accurately identifying "what the observer is looking at" and realizing deep semantic analysis of visual attention content.

[0019] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] In one exemplary embodiment, a deep learning-based method for generating defocused images and recognizing gazed targets is provided, such as... Figure 1 As shown, this deep learning-based method for generating defocused images and recognizing gazed targets includes: Step S1: Obtain a single RGB image to be processed.

[0021] In this embodiment, the single RGB (color) image to be processed To characterize the scene image observed by the human eye, its resolution is , Indicates the image width. This indicates the image height. The image can come from an internet database, local storage, or be captured by a monocular camera, without requiring additional depth sensor or eye tracker hardware input.

[0022] Step S2: Input the RGB image into the pre-trained monocular metric depth estimation model, and use the pre-trained monocular metric depth estimation model to predict the absolute depth map that reflects the real physical distance of the scene.

[0023] In this embodiment, the monocular depth estimation model adopts the Depth Anything v2 model. This model can overcome the scale uncertainty problem in traditional monocular depth estimation and output an absolute depth map that reflects the true physical distance of the scene. .in, For pixels The absolute depth, representing the pixel. The physical distance from the camera, in meters.

[0024] The pre-training process of the monocular metric depth estimation model includes: using publicly available large-scale monocular RGB image data, synthetic scene data, and RGB-D data with camera calibration or depth sensor labels as training data; during training, using a single RGB image / synthetic scene as input and the corresponding metric depth map or a depth label obtained through calibration as supervised output; constraining the consistency between predicted depth and true depth through depth regression loss, and combining scale consistency loss, edge smoothing loss, and gradient consistency loss to maintain the alignment of depth boundaries with image structure. After pre-training, the model can output absolute depth estimation results with physical scale significance for ordinary monocular RGB images without a depth sensor.

[0025] Step S3: Input the RGB image into the pre-trained visual saliency detection model, and use the pre-trained visual saliency detection model to perform feature analysis on the RGB image to generate a pixel-level saliency map.

[0026] In this embodiment, the visual saliency detection model adopts the DeepGaze IIIE model, which can simulate the human visual system's response to color, contrast, and semantic targets in a scene, and output pixel-level saliency maps. .in, Represents pixels The significance value, with a range of values ​​being [value range missing]. It is used to simulate the visual attention mechanism of the human eye, identify the regions in an image that the human eye is most likely to focus on, and the higher the salience value, the greater the likelihood that the region will attract visual attention.

[0027] The pre-training process of the visual saliency detection model includes: using natural scene images from the SALICON dataset and pseudo-foveation point data collected by the observer's mouse trajectory as pre-training data, and fine-tuning on real eye-tracking foveation point data from the MIT1003 dataset; the model uses a CNN backbone network pre-trained on ImageNet to extract image features, and the backbone network parameters remain fixed during the saliency training phase, training only the subsequent readout network, blur kernel parameters, and center bias weights; during training, RGB images are used as input, and a two-dimensional foveation probability density distribution is output through softmax, with training labels being manually labeled foveation point coordinates; the loss function uses negative log-likelihood loss, which minimizes the negative log probability of the artificial foveation point in the model's predicted foveation probability density distribution, enabling the model to assign a higher prediction probability at the real foveation point location.

[0028] Based on the above, steps S2 and S3 achieve parallel prediction of depth and saliency based on a deep learning model.

[0029] Step S4: Based on the absolute depth map and saliency map, calculate the defocus distribution of the entire image according to the principle of geometric optics imaging to obtain the defocused image.

[0030] In this embodiment, "full image" refers to the entire RGB image, and the RGB image, absolute depth map, and saliency map are all the same size.

[0031] Step S4 includes: calculating the distance to the gaze plane and calculating the pixel-level defocus amount.

[0032] (1) Calculation of distance to the gaze plane.

[0033] Step 1, Adaptive Thresholding: Calculate the saliency map The maximum significance value in With the maximum significance value 80% as the threshold for significance plot Perform binarization to generate the first binarized mask. This mask is used to determine the gaze plane in subsequent defocus calculations.

[0034] The second step is connected component separation: This involves applying the first binarization mask... Perform connected component analysis algorithm to transform the first binarized mask. The foreground pixels in the image are separated into multiple independent connected regions (i.e., highly salient regions).

[0035] .

[0036] In the formula, It is a set of connected components; For the first i Connected components, The number of connected components.

[0037] The third step is to calculate the overall saliency within each connected component (i.e., the sum of the saliency values ​​of all pixels within the connected component), and select the connected component with the highest overall saliency as the final fixation region. .

[0038] .

[0039] In the formula, For the first i Connected components Overall saliency. This step ensures that the selected region not only contains highly salient points, but also possesses overall visual attention.

[0040] Fourth step, in the fixation area Inside, using absolute depth maps Significance plot Calculate the gaze area The depth-weighted average value is used as the final gaze distance.

[0041] .

[0042] In the formula, The fixation distance is the physical distance that corresponds to the plane of the human eye's current gaze.

[0043] (2) Pixel-level defocus calculation.

[0044] Based on absolute depth map and gaze distance Based on the principles of geometric optics imaging, the defocus amount of each pixel in the entire image is calculated to obtain the defocused image. .

[0045] .

[0046] In the formula, For pixels The amount of defocus, also known as the number of pixels in the scene. Difference in refractive power relative to the gaze plane.

[0047] The final defocused image It can be used to determine the degree of blur of pixels on the retina in the current scene. For out-of-focus images. Pixel The defocus amount; when this value is positive, it indicates hyperopic defocus, and the number of pixels... Located in front of the gaze plane; when this value is negative, it indicates myopic defocus, pixel point Located behind the gaze plane; the larger the absolute value of this value, the more pixels there are. The more blurred the image on the retina.

[0048] Step S5: Determine the set of coordinates of highly saliency regions based on the saliency map, and draw bounding boxes on the RGB image using the set of coordinates of highly saliency regions to obtain the labeled image.

[0049] In this embodiment, step S5 is as follows.

[0050] Step 1, binarization: using the maximum significance value 55% as the threshold for significance plot Perform binarization processing to generate a second binarized mask. This step aims to preserve regions with high saliency in the image as candidate saliency regions for subsequent contour extraction, while the second binarization mask... Used for subsequent gaze target recognition, in conjunction with the aforementioned first binarized mask used to determine the gaze plane. Distinguish between them.

[0051] The second step is contour extraction: This involves applying the second binarized mask... Perform a contour search algorithm to extract the outer contours of all connected components. Sort the outer contours of all connected components in descending order based on the area enclosed by their respective outer contours, and retain the top ones. k The outer contour. This step is mainly to focus on the primary object of gaze. k It can be preset according to the application scenario.

[0052] The third step is to iterate through the previous steps. k For each outer contour, calculate the bounding rectangle of each outer contour and use it as the bounding box (rectangle). Retain areas whose area percentage exceeds a preset threshold. The bounding box is used to obtain a set of coordinates of highly salient regions.

[0053] .

[0054] .

[0055] In the formula, This is the set of coordinates for highly significant regions. For the first i The coordinates of the bounding box, also known as the coordinates of the highly salient region; For the first i The coordinates of the top left corner of the bounding box; and The first i The width and height of each bounding box; This represents the number of valid bounding boxes. Additionally, the area percentage is the ratio of the area of ​​the circumscribed rectangle to the total area of ​​the map.

[0056] The fourth step is to use the set of coordinates of highly significant regions. In RGB images Draw a bounding box on top to obtain the labeled image. .

[0057] Step S6: Input the labeled image and specific instructions into the visual language model, and use the visual language model to identify and output the object name or semantic description of the gaze target.

[0058] In this embodiment, the visual language model adopts Qwen-VL, which is used to annotate the image. While providing visual input to Qwen-VL, it is also necessary to construct text prompts for specific instructions in order to enable the annotation of the image. Semantic parsing of each gaze region (i.e., bounding box). The specific instructions here mainly concern the textual requirements for the final semantic parsing, such as "keep the textual description simple" or "use words to summarize as much as possible."

[0059] In summary, this application has the following main advantages.

[0060] (1) Existing technologies typically rely on eye trackers to capture gaze points and depth cameras to obtain scene depth to simulate human vision. These devices are expensive and have poor portability. This application innovatively proposes a purely software-based alternative based on monocular depth estimation and saliency detection. By using a deep learning model to predict physical depth and gaze point location from a single RGB image, it realizes the complete process of reproducing human physical imaging at low cost on ordinary consumer devices (such as mobile phones and monocular cameras).

[0061] (2) This application innovatively introduces VLM and proposes a semantic recognition technology based on saliency cues, which not only simulates the physical perception of the human eye, but also uses the generated visual tags to guide the large model to perform semantic understanding and accurately identify object information in highly saliency regions. This design simulates the complete link from photophysical imaging to brain semantic cognition, filling the gap in the semantic understanding level of existing visual simulation technology.

[0062] In another exemplary embodiment, a computer system is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 2As shown, the computer system includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores RGB images. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When executed by the processor, the computer program implements a deep learning-based method for generating defocused images and recognizing gazed targets.

[0063] Those skilled in the art will understand that Figure 2 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer system to which the present application is applied. A specific computer system may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0064] In another exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0065] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0066] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0067] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0068] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0069] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for generating defocused images and recognizing gazed targets based on deep learning, characterized in that, The deep learning-based method for generating defocused images and recognizing gazed targets includes: Acquire a single RGB image to be processed; the RGB image is a scene image representing human visual observation; The RGB image is input into a pre-trained monocular metric depth estimation model, and the pre-trained monocular metric depth estimation model is used to predict an absolute depth map that reflects the true physical distance of the scene. The RGB image is input into a pre-trained visual saliency detection model, and the pre-trained visual saliency detection model is used to perform feature analysis on the RGB image to generate a pixel-level saliency map; both the monocular metric depth estimation model and the visual saliency detection model are deep learning models. Based on the absolute depth map and the saliency map, the defocus distribution of the entire image is calculated according to the geometric optics imaging principle to obtain the defocused image; Based on the saliency map, a set of coordinates of highly saliency regions is determined, and bounding boxes are drawn on the RGB image using the set of coordinates of highly saliency regions to obtain an annotated image; The labeled image and specific instructions are input into the visual language model, which is then used to identify and output the object name or semantic description of the gaze target.

2. The method for generating defocused images and recognizing gazed targets based on deep learning according to claim 1, characterized in that, Based on the absolute depth map and the saliency map, the defocus distribution of the entire image is calculated according to the geometric optics imaging principle to obtain a defocused image, specifically including: Calculate the maximum saliency value in the saliency map, and use 80% of the maximum saliency value as a threshold to binarize the saliency map to generate a first binarization mask; A connected component analysis algorithm is performed on the first binarized mask to separate the foreground pixels in the first binarized mask into multiple independent connected components; Calculate the overall saliency within each connected component, and select the connected component with the largest overall saliency as the final fixation region; Using the absolute depth map and the saliency map, the depth-weighted average value of the gaze region is calculated and used as the final gaze distance; Based on the absolute depth map and the gaze distance, the defocus amount of each pixel in the entire image is calculated according to the geometric optics imaging principle to obtain a defocused image.

3. The method for generating defocused images and recognizing gazed targets based on deep learning according to claim 2, characterized in that, The formula for calculating the overall significance is: ； In the formula, For the first i Connected components The overall significance; For pixels The significance value.

4. The method for generating defocused images and recognizing gazed targets based on deep learning according to claim 2, characterized in that, The formula for calculating the gaze distance is: ； In the formula, The viewing distance; For pixels The absolute depth; For pixels The significance value; The area to be gazed at.

5. The method for generating defocused images and recognizing gazed targets based on deep learning according to claim 2, characterized in that, The formula for calculating the defocus amount is: ； In the formula, For pixels The amount of defocus; For pixels The absolute depth; The viewing distance.

6. The method for generating defocused images and recognizing gazed targets based on deep learning according to claim 2, characterized in that, Based on the saliency map, a set of coordinates for highly saliency regions is determined. Bounding boxes are then drawn on the RGB image using these coordinates to obtain a labeled image. Specifically, this includes: The saliency map is binarized using 55% of the maximum saliency value as a threshold to generate a second binarization mask; A contour-finding algorithm is applied to the second binarized mask to extract the outer contours of all connected components. The outer contours of all connected components are then sorted in descending order based on the area enclosed by their respective outer contours, and the first few contours are retained. k The outer contour; Before traversal k For each outer contour, calculate the bounding rectangle of each outer contour and use it as a bounding box. Retain the bounding boxes whose area ratio exceeds a preset threshold to obtain a set of highly saliency region coordinates; the area ratio is the ratio of the area of ​​the bounding rectangle to the area of ​​the entire image. Based on the set of coordinates of the highly saliency regions, a bounding box is drawn on the RGB image to obtain the labeled image.

7. The method for generating defocused images and recognizing gazed targets based on deep learning according to claim 6, characterized in that, The expression for the set of coordinates of the highly significant region is: ； ； In the formula, This is the set of coordinates for highly significant regions. For the first i The coordinates of the bounding box; For the first i The coordinates of the top left corner of the bounding box; and The first i The width and height of each bounding box; The number of valid bounding boxes.

8. The method for generating defocused images and recognizing gazed targets based on deep learning according to claim 1, characterized in that, The monocular depth estimation model is the Depth Anything v2 model; the visual saliency detection model is the DeepGaze IIE model.

9. A computer system, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the deep learning-based method for generating defocused images and recognizing gazed targets according to any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the deep learning-based method for generating defocused images and recognizing gazed targets as described in any one of claims 1-8.

Images