Systems and methods for masking query input driven visual search

Abstract

An image masking system (100) is disclosed. The system (100) includes a segmentation module (130) configured to extract one or more regions of interest by processing an input query image using a segmentation network. The segmentation module (130) is further configured to select one or more segments from the one or more extracted regions of interest based on a user indicator. The system (100) includes a mask generation module (140) configured to generate a binary mask for the selected segments on the input query image. The system (100) further includes an image transformer module (150) configured to generate a masked image by performing an arithmetic operation on the binary mask and the input query image. The image transformer module (150) is further configured to extract an embedding of the masked image using one of a convolutional neural network architecture or a Transformer architecture.

Description

Technical Field

[0001] Embodiments of this disclosure relate to an image-based information retrieval system, and more specifically, to an image masking system and method thereof. Background Technology

[0002] In image-based search engines, the retrieval process primarily relies on users providing images as input. The system then attempts to find visually similar images from a database. Such systems have been deployed in various applications, including multimedia search, surveillance, medical imaging, and autonomous driving. However, the effectiveness of these systems is often limited by their inability to interpret the specific intent of users when uploading images.

[0003] Current image search systems process the input image holistically, searching for visually similar images without distinguishing specific regions or features of interest. However, in this approach, the search engine treats all objects in the image as equally important, leading to irrelevant results.

[0004] Many use cases, such as autonomous driving, require the identification of specific aspects of an image (e.g., lane markings or shadows). However, existing systems lack the ability to focus on these specific areas because they cannot generate region-specific masks. Additional objects in the image increase computational inefficiency and reduce the relevance of retrieval results.

[0005] Therefore, an improved image masking system and method are needed to solve the above problems. Summary of the Invention

[0006] According to embodiments of this disclosure, a system and method for masking images are provided. The system provides a target visual input image for a search process by masking irrelevant regions. The system receives an input query image. Furthermore, the image is segmented and the system extracts regions of interest (ROIs). The system then selects segments from the extracted ROIs based on user indicators and generates a binary mask for the selected segments. The system also generates a masked image and extracts the embeddings of the masked image using at least one of a convolutional neural network architecture or a Transformer architecture. Attached Figure Description

[0007] Embodiments of the present invention are described with reference to the following figures: Figure 1 This is a block diagram representation of an image masking system according to an embodiment of the present disclosure; Figure 2 According to embodiments of this disclosure Figure 1 Exemplary embodiments of a system for masking images; and Figure 3This is a flowchart illustrating the steps included in a method for masking an image according to an embodiment of the present disclosure. Detailed Implementation

[0008] The invention will now be described by way of example with reference to the accompanying drawings. Throughout the drawings, the same or corresponding elements are generally indicated by the same reference numerals. These illustrated embodiments should be understood as illustrative of the invention and not as limiting the invention in any way. It should also be understood that these drawings are not necessarily drawn to scale, and these embodiments are sometimes shown using graphic symbols, dashed lines, schematic diagrams, and partial views. In certain circumstances, details that are not essential for understanding the invention or that would obscure other details may be omitted.

[0009] Embodiments of this disclosure can be implemented by software (e.g., a program or application) containing one or more instructions stored in a storage medium (e.g., memory) that are machine-readable. For example, a processor of the machine can invoke at least one of the one or more instructions stored in the storage medium and can execute that instruction. This makes it possible for at least one function to be performed according to the invoked at least one instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transient storage medium. Here, "non-transient storage medium" means only that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), and for the purposes of this terminology, there is no distinction between the case where data is semi-permanently stored in a storage medium and the case where data is temporarily stored in a storage medium.

[0010] Embodiments of this disclosure relate to an image masking system and method. As used herein, "image masking" is a technique used to separate or isolate specific regions of an image from the rest, thereby providing more accurate results for visual search. Essentially, it is equivalent to placing a "mask" on certain portions of an input image to protect or hide those regions while exposing other regions. In this disclosure, image masking incorporates two main concepts: image segmentation and object extraction, which are combined to obtain the image mask. As used herein, image segmentation generally refers to the process of dividing a digital image into multiple image segments, image regions, or image objects (sets of pixels). Similarly, object extraction generally refers to the process of extracting foreground objects from a segmented image. According to embodiments of this disclosure, image masking is described in detail below with reference to these concepts.

[0011] Figure 1This is a block diagram representation of an image masking system (100) according to an embodiment of the present disclosure. The system (100) includes a processing subsystem (110). As used herein, the processing subsystem (110) refers to any type of computing circuitry, such as, but not limited to, a microprocessor, microcontroller, complex instruction set computing microprocessor, simplified instruction set computing microprocessor, very long instruction word microprocessor, explicit parallel instruction computing microprocessor, digital signal processor, or any other type of processing circuitry, or combinations thereof. The processing subsystem (110) is hosted on a server (120). In one embodiment, the server (120) may include a cloud server. In another embodiment, the server (120) may include a local server. The processing subsystem (110) is configured to operate on a network ( Figure 1 The network (not shown) is executed on a device to control bidirectional communication between multiple modules. In one embodiment, the network may include a wired network, such as a local area network (LAN). In another embodiment, the network may include a wireless network, such as Wi-Fi.

[0012] The processing subsystem (110) includes a segmentation module (130) configured to extract one or more regions of interest (ROIs) by processing an input query image using a segmentation network. As used herein, a “region of interest” (ROI) refers to a specific area within an image or dataset selected for detailed analysis. In some embodiments, these regions may be specific objects (e.g., pedestrians, vehicles) or textures (e.g., road markings or shadows). Specifically, the input query image is fed into the segmentation network, which processes the input query image to identify different regions. In one embodiment, the segmentation network is a deep learning architecture specifically designed to identify and label regions in an image based on predefined categories or user-specified criteria. After processing, the segmentation network generates a segmentation map of the input image. Each pixel in the input query image is classified into a predefined category. These categories represent regions of interest that can be adjusted based on user input. Specifically, the deep learning architecture identifies and labels regions in an image based on predefined categories or user-specified criteria by employing semantic segmentation or instance segmentation techniques. Such techniques use a neural network trained on a labeled dataset to predict the category of each pixel in the image.

[0013] The segmentation module (130) is configured to select one or more segments from one or more extracted regions of interest based on user indicators. If user-specified criteria are required, the model can be fine-tuned using a smaller, domain-specific dataset. The user can provide labels or bounding boxes for the regions of interest. The model adjusts its segmentation based on such user input. After segmentation, categories irrelevant to the user's criteria can be excluded by filtering the output.

[0014] Furthermore, the processing subsystem (110) includes a mask generation module (140) configured to generate a binary mask for selected segments on the input query image. Specifically, a binary mask is created based on the segmentation map by assigning a value of 1 to pixels corresponding to a selected region of interest and assigning a value of 0 to all other pixels. The binary mask is crucial for focusing on a specific region. In some embodiments, the user can specify the region of interest. The mask generation module (140) filters the segmentation map to generate a binary mask only for the user-specified region. For time-critical applications such as autonomous driving, the mask generation module (140) can be optimized using techniques such as lightweight segmentation models or hardware acceleration to achieve real-time operation. In one embodiment, the user can choose to manually mask irrelevant regions using masking features. The user manually erases irrelevant objects through a user interface such as a drawing application. In another embodiment, the system can automatically provide a segmented and masked image. As described above, for automatic segmentation, the image is processed by a deep neural network (DNN) that takes the image as input, generates all individual segments, and masks the others.

[0015] Furthermore, the processing subsystem (110) includes an image converter module (150) configured to generate a masked image by performing arithmetic operations on a binary mask and an input query image. The image converter module (150) is also configured to extract the embedding of the masked image using either a convolutional neural network (CNN) architecture or a Transformer architecture. In one embodiment, the Transformer architecture is configured to apply a masking attention mechanism to extract the embedding, wherein the masking attention focuses on an image segment containing at least one active pixel corresponding to a value of 1 in the binary mask. In this embodiment, the image segment containing at least one active pixel employs a positional encoding mechanism to preserve the spatial relationships between image segments.

[0016] Specifically, the binary mask is a pixel-level mask generated by a segmentation network. The input query image is a multi-channel image, such as an RGB image. In one embodiment, arithmetic operations may include multiplication, convolution, etc. In a specific embodiment, the masked image is obtained by multiplying the binary mask by the input query image. If the value in the binary mask is 1, the corresponding pixel in the image remains unchanged. If the value in the binary mask is 0, the corresponding pixel in the image is set to 0, effectively blacking it out. Furthermore, if the input query image has multiple channels, the multiplication is applied to all channels.

[0017] Masking image (B×I) = Binary mask (x,y) × Input query image (x,y,c) Here, B: binary mask, I: input query image, x and y: pixel coordinates, and c: channel index.

[0018] This results in a masked image where only regions corresponding to 1s in the binary mask are visible, while all other regions are either blacked out or blurred. This ensures that only the regions of interest from the input query image are preserved for subsequent processing.

[0019] Once the masked image is generated, it is passed through a neural network to extract semantic embeddings of the image, such as compact high-dimensional features. In one embodiment, a convolutional neural network (CNN) is used to obtain the embeddings of the masked image. In this embodiment, the CNN extracts hierarchical features from the masked image, where convolutional layers extract low-level features such as edges, textures, and colors, and apply filters to detect specific patterns in the image. In the pooling layers of the CNN, it uses, for example, one of max pooling or average pooling techniques to reduce the spatial dimension of the feature maps while preserving important information. Furthermore, deeper convolutional layers capture high-level semantic features, such as object shapes or contextual information about the region of interest. The output of the CNN is a high-dimensional feature embedding representing the masked image. These embeddings capture the essence of the region of interest while ignoring irrelevant parts of the input query image.

[0020] In another embodiment, a Transformer-based architecture is used to obtain the embedding of the masked image. In this architecture, the masked image, along with a binary mask, is divided into smaller, non-overlapping image patches and fed into a transformer. Each patch is flattened into a vector. Only masked image patches whose binary masks contain at least one '1' are processed through masking attention; the remaining patches are not involved in embedding generation. Each vector is linearly projected into a fixed-dimensional embedding space. Furthermore, positional information is added to each patch embedding to ensure the model understands the spatial arrangement of the patches. Each patch interacts with every other patch in the image through self-attention, which calculates the importance of each patch relative to the others. This attention mechanism allows the model to focus on relevant regions (highlighted by the binary mask) while ignoring the rest. After self-attention processing, the embedding is further refined through fully connected layers. The transformer generates feature vectors or embeddings that encapsulate the global context of the masked image. In one embodiment, the image converter module is configured to use a similarity metric to compare the embeddings from the input query image with the embeddings of a stored reference image for masked query-driven visual search.

[0021] Figure 2This is an exemplary embodiment of a system (200) for masking images according to embodiments of the present disclosure. Consider a non-limiting example of lane detection and scene retrieval for autonomous driving, where identifying and understanding specific regions in a road scene is crucial for tasks such as lane detection, pedestrian safety, and obstacle avoidance. Consider a scenario where an autonomous vehicle is driving on a busy urban road and needs to identify lane markings to maintain its course. The scene is presented using an input query image (210), which contains several irrelevant features, such as parked vehicles, pedestrians, traffic signs, and shadows, which could confuse conventional image retrieval systems.

[0022] The segmentation module (130) of the system (100) receives the input query image (210). This segmentation module uses a segmentation network to identify and extract specific regions (220), such as lane markings or pedestrian crossings, while ignoring other regions, such as trees, shadows, or parked vehicles. The system's mask generation module (140) generates a binary mask (230), in which lane markings are labeled with a value of 1 and the rest of the scene is labeled with a value of 0. This labeling ensures that only lane areas are highlighted. The binary mask (230) is then combined with the input query image (210) to create a masked image (240). In the final masked image (240), only lane markings are visible, and all other regions or areas are blurred. This isolates the key regions of interest and eliminates interference from the background.

[0023] Furthermore, the masked image is then fed into a transformer module (150) using a CNN or Transformer architecture. This network extracts embeddings (250), such as high-dimensional feature vectors representing the shape, curvature, and spatial relationships of lane markings. If a Transformer architecture is used, the mask attention mechanism ensures that only image patches corresponding to lane markings are considered. This global attention mechanism is capable of handling complex lane structures, such as curves, merging lanes, or intersections.

[0024] Therefore, the extracted embedding (250) can be used to match the current scene with a database of known lane patterns for similar scenes, such as winding highways or city streets. The matching results will then be fed into the vehicle's navigation system to make real-time decisions, such as keeping in the lane or merging safely.

[0025] Figure 3This is a flowchart illustrating the steps included in a method (300) for masking an image according to an embodiment of the present disclosure. The method (300) includes extracting one or more regions of interest in step 310 by processing an input query image using a segmentation network. In one embodiment, extracting the one or more regions of interest may include extracting the one or more regions by a segmentation module. In some embodiments, the segmentation network may identify and label regions in the image based on predefined categories or user-specified criteria.

[0026] The method (300) further includes selecting one or more segments from one or more extracted regions of interest (ROIs) in step 320 based on a user indicator. In one embodiment, selecting one or more segments may include selection via a segmentation module. ROIs associated with the user intent are segmented and classified. One or more ROIs are selected from the segmented regions based on the user selection. In one case, the user can select manually. In another case, a deep learning model can be trained to automatically select the desired region. The method (300) further includes generating a binary mask for the selected segments on the input query image in step 330. In one embodiment, generating the binary mask may include generation by a mask generation module. Once the ROIs are identified, a binary mask is created. The mask assigns a value of 1 to pixels belonging to the selected region and a value of 0 to the remaining pixels.

[0027] Furthermore, the method (300) also includes generating a masking image in step 340 by performing arithmetic operations on a binary mask and an input query image. In one embodiment, generating the masking image may include generation by an image converter module. In some embodiments, performing arithmetic operations may include performing multiplication, convolution, etc. Specifically, each pixel in the input query image is multiplied by the corresponding value in the binary mask. If the mask value is 1, the original pixel value is preserved. If the mask value is 0, the pixel value is set to the black intensity corresponding to the value 0. The resulting masking image effectively isolates the region of interest.

[0028] Furthermore, method (300) includes extracting embeddings of the masked image in step 350 using either a convolutional neural network architecture or a Transformer architecture. In one embodiment, embedding extraction may include extraction via an image converter module. The masked image is fed into a CNN to extract feature embeddings, which are high-dimensional vectors representing region features. Specifically, the masked image is fed into a CNN consisting of convolutional layers, pooling layers, and fully connected layers. The convolutional layers extract spatial features, such as edges, textures, and patterns, from the masked image. The pooling layers reduce dimensionality, retaining only the most salient features. The final output is a set of feature embeddings that encode unique features of the region, such as the shape and structure of objects in the image. In another embodiment, the Transformer-based architecture may employ a mask attention mechanism to focus only on relevant regions identified by the binary mask.

[0029] Specifically, the system presents the user with an input query image, offering two options: manual and automatic. In the manual option, the user interface includes a masking tool or brush and eraser to mask irrelevant areas, retaining only the important areas relevant to the user's intent. The final masked image is used in the search process. In the automatic option, a deep neural network segments the image into multiple regions, taking the image as input and presenting these regions to the user. The user selects regions from this visualization and submits them to the search process. The deep neural network is trained using a large amount of data to segment into important regions. Feature embeddings extracted by a convolutional neural network (CNN) or transformer are used for applications such as image retrieval, object classification, object detection, scene understanding, or navigation.

[0030] The various embodiments of this disclosure described above can effectively focus on relevant regions. Binary masks ensure that only user-specified regions are processed, thereby reducing noise and irrelevant information. The system also offers a variety of architectural options by supporting both convolutional neural networks (CNNs) for local feature extraction and Transformers for capturing global relationships, making it adaptable to different use cases. The system improves accuracy by isolating and focusing on specific regions of interest. This system improves accuracy in tasks such as image retrieval and object detection.

[0031] This system improves the accuracy and relevance of extracted features, reduces computational overhead, and enhances performance in complex visual scenes. It integrates segmentation and retrieval into a unified workflow. This integrated workflow ensures that downstream tasks focus on user-specified criteria, enabling the system to be highly adaptable to various use cases, such as lane detection in autonomous driving or pedestrian tracking in surveillance.

[0032] Element-wise multiplication with the binary mask ensures lightweight preprocessing, enabling real-time or near-real-time applications in speed- and accuracy-critical systems such as autonomous vehicle navigation. Furthermore, embeddings generated from the masked image regions allow for targeted retrieval of visually similar images from a database. This ensures that the retrieval is guided by user intent, rather than by irrelevant objects in the images. This feature reduces false positives in the retrieval results and improves the system's ability to locate and retrieve context-similar images.

[0033] Furthermore, the system achieves noise reduction and robustness by masking irrelevant regions. It reduces the impact of noise in the input query image, such as background clutter and shadows, leading to more robust feature extraction and retrieval results. This feature improves the system's reliability in complex environments such as poorly lit roads or occluded scenes.

[0034] It should be understood that the embodiments described above are merely illustrative and do not limit the scope of the invention. Many such embodiments, as well as other modifications and variations to the above embodiments, are contemplated. The scope of the invention is limited only by the claims.

Claims

1. An image masking system (100), comprising: The processing subsystem (110) includes: The segmentation module (130) is configured for: One or more regions of interest are extracted by processing the input query image using a segmentation network; Select one or more segments from one or more extracted regions of interest based on user indicators; Its features are, A mask generation module (140) is configured to generate a binary mask for a selected segment on the input query image; Image converter module (150), configured for: A masking image is generated by performing arithmetic operations on the binary mask and the input query image; and Use either a convolutional neural network architecture or a Transformer architecture to extract the embeddings of the masked image.

2. The system (100) of claim 1, wherein the segmentation network includes semantic segmentation technology or instance segmentation technology for identifying the one or more regions of interest.

3. The system (100) of claim 1, wherein the binary mask generation module (140) is configured to assign a value of 1 to pixels corresponding to selected segments and a value of 0 to unselected segments.

4. The system (100) of claim 1, wherein the Transformer architecture is configured to apply a masking attention mechanism to extract the embedding, wherein the masking attention focuses on an image segment containing at least one active pixel corresponding to a value of 1 in the binary mask.

5. The system (100) of claim 4, wherein image segments containing at least one active pixel use a positional encoding mechanism to maintain spatial relationships between the image segments.

6. The system (100) of claim 1, wherein the image converter module (150) is configured to generate the masking image using a deep neural network (DNN), wherein the deep neural network is trained on a plurality of images and labeled data to predict masking regions.

7. The system (100) of claim 1, wherein the image converter module (150) is configured to compare the embedding from the input query image with the embedding of a stored reference image using a similarity metric for masked query-driven visual search.

8. A method (300) for generating a masking image, comprising: The segmentation module extracts one or more regions of interest by processing the input query image using a segmentation network (step 310). The segmentation module selects one or more segments from one or more extracted regions of interest based on user indicators (step 320). Its features are, The mask generation module generates a binary mask for the selected segment on the input query image (step 330). The image converter module generates a masking image by performing arithmetic operations on the binary mask and the input query image (step 340); and The image converter module extracts the embedding of the masked image using either a convolutional neural network architecture or a Transformer architecture (step 350).

9. The method (300) of claim 8, wherein generating the binary mask includes simultaneously selecting multiple regions of interest using a multi-class binary mask, wherein each region of interest is assigned a unique identifier from the binary mask.

10. The method (300) of claim 8, wherein generating a binary mask includes generating one or more masks at one or more granular levels using hierarchical masking, wherein the hierarchical masking includes pixel-level masks, object-level masks, and scene-level masks.

Images