A zero-shot representation understanding method based on a differentiable fourier descriptor

Abstract

The application discloses a zero-shot referent representation understanding method based on a differentiable Fourier descriptor. The contour of a target object is modeled as a set of learnable Fourier coefficients, which are converted into soft masks through differentiable rendering, and the semantic alignment signal of a pre-trained visual-language model is used as guidance to directly optimize the Fourier coefficients in a gradient descent manner, so as to locate the object boundary referred to by the language. The method disclosed by the application discards the traditional two-stage target detection process, realizes generation of a high-quality instance mask with consistent semantics and geometric continuity only by means of a language instruction without fine-tuning for a specific task, and provides a new technical path with accuracy, flexibility and generalization for a referent representation understanding task.

Description

Technical Field

[0001] This invention relates to a zero-sample index expression understanding method based on differentiable Fourier descriptors, belonging to the field of artificial intelligence technology. Background Technology

[0002] In the information age, images, as an important carrier of information dissemination, have become a significant driving force for the development and application of artificial intelligence through their automatic understanding and efficient utilization. Referring Expression Comprehension (REC) is a core task in the intersection of computer vision and natural language processing, aiming to accurately locate the target object in an image based on a given natural language description.

[0003] Traditional methods for understanding language points typically rely on supervised training using a large number of manually annotated bounding boxes or masks, combined with object detectors to extract candidate regions, and then filtering them through a vision-language matching mechanism. However, these methods are not only limited by the performance bottlenecks of the detectors and category priors, but also have difficulty generalizing to open vocabulary or unseen category scenarios.

[0004] Existing technologies have proposed zero-shot indexing expression understanding methods, attempting to break free from the dependence on task-specific annotations and utilize pre-trained visual language models (such as CLIP) to achieve cross-modal alignment. However, their localization accuracy is still limited by coarse-grained spatial priors (such as Grad-CAM heatmaps), making it difficult to generate accurate and continuous instance-level masks. Especially for targets with complex shapes, irregular boundaries, or partial occlusion, existing methods often fail to provide pixel-level accurate responses.

[0005] Therefore, it is necessary to conduct a more in-depth study of existing methods for understanding reference in order to solve the above problems. Summary of the Invention

[0006] To overcome the above problems, in-depth research was conducted, and a zero-sample referencing expression understanding method based on differentiable Fourier descriptors was proposed. By modeling the contour of the target object as a set of learnable Fourier coefficients, converting them into a soft mask through differentiable rendering, and using the semantic alignment signal of the pre-trained vision-language model as guidance, the Fourier coefficients are directly optimized by gradient descent, thereby locating the boundary of the object referred to by the language.

[0007] In a preferred embodiment, the method includes the following steps:

[0008] S1. For each input original image and its corresponding natural language representation, an initial semantic heatmap is generated using a pre-trained visual-language model. Based on this heatmap, a set of Fourier coefficients is initialized as a parameterized representation of the target contour.

[0009] S2. Reconstruct the Fourier coefficients into an ordered sequence of contour points on a two-dimensional plane. Convert the contour point sequence into a soft mask through differentiable rendering. Process the original image based on the soft mask to generate a prompt image.

[0010] S3. Input the prompt image and the natural language instruction into the pre-trained visual-language model and obtain the loss;

[0011] S4. The lost gradient is backpropagated through the automatic differentiation mechanism and then fed back to the Fourier coefficients via the differentiable renderer.

[0012] S5. Repeat S2~S4, iterating and updating the Fourier coefficients multiple times until the loss converges or the preset number of iterations is reached. Generate a soft mask based on the final Fourier coefficients. This soft mask is the target mask.

[0013] In a preferred embodiment, in S1, a pre-trained visual-language model is used to extract the visual features of the image and the semantic features of the natural language index text, respectively. The cross-modal similarity between the image and the natural language index text is calculated, and an initial semantic heatmap is generated through a weakly supervised method.

[0014] In a preferred embodiment, in S2, the Fourier coefficients are reconstructed into a two-dimensional contour point sequence via inverse discrete Fourier transform, as follows:

[0015]

[0016] in, This represents the sequence number, where N is the total number of sequences. Represents a contour sequence. For two-dimensional contour points in the sequence, These are the Fourier coefficients. To represent a complex number, The sequence number representing the Fourier coefficients. This indicates the truncation order of the Fourier coefficients.

[0017] In a preferred embodiment, the step of processing the original image based on a soft mask to generate a prompt image is represented as follows:

[0018]

[0019] in, This indicates a prompt image. Indicates a soft mask. Represents the original image. Represents a mask image. This represents the Hadamard product operation.

[0020] In a preferred embodiment, in S3, the loss includes a semantic alignment loss, which is used to characterize the similarity between the visual representation generated by the prompt image and the text representation expressed by the natural language pointer. The semantic alignment loss is obtained by inputting the prompt image and the natural language pointer into a pre-trained visual-language model, extracting the corresponding normalized image embedding and text embedding, and calculating their similarity.

[0021] In a preferred embodiment, the loss further includes an influence maximization loss, used to characterize the spatial overlap between the soft mask and the gradient heatmap, expressed as:

[0022]

[0023] in, This indicates the maximum impact loss. Indicates different pixel positions, It is the set of all pixels in the image. Indicates the soft mask at the pixel level. The value at that location, This represents the semantic activation heatmap extracted from a pre-trained visual-language model using the gradient CAM method at the pixel level. The response intensity at that point Represents the total number of pixels in an image, operators This represents element-wise product.

[0024] In a preferred embodiment, the loss further includes a compactness loss to characterize the profile compactness, expressed as:

[0025]

[0026] in, This indicates a loss of compactness.

[0027] The present invention also provides an electronic device, comprising:

[0028] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the methods described above.

[0029] The present invention also provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in any of the preceding claims.

[0030] The beneficial effects of this invention include:

[0031] (1) It abandons the traditional two-stage target detection process and realizes the generation of high-quality instance masks with semantic consistency and geometric coherence based solely on language instructions without the need for fine-tuning for specific tasks. This provides a new technical path for instruction expression understanding tasks that combines accuracy, flexibility and generalization.

[0032] (2) By modeling the object region as a continuous learnable Fourier descriptor, rather than a rough bounding box or basic geometry such as an ellipse.

[0033] (3) By parameterizing the contour using Fourier descriptors, the discrete, high-dimensional pixel optimization problem in shape space is transformed into an optimization problem for low-dimensional, continuous Fourier coefficients. This representation naturally has smoothness and closure, which can effectively characterize complex non-convex shapes, while greatly reducing the optimization difficulty and computational cost.

[0034] (4) The optimization is carried out by combining semantic alignment loss, influence maximization loss and compactness loss. This not only ensures global semantic consistency, but also constrains the accuracy of local localization and the geometric rationality of the contour, avoiding the optimization process from getting trapped in trivial solutions or local optima, thus obtaining more stable and accurate optimization results.

[0035] (5) It does not require training on a specific dataset or object category and can be directly applied to any natural language description and image in the open world, providing a strong technical foundation for real-world applications that require precise reference, such as robot interaction, image editing, and assistance for the blind. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of a zero-sample index expression understanding method based on a differentiable Fourier descriptor according to a preferred embodiment of the present invention.

[0037] Figure 2 This is a schematic diagram illustrating the iterative update of a zero-sample index expression understanding method based on a differentiable Fourier descriptor according to a preferred embodiment of the present invention.

[0038] Figure 3 To improve the positioning accuracy of the zero-sample index expression understanding method based on differentiable Fourier descriptors according to a preferred embodiment of the present invention. Detailed Implementation

[0039] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Through these descriptions, the features and advantages of the present invention will become clearer and more apparent.

[0040] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments. Although various aspects of embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless specifically indicated otherwise.

[0041] The method provided in this invention aims to overcome two key bottlenecks in current reference-region understanding tasks: insufficient precise localization capability and high dependence on labeled data. The former is manifested in the fact that existing methods are mostly limited to bounding boxes or coarse-grained heatmaps, making it difficult to generate pixel-level masks consistent with language semantics; the latter is reflected in the fact that most models need to be trained on a large number of manually labeled reference-region pairs, which severely restricts their generalization ability in open scenes and new categories.

[0042] This invention provides a zero-shot indicative expression understanding method based on differentiable Fourier descriptors. By modeling the contour of the target object as a set of learnable Fourier coefficients, converting them into a soft mask through differentiable rendering, and using the semantic alignment signal of a pre-trained vision-language model as guidance, the Fourier coefficients are directly optimized by gradient descent, thereby locating the boundary of the object referred to by the language.

[0043] Furthermore, the method includes the following steps:

[0044] S1. For each input original image and its corresponding natural language representation, an initial semantic heatmap is generated using a pre-trained visual-language model. Based on this heatmap, a set of Fourier coefficients is initialized as a parameterized representation of the target contour.

[0045] S2. Reconstruct the Fourier coefficients into an ordered sequence of contour points on a two-dimensional plane. Convert the contour point sequence into a soft mask through differentiable rendering. Process the original image based on the soft mask to generate a prompt image.

[0046] S3. Input the prompt image and the natural language instruction into the pre-trained visual-language model and obtain the loss;

[0047] S4. The lost gradient is backpropagated through the automatic differentiation mechanism and then fed back to the Fourier coefficients via the differentiable renderer.

[0048] S5. Repeat S2~S4, iterating and updating the Fourier coefficients multiple times until the loss converges or the preset number of iterations is reached. Generate a soft mask based on the final Fourier coefficients. This soft mask is the target mask.

[0049] In S1, the specific structure of the visual-language model is not limited. Those skilled in the art can use any known visual-language model, such as CLIP.

[0050] Furthermore, a pre-trained vision-language model is used to extract the visual features of the image and the semantic features of the natural language indexed text, respectively. The cross-modal similarity between the image and the natural language indexed text is calculated, and an initial semantic heatmap is generated through a weakly supervised method. This heatmap reflects the coarse region of interest of the language expression in the image space.

[0051] In this invention, any weak supervision method can be used, such as Grad-CAM.

[0052] According to the present invention, a bounding box is extracted based on the maximum response region of the heatmap, thereby initializing a set of Fourier coefficients. These coefficients constitute a parameterized representation of the closed contour of the target object, which is compact in form (requiring only...). It is a complex number, geometrically smooth, and fully differentiable, which can effectively characterize the boundaries of complex, non-convex, and even fine-grained objects.

[0053] Typically, the cutoff order of the Fourier coefficients is set to... or .

[0054] In S2, preferably, the Fourier coefficients are reconstructed into a two-dimensional contour point sequence via inverse discrete Fourier transform (IDFT), as follows:

[0055]

[0056] in, This represents the sequence number, where N is the total number of sequences. Represents a contour sequence. For two-dimensional contour points in the sequence, These are the Fourier coefficients. To represent a complex number, The sequence number representing the Fourier coefficients. This indicates the truncation order of the Fourier coefficients.

[0057] According to the present invention, a two-dimensional contour point sequence This forms a closed, smooth, parametric path. Using differentiable rendering, such as the differentiable vector graphics renderer DiffVG, this path can be rasterized into a continuous soft mask. This soft mask has anti-aliasing properties and is completely differentiable with respect to the Fourier coefficients.

[0058] By fusing the soft mask with the original image, a prompt image can be generated, represented as follows:

[0059]

[0060] in, This indicates a prompt image. Indicates a soft mask. Represents the original image. The mask image is obtained by directly applying Gaussian blur to the original input image I. This represents the Hadamard product operation.

[0061] In this invention, a mask image is incorporated into the prompt image, which can smooth out details. By smoothing out the abrupt color transitions in the image and softening sharp edges, background suppression is achieved, allowing the model to focus on key areas and facilitating mask overlay.

[0062] In S3, the loss includes semantic alignment loss, which is used to characterize the similarity between the visual representation generated by the cue image and the textual representation expressed by the natural language cue.

[0063] By inputting the cue image and the natural language pointer into a pre-trained visual-language model, respectively, the corresponding normalized image embeddings and text embeddings are extracted, and their similarity is calculated to obtain the semantic alignment loss, which is expressed as:

[0064]

[0065] in, Represents semantic alignment loss. Natural language refers to the expression of meaning. This indicates that the pre-trained visual-language model extracts the corresponding normalized image embeddings and text embeddings and then calculates the similarity.

[0066] Preferably, the similarity is obtained by performing a dot product on the image embedding and the text embedding.

[0067] While semantic alignment loss provides global alignment guidance, its gradient information may not be sufficient to accurately anchor the contour to a visually prominent target region.

[0068] In a preferred embodiment, the loss further includes an influence maximization loss, used to characterize the spatial overlap between the soft mask and the gradient heatmap, expressed as:

[0069]

[0070] in, This indicates the maximum impact loss. Indicates different pixel positions, It is the set of all pixels in the image. Indicates the soft mask at the pixel level. The value at that location, , This represents the semantic activation heatmap extracted from a pre-trained visual-language model using the gradient CAM method at the pixel level. The response intensity at that point , Represents the total number of pixels in an image, operators This represents element-wise product.

[0071] According to the present invention, the influence maximization loss can encourage the generated soft mask to cover highly activated regions, thereby enhancing the model's responsiveness to key visual features and improving the stability of the optimization process.

[0072] In a preferred embodiment, the loss further includes a compactness loss to characterize the profile compactness, expressed as:

[0073]

[0074] in, This indicates a loss of compactness.

[0075] According to the present invention, the compactness loss penalizes masks that cover large areas with low semantic or visual relevance, thereby encouraging the contour to remain focused on the most informative part of the object, suppressing excessive oscillations of Fourier coefficients, and avoiding the generation of unreasonable boundary shapes.

[0076] Preferably, the loss is the sum of semantic alignment loss, influence maximization loss, and compactness loss, expressed as:

[0077]

[0078] in, This indicates the total loss.

[0079] In S4, the loss gradient is backpropagated to the Fourier coefficients along the differentiable rendering path and the inverse path of the inverse discrete Fourier transform using an automatic differentiation mechanism. In S5, optimizers such as Adam are used for iterative updates until convergence. After constructing an ordered sequence of contour points based on the converged Fourier coefficients, the sequence is converted into a soft mask through differentiable rendering. This soft mask is the target mask. Figure 2 As shown, starting from Step 0, the model generates a coarse outline (such as the green area) based on the initial semantic heatmap, and then progressively optimizes it using differentiable Fourier descriptors. At each step, the Fourier coefficients are adjusted under the drive of the visual-language alignment score. As the number of optimization steps increases, the outline gradually converges to a precise boundary consistent with the semantic meaning expressed in the language: for example, "Black Shirt" ultimately accurately covers the area of ​​the person's shirt, while "Big Sheep" completely surrounds the larger sheep in the foreground. This process demonstrates that the model can achieve high-precision, geometrically coherent instance-level localization solely through language guidance without the need for labeled masks.

[0080] The target mask obtained in this invention has good generalization ability and positioning accuracy in complex scenarios, such as... Figure 3 As shown, in the first row, the expression "The Girl On the right" contains relative position information. This invention successfully identified and generated a closed contour, outperforming the elliptical approximation of the traditional method (Tune-An-Ellipse). In the second row, "LeftBroccoli" requires distinguishing similar-looking objects. This invention can accurately locate the broccoli on the left, avoiding the misselection of similar objects on the right. In the third row, "Front of horse" involves non-convex, irregular regions. This invention generates a mask that conforms to the real shape through differentiable rendering, significantly outperforming the simplified representations of other methods. Furthermore, the mask generated by this invention performs excellently in terms of boundary details and shape integrity, verifying its robustness and accuracy in handling challenging scenarios such as occlusion, multiple targets, and fine-grained semantics.

[0081] Various embodiments of the methods described above in this invention can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0082] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0083] Example

[0084] Example 1

[0085] Pointer representation understanding experiments were conducted on the benchmark datasets RefCOCO, RefCOCO+, and RefCOCOg. RefCOCO, RefCOCO+, and RefCOCOg are commonly used benchmark datasets for pointer representation understanding. The RefCOCO dataset contains 19,994 images from MS COCO, partitioned using the UNC standard. The experiment selected a validation set (val) of 2,985 images and test subsets A and B (2,629 and 2,349 images, respectively) for model training and evaluation. This dataset contains 142,210 pointer representations, each corresponding to the bounding box of a specific target in the image, supporting the use of location information to complete target reference description. RefCOCO+ is an extended version of RefCOCO, also built on MS COCO images and partitioned using the UNC standard. Experiments were conducted using its validation set (val) and test subsets (testA and testB). The dataset contains 19,992 images and 141,564 index expressions. Its core feature is the disabling of relative positional words, relying solely on absolute position and appearance semantics (such as color, shape, and category) to achieve target referencing. RefCOCOg focuses on longer and more complex natural language index expressions, containing 25,799 images and 95,010 index expressions. This dataset uses the UMD standard for partitioning. Experiments were conducted using its validation set (val) and test subsets to validate model performance. Its annotation characteristics are that each image typically only labels one target, and multiple diverse index expressions are configured for that target.

[0086] In the experiment, the method of understanding the meaning of the target object is to model the outline of the target object as a set of learnable Fourier coefficients, convert them into a soft mask through differentiable rendering, and use the semantic alignment signal of the pre-trained vision-language model as guidance to directly optimize the Fourier coefficients in the manner of gradient descent, thereby locating the boundary of the object referred to by the language.

[0087] The method includes the following steps:

[0088] S1. For each input original image and its corresponding natural language representation, an initial semantic heatmap is generated using a pre-trained visual-language model. Based on this heatmap, a set of Fourier coefficients is initialized as a parameterized representation of the target contour.

[0089] S2. Reconstruct the Fourier coefficients into an ordered sequence of contour points on a two-dimensional plane. Convert the contour point sequence into a soft mask through differentiable rendering. Process the original image based on the soft mask to generate a prompt image.

[0090] S3. Input the prompt image and the natural language instruction into the pre-trained visual-language model and obtain the loss;

[0091] S4. The lost gradient is backpropagated through the automatic differentiation mechanism and then fed back to the Fourier coefficients via the differentiable renderer.

[0092] S5. Repeat S2~S4, iterating and updating the Fourier coefficients multiple times until the loss converges or the preset number of iterations is reached. Generate a soft mask based on the final Fourier coefficients. This soft mask is the target mask.

[0093] In S1, a pre-trained vision-language model is used to extract the visual features of the image and the semantic features of the natural language index text, respectively. The cross-modal similarity between the image and the natural language index text is calculated, and an initial semantic heatmap is generated through a weakly supervised method (Grad-CAM). This heatmap reflects the coarse region of interest of the language expression in the image space.

[0094] In S2, the Fourier coefficients are reconstructed into a two-dimensional contour point sequence using the inverse discrete Fourier transform (IDFT), as follows:

[0095]

[0096] in , .

[0097] The soft mask is merged with the original image to generate the prompt image, as shown below:

[0098]

[0099] In S3, the loss is:

[0100]

[0101]

[0102]

[0103]

[0104] In S4, the loss gradient is backpropagated to the Fourier coefficients along the differentiable rendering path and the inverse path of the inverse discrete Fourier transform through the automatic differentiation mechanism. In S5, the Adam optimizer is used to iteratively update until convergence. A soft mask is formed based on the converged Fourier coefficients, which is the target mask.

[0105] Comparative Example 1

[0106] The same experiments as in Example 1 were conducted, except that the ReCLIP, RedCircle, TAE, and Grad-CAM methods were used respectively. The ReCLIP, RedCircle, TAE, and Grad-CAM methods are all state-of-the-art methods for zero-shot indexing expression understanding.

[0107] For ReCLIP, see the reference "Subramanian S, Merrill W, Darrell T, et al. Reclip: A strong zero-shot baseline for referring expression comprehension. arXiv, 2022.";

[0108] For RedCircle, see the literature "Shtedritski A, Rupprecht C, Vedaldi A. What doesclip know about a red circle? visual prompt engineering for vlms. In IEEEICCV, 2023.";

[0109] For TAE, see the paper "Yanxin Long, Youpeng Wen, Jianhua Han, Hang Xu, Pengzhen Ren, Wei Zhang, Shen Zhao, and Xiaodan Liang. Capdet: Unifying dense captioning and open-world detection pretraining. In IEEE CVPR, 2023.";

[0110] See the paper “Selvaraju RR, Cogswell M, Das A, et al. Grad-cam: Visual explanations from deep networks via gradient-based localization. In IEEE CVPR, 2017.” for Grad-CAM.

[0111] The results of comparing Example 1 and Comparative Example 1 are shown in Tables 1 to 3.

[0112] Table 1 Comparison of test performance ACC (%) on the RefCOCO dataset

[0113] method testA testB val avg Comparative Example 1 - ReCLIP 6.59 7.69 7.35 7.21 Comparative Example 1 - RedCircle 6.56 10.2 8.34 8.37 Comparative Example 1-TAE 32.3 19.8 26.3 26.1 Example 1 31.3 23.6 27.2 27.4

[0114] Table 1 shows the performance comparison results of different methods in Example 1 and Comparative Example 1 on the RefCOCO dataset. The overall localization accuracy (ACC,%) is used as the evaluation index, and the intersection-union ratio (IoU) between the predicted mask and the real target region is considered to be a correct prediction when it is greater than 0.5.

[0115] As shown in Table 1, the method presented in Example 1 achieved accuracies of 31.3%, 23.6%, and 27.2% on the three subsets testA, testB, and val, respectively, with an average performance of 27.4%, significantly outperforming ReCLIP (7.21%), RedCircle (8.37%), and TAE (26.1%). Example 1 performed particularly well on testB, showing a 3.8 percentage point improvement over TAE, indicating stronger robustness to complex semantic representations. These results fully demonstrate that the method in Example 1 can achieve high-precision indexing and expression understanding even without annotations or detectors, possessing good practical value and technological advancement.

[0116] Table 2 Comparison of test performance ACC (%) on the RefCOCO+ dataset

[0117] method testA testB val avg Comparative Example 1 - ReCLIP 6.88 8.99 7.90 7.92 Comparative Example 1 - RedCircle 6.85 11.3 8.94 9.03 Comparative Example 1-TAE 32.7 22.1 27.8 27.5 Example 1 31.1 26.0 28.4 28.5

[0118] Table 2 shows the performance comparison results of different methods in Example 1 and Comparative Example 1 on the RefCOCO+ dataset, with overall localization accuracy (ACC,%) as the evaluation metric. This dataset prohibits the use of absolute or relative positional terms (such as "left" or "front"), emphasizing the model's ability to understand the semantics of the target's appearance.

[0119] As shown in Table 2, the method in Example 1 achieved accuracies of 31.1%, 26.0%, and 28.4% on the three subsets testA, testB, and val, respectively, with an average performance of 28.5%, significantly outperforming ReCLIP (7.92%), RedCircle (9.03%), and TAE (27.5%). Specifically, the method in Example 1 improved performance by 1.5 percentage points on testB and 0.6 percentage points on the val set compared to TAE, indicating stronger robustness under complex semantic constraints. These results fully demonstrate that the method in Example 1 can achieve high-precision index expression understanding even without annotations or detectors, possessing good practical value and technological advancement.

[0120] Table 3 Comparison of test performance ACC (%) on the RefCOCOg dataset

[0121] method test val avg Comparative Example 1 - ReCLIP 12.1 11.7 16.4 Comparative Example 1 - RedCircle 16.0 15.1 15.6 Comparative Example 1-TAE 29.9 29.0 29.5 Example 1 32.6 34.4 33.5

[0122] Table 3 shows the performance comparison results of different methods in Example 1 and Comparative Example 1 on the RefCOCOg dataset, with the overall localization accuracy (ACC,%) as the evaluation metric. This dataset uses longer and more complex natural language sentences, requiring the model to have stronger semantic parsing and contextual reasoning capabilities.

[0123] As shown in Table 3, the method in Example 1 achieved 28.4% and 28.7% accuracy on the test and val subsets, respectively, with an average performance of 28.5%, significantly outperforming ReCLIP (16.4%) and RedCircle (15.6%). This result demonstrates that the method in Example 1 still possesses strong generalization and semantic understanding capabilities when facing long sentences and multi-conditional reference tasks, providing a new technical path for zero-shot reference expression understanding.

[0124] The present invention has been described above with reference to preferred embodiments; however, these embodiments are merely exemplary and illustrative. Various substitutions and modifications can be made to the present invention based on these embodiments, all of which fall within the scope of protection of the present invention.

Claims

1. A zero-sample indexing expression method based on differentiable Fourier descriptors, characterized in that, By modeling the outline of the target object as a set of learnable Fourier coefficients, converting them into a soft mask through differentiable rendering, and using the semantic alignment signal of a pre-trained vision-language model as guidance, the Fourier coefficients are directly optimized in a gradient descent manner, thereby locating the boundary of the object referred to by the language.

2. The zero-sample indexing expression method based on differentiable Fourier descriptors according to claim 1, characterized in that, The method includes the following steps: S1. For each input original image and its corresponding natural language representation, an initial semantic heatmap is generated using a pre-trained visual-language model. Based on this heatmap, a set of Fourier coefficients is initialized as a parameterized representation of the target contour. S2. Reconstruct the Fourier coefficients into an ordered sequence of contour points on a two-dimensional plane. Convert the contour point sequence into a soft mask through differentiable rendering. Process the original image based on the soft mask to generate a prompt image. S3. Input the prompt image and the natural language instruction into the pre-trained visual-language model and obtain the loss; S4. The lost gradient is backpropagated through the automatic differentiation mechanism and then fed back to the Fourier coefficients via the differentiable renderer. S5. Repeat S2~S4, iterating and updating the Fourier coefficients multiple times until the loss converges or the preset number of iterations is reached. Generate a soft mask based on the final Fourier coefficients. This soft mask is the target mask.

3. The zero-sample indexing expression method based on differentiable Fourier descriptors according to claim 2, characterized in that, In S1, a pre-trained vision-language model is used to extract the visual features of the image and the semantic features of the natural language indexed text, respectively. The cross-modal similarity between the image and the natural language indexed text is calculated, and an initial semantic heatmap is generated through a weakly supervised method.

4. The zero-sample indexing expression method based on differentiable Fourier descriptors according to claim 2, characterized in that, In S2, the Fourier coefficients are reconstructed into a two-dimensional contour point sequence through the inverse discrete Fourier transform, as follows: in, This represents the sequence number, where N is the total number of sequences. Represents a contour sequence. For two-dimensional contour points in the sequence, These are the Fourier coefficients. To represent a complex number, The sequence number representing the Fourier coefficients. This indicates the truncation order of the Fourier coefficients.

5. The zero-sample indexing expression method based on differentiable Fourier descriptors according to claim 2, characterized in that, The process of processing the original image based on a soft mask to generate a prompt image is as follows: in, This indicates a prompt image. Indicates a soft mask. Represents the original image. Represents a mask image. This represents the Hadamard product operation.

6. The zero-sample indexing expression method based on differentiable Fourier descriptors according to claim 2, characterized in that, In S3, the loss includes semantic alignment loss, which is used to characterize the similarity between the visual representation generated by the prompt image and the text representation expressed by the natural language pointer. By inputting the prompt image and the natural language pointer into the pre-trained visual-language model respectively, the corresponding normalized image embedding and text embedding are extracted, and their similarity is calculated to obtain the semantic alignment loss.

7. The zero-sample indexing expression method based on differentiable Fourier descriptors according to claim 6, characterized in that, The loss also includes an influence maximization loss, used to characterize the spatial overlap between the soft mask and the gradient heatmap, expressed as: in, This indicates the maximum impact loss. Indicates different pixel positions, It is the set of all pixels in the image. Indicates the soft mask at the pixel level. The value at that location, This represents the semantic activation heatmap extracted from a pre-trained visual-language model using the gradient CAM method at the pixel level. The response intensity at that point Represents the total number of pixels in an image, operators This represents element-wise product.

8. The zero-sample indexing expression method based on differentiable Fourier descriptors according to claim 6, characterized in that, The loss also includes a compactness loss, used to characterize the compactness of the contour, expressed as: in, This indicates a loss of compactness.

9. An electronic device, comprising: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

10. A computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

Images