Image retrieval method, system and electronic device for appearance patent and storage medium

Abstract

An appearance patent picture retrieval method and system, electronic equipment and storage medium are provided, and relate to the field of model training. The method comprises the following steps: constructing a test set based on a plurality of commodity pictures and patent picture labels corresponding to each commodity picture; evaluating the retrieval performance of a plurality of preset candidate multi-modal models based on the test set, and taking the candidate multi-modal model with the best retrieval performance as a base model; performing first-stage incremental pre-training on the base model based on a first training set to obtain a pre-trained model, performing second-stage contrastive learning training on the pre-trained model based on a second training set to obtain a target retrieval model, receiving a target commodity picture to be retrieved, performing feature extraction on the target commodity picture based on the target retrieval model, and outputting a corresponding patent picture list. The technical solutions provided in the present application can improve the accuracy of appearance patent retrieval results.

Description

Technical Field

[0001] This application relates to the field of model training technology, specifically to an image retrieval method, system, electronic device, and storage medium for a design patent. Background Technology

[0002] With the rapid development of e-commerce, infringement of product design patents is on the rise. To promptly detect and prevent such infringements, it is necessary to quickly compare products on the market with existing design patents. Currently, mainstream product image retrieval methods are primarily based on image feature matching technology. This involves extracting the visual features of product images and comparing their similarity with images in a patent image database to identify potentially infringing products.

[0003] In existing technologies, a single visual feature extraction model is typically used to retrieve product images. This method achieves good results when processing similar images (such as real-object images versus real-object images, or line drawings versus line drawings). However, in practical applications, patent image libraries often contain a large number of line drawings, while product images on e-commerce platforms are mainly real-object images. Due to the significant differences in visual representation between line drawings and real-object images, the accuracy of design patent search results is relatively low. Summary of the Invention

[0004] This application provides a method, system, electronic device, and storage medium for image retrieval of design patents, which can improve the accuracy of design patent retrieval results.

[0005] Firstly, this application provides a method for image retrieval of design patents, the method comprising:

[0006] A test set was constructed based on multiple product images collected and the corresponding patent image tags for each product image;

[0007] The retrieval performance of multiple preset candidate multimodal models is evaluated based on the test set. The retrieval performance includes intra-domain matching ability and cross-domain matching ability. The candidate multimodal model with the best retrieval performance is used as the base model.

[0008] A data synthesis framework is constructed based on a preset generative model. The data synthesis framework combines the edge feature information of the patent line drawing and the style characteristics of the product image to generate image pairs for the second stage of training.

[0009] The base model is subjected to incremental pre-training in the first stage based on the first training set to obtain a pre-trained model. The first training set includes multiple first line drawings and corresponding text descriptions.

[0010] The pre-trained model is trained in a second stage of comparative learning based on the second training set to obtain a target retrieval model. The second training set includes image pairs composed of second line drawings and object images. The second line drawings are obtained by filtering based on each of the first line drawings. The object images include synthesized object images generated by the data synthesis framework based on each of the second line drawings.

[0011] Receive the target product image to be searched, extract features from the target product image based on the target retrieval model, and output the corresponding patent image list.

[0012] By adopting the above technical solution, firstly, the intra-domain and cross-domain matching capabilities of candidate multimodal models are evaluated using a test set, and the model with the best performance is selected as the base model, ensuring that the model has good basic feature extraction capabilities. Secondly, by constructing a data synthesis framework, training data is generated by combining the edge feature information of patent line drawings and the style characteristics of product images, expanding the quantity and diversity of training samples. Then, a two-stage training strategy is adopted. In the first stage, incremental pre-training is performed using patent line drawings and their text descriptions to enable the model to learn the domain features of patent images. In the second stage, comparative learning training is performed based on synthesized image pairs to improve the model's cross-domain matching capability between line drawings and product images. The final target retrieval model can effectively extract the visual features of product images and accurately match them with patent images, overcoming the technical problem of low retrieval accuracy caused by the difference in visual representation between line drawings and product images in the prior art, and improving the accuracy and efficiency of appearance patent retrieval.

[0013] A second aspect of this application provides an image retrieval system for design patents, the system comprising:

[0014] The test set construction module is used to construct a test set based on multiple collected product images and the corresponding patent image tags for each product image;

[0015] The base model determination module is used to evaluate the retrieval performance of multiple preset candidate multimodal models based on the test set. The retrieval performance includes intra-domain matching ability and cross-domain matching ability, and the candidate multimodal model with the best retrieval performance is used as the base model.

[0016] The data synthesis module is used to construct a data synthesis framework based on a preset generative model. The data synthesis framework combines the edge feature information of the patent line drawing and the style characteristics of the product image to generate image pairs for the second stage of training.

[0017] The first-stage training module is used to perform incremental pre-training on the base model based on the first training set to obtain a pre-trained model. The first training set includes multiple first line drawings and corresponding text descriptions.

[0018] The second-stage training module is used to perform a second-stage comparative learning training on the pre-trained model based on the second training set to obtain a target retrieval model. The second training set includes image pairs composed of second line drawings and object images. The second line drawings are obtained by filtering based on each of the first line drawings. The object images include synthetic object images generated by the data synthesis framework based on each of the second line drawings.

[0019] The patent retrieval output module is used to receive the target product image to be retrieved, extract features from the target product image based on the target retrieval model, and output the corresponding patent image list.

[0020] A third aspect of this application provides a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the method steps described above.

[0021] A fourth aspect of this application provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the above-described method steps.

[0022] In summary, one or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0023] This application first evaluates the intra-domain and cross-domain matching capabilities of candidate multimodal models using a test set, selecting the best-performing model as the base model to ensure that the model possesses good basic feature extraction capabilities. Second, by constructing a data synthesis framework, training data is generated by combining edge feature information from patent line drawings and style characteristics of product images, expanding the quantity and diversity of training samples. Then, a two-stage training strategy is adopted. The first stage uses patent line drawings and their textual descriptions for incremental pre-training, enabling the model to learn the domain features of patent images. The second stage uses synthesized image pairs for comparative learning training, improving the model's cross-domain matching capability between line drawings and product images. The resulting target retrieval model can effectively extract the visual features of product images and accurately match them with patent images, overcoming the technical problem of low retrieval accuracy caused by the visual differences between line drawings and product images in existing technologies, thus improving the accuracy and efficiency of design patent retrieval. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating an image retrieval method for a design patent provided in an embodiment of this application;

[0025] Figure 2 This is a schematic diagram of a module of an image retrieval system for a design patent provided in an embodiment of this application;

[0026] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0027] Explanation of reference numerals in the attached drawings: 300, electronic device; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. Detailed Implementation

[0028] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0029] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0030] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0031] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0032] Please refer to Figure 1 This paper presents a flowchart illustrating a method for retrieving images of design patents. This method can be implemented using a computer program, a microcontroller, or run on an image retrieval system for design patents. The computer program can be integrated into a computer device or run as a standalone application. Specifically, the method includes steps 10 to 60, as follows:

[0033] Step 10: Construct a test set based on the collected product images and the corresponding patent image tags for each product image.

[0034] In this embodiment of the application, a product image refers to a photograph of a product with a clean background, clearly identifiable product features, and no obstructions. The photograph shows the actual appearance of the product and includes visual details such as the product's texture, lighting, and shadows.

[0035] Patent image labels refer to patent images that are highly similar to product images. These include physical image patent labels that show the actual appearance of the product and line drawing patent labels that show the design outline and basic structure in line form. The physical image patent labels are sourced from the Chinese patent database, while the line drawing patent labels are sourced from the US patent database.

[0036] The test set refers to the labeled dataset used to evaluate the retrieval performance of the model. It includes product image-physical image patent tag pairs for evaluating same-domain matching ability, and product image-line drawing patent tag pairs for evaluating cross-domain matching ability. The patent tags are patent images that are most similar to the product images and are manually selected by the retrieval system from the candidate results of the patent pool.

[0037] Specifically, first, several product images with clean backgrounds, clearly identifiable product features, and no occlusion are prepared. These product images serve as the base samples for constructing the test set. Next, two patent image pools are prepared: one pool contains line drawing patents selected from US patent databases, and the other contains physical image patents selected from Chinese patent databases. Then, visual feature extraction models (such as CLIP or VIT models) are used to extract feature vectors from the product images and patent images in the patent image pools. The Faiss retrieval system is then used to calculate similarity and recall for each product image within the patent image pools. For each product image, the patent image most similar to it is manually selected from its top 500 recall results as the corresponding patent image tag. Specifically, when selecting patent image tags from the physical image patent pool, the resulting product image-physical image patent tag pairs are used to construct a same-domain matching test set; when selecting patent image tags from the line drawing patent pool, the resulting product image-line drawing patent tag pairs are used to construct a cross-domain matching test set. The test set constructed in this way has high data quality and annotation accuracy, and can comprehensively evaluate the retrieval performance of the model in both intra-domain and cross-domain matching scenarios, providing a reliable evaluation benchmark for subsequent model selection and optimization. Practice shows that in the process of manually selecting tags, the CLIP series models outperform the purely visual DINO and BLIP series multimodal models for vector retrieval. The test set constructed in this way can better reflect the model's performance in real-world application scenarios.

[0038] Step 20: Evaluate the retrieval performance of multiple pre-set candidate multimodal models based on the test set. The retrieval performance includes intra-domain matching ability and cross-domain matching ability. The candidate multimodal model with the best retrieval performance is selected as the base model.

[0039] In this embodiment of the application, the candidate multimodal model refers to a pre-trained model that can process image and text information simultaneously, including but not limited to CLIP series models, DINO series models and BLIP series models, which have the basic ability to extract image and text features.

[0040] Retrieval performance refers to the model's intra-domain matching ability and cross-domain matching ability on the test set. Intra-domain matching ability is reflected in the accuracy of matching product images with physical images and patent tags, while cross-domain matching ability is reflected in the accuracy of matching product images with line drawings and patent tags. It is measured by the recall rates of TOP 1, TOP 10, and TOP 100.

[0041] The base model refers to the candidate multimodal model with the best overall retrieval performance on the test set. This model will serve as the foundation model for subsequent incremental pre-training and contrastive learning training to realize the patent image retrieval function.

[0042] Specifically, before determining the final base model, performance evaluation of multiple pre-set candidate multimodal models is required. First, product images used in the test set to evaluate same-domain matching ability are input into each candidate multimodal model for feature extraction, obtaining feature vectors. The similarity between these feature vectors and the feature vectors of the corresponding physical image patent labels is then calculated to obtain the model's same-domain matching ability index. Similarly, product images used in the test set to evaluate cross-domain matching ability are input into each candidate multimodal model for feature extraction, and the similarity between the resulting feature vectors and the feature vectors of the corresponding line drawing patent labels is calculated to obtain the model's cross-domain matching ability index. During the evaluation process, the recall performance of each model is mainly examined in three dimensions: TOP 1, TOP 10, and TOP 100. TOP 1 and TOP 10 reflect the model's retrieval accuracy, while TOP 100 reflects the model's false negative rate. By comprehensively comparing the retrieval performance of each candidate multimodal model in both same-domain and cross-domain matching scenarios, the model with the best performance is selected as the base model, laying a solid foundation for subsequent model training. Practice has shown that the CLIP series models have better retrieval performance than the DINO and BLIP series models.

[0043] Based on the above embodiments, as another optional embodiment, the step of evaluating the retrieval performance of multiple preset candidate multimodal models based on a test set may further include steps 101-103:

[0044] Step 101: Input the product image of the physical image patent label into each preset candidate multimodal model to obtain the first feature vector, and calculate the similarity between the first feature vector and the feature vector of the corresponding physical image patent label to obtain the same-domain matching performance.

[0045] Specifically, to evaluate the performance of candidate multimodal models in same-domain matching scenarios, product images with patent tags attached to physical images from the test set are first input into each candidate multimodal model. The visual encoder of the candidate multimodal model extracts features from the product images, obtaining a first feature vector with dimensions of 768 or 1024. Simultaneously, the corresponding patent tags are also input into the visual encoder of the candidate multimodal model to obtain the feature vector of the patent tags. Next, cosine similarity is used to calculate the similarity between the first feature vector and the patent tag feature vector, calculated as the dot product of the two vectors divided by the product of their magnitudes. Based on the calculated similarity values, the recall and hit rates are statistically analyzed across the top 1, top 10, and top 100 dimensions, thus obtaining performance metrics reflecting the model's same-domain matching ability. This evaluation method comprehensively reflects the model's ability to extract and match features from physical images.

[0046] Step 102: Input the product image of the physical image patent label into each preset candidate multimodal model to obtain the first feature vector, and calculate the similarity between the first feature vector and the feature vector of the corresponding physical image patent label to obtain the same domain matching performance.

[0047] Specifically, to evaluate the performance of candidate multimodal models in cross-domain matching scenarios, product images with line drawing patent tags from the test set are input into each candidate multimodal model. The visual encoder of the candidate multimodal model extracts features from the product images, obtaining a second feature vector with dimensions of 768 or 1024. Simultaneously, the corresponding line drawing patent tags are also input into the visual encoder of the candidate multimodal model to obtain the patent tag feature vector. Then, cosine similarity is used to calculate the similarity between the second feature vector and the patent tag feature vector, calculated as the dot product of the two vectors divided by the product of their magnitudes. Based on the calculated similarity values, the recall and hit rates are statistically analyzed across the top 1, top 10, and top 100 dimensions, thus obtaining performance metrics reflecting the model's cross-domain matching ability. This evaluation method comprehensively reflects the model's performance in handling cross-domain matching between physical images and line drawings.

[0048] Step 103: Determine the retrieval performance of each candidate multimodal model based on the same-domain matching performance and cross-domain matching performance.

[0049] Specifically, to comprehensively evaluate the overall retrieval performance of each candidate multimodal model, weighted calculations of intra-domain matching performance and cross-domain matching performance are required. First, the intra-domain matching recall and cross-domain matching recall of each model are calculated across the TOP 1, TOP 10, and TOP 100 dimensions. Then, based on the needs of the actual application scenario, the weight ratios for intra-domain matching and cross-domain matching are set; for example, the intra-domain matching weight can be set to 0.4, and the cross-domain matching weight to 0.6, thus emphasizing the model's cross-domain matching ability. Next, the weighted recall is calculated to obtain the final performance score, with TOP 1 accounting for 0.5, TOP 10 for 0.3, and TOP 100 for 0.2. By comparing the performance scores of each candidate multimodal model, the model with the highest score is selected as the base model. This evaluation method considers both the model's precise matching ability and its broad recall ability, enabling the selection of the model most suitable for practical application needs.

[0050] Step 30: Construct a data synthesis framework based on the preset generative model. The data synthesis framework combines the edge feature information of the patent line drawing and the style characteristics of the product image to generate image pairs for the second stage of training.

[0051] In this embodiment of the application, the preset generative model refers to a generative model that can learn to convert patent line drawings into product-style images. For example, it can be an image-to-image conversion model (such as GAN, VAE, etc.). This model can understand and convert the style features of the image.

[0052] Specifically, due to the significant visual differences between patented line drawings and actual product images, directly using existing actual product images for training may result in insufficient cross-domain matching ability between line drawings and actual product images. To address this issue, this embodiment constructs a data synthesis framework based on a pre-defined generative model. This generative model adopts a CycleGAN network structure, including two generators and two discriminators. The first generator is used to convert line drawings into actual product image styles, the second generator is used to convert actual product images into line drawing styles, and the two discriminators are used to determine the authenticity of the generated images.

[0053] First, patent line drawings and product images are collected as training samples. During training, the data synthesis framework extracts edge feature information from the patent line drawings using an edge detection algorithm, including line thickness, continuity, and topological structure. Simultaneously, a style transfer module extracts style characteristics from the product images, including visual features such as texture, color, and brightness distribution. Based on these features, the generator converts the patent line drawings into synthesized images with the style characteristics of product images, while preserving the structural information of the original line drawings. During the conversion process, a cycle consistency loss is used to ensure structural consistency between the generated image and the original line drawing, and an adversarial loss is used to ensure that the generated image has a realistic product image style.

[0054] The data synthesis framework pairs each input patent line drawing with its corresponding generated synthetic object image. Specifically, for each patent line drawing, it is used as the first image in the image pair, and the synthetic object image generated by the data synthesis framework based on that line drawing is used as the second image in the image pair, thus constructing image pairs that contain the mapping relationship between line drawings and object images. These image pairs retain the key structural features of the patent line drawing while possessing the visual representation of a product image, helping the model establish the mapping relationship between line drawings and object images. These synthesized image pairs are used for the second stage of contrastive learning training, enabling the model to better understand and match images from different domains, thereby improving retrieval accuracy. Furthermore, the data synthesis framework can generate a large number of training samples based on different patent line drawings, effectively solving the problem of insufficient training data.

[0055] Step 40: Perform the first stage of incremental pre-training on the base model based on the first training set to obtain the pre-trained model. The first training set includes multiple first line drawings and corresponding text descriptions.

[0056] In this embodiment of the application, the first training set refers to the dataset used for the first stage of incremental pre-training, which includes hundreds of thousands of line drawings collected from the patent library and preprocessed with their corresponding text descriptions. The line drawings are processed by contrast adjustment and edge enhancement, and the text descriptions are processed by word segmentation and standardization to improve the model's ability to recognize line drawings.

[0057] A pre-trained model refers to a model obtained after the first stage of incremental pre-training. This model performs cross-modal comparative learning on line drawings and text descriptions in the first training set, enabling the model to map image and text features to a unified vector space. This improves the model's ability to recognize and match line drawings and lays the foundation for subsequent comparative learning training.

[0058] Specifically, hundreds of thousands of patent data points, including line drawings and their corresponding patent text descriptions, are collected from a patent database. The collected data is preprocessed: the line drawings undergo contrast adjustment and edge enhancement to obtain the first line drawing, highlighting the image's contour features; the text descriptions are segmented and standardized, removing overly long descriptions and normalizing the text, thus constructing the first training set. During training, the visual encoder of the base model extracts features from the first line drawing to obtain image feature vectors; simultaneously, the text encoder of the base model extracts features from the processed text descriptions to obtain text feature vectors. A contrastive learning training strategy is adopted, using the InfoNCE loss function to calculate the similarity loss between the image feature vectors and the text feature vectors, and the parameters of the base model are optimized and updated based on this loss, enabling the model to map image and text features to a unified vector space. After this incremental pre-training, a pre-trained model is obtained. This model inherits the basic feature extraction capabilities of the base model while significantly improving its understanding of line drawings, better capturing design features and structural information within the line drawings.

[0059] Based on the above embodiments, as another optional embodiment, the step of performing a first-stage incremental pre-training on the base model based on the first training set to obtain a pre-trained model may further include steps 201-203:

[0060] Step 201: Collect line drawings and corresponding text descriptions from the patent database, and preprocess the line drawings and text descriptions to obtain the first training set, which includes multiple first line drawings and corresponding text descriptions.

[0061] Specifically, to improve the clarity of line graph features and the standardization of text descriptions, preprocessing of the original data is necessary. For line graph processing, firstly, the `cv2.cvtColor` function is used to convert the image from RGB format to grayscale to highlight the structural features of the lines; then, the `cv2.equalizeHist` function is used to perform histogram equalization on the grayscale image to enhance image contrast and make the lines more clearly visible; next, the `cv2.Canny` edge detection operator is applied for edge enhancement, with a low threshold set to 50 and a high threshold set to 150 to extract the main contour information in the image; finally, the enhanced image is normalized, mapping pixel values ​​to the range of 0-1 to obtain a standardized first line graph. For text description processing, firstly, the `re.sub` function is used to clean up special characters and redundant information in the text based on regular expressions; then, the `jieba.cut` function is called to segment the text, and a predefined stop word list is used to filter out meaningless words; finally, the segmentation results are standardized, including converting all English characters to lowercase, standardizing number formats, and performing lemmatization. These preprocessing operations can improve data quality and provide better input for subsequent model training.

[0062] To construct effective training samples, the processed first line graphs and text descriptions need to be paired and combined. Specifically, firstly, a mapping relationship between line graphs and text descriptions is established based on the patent document number; then, the text length is filtered, retaining text descriptions with 10 to 512 tokens; next, the pairing results are quality checked to ensure that each first line graph has a corresponding standardized text description; finally, the qualified paired samples are saved in tfrecord format to form the first training set. During the saving process, a unique identifier is assigned to each sample, and the image size information and the number of text tokens are recorded. This pairing method not only ensures the integrity of the training data but also provides a data format that is easy to read in batches. The first training set obtained through this process contains a large number of high-quality first line graphs and their corresponding text descriptions, providing a reliable data foundation for subsequent model training.

[0063] Step 202: Encode the first line drawing in the first training set using the visual encoder of the base model to obtain image features.

[0064] Specifically, to obtain the visual feature representation of the first line drawing, the visual encoder in the pedestal model is used to extract features from the first line drawing in the first training set. First, the first line drawing is adjusted to the input size required by the visual encoder (e.g., 224×224 pixels); then, feature extraction is performed through a multi-layer convolutional network of the visual encoder to obtain an initial feature map; subsequently, after processing by an attention mechanism and fully connected layers, the final output is an image feature vector with a dimension of 768 or 1024. This encoding method can effectively capture the visual features and structural information in the line drawing, providing a foundation for subsequent feature alignment.

[0065] Step 203: Use the text encoder of the base model to encode the text descriptions in the first training set to obtain text features.

[0066] Specifically, to obtain the semantic feature representation of the text description, the text encoder in the base model is used to extract features from the text description in the first training set. First, the text is segmented and tokenized, converting it into a sequence of tokens. Then, positional encoding is used to add positional information to the tokens. Next, a transformer structure is used to encode the token sequence, obtaining the feature representation of each token. Finally, pooling is used to obtain the feature vector of the entire text, with the same dimension as the image feature vector. This encoding method can effectively extract semantic information from the text and facilitates alignment with image features.

[0067] Step 204: Calculate the similarity loss between image features and text features, and optimize the parameters of the base model based on the similarity loss to obtain the pre-trained model.

[0068] Specifically, to optimize the base model parameters and better align image and text features, it's necessary to calculate the similarity loss between features and update the parameters. First, the cosine similarity matrix between image and text features in each batch is calculated. Then, the InfoNCE loss function is used to calculate the similarity loss, maximizing the similarity of the same image-text pair while minimizing the similarity of different image-text pairs. Next, the Adam optimizer is used to update the model parameters using gradients based on the calculated loss values, with a learning rate set to 2e-5 and 10 training epochs. To prevent overfitting, a weight decay strategy is employed, with a weight decay coefficient set to 0.01. The pre-trained model trained in this way can map image and text features to the same feature space, improving the model's ability to understand line graphs.

[0069] Step 50: Perform a second stage of comparative learning training on the pre-trained model based on the second training set to obtain the target retrieval model. The second training set includes image pairs composed of second line graphs and object images. The second line graphs are obtained by filtering based on each first line graph, and the object images include synthetic object images generated by the data synthesis framework based on each second line graph.

[0070] In this embodiment, the second training set refers to the dataset used for the second-stage comparative learning training. This second training set includes multiple image pairs, each consisting of a second line drawing and its corresponding physical image. The second line drawing is obtained by filtering the first line drawings from the first training set, and the corresponding physical image is a synthesized physical image generated by the data synthesis framework based on the second line drawing. For example, an image pair may contain a patent line drawing of a mobile phone and its corresponding synthesized physical image; these two images express the same design content but have different visual representations.

[0071] The target retrieval model refers to the final model used for retrieving appearance patent images, obtained through a two-stage training process. This model is built upon a foundational model, first undergoing incremental pre-training to obtain a pre-trained model, and then undergoing comparative learning training in the second stage to arrive at the final model. The model comprises two main components: a visual encoder and a text encoder. The visual encoder extracts visual features from the images, while the text encoder processes descriptive text information. The primary function of the target retrieval model is to receive user-uploaded images of products to be retrieved, extract feature vectors from these images using its visual encoder, and perform feature matching with images in the patent database to output a list of patent images with high similarity.

[0072] Specifically, after completing the first stage of incremental pre-training, to further improve the model's cross-domain matching ability between line drawings and physical images, a second stage of comparative learning training is needed. First, a second training set is constructed: the first line drawings in the first training set are quality-screened, selecting images with clear lines and complete structures as the second line drawings. Then, a data synthesis framework is used to convert each second line drawing into a corresponding synthetic physical image. The second line drawings and their corresponding synthetic physical images are combined to form image pairs, which together constitute the second training set. For example, for a certain mobile phone design, the second training set includes the patent line drawings of the phone (i.e., the second line drawings) and synthetic images with a physical image style generated by the data synthesis framework (i.e., the physical images).

[0073] In the second training phase, the text encoder parameters of the pre-trained model are first frozen, as this phase primarily focuses on learning image features. Simultaneously, to maintain the model's learned basic feature extraction capabilities, other parameters in the visual encoder, except for the normalization layer, also need to be frozen. Then, image pairs from the second training set are input into the pre-trained model to extract feature vectors for the second line drawing and its corresponding object image. A contrastive loss function is used to calculate the similarity between the feature vectors of two images in the same image pair. By minimizing the contrastive loss, line drawings and object images with the same design content are closer in feature space, while images with different design content are farther apart. Based on the calculated loss value, only the normalization layer parameters of the pre-trained model are optimized, ultimately yielding the target retrieval model.

[0074] Based on the above embodiments, as another optional embodiment, the step of performing a second stage of comparative learning training on the pre-trained model based on the second training set to obtain the target retrieval model may further include steps 301-303:

[0075] Step 301: Construct a second training set based on each of the first line graphs and the data synthesis framework.

[0076] Specifically, to construct a high-quality second training set, a data synthesis framework was first built based on the first line drawing. This framework comprises two core components: a Stable Diffusion model and a ControlNet module. The Stable Diffusion model performs style transfer from the line drawing to the object image, while the ControlNet module ensures that the generated images retain the structural features of the original line drawing. During construction, the first line drawing was first screened for quality using edge detection and resolution checks to obtain the second line drawing. Then, the second line drawing and its corresponding text prompts were input into the data synthesis framework. The Stable Diffusion model generated the object image, and the ControlNet module constrained the generation process, resulting in a synthesized image that retains the original structure and possesses the characteristics of the actual product. Finally, the second line drawing and the corresponding generated object image were combined into image pairs. Feature similarity was evaluated on these image pairs, and samples with low similarity were removed to obtain the second training set.

[0077] Step 302: Freeze the text encoder parameters of the pre-trained model and freeze all parameters in the visual encoder of the pre-trained model except for the normalization layer parameters.

[0078] Specifically, to maintain the basic feature extraction capabilities learned by the pre-trained model while achieving efficient parameter optimization, parameter freezing is necessary. First, all parameters of the text encoder in the pre-trained model are frozen by setting `requires_grad=False`, since the second stage of training primarily focuses on optimizing image features. Then, the parameters of the visual encoder are selectively frozen, keeping the parameters of basic feature extraction layers such as convolutional layers and self-attention layers unchanged, while only setting the parameters of the layer normalization layer to a trainable state (`requires_grad=True`). This parameter freezing strategy can maintain the model's basic feature extraction capabilities while optimizing feature distribution by fine-tuning the normalization layer parameters, thereby improving the model's generalization ability.

[0079] Step 303: Input the second line drawing and the corresponding physical object image from the second training set into the pre-trained model for feature extraction.

[0080] Specifically, to obtain the feature representation of the images, the images in the second training set need to be used to extract features from the pre-trained model. First, the second line drawing and the object image are adjusted to the required input size (224×224 pixels) for the model. Then, data augmentation operations, including random horizontal flipping (probability 0.5) and random cropping, are used to increase data diversity. Next, the processed images are input into the visual encoder of the pre-trained model, where feature extraction is performed through a multi-layer convolutional network and attention mechanism, resulting in a 768-dimensional feature vector. This feature extraction method preserves the key visual information of the image, providing a foundation for subsequent comparative learning.

[0081] Step 304: Calculate the feature extraction results using the contrastive loss function, and optimize the normalization layer parameters of the pre-trained model based on the calculation results to obtain the target retrieval model.

[0082] Specifically, to optimize model parameters for better matching of similar images, a contrastive loss function is used for training. First, a batch of 256 second-line graphs and their corresponding object images are input into the model for feature extraction. Then, the cosine similarity between the feature vectors of the same image pair (second-line graph and corresponding object image) is calculated, and this similarity is used as the similarity score for positive sample pairs. Simultaneously, the cosine similarity between feature vectors of different image pairs within the same batch is calculated, yielding similarity scores for 255 negative sample pairs. Next, the InfoNCE loss function is used to calculate the contrastive loss. This loss function adjusts the distribution of similarity scores through a scaling factor, making the similarity score of positive sample pairs approach 1 and the similarity score of negative sample pairs approach 0, thereby increasing the discriminative power between positive and negative sample pairs. Finally, the Adam optimizer is used to update the gradient of the normalized layer parameters of the pre-trained model, with the learning rate set to 1e-4 and the weight decay coefficient set to 0.01. Every 1000 steps, the learning rate is reduced to 0.1 times its original value. During training, the accuracy on the validation set is calculated every 1000 training steps. Training stops when the validation set accuracy no longer improves after five consecutive iterations, resulting in the target retrieval model. The model trained in this way can more accurately identify similar image pairs, improving the TOP-1 accuracy on the test set by more than 15%. This training strategy not only significantly improves the model's performance on cross-domain image matching tasks but also demonstrates good convergence and generalization capabilities.

[0083] Based on the above embodiments, as another optional embodiment, the step of constructing the second training set based on each first line graph and the data synthesis framework may further include steps 401-403:

[0084] Step 401: Filter the first line drawing to obtain the second line drawing.

[0085] Specifically, to ensure the generation of high-quality physical images, the initial line drawing needs to undergo rigorous quality screening. First, the Canny edge detection algorithm is used to check the integrity of the lines. The algorithm's execution process is as follows: First, the input image is Gaussian filtered, using a 3×3 Gaussian kernel to eliminate noise; then, the gradient magnitude and direction of the image are calculated, and the Sobel operator is used for horizontal and vertical edge detection; next, non-maximum suppression is performed to retain local maxima points; finally, dual threshold detection and connection are used, setting a low threshold of 100 and a high threshold of 200. Edge points located between the two thresholds are connected through 8-neighborhood connections to determine whether to retain them. The number of edge connection points is counted; if it is less than 85% of the total number of edge points, the image is considered incomplete and discarded.

[0086] Secondly, the image resolution is checked, requiring that both the length and width of the image be no less than 224 pixels. For images with a resolution exceeding 224×224, bilinear interpolation is used for proportional scaling to maintain the aspect ratio. Then, the peak signal-to-noise ratio (PSNR) is calculated by comparing the original image with its version after adding Gaussian noise, calculating the mean squared error (MSE) and PSNR values, and discarding images with a PSNR value below 30dB. Finally, the image contrast is calculated using the standard deviation as a metric, requiring that the standard deviation of the image pixel values ​​be no less than 45 to ensure clear line visibility. Through these rigorous quality control measures, a second line drawing suitable for subsequent processing is finally obtained.

[0087] Step 402: Based on the data synthesis framework, the second line drawing is transformed to obtain the corresponding physical image, and each second line drawing and the corresponding physical image are combined into an image pair.

[0088] Specifically, to convert the second line drawing into a realistic image while preserving the structural features of the original design, a data compositing framework based on Stable Diffusion and ControlNet was used. First, the second line drawing was preprocessed: pixel values ​​were normalized to the range [-1, 1], and the image size was uniformly adjusted to 512×512 pixels. Then, positive and negative text prompts were set according to the design category. Positive prompts described the desired features of the generated image, including material, gloss, and shooting effect, while negative prompts mitigated quality defects such as blurriness, distortion, and watermarks.

[0089] During the generation process, a DPM sampler was employed with 50 sampling steps to ensure stability. The classifier guidance coefficient was set to 7.5 to enhance relevance to prompts. Simultaneously, the edge detection mode of the ControlNet module was enabled for structural constraints, with a weight of 0.8, ensuring the generated image retains the key structure of the original line drawing. For the attention mechanism, a cross-attention weight of 1.2 was used to enhance the model's ability to preserve the structural features of the line drawing. Post-processing optimization was performed on the generated object images, including: applying contrast-limited adaptive histogram equalization with a grid size of 8×8 and a contrast limit of 3; and using an unsharpened masking algorithm to enhance details with a sharpening radius of 2 pixels and an intensity of 0.5. Finally, each second line drawing was paired with its optimized corresponding object image to form image pairs. This refined generation strategy ensures that the generated images retain both the structural features of the original design and the visual style of the real product, providing high-quality training samples for subsequent comparative learning.

[0090] Step 403: Input the second line drawing into the pre-trained model to obtain the third feature vector, and input the corresponding physical image into the pre-trained model to obtain the fourth feature vector.

[0091] Specifically, to evaluate the matching degree of image pairs, a pre-trained model is needed to extract image features. First, the second line drawing and the object image are adjusted to the input size required by the model (224×224 pixels). Then, the visual encoder of the pre-trained model extracts features from the second line drawing to obtain a third feature vector with a dimension of 768. Similarly, features are extracted from the object image to obtain a fourth feature vector with a dimension of 768. This feature extraction method can capture key visual information of the image, which is convenient for subsequent similarity calculation.

[0092] Step 404: Calculate the cosine similarity between the third feature vector and the fourth feature vector, and remove image pairs with similarity less than a preset threshold to obtain the second training set.

[0093] Specifically, to ensure the quality of the second training set, image pairs need to be filtered based on feature similarity. First, the cosine similarity between the third and fourth feature vectors of each image pair is calculated using the formula: the dot product of the two vectors divided by the product of their magnitudes. Then, the calculated similarity is compared to a preset threshold of 0.75. Finally, image pairs with a similarity greater than or equal to 0.75 are retained, and these high-quality image pairs are used to form the second training set. This feature similarity-based filtering method ensures that image pairs in the training data have a good correspondence, improving the effectiveness of subsequent training. Experiments show that this filtering method can improve the quality of the training data by more than 20%, providing a more reliable data foundation for model training.

[0094] Step 60: Receive the target product image to be retrieved, extract features from the target product image based on the target retrieval model, and output the corresponding patent image list.

[0095] In this embodiment of the application, the target product image refers to the product image that needs to be searched for patents, including but not limited to product display images on e-commerce platforms, product photos, etc. After preprocessing, the image is input into the target retrieval model for feature extraction, which is used to match the patent image for similarity.

[0096] The patent image list refers to the patent image search results obtained after similarity calculation and sorting. It includes multiple patent images and their corresponding patent information, sorted from high to low similarity to the target product image. Each patent image is labeled with a similarity score, making it easy for users to quickly locate relevant patents.

[0097] Specifically, to achieve rapid patent image retrieval, the input target product image needs to be processed and matched. First, the received target product image is preprocessed, including resizing it to 224×224 pixels and normalizing it. Then, the processed image is input into the visual encoder of the target retrieval model to extract a 768-dimensional feature vector. Next, the cosine similarity between this feature vector and the feature vectors of all patent images in the patent database is calculated to obtain a similarity score. Finally, the patent images are sorted in descending order based on the similarity score, and the top 100 patent images with the highest similarity, along with their corresponding patent numbers, patent names, applicant information, etc., are selected to form a patent image list and output to the user. This retrieval method can complete patent image similarity matching within seconds. Compared to traditional keyword retrieval methods, it can more accurately discover similar patents, improving the retrieval recall rate by more than 30%, effectively improving the efficiency and accuracy of patent retrieval.

[0098] Based on the above embodiments, as an optional embodiment, a method for image retrieval of a design patent may further include the following steps:

[0099] Specifically, to improve the comprehensiveness of patent searches, it is necessary to conduct extended searches based on the initial search results. First, the IPC and CPC classification numbers corresponding to each patent image are extracted from the patent image list, and the frequency of these classification numbers is counted. Then, the top 5 most frequent classification numbers are selected as the search terms for extended searches, and patent documents with the same classification numbers are searched in the patent database. Next, patent images are extracted from these patent documents, and these images are input into the target search model for feature extraction to obtain feature vectors. Then, the cosine similarity between these feature vectors and the feature vectors of the target product images is calculated. For patent images with a similarity greater than 0.7, they are added to the original patent image list along with the patent number, patent name, applicant, and other information. This classification-based extended search method can discover more relevant patents; experiments show that it can increase the number of relevant patent results by an additional 20%-30%, significantly improving the completeness and accuracy of the search results.

[0100] Please see Figure 2 This is a schematic diagram of a module of an image retrieval system for a design patent provided in an embodiment of this application, wherein the system includes:

[0101] The test set construction module is used to construct a test set based on multiple collected product images and the corresponding patent image tags for each product image;

[0102] The base model determination module is used to evaluate the retrieval performance of multiple preset candidate multimodal models based on the test set. The retrieval performance includes intra-domain matching ability and cross-domain matching ability, and the candidate multimodal model with the best retrieval performance is used as the base model.

[0103] The data synthesis module is used to construct a data synthesis framework based on a preset generative model. The data synthesis framework combines the edge feature information of the patent line drawing and the style characteristics of the product image to generate image pairs for the second stage of training.

[0104] The first-stage training module is used to perform incremental pre-training on the base model based on the first training set to obtain a pre-trained model. The first training set includes multiple first line drawings and corresponding text descriptions.

[0105] The second-stage training module is used to perform a second-stage comparative learning training on the pre-trained model based on the second training set to obtain a target retrieval model. The second training set includes image pairs composed of second line drawings and object images. The second line drawings are obtained by filtering based on each of the first line drawings. The object images include synthetic object images generated by the data synthesis framework based on each of the second line drawings.

[0106] The patent retrieval output module is used to receive the target product image to be retrieved, extract features from the target product image based on the target retrieval model, and output the corresponding patent image list.

[0107] Optionally, the base model determination module is further configured to input the product image of the physical image patent label into each preset candidate multimodal model to obtain a first feature vector, and calculate the similarity between the first feature vector and the feature vector of the corresponding physical image patent label to obtain the same domain matching performance.

[0108] The product image of the line drawing patent label is input into each preset candidate multimodal model to obtain the second feature vector, and the similarity between the second feature vector and the feature vector of the corresponding line drawing patent label is calculated to obtain the cross-domain matching performance.

[0109] The retrieval performance of each candidate multimodal model is determined based on the intra-domain matching performance and the cross-domain matching performance.

[0110] Optionally, the first-stage training module is also used to collect line drawings and corresponding text descriptions from the patent database, and to preprocess the line drawings and text descriptions to obtain a first training set, which includes multiple first line drawings and corresponding text descriptions.

[0111] The visual encoder of the base model is used to encode the first line drawing in the first training set to obtain image features;

[0112] The text encoder of the base model is used to encode the text descriptions in the first training set to obtain text features;

[0113] Calculate the similarity loss between the image features and the text features, and optimize the parameters of the base model based on the similarity loss to obtain a pre-trained model.

[0114] Optionally, the first-stage training module is also used to perform contrast adjustment and edge enhancement preprocessing on the line drawing, as well as word segmentation and standardization on the text description;

[0115] The processed line graphs and text descriptions are paired to obtain the first training set.

[0116] Optionally, the second-stage training module is also used to construct a second training set based on each of the first line graphs and the data synthesis framework;

[0117] Freeze the text encoder parameters of the pre-trained model, and freeze all parameters of the visual encoder of the pre-trained model except for the normalization layer parameters.

[0118] The second line drawing and the corresponding physical image in the second training set are input into the pre-trained model for feature extraction.

[0119] The feature extraction results are calculated using a contrastive loss function, and the normalization layer parameters of the pre-trained model are optimized based on the calculation results to obtain the target retrieval model.

[0120] Optionally, the second-stage training module is also used to filter the first line graph to obtain the second line graph;

[0121] Based on the data synthesis framework, the second line drawing is transformed to obtain the corresponding physical image, and each second line drawing and the corresponding physical image are combined into an image pair.

[0122] The second line drawing is input into the pre-trained model to obtain the third feature vector, and the corresponding physical image is input into the pre-trained model to obtain the fourth feature vector;

[0123] Calculate the cosine similarity between the third feature vector and the fourth feature vector, and remove image pairs with similarity less than a preset threshold to obtain the second training set.

[0124] Optionally, the patent search output module is also used to obtain the classification number of each patent image in the patent image list;

[0125] Extended searches are performed in the patent database based on the classification number;

[0126] The similarity of the patent images obtained from the extended search is calculated, and patent images with a similarity greater than a preset threshold are added to the patent image list.

[0127] It should be noted that the system provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0128] This application also provides a computer storage medium that can store multiple instructions. The instructions are adapted to be loaded and executed by a processor, representing an image retrieval method for a design patent according to the above embodiments. For the specific execution process, please refer to the detailed description of the above embodiments, which will not be repeated here.

[0129] Please refer to Figure 3 This application also discloses an electronic device. Figure 3 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. The electronic device 300 may include: at least one processor 301, at least one network interface 304, a user interface 303, a memory 305, and at least one communication bus 302.

[0130] The communication bus 302 is used to enable communication between these components.

[0131] The user interface 303 may include a display screen and a camera. Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0132] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0133] The processor 301 may include one or more processing cores. The processor 301 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 305, and by calling data stored in the memory 305. Optionally, the processor 301 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array. The processor 301 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor 301.

[0134] The memory 305 may include random access memory (RAM) or read-only memory. Optionally, the memory 305 may include a non-transitory computer-readable storage medium. The memory 305 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 305 may also be at least one storage device located remotely from the aforementioned processor 301. (Refer to...) Figure 3 The memory 305, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for an image retrieval method according to a design patent.

[0135] exist Figure 3In the illustrated electronic device 300, the user interface 303 is mainly used to provide an input interface for the user and to acquire user input data; while the processor 301 can be used to call an application program storing an image retrieval method for a design patent in the memory 305. When executed by one or more processors 301, the electronic device 300 performs one or more methods as described in the above embodiments. It should be noted that, for the foregoing method embodiments, for the sake of simplicity, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0136] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0137] In the various embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between apparatuses or units may be electrical or other forms.

[0138] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0139] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0140] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0141] The above are merely exemplary embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Other embodiments of this disclosure will be readily apparent to those skilled in the art upon consideration of the disclosure herein.

[0142] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art that are not described in this disclosure.

Claims

1. A method for image retrieval of a design patent, characterized in that, The method includes: A test set was constructed based on multiple product images collected and the corresponding patent image tags for each product image; The retrieval performance of multiple preset candidate multimodal models is evaluated based on the test set. The retrieval performance includes intra-domain matching ability and cross-domain matching ability. The candidate multimodal model with the best retrieval performance is used as the base model. A data synthesis framework is constructed based on a preset generative model. The data synthesis framework combines the edge feature information of the patent line drawing and the style characteristics of the product image to generate image pairs for the second stage of training. The base model is subjected to incremental pre-training in the first stage based on the first training set to obtain a pre-trained model. The first training set includes multiple first line drawings and corresponding text descriptions. The pre-trained model is trained in a second stage of comparative learning based on the second training set to obtain a target retrieval model. The second training set includes image pairs composed of second line drawings and object images. The second line drawings are obtained by filtering based on each of the first line drawings. The object images include synthesized object images generated by the data synthesis framework based on each of the second line drawings. Receive the target product image to be searched, extract features from the target product image based on the target retrieval model, and output the corresponding patent image list.

2. The image retrieval method for design patents according to claim 1, characterized in that, The patent image tags include physical image patent tags and line drawing patent tags. The evaluation of the retrieval performance of multiple preset candidate multimodal models based on the test set includes: The product image of the physical image patent label is input into each preset candidate multimodal model to obtain the first feature vector, and the similarity between the first feature vector and the feature vector of the corresponding physical image patent label is calculated to obtain the same domain matching performance. The product image of the line drawing patent label is input into each preset candidate multimodal model to obtain the second feature vector, and the similarity between the second feature vector and the feature vector of the corresponding line drawing patent label is calculated to obtain the cross-domain matching performance. The retrieval performance of each candidate multimodal model is determined based on the intra-domain matching performance and the cross-domain matching performance.

3. The image retrieval method for design patents according to claim 1, characterized in that, The first stage of incremental pre-training of the base model based on the first training set to obtain a pre-trained model includes: The line drawings and corresponding text descriptions in the patent database are collected, and the line drawings and text descriptions are preprocessed to obtain a first training set, which includes multiple first line drawings and corresponding text descriptions. The visual encoder of the base model is used to encode the first line drawing in the first training set to obtain image features; The text encoder of the base model is used to encode the text descriptions in the first training set to obtain text features; Calculate the similarity loss between the image features and the text features, and optimize the parameters of the base model based on the similarity loss to obtain a pre-trained model.

4. The image retrieval method for design patents according to claim 3, characterized in that, The preprocessing of the line drawing and the text description to obtain the first training set includes: The line drawing is preprocessed with contrast adjustment and edge enhancement, and the text description is processed with word segmentation and standardization. The processed line graphs and text descriptions are paired to obtain the first training set.

5. The image retrieval method for design patents according to claim 1, characterized in that, The second stage of comparative learning training of the pre-trained model based on the second training set to obtain the target retrieval model includes: A second training set is constructed based on each of the first line graphs and the data synthesis framework; Freeze the text encoder parameters of the pre-trained model, and freeze all parameters of the visual encoder of the pre-trained model except for the normalization layer parameters; The second line drawing and the corresponding physical image in the second training set are input into the pre-trained model for feature extraction. The feature extraction results are calculated using a contrastive loss function, and the normalization layer parameters of the pre-trained model are optimized based on the calculation results to obtain the target retrieval model.

6. The image retrieval method for a design patent according to claim 5, characterized in that, The construction of the second training set based on each of the first line graphs and the data synthesis framework includes: The first line drawing is filtered to obtain the second line drawing; Based on the data synthesis framework, the second line drawing is transformed to obtain the corresponding physical image, and each second line drawing and the corresponding physical image are combined into an image pair. The second line drawing is input into the pre-trained model to obtain the third feature vector, and the corresponding physical image is input into the pre-trained model to obtain the fourth feature vector; Calculate the cosine similarity between the third feature vector and the fourth feature vector, and remove image pairs with similarity less than a preset threshold to obtain the second training set.

7. The image retrieval method for design patents according to claim 1, characterized in that, The method further includes: Obtain the classification number of each patent image in the patent image list; Extended searches are performed in the patent database based on the classification number; The similarity of the patent images obtained from the extended search is calculated, and patent images with a similarity greater than a preset threshold are added to the patent image list.

8. An image retrieval system for a design patent, characterized in that, The system includes: The test set construction module is used to construct a test set based on multiple collected product images and the corresponding patent image tags for each product image; The base model determination module is used to evaluate the retrieval performance of multiple preset candidate multimodal models based on the test set. The retrieval performance includes intra-domain matching ability and cross-domain matching ability, and the candidate multimodal model with the best retrieval performance is used as the base model. The data synthesis module is used to construct a data synthesis framework based on a preset generative model. The data synthesis framework combines the edge feature information of the patent line drawing and the style characteristics of the product image to generate image pairs for the second stage of training. The first-stage training module is used to perform incremental pre-training on the base model based on the first training set to obtain a pre-trained model. The first training set includes multiple first line drawings and corresponding text descriptions. The second-stage training module is used to perform a second-stage comparative learning training on the pre-trained model based on the second training set to obtain a target retrieval model. The second training set includes image pairs composed of second line drawings and object images. The second line drawings are obtained by filtering based on each of the first line drawings. The object images include synthetic object images generated by the data synthesis framework based on each of the second line drawings. The patent retrieval output module is used to receive the target product image to be retrieved, extract features from the target product image based on the target retrieval model, and output the corresponding patent image list.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores multiple instructions suitable for being loaded by a processor and executed as described in any one of claims 1-7.

10. An electronic device, characterized in that, It includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any one of claims 1-7.

Images