An appearance design patent image retrieval method based on multi-modal classification focusing

Abstract

The application discloses an appearance design patent image retrieval method based on multi-modal classification focusing. The method first constructs training data based on Logano product items, and obtains a product item identification model by using a lightweight multi-modal large model and LoRA fine-tuning. After product item labeling of all library images, image and text double-modal encoding is performed to construct vector indexes. During retrieval, a double-stage strategy of image encoding recall and text encoding rearrangement is adopted, and linear weighting fusion of image similarity and text similarity is performed to realize result rearrangement. The three-level technical architecture of classification focusing first, encoding retrieval second, and rearrangement optimization third is adopted to significantly improve the retrieval effect.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence and intellectual property technology, and in particular to a method for retrieving design patent images based on multimodal classification and focusing. Background Technology

[0002] Currently, design patent image retrieval mainly relies on unimodal image similarity matching methods. This involves using pre-trained image encoders (such as ResNet and ViT) to encode the design patent drawings, generating feature vectors, and then performing similarity searches in a vector database. This method takes the entire image as input and performs end-to-end similarity calculations using global features. However, in design patent retrieval scenarios, many product local features (such as displays, buttons, and indicator lights) are widely present in different product categories. This makes unimodal retrieval susceptible to interference from these shared local features, resulting in noisy cross-category results.

[0003] With the development of multimodal technologies, multimodal coding models such as CLIP and Qwen3-VL-8B-Embedding have been introduced into the field of image retrieval. However, these models still suffer from the problem of local feature similarity outweighing global category features in the context of design patent retrieval, failing to effectively distinguish between "intra-category similarity" and "cross-category local similarity," resulting in the top-K positions of the retrieval results containing a large amount of cross-category noise.

[0004] Furthermore, there is significant inconsistency in the title information of design patents. The same type of product may be named by multiple different names (such as "laser tape measure", "laser rangefinder", "laser range measuring tape", "steel tape measure laser rangefinder", etc.), making it difficult to use the title information directly for search and focusing. Summary of the Invention

[0005] Based on this, the embodiments of this application provide a design patent image retrieval method based on multimodal classification and focusing, which achieves a significant improvement in retrieval performance through a three-level technical architecture of "classification and focusing first, encoding retrieval second, and rearrangement optimization third".

[0006] Firstly, a method for retrieving design patent images based on multimodal classification and focusing is provided, the method comprising:

[0007] S1. Construct training data based on Logano product items: For each design patent drawing, obtain the secondary classification number in its patent bibliographic information and match it with the corresponding product item description in the Logano classification table to construct image-text pair training data of image and product item description text.

[0008] S2. Training the multimodal classification model: Based on the lightweight multimodal large model, the training data is fine-tuned using the aforementioned images and text to obtain the product item recognition model;

[0009] S3. Full-database classification and labeling: Using the product item recognition model, perform batch reasoning on the design patent drawings in the patent image database to generate corresponding product item description text for each image;

[0010] S4. Bimodal Encoding and Index Construction: The product item description text and its corresponding design patent drawings are input into a multimodal encoding model for bimodal encoding to obtain text vector representation and image vector representation respectively, and a bimodal vector index is constructed based on the text vector representation and image vector representation;

[0011] S5. Retrieval and Rearrangement: Receive a query image, generate product item description text for the query image using the product item recognition model, and encode the query image and its product item description text to obtain a query image vector and a query text vector, respectively; perform similarity retrieval in the bimodal vector index based on the query image vector to obtain a candidate result set, calculate the similarity between the text vector corresponding to each result in the candidate result set and the query text vector, and rearrange the candidate result set according to a linear weighted fusion strategy to obtain the retrieval results.

[0012] Optionally, the construction of training data based on Logano product items further includes:

[0013] A multi-model cross-validation strategy is adopted to verify the judgment results of the multimodal large model. Specifically, two or more multimodal large models are used to make judgments respectively. When the judgment results of each model are consistent, the judgment result is included in the training data as a high-confidence sample, and the training data is manually sampled.

[0014] Optionally, the LoRA fine-tuning achieves efficient parameter fine-tuning by injecting a low-rank update matrix alongside the original weight matrix, satisfying the formula:

[0015]

[0016]

[0017] Where W is the original pre-trained weight matrix, ΔW is the low-rank update amount, B and A are the low-rank decomposition matrices, d and k are the dimensions of the original matrix, and r is the rank, set to between 16 and 64.

[0018] Optionally, the dual-modal coding satisfies the formula:

[0019]

[0020]

[0021] in, For the dth patent drawing, For its corresponding product item description text, and These are image encoders and text encoders, respectively, in a multimodal coding model. and These are the output image vector and text vector, respectively.

[0022] Optionally, the linear weighted fusion strategy satisfies the formula:

[0023]

[0024] Where d is a result in the candidate set. and These are the image vectors of the query image and the candidate image, respectively. and These are the product item description text vectors for the query image and the candidate images, respectively. and These represent image similarity and text similarity, respectively, with α and β being adjustable weight parameters and α+β=1. This is the result of linear weighted fusion.

[0025] Optionally, both image similarity and text similarity are calculated using cosine similarity, satisfying the formula:

[0026]

[0027] Here, A and B are two vectors, A·B is the vector dot product, and ||A|| and ||B|| are the L2 norms of the vectors.

[0028] Optionally, the linear weighted fusion strategy achieves cross-class noise suppression based on the class consistency assumption, which satisfies the formula:

[0029]

[0030] in, For text vectors of products in the same category, For text vectors of cross-category products, This is the result of the category consistency assumption.

[0031] Secondly, a design patent image retrieval system based on multimodal classification and focusing is provided, the system comprising:

[0032] Module for constructing training data: For each design patent drawing, obtain the secondary classification number from its patent bibliographic information and match it with the corresponding product item description in the Logano classification table to construct image-text pair training data of image and product item description text;

[0033] Training multimodal classification model module: used to fine-tune the training data using the images and text based on a lightweight multimodal large model to obtain a product item recognition model;

[0034] Full-database classification and labeling module: used to perform batch reasoning on the design patent drawings in the patent image library using the product item recognition model, and generate corresponding product item description text for each image;

[0035] The bimodal coding and index construction module is used to input the product item description text and its corresponding appearance design patent drawings into the multimodal coding model for bimodal coding, thereby obtaining text vector representation and image vector representation respectively, and constructing a bimodal vector index based on the text vector representation and image vector representation;

[0036] The retrieval and rearrangement module is used to receive a query image, generate product item description text for the query image using the product item recognition model, and encode the query image and its product item description text to obtain a query image vector and a query text vector, respectively; perform similarity retrieval in the bimodal vector index based on the query image vector to obtain a candidate result set, calculate the similarity between the text vector corresponding to each result in the candidate result set and the query text vector, and rearrange the candidate result set according to a linear weighted fusion strategy to obtain the retrieval result.

[0037] Thirdly, an electronic device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-mentioned design patent image retrieval method based on multimodal classification and focusing.

[0038] Fourthly, a computer program product is provided, including a computer program / instruction that, when executed by a processor, implements the steps of the above-described design patent image retrieval method based on multimodal classification and focusing.

[0039] The beneficial effects of the technical solutions provided in this application include at least the following:

[0040] (1) A unified and standardized category description system was constructed to eliminate semantic ambiguity: By adopting Logano product items as image semantic tags and using a semi-automated process of retrieving candidate product item sets based on secondary classification numbers to perform multimodal large model judgment, the internationally accepted design classification standard was transformed into standardized category description text, replacing the non-standard patent titles in the prior art. This provided a unified and standardized category semantic benchmark for the retrieval system, enabling the retrieval to perform accurate similarity matching within the correct category range and avoiding retrieval deviations caused by title ambiguity.

[0041] (2) The technical bottleneck of missing fine-grained classification numbers in bibliographic entries is solved at a low cost: by adopting a lightweight multimodal large model combined with low-rank adaptive fine-tuning technology, a low-rank update matrix is ​​injected next to the original weight matrix to achieve efficient parameter fine-tuning. With extremely low training cost, accurate automatic labeling of product items for massive patent images is achieved, overcoming the difficulty of retrieval focus caused by the fact that the bibliographic entries of design patent publications only contain secondary classification numbers and the granularity is too coarse. It supports efficient parallel processing of the entire patent image library through a batch inference framework.

[0042] (3) A joint indexing system of images and semantics was established through bimodal coding: The attached drawings of the design patent and the corresponding product item description text were input into the multimodal coding model for bimodal coding, and image vector index and text vector index were constructed respectively, forming a vector retrieval basis that supports category semantic constraints, so that the retrieval system can use image visual features and category semantic features to calculate similarity at the same time.

[0043] (4) The two-stage retrieval strategy significantly suppresses cross-category noise and improves the quality of head retrieval: The two-stage strategy of combining image encoding recall and text encoding reordering is adopted. The candidate results are reordered by linearly weighted fusion of image similarity and text similarity, and the category semantics are encoded as independent constraint signals to participate in the ranking. Based on the category consistency assumption that the text vector similarity of products in the same category is significantly greater than that of cross-category products, cross-category noise is penalized, which effectively suppresses the ranking of cross-category noise caused by local feature interference in the head results, significantly improves the category consistency of the head retrieval results, and greatly reduces the workload of manually screening noise results.

[0044] (5) A retrieval paradigm shift of focusing on categories first and then matching details has been realized: The category semantics of images are explicitly extracted and encoded into independent constraint signals through a multimodal generation model, transforming the traditional end-to-end single-modal image matching into a dual-modal retrieval under explicit category constraints. This fundamentally solves the technical problem that local feature similarity outweighs global category feature matching score. Even in query scenarios with high cross-class interference, it can still maintain stable category focusing ability, improving the automation level and examination efficiency of design patent retrieval. Attached Figure Description

[0045] To more clearly illustrate the embodiments of this application or the technical solutions in the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0046] Figure 1 A flowchart illustrating the steps of a design patent image retrieval method based on multimodal classification and focusing, as provided in this application embodiment;

[0047] Figure 2 A flowchart illustrating the training data construction process provided in this application embodiment;

[0048] Figure 3 A flowchart of the classification model training process provided in this application embodiment;

[0049] Figure 4 A flowchart of the full-library annotation and encoding process provided for embodiments of this application;

[0050] Figure 5 The retrieval and rearrangement flowchart provided for embodiments of this application;

[0051] Figure 6 A system block diagram of a design patent image retrieval method based on multimodal classification and focusing, provided for embodiments of this application;

[0052] Figure 7 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] In the description of this application, the terms "comprising," "having," and any variations thereof are intended to cover non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those steps or units that are expressly listed, but may also include other steps or units that are not expressly listed but are inherent to these processes, methods, products, or apparatuses, or steps or units added based on further optimizations conceived in this application.

[0055] This application presents a multimodal classification-focused image retrieval method for design patents. It achieves a significant improvement in retrieval performance through a three-tiered technical architecture: "classification-focusing first, then encoding-retrieval, and finally reordering-optimization." The entire invention process includes: 1. Constructing training data based on Logano product items; 2. Training a multimodal generative model for product item recognition; 3. Labeling all design patent drawings with product items; 4. Inputting the images and corresponding product item description text into the encoding model for encoding and establishing a vector index; 5. Employing a two-stage strategy of image encoding recall + text encoding reordering during the retrieval phase.

[0056] The accompanying drawings for the design patent mentioned in this application refer to the "design pictures or photographs" in the design patent documents.

[0057] Please refer to Figure 1 The document illustrates a flowchart of a design patent image retrieval method based on multimodal classification focusing, as provided in an embodiment of this application. This method may include the following steps:

[0058] S1. Construct training data based on Logano product items: Obtain the secondary classification number of the design patent drawings, retrieve the corresponding product item description set from the Logano classification table according to the secondary classification number as the candidate set, input the design patent drawings, patent title and candidate set into the multimodal large model for matching and judgment, and construct image-text pairs training data of images and product item description texts.

[0059] This step involved constructing training data based on Locarno product categories: The Locarno International Industrial Design Classification is an internationally recognized standard for industrial design classification. According to the International Industrial Design Classification (15th edition) (effective January 1, 2025), this classification system includes major categories, subcategories, and more specific product categories (use categories), as shown in Table 1:

[0060] Table 1. Examples of Design Classification

[0061] Classification number Serial Number Product Item (Application Category) 01-02 104598 Vegetable crisps, dried vegetables 01-02 104840 Fruit crisps, dried fruit 01-02 105043 Vegetable pre-process

[0062] In Table 1, "01" represents the major category, "02" represents the minor category, and "vegetable chips and dried vegetables" represents more specific product items (use categories).

[0063] The bibliographic information disclosed in design patents only includes the secondary classification number and does not include product item information, making it impossible to directly obtain the product item label corresponding to each design patent drawing. This application adopts the following process to automatically construct training data: For each design patent drawing, its secondary classification number (existing information) is obtained from the patent bibliographic information; based on the secondary classification number, the descriptive text of all product items under that secondary classification number is retrieved from the International Classification of Industrial Designs (15th Edition) to form a candidate set; the design patent drawing, patent title, and candidate product item description set are input together into an open-source multimodal large model (such as Qwen3.5-27B or a similar model), and the large model determines the most matching product item description from the candidate set; the determination result and the corresponding image are combined to form an "image-product item description text" image-text pair as training data. To ensure data quality, multi-model cross-validation (where two or more large models are used to make independent judgments, and consistent results are taken as high-confidence samples) and manual sampling strategies can be adopted.

[0064] like Figure 2 A flowchart for constructing the training data is provided. Specifically, using the attached drawings of a design patent as input, the process first obtains its corresponding secondary classification number, and then retrieves the set of candidate product descriptions under that secondary classification number from the International Classification of Industrial Designs (ICIC). Subsequently, the attached drawings of the design patent, the patent title, and the set of candidate product descriptions are input into an open-source multimodal large model (such as Qwen3.5-27B) for matching and judgment, outputting image-text pairs consisting of images and product description text. Finally, through multi-model cross-validation and quality control steps such as manual sampling / proofreading, a high-quality training dataset is efficiently constructed, providing a data foundation for subsequent LoRA fine-tuning of the multimodal classification model.

[0065] S2. Training the multimodal classification model: Based on a lightweight multimodal large model, LoRA fine-tuning is performed on the training data using images and text to obtain a product item recognition model.

[0066] This step trains a multimodal generative model for product item recognition: a lightweight multimodal large model (e.g., Qwen3.5-2B) is selected as the base, and the LoRA (Low-Rank Adaptation) method is used for efficient parameter fine-tuning. The model input is an attached image of a design patent, along with an instruction template (e.g., "Please identify the Logano product item description of this product image"). The model output is the corresponding product item description text (e.g., "measuring tape"). The main training configuration includes: LoRARank values ​​of 16~64 (32 recommended), LoRAAlpha of 2×Rank, target module coverage of QKV+O+MLP, training epochs of 3~5, and a learning rate of 1e-4~2e-4. The lightweight model is chosen because of its low inference cost, making it suitable for batch classification and annotation of massive amounts of patent images; fine-tuning based on the Instruct version can inherit existing instruction compliance and visual understanding capabilities. Note: This application does not involve optimization at the model structure level, but only uses labeled data for LoRA fine-tuning on existing open-source models.

[0067] The core principle of LoRA fine-tuning is low-rank decomposition. Efficient parameter fine-tuning is achieved by injecting a low-rank update matrix next to the original weight matrix, as shown in the following formula:

[0068]

[0069]

[0070] Where W is the original pre-trained weight matrix, ΔW is the low-rank update amount, B and A are the low-rank decomposition matrices, d and k are the dimensions of the original matrix, and r is the rank (in this application, the value is 16~64, and 32 is recommended). r is much smaller than d and k, thus greatly reducing the number of trainable parameters.

[0071] like Figure 3 A flowchart of the classification model training process is presented. Specifically, the high-quality training dataset constructed in S1 is used to efficiently fine-tune the parameters through the LoRA (low-rank adaptation) method. The fine-tuned model receives the attached drawings of the appearance design patent and the instruction template as input, and outputs the product item description text (such as a measuring tape) corresponding to the image after inference. Finally, a product item recognition model that can automatically identify the product item is formed, which provides a model foundation for subsequent full-database classification and labeling.

[0072] S3. Full-database classification and labeling: Using a product item recognition model, batch reasoning is performed on the design patent illustrations in the patent image database to generate corresponding product item description text for each image.

[0073] This step achieves full-database classification and annotation: using the LoRA fine-tuned model trained in step S2, batch inference is performed on all design patent illustrations in the patent image database to generate corresponding product description text for each image. This process can be accelerated using a batch inference framework (such as vLLM) and supports large-scale parallel processing.

[0074] S4. Bimodal Encoding and Index Construction: The product item description text and its corresponding design patent drawings are input into the multimodal encoding model for bimodal encoding to obtain text vector representation and image vector representation respectively, and a bimodal vector index is constructed based on the text vector representation and image vector representation.

[0075] This step implements bimodal encoding and vector index construction: For each patent drawing with labeled product descriptions, the following encoding operations are performed: ① Image encoding: The patent drawing is fed into a multimodal encoding model (such as Qwen3-VL-8B-Embedding or CLIP) to obtain an image vector representation; ② Text encoding: The product description text is fed into a text encoder of the same encoding model to obtain a text vector representation; ③ Image vector index and text vector index are established separately (such as based on FAISS or Milvus) to support efficient retrieval.

[0076] The mathematical expression of the bimodal coding process is as follows:

[0077]

[0078]

[0079] in, For the dth patent drawing, For its corresponding product item description text, and These are image encoders and text encoders, respectively, in a multimodal coding model. and These are the output image vector and text vector, respectively.

[0080] like Figure 4 The flowchart for full-database annotation and encoding is as follows. Specifically, this stage uses the full patent image database as input. First, the obtained product item recognition model is used for batch inference to generate corresponding product item description text for each patent image. Then, the images and their product item description texts are respectively fed into a multimodal encoding model (such as Qwen3-VL-8B-Embedding) for bimodal encoding. The image encoding branch generates image vectors and constructs image vector indexes (based on FAISS or Milvus), while the text encoding branch generates text vectors and constructs text vector indexes (based on FAISS or Milvus).

[0081] S5. Retrieval and Reordering: Receive the query image, generate product description text for the query image using a product item recognition model, and encode the query image and its product description text to obtain query image vector and query text vector respectively; perform similarity retrieval in a bimodal vector index based on the query image vector to obtain a candidate result set, calculate the similarity between the text vector corresponding to each result in the candidate result set and the query text vector, and reorder the candidate result set according to a linear weighted fusion strategy to obtain the retrieval results.

[0082] This step implements retrieval and rearrangement: When a user submits an image to be retrieved, the system executes the following process:

[0083] 1): Input the image to be inspected into the classification model to generate product item description text T_q;

[0084] 2): The image to be inspected is fed into the coding model to obtain the image vector V_q;

[0085] 3): Feed the category text T_q into the encoding model to obtain the text vector E_q;

[0086] 4): Use the image vector V_q to search in the image vector index to obtain the candidate set C (Top-N);

[0087] 5) For each result in the candidate set C, calculate the similarity between the text vector and E_q, and then use a linear weighted fusion strategy for re-ranking: Final score = α × image similarity + β × text similarity, where α and β are adjustable parameters. The optimal weight ratio is determined through validation set optimization to obtain the re-ranked result R (Top-K). (This application recommends α=0.3, β=0.7)

[0088] Both image similarity and text similarity are calculated using cosine similarity, as shown in the following formula:

[0089]

[0090] Here, A and B are two vectors, A·B is the vector dot product, and ||A|| and ||B|| are the L2 norms of the vectors.

[0091] The final rearrangement score is calculated using the following formula:

[0092]

[0093] Where d is a result in the candidate set. and These are the image vectors of the query image and the candidate image, respectively. and These are the product item description text vectors for the query image and the candidate images, respectively. and These are image similarity and text similarity (cosine similarity), respectively, where α and β are adjustable weight parameters and α+β=1.

[0094] The effectiveness of the above mechanism is based on the following category consistency assumption:

[0095]

[0096] in, For text vectors of products in the same category, This refers to the text vectors of products across different categories. It also represents the text vector similarity of products within the same category. Significantly greater than the text vector similarity of cross-category products Therefore, text similarity scores can effectively distinguish between candidate results of the same category and those of different categories, and penalize cross-category noise.

[0097] The introduction of text similarity is equivalent to imposing a category consistency constraint on candidate results—the text vectors of products in the same category are closer together, while the text vectors of products across categories are farther apart, thus effectively suppressing cross-category noise in the ranking of the top results.

[0098] like Figure 5 The flowchart for retrieval and reordering is as follows: Specifically, the image to be examined is used as input and processed through two parallel paths: one path generates product category text Tq through a classification model, and then generates text vector Eq through an encoding model; the other path generates image vector Vq through an encoding model, and retrieves candidate set C (Top-N) based on Vq in the image vector index; then, the text similarity between the text vectors in the candidate set and Eq is calculated, and linear weighted fusion is performed with the image similarity (Score = α×image similarity + β×text similarity), and finally, the reordered retrieval result R (Top-K) is output, realizing result optimization based on category consistency.

[0099] As can be seen, this application does not involve optimization at the model structure level. Instead, it introduces Logano product item descriptions as normalized semantic labels, trains a multimodal generative model to achieve automated category recognition, and encodes category semantics into independent constraint signals to participate in retrieval ranking, thus realizing a paradigm shift in retrieval that prioritizes categories before matching details. Specifically, this includes the following key points:

[0100] a. Existing technologies lack standardized category descriptions for design patent images, making it impossible to perform similarity matching within the correct category during retrieval. Design patent titles exhibit significant inconsistencies, with the same product potentially having multiple different names. This application addresses this by introducing Logano product item descriptions as standardized image semantic tags, eliminating semantic ambiguity caused by inconsistent titles and providing a unified standard for focused retrieval. (Explanation: The Logano classification system is an internationally recognized standard for design patent classification; its product item descriptions are sufficiently granular to accurately depict the category semantics of a type of design product.) Compared to pure image encoding retrieval methods, this application's multimodal classification-focused retrieval method is expected to improve Top-10 retrieval accuracy by 15% to 25%, with particularly significant results in query scenarios containing high levels of cross-category interference.

[0101] b. Existing technologies employ a single global feature encoding for end-to-end retrieval, where category information is only used as an implicit feature dimension in the calculation. This leads to local feature similarity often outweighing the matching score of global category features. For example, using an image of a laser measuring tape (which has a tape measure overall but includes a display screen and buttons on the side) for retrieval might detect products with similar displays or buttons but completely different categories (such as calculators and remote controls). This application explicitly extracts the category semantics of images by training a multimodal generative model and encodes it into independent constraint signals to participate in retrieval ranking, achieving a retrieval strategy of "focusing on categories first, then matching details."

[0102] c. Addressing the lack of a fine-grained category classification labeling system in existing technologies. Although the International Classification of Industrial Designs (15th Edition) has established a complete classification standard including major categories, subcategories, and product items, the bibliographic information disclosed in design patents only includes secondary classification numbers and does not include product items, making it impossible to directly use product items for category focusing in image retrieval. Existing secondary classification numbers are too coarse-grained; a single secondary classification number may contain up to 125 product items, covering a large number of significantly different products. This application solves the problem of missing product items in the bibliographic information of design patents through a semi-automated process of "known secondary classification numbers + candidate product item set + large model judgment," automatically labeling product item descriptions for each patent drawing and constructing a high-quality, fine-grained classification labeling system. (Explanation: For example, secondary classification number 10-04 is for measuring, detection and signal instruments, which includes products with huge differences such as tape measures, thermometers, and stopwatches. Relying solely on secondary classification numbers cannot achieve effective focusing.) This application introduces the Logano product item as a standardized image semantic tag, which eliminates the semantic ambiguity caused by non-standard patent titles and provides a unified and standardized category description system for the retrieval system. In addition, the classification model inference latency is less than 100ms / image (based on the 2B parameter model), which does not affect the online response speed of the retrieval system.

[0103] d. Existing technologies lack category consistency constraints during the retrieval phase, resulting in a large amount of cross-category noise in the header search results, increasing the workload of manual review. This application introduces a text encoding reordering mechanism during the retrieval phase, utilizing the semantic information of product item descriptions to reorder candidate results for image retrieval, effectively suppressing cross-category noise in the header ranking and significantly improving header retrieval accuracy. The proportion of products of the same category in the header results is expected to increase from the current 60%~70% to 85%~95%, significantly reducing the interference of cross-category noise.

[0104] like Figure 6 As shown in the embodiments of this application, a design patent image retrieval system based on multimodal classification and focusing is also provided, which may include:

[0105] The training data construction module is used to obtain the secondary classification number of the design patent drawings. Based on the secondary classification number, the corresponding product item description set is retrieved from the Logano classification table as a candidate set. The design patent drawings, patent title and candidate set are input into the multimodal large model for matching and judgment, and image-text pair training data of image and product item description text are constructed.

[0106] Training multimodal classification model module: Used to fine-tune the training data using LoRA based on a lightweight multimodal large model, and obtain a product item recognition model;

[0107] Full-database classification and annotation module: Used to perform batch reasoning on the design patent illustrations in the patent image library using a product item recognition model, and generate corresponding product item description text for each image;

[0108] The bimodal coding and index construction module is used to input the product item description text and its corresponding design patent drawings into the multimodal coding model for bimodal coding, thereby obtaining text vector representation and image vector representation respectively, and constructing a bimodal vector index based on the text vector representation and image vector representation;

[0109] The retrieval and rearrangement module receives the query image, generates product description text for the query image using a product item recognition model, and encodes the query image and its product description text to obtain query image vector and query text vector, respectively. Based on the query image vector, it performs similarity retrieval in a bimodal vector index to obtain a candidate result set, calculates the similarity between the text vector corresponding to each result in the candidate result set and the query text vector, and rearranges the candidate result set according to a linear weighted fusion strategy to obtain the retrieval results.

[0110] Specific limitations regarding the design patent image retrieval system based on multimodal classification and focusing can be found in the limitations of the design patent image retrieval method based on multimodal classification and focusing described above, and will not be repeated here. Each module in the aforementioned design patent image retrieval system based on multimodal classification and focusing can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0111] In one embodiment, an electronic device is provided, which may be a computer, and its internal structure diagram may be as follows: Figure 7 As shown, the electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for retrieving design patent images. The network interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a design patent image retrieval method.

[0112] Those skilled in the art will understand that, Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0113] In one embodiment of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described design patent image retrieval method based on multimodal classification and focusing.

[0114] In one embodiment of this application, a computer program product is provided, including a computer program / instructions, which, when executed by a processor, implements the steps of the above-described design patent image retrieval method based on multimodal classification and focusing.

[0115] The computer-readable storage medium and computer program product provided in this embodiment are similar in implementation principle and technical effect to the above method embodiments, and will not be repeated here.

[0116] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods.

[0117] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0118] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for retrieving appearance design patent images based on multimodal classification and focusing, characterized in that, The method includes: S1. Construct training data based on Logano product items: For each design patent drawing, obtain the secondary classification number in its patent bibliographic information and match it with the corresponding product item description in the Logano classification table to construct image-text pair training data of image and product item description text. S2. Training the multimodal classification model: Based on the lightweight multimodal large model, the training data is fine-tuned using the aforementioned images and text to obtain the product item recognition model; S3. Full-database classification and labeling: Using the product item recognition model, perform batch reasoning on the design patent drawings in the patent image database to generate corresponding product item description text for each image; S4. Bimodal Encoding and Index Construction: The product item description text and its corresponding design patent drawings are input into a multimodal encoding model for bimodal encoding to obtain text vector representation and image vector representation respectively, and a bimodal vector index is constructed based on the text vector representation and image vector representation; S5. Retrieval and Rearrangement: Receive a query image, generate product item description text for the query image using the product item recognition model, and encode the query image and its product item description text to obtain a query image vector and a query text vector, respectively; perform similarity retrieval in the bimodal vector index based on the query image vector to obtain a candidate result set, calculate the similarity between the text vector corresponding to each result in the candidate result set and the query text vector, and rearrange the candidate result set according to a linear weighted fusion strategy to obtain the retrieval results.

2. The method according to claim 1, characterized in that, The construction of training data based on Logano product items also includes: A multi-model cross-validation strategy is adopted to verify the judgment results of the multimodal large model. Specifically, two or more multimodal large models are used to make judgments respectively. When the judgment results of each model are consistent, the judgment result is included in the training data as a high-confidence sample, and the training data is manually sampled.

3. The method according to claim 1, characterized in that, The LoRA fine-tuning achieves efficient parameter fine-tuning by injecting a low-rank update matrix next to the original weight matrix, satisfying the formula: Where W is the original pre-trained weight matrix, ΔW is the low-rank update amount, B and A are the low-rank decomposition matrices, d and k are the dimensions of the original matrix, and r is the rank, set to between 16 and 64.

4. The method according to claim 1, characterized in that, The dual-modal coding satisfies the formula: in, For the dth patent drawing, For its corresponding product item description text, and These are image encoders and text encoders, respectively, in a multimodal coding model. and These are the output image vector and text vector, respectively.

5. The method according to claim 1, characterized in that, The linear weighted fusion strategy satisfies the following formula: Where d is a result in the candidate set. and These are the image vectors of the query image and the candidate image, respectively. and These are the product item description text vectors for the query image and the candidate images, respectively. and These represent image similarity and text similarity, respectively, with α and β being adjustable weight parameters and α+β=1. This is the result of linear weighted fusion.

6. The method according to claim 5, characterized in that, Both image similarity and text similarity are calculated using cosine similarity, satisfying the formula: Here, A and B are two vectors, A·B is the vector dot product, and ||A|| and ||B|| are the L2 norms of the vectors.

7. The method according to claim 5, characterized in that, The linear weighted fusion strategy achieves cross-class noise suppression based on the class consistency assumption, which satisfies the following formula: in, For text vectors of products in the same category, For text vectors of cross-category products, This is the result of the category consistency assumption.

8. A design patent image retrieval system based on multimodal classification and focusing, characterized in that, The system includes: Module for constructing training data: For each design patent drawing, obtain the secondary classification number from its patent bibliographic information and match it with the corresponding product item description in the Logano classification table to construct image-text pair training data of image and product item description text; Training multimodal classification model module: used to fine-tune the training data using the images and text based on a lightweight multimodal large model to obtain a product item recognition model; Full-database classification and labeling module: used to perform batch reasoning on the design patent drawings in the patent image library using the product item recognition model, and generate corresponding product item description text for each image; The bimodal coding and index construction module is used to input the product item description text and its corresponding appearance design patent drawings into the multimodal coding model for bimodal coding, thereby obtaining text vector representation and image vector representation respectively, and constructing a bimodal vector index based on the text vector representation and image vector representation; The retrieval and rearrangement module is used to receive a query image, generate product item description text for the query image using the product item recognition model, and encode the query image and its product item description text to obtain a query image vector and a query text vector, respectively; perform similarity retrieval in the bimodal vector index based on the query image vector to obtain a candidate result set, calculate the similarity between the text vector corresponding to each result in the candidate result set and the query text vector, and rearrange the candidate result set according to a linear weighted fusion strategy to obtain the retrieval result.

9. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the method as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

Images