Line drawing query method, device, equipment, medium and program product

Abstract

One or more embodiments of the specification provide a line drawing query method, device, equipment, medium and program product. The method comprises: acquiring a to-be-queried line drawing and to-be-queried text; generating a description text for the to-be-queried line drawing; fusing the to-be-queried text and the description text to obtain a fused text; performing feature extraction on the to-be-queried line drawing and the fused text by using a multi-modal model to obtain image features, text features and image-text fused features; querying a plurality of candidate line drawings in a line drawing database based on the image features, the text features and the image-text fused features; and determining a target line drawing as a query result from the candidate line drawings.

Description

Technical Field

[0001] This specification relates to the field of image processing technology, and more particularly to a method, apparatus, device, medium, and program product for querying line drawings. Background Technology

[0002] Line art, primarily composed of lines, plays a crucial role in design, accurately conveying key information such as structure, proportion, and craftsmanship details. However, designers often face numerous technical obstacles when searching for and referencing past line art. Existing management methods rely mainly on manually set tags for classification and searching, which is prone to human error or omissions, leading to misclassification and difficulty in finding relevant lines art. Furthermore, traditional RGB image feature-based search techniques focus more on image color, texture, and overall appearance, failing to accurately capture the geometric structure and design intent of line art, often resulting in biased search results. These technical limitations make searching line art difficult, significantly restricting design efficiency. Summary of the Invention

[0003] In view of the above, one or more embodiments of this specification provide the following technical solutions:

[0004] According to a first aspect of one or more embodiments of this specification, a method for querying line art diagrams is proposed, comprising:

[0005] Retrieve the line drawing and text to be queried;

[0006] Generate descriptive text for the line drawing to be queried;

[0007] The queried text and the descriptive text are combined to obtain the fused text;

[0008] A multimodal model is used to extract features from the line drawing to be queried and the fused text to obtain image features, text features and image-text fusion features;

[0009] Based on the image features, text features, and image-text fusion features, several candidate line drawings were obtained by querying the online manuscript database;

[0010] The target line drawing is selected from the candidate line drawing diagrams as the query result.

[0011] Optionally, the process of determining the target line drawing as the query result from the candidate line drawing images includes:

[0012] Extract the outline features of the line drawing to be queried;

[0013] For each candidate line drawing, extract the candidate contour features of the candidate line drawing;

[0014] The contour similarity between the candidate line drawing and the line drawing to be queried is determined based on the contour features to be queried and the candidate contour features.

[0015] The target line drawing is determined from the candidate line drawing based on the contour similarity.

[0016] Optionally, the process of determining the target line drawing based on the contour similarity in the candidate line drawing includes:

[0017] For each candidate line drawing, a comprehensive similarity between the candidate line drawing and the line drawing to be queried is determined based on the feature similarity and the contour similarity between the candidate line drawing and the line drawing to be queried.

[0018] Candidate line drawings whose comprehensive similarity satisfies the comprehensive constraint conditions are determined as the target line drawings.

[0019] Optionally, the process of determining the target line drawing based on the contour similarity in the candidate line drawing includes:

[0020] Candidate line drawings whose contour similarity satisfies the contour constraint conditions are determined as the target line drawings.

[0021] Optionally, the contour features include global contour features and local contour features;

[0022] The process of determining the contour similarity between the candidate line drawing and the line drawing to be queried based on the contour features to be queried and the candidate contour features, and determining the target line drawing from the candidate line drawing based on the contour similarity, includes:

[0023] The global contour similarity between each candidate line drawing and the line drawing to be queried is determined based on the global contour features to be queried and the candidate global contour features.

[0024] Candidate line drawings whose global contour similarity satisfies the global contour constraint are selected as initial screening line drawings.

[0025] For each initial screening line drawing, the local contour similarity between the initial screening line drawing and the line drawing to be queried is determined based on the candidate local contour features of the initial screening line drawing and the query local contour features of the line drawing to be queried.

[0026] The target line drawing is determined from the initial screening line drawing based on the local contour similarity.

[0027] Optionally, the multimodal model includes an image encoder and a text encoder, and the training process includes:

[0028] Obtain the line drawing sample and the corresponding design text;

[0029] Generate descriptive text for the line drawing sample;

[0030] By combining the descriptive text of the line drawing sample and the design text, a text sample corresponding to the line drawing sample is obtained;

[0031] The line drawing samples and the text samples are input into a multimodal model. The image encoder in the multimodal model extracts the image features of the line drawing samples, and the text encoder extracts the text features of the text samples. The goal is to train the multimodal model to maximize the similarity of correct image-text pairings while minimizing the similarity of incorrect image-text pairings.

[0032] Optional, also includes:

[0033] Extract the standard contour features of the line drawing sample;

[0034] The process of training a multimodal model with the goal of maximizing the similarity of correct image-text pairings while minimizing the similarity of incorrect image-text pairings includes:

[0035] The goal is to train a multimodal model that maximizes the similarity of correct image-text pairings, minimizes the similarity of incorrect image-text pairings, and minimizes the loss of contour features.

[0036] The contour feature loss is determined based on the standard contour features and the contour features of the line drawing sample extracted by the image encoder.

[0037] Optionally, the process of querying the online manuscript database to obtain several candidate line drawings based on the image features, the text features, and the image-text fusion features includes:

[0038] Using the image features, text features, and image-text fusion features as query conditions, similar initial line drawings are searched in the line drawing database.

[0039] The candidate line drawing is obtained by summarizing the initial line drawing obtained from the query.

[0040] Optionally, the process of fusing the query text and the description text to obtain the fused text includes:

[0041] The query text and the description text are input into a large language model, and the query text and the description text are optimized and fused by the large language model to obtain the fused text;

[0042] The large language model is obtained by fine-tuning a pre-trained language model by combining a corpus and knowledge graph related to the domain to which the line drawing to be queried belongs.

[0043] According to a second aspect of one or more embodiments of this specification, a line drawing query device is provided, comprising:

[0044] The query retrieval unit retrieves the line drawing and text to be queried.

[0045] The text generation unit generates descriptive text for the line drawing to be queried;

[0046] The text fusion unit merges the query text and the description text to obtain fused text;

[0047] The feature extraction unit uses a multimodal model to extract features from the line drawing to be queried and the fused text, obtaining image features, text features and image-text fusion features;

[0048] The candidate query unit retrieves several candidate line drawings from the online drawing database based on the image features, the text features, and the image-text fusion features.

[0049] The target determination unit determines the target line drawing as the query result from the candidate line drawing drawings.

[0050] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described above by executing the executable instructions.

[0051] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method described above.

[0052] According to a fifth aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as described above.

[0053] As can be seen from the above description, the technical solution provided in this specification can generate descriptive text for the line drawing to be queried, and merge the descriptive text and the text to be queried to obtain fused text. Then, a multimodal model can be used to extract features from the line drawing to be queried and the fused text to obtain multimodal features such as image features, text features, and image-text fusion features. Then, based on these multimodal features, several candidate line drawings can be obtained by querying the line drawing database, and the query result can be determined from the candidate line drawings.

[0054] Therefore, this manual supports multimodal query information when querying line drawings, eliminating the need for manual labeling and effectively avoiding inaccurate query results due to human error or omission. Furthermore, compared to query techniques based on RGB image features, this manual can accurately capture the geometric structure and design intent of line drawings through multimodal information, making the query results more accurate and more in line with expectations, thereby improving design efficiency. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the architecture of a line drawing query service system provided in an exemplary embodiment.

[0056] Figure 2 This is a flowchart of a line drawing query method provided in an exemplary embodiment.

[0057] Figure 3 This is a flowchart of a multimodal model training method provided in an exemplary embodiment.

[0058] Figure 4 This is a schematic diagram of a multimodal model training process provided in an exemplary embodiment.

[0059] Figure 5 This is a schematic diagram illustrating a method for storing and querying line drawings, provided as an exemplary embodiment.

[0060] Figure 6 This is a flowchart of a method for optimizing query results provided in an exemplary embodiment.

[0061] Figure 7 This is a flowchart of another method for optimizing query results provided in an exemplary embodiment.

[0062] Figure 8 This is a schematic diagram illustrating an example of querying a line drawing, provided as an exemplary embodiment.

[0063] Figure 9 This is a schematic diagram of the structure of a device provided in an exemplary embodiment.

[0064] Figure 10 This is a block diagram of a line drawing query device provided in an exemplary embodiment. Detailed Implementation

[0065] Line art, primarily composed of lines, plays a crucial role in design, accurately conveying key information such as structure, proportion, and craftsmanship details. However, designers often face numerous technical obstacles when searching for and referencing past line art. Existing management methods rely mainly on manually set tags for classification and searching, which is prone to human error or omissions, leading to misclassification and difficulty in finding relevant lines art. Furthermore, traditional RGB image feature-based search techniques focus more on image color, texture, and overall appearance, failing to accurately capture the geometric structure and design intent of line art, often resulting in biased search results. These technical limitations make searching line art difficult, significantly restricting design efficiency.

[0066] This manual provides a line drawing query scheme that allows users to input multimodal information to query line drawings without manually setting tags, resulting in higher query accuracy. Compared to query techniques based on RGB image features, this manual can accurately capture the geometric structure and design intent of line drawings through multimodal information, making the query results more in line with expectations and thus improving design efficiency.

[0067] The line drawings described in this manual can be clothing line drawings in the field of clothing design, bag line drawings in the field of bag design, line drawings in the field of industrial design, line drawings in the field of game design, etc. This manual does not impose any special restrictions on them.

[0068] Figure 1 This is a schematic diagram of the architecture of a line drawing query system provided in an exemplary embodiment. For example... Figure 1 As shown, the system may include a server 11, a network 12, and several electronic devices, such as a PC (Personal Computer) 13, a mobile phone 14, etc.

[0069] Server 11 can be a physical server containing an independent host, or it can be a virtual server hosted in a host cluster. During operation, server 11 can run server-side programs for a certain application to implement the relevant functions of that application. For example, when server 11 runs a line drawing query service program, it can function as a corresponding line drawing query service platform.

[0070] PC13 and mobile phone 14 are just some of the types of electronic devices that users can use. In reality, users can obviously also use electronic devices such as tablets, laptops, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smartwatches, etc.), etc., and one or more embodiments in this specification do not limit this. During operation, the electronic device can run a client-side program of an application to implement the relevant functions of that application. For example, when the electronic device runs a line drawing query service program, it can act as a client for that line drawing query service. Specifically, the client can send the line drawing and / or text description entered by the user to the server 11, which will then perform the line drawing query and display the query results returned by the server 11 to the user. The aforementioned line drawing query service client application can be launched and run on the electronic device. This client-side program can be a native application installed on the electronic device, or it can be a mini-program, quick app, or other similar form. Of course, when using web technologies such as HTML5 or similar, the relevant functions can be achieved through the page displayed by the browser. The browser here can be a standalone browser application or a browser module embedded in some applications.

[0071] As for the network 12 that enables interaction between electronic devices such as PC13 and mobile phone 14 and server 11, communication can be achieved using either wired or wireless networks, depending on the communication methods supported by the respective electronic devices. This specification does not impose any restrictions on this. For example, PC13 can support both wired and wireless communication, so it can use either wired or wireless networks as needed. Mobile phone 14 typically only supports wireless communication, so it can use a wireless network for communication.

[0072] Of course, in other examples, the line drawing query method provided in this manual can also be applied only to electronic devices, where the electronic device performs the line drawing query, and this manual does not impose any special restrictions on this.

[0073] Figure 2 This is a flowchart of a line drawing query method provided in an exemplary embodiment.

[0074] Please refer to Figure 2 The line drawing query method provided in this embodiment can be applied to the aforementioned Figure 1 The server shown includes the following steps:

[0075] Step 202: Obtain the line drawing and text to be queried.

[0076] In this embodiment, when designers are conceiving new designs, they may need to find similar designs from the past as references. They can input existing line drawings as the line drawings to be queried, or they can input some design concept text as the text to be queried.

[0077] Step 204: Generate descriptive text for the line drawing to be queried.

[0078] In this embodiment, for the line drawing to be queried, a corresponding descriptive text can be generated to generate corresponding high-level semantic information for the line drawing to be queried.

[0079] In one example, a deep learning algorithm can be used to generate the descriptive text. For instance, a trained deep learning model including an encoder-decoder architecture can be used to generate descriptive text for the line drawing to be queried.

[0080] The encoder in the deep learning model can extract the image features of the line drawing to be queried through a convolutional neural network (CNN) and convert them into a fixed-length vector representation. Then, a descriptive text can be generated based on the vector representation output by the encoder through a recurrent neural network (RNN) or a long short-term memory network (LSTM).

[0081] The deep learning model can be trained using several line drawings and their corresponding descriptive text.

[0082] In another example, the descriptive text can also be generated based on a visual large model approach. For instance, effective prompts can be designed to guide the visual large model in generating high-quality descriptive text.

[0083] The effective prompts may include: basic information about the line drawing. Taking a clothing line drawing as an example, it could be "Describe the main design points and features of the clothing in this image." The prompts may also include specific requirements; again using a clothing line drawing as an example, it could be "Please provide design information and emotional description of the clothing in this image," etc.

[0084] After the valid prompts and line drawings are input into the visual model, the visual model can output the corresponding descriptive text.

[0085] Furthermore, the parameters of the visual model can be adjusted as needed to control the level of detail in the descriptive text, thereby obtaining outputs with different styles and depths. Taking clothing necklines as an example, the visual model can be controlled to output broader descriptive text such as neckline labels (V-neck, round neck, etc.), or it can be controlled to output more detailed descriptive text such as detailed features of the V-neck. This manual does not impose any special restrictions on this.

[0086] The large visual model can be GPT-4 (Generative Pre-trained Transformer-4Vision, the fourth generation of generative pre-trained transformer model).

[0087] In this embodiment, a large visual model can be used to perform multi-level analysis of the content of the line drawing to be queried, thereby accurately extracting key design details, such as clothing pattern, outline, print pattern, structural splicing, and decorative elements. For example, for a line drawing of a dress containing a complex original design, details such as neckline shape (e.g., V-neck, round neck), sleeve type (e.g., puff sleeve, bell sleeve), and skirt style (e.g., umbrella skirt, fishtail skirt) can be automatically segmented and identified. At the same time, combined with multimodal analysis capabilities, these features can be converted into standardized descriptive text, which is convenient for subsequent text fusion.

[0088] In this embodiment, taking clothing line drawing as an example, the ability to identify different clothing styles can be enhanced through comparison learning and style transfer technology. For example, when identifying a retro-style suit, key information such as its shoulder structure, button arrangement, and fabric texture can be extracted, and designs with similar styles can be automatically matched to generate the descriptive text.

[0089] Step 206: Merge the text to be queried and the description text to obtain the merged text.

[0090] In this embodiment, the query text and the description text can be input into a large language model, and the query text and the description text can be optimized and fused by the large language model to obtain the fused text.

[0091] The large language model is obtained by fine-tuning a pre-trained language model by combining a corpus and knowledge graph related to the domain to which the line drawing to be queried belongs.

[0092] Taking clothing line drawings as an example, the pre-trained language model can be fine-tuned by combining a clothing style corpus and a professional knowledge graph in the field of clothing design. The fine-tuned large language model can then be used to fuse the query text and the description text.

[0093] In this embodiment, the large language model can optimize the query text and the description text by using semantic expansion, detail supplementation and normalization reconstruction to achieve optimized fusion of the query text and the description text, thereby making the generated fused text more accurate and more in line with the design expression requirements.

[0094] For example, suppose the query text includes "loose T-shirt, minimalist style". The large language model can identify the characteristics of loose fit and then supplement it with more detailed design descriptions, such as "loose cut, dropped shoulder design, solid color or minimalist print, overall simple and elegant style", etc.

[0095] In this embodiment, the large language model can also understand the user's potential needs based on the context, and reason and complete incomplete or ambiguous descriptions.

[0096] For example, assuming the query text includes "a dress with a retro feel", the large language model can supplement it with "may have an A-line skirt, high waist design, and combine polka dots or retro print elements to enhance the retro style characteristics", etc.

[0097] In this embodiment, the large language model can also optimize the corresponding description. For example, "short top" can be optimized to "the length of the top is about to the waist or slightly shorter, so as to highlight the body proportions".

[0098] In this embodiment, the large language model also ensures that the optimized fused text conforms to industry terminology standards, while enhancing the accuracy of professional expressions. The fused text can be standardized into structured information, making subsequent query results more aligned with users' actual needs and improving the user's interactive experience.

[0099] In this embodiment, before using the large language model to fuse and optimize the query text and the description text, these texts can be preprocessed, such as removing redundant words and correcting grammatical errors. The processing and implementation of this part can be referred to relevant technologies, and will not be described in detail here.

[0100] Step 208: Use a multimodal model to extract features from the line drawing to be queried and the fused text to obtain image features, text features and image-text fusion features.

[0101] In this embodiment, the line drawing to be queried and the fused text can be input into a trained multimodal model. Feature extraction is performed through the multimodal model to obtain image features of a uniform length corresponding to the line drawing to be queried, text features corresponding to the fused text, and image-text fusion features after the two are fused.

[0102] The multimodal model may include an image encoder and a text encoder. The image encoder can be used to extract features from the line drawing to be queried to obtain the image features, and the text encoder can be used to extract features from the fused text to obtain the text features. The multimodal model may also include a fusion encoder, which can be used to extract the image-text fusion features.

[0103] The training process of the multimodal model will be described in detail in subsequent embodiments.

[0104] Step 210: Based on the image features, the text features, and the image-text fusion features, several candidate line drawings are obtained by querying the online drawing database.

[0105] In this embodiment, the line drawing database can store line drawings of historical designs, design text corresponding to the line drawings, image features of the line drawings, text features corresponding to the design text and the line drawing description text, and image-text fusion features corresponding to the line drawings and their corresponding design text.

[0106] In this embodiment, the image features, text features, and image-text fusion features can be used as query conditions to search for similar line art images in the line art database as initial line art images. Three sets of initial line art images can be obtained, and then these three sets of initial line art images can be summarized to obtain candidate line art images.

[0107] For example, taking image features as an example, the similarity between the image features of each line drawing image in the line drawing database and the image features (query conditions) of the line drawing image to be queried can be calculated. Then, a specified number of line drawing images are selected in descending order of similarity as a set of initial line drawing images obtained by querying the image features. The specified number can be preset, such as 10, 20, etc., and this specification does not impose any special restrictions on it.

[0108] Taking the example of retrieving 10 initial line drawings for each query condition, a total of 30 candidate line drawings can be obtained.

[0109] Step 212: Determine the target line drawing from the candidate line drawing diagrams as the query result.

[0110] In this embodiment, the target line drawing can be determined from the candidate line drawing diagrams as the query result corresponding to the line drawing diagram to be queried and the text to be queried.

[0111] In one example, all candidate line drawings can be identified as the target line drawing.

[0112] In another example, a subset of candidate line drawings can be identified as the target line drawing. For instance, since line drawings have significant characteristics of sparse data representation—that is, the lines in a line drawing are simple and the information density is low—to improve the accuracy of the query results, the target line drawing can be determined based on the similarity between the contour features of the candidate line drawings and the contour features of the image to be queried. This will be described in detail in subsequent embodiments.

[0113] In this embodiment, after the query results are determined, the query results can be returned and displayed through a visual interface for designers to refer to.

[0114] As can be seen from the above description, the technical solution provided in this specification can generate descriptive text for the line drawing to be queried, and merge the descriptive text and the text to be queried to obtain fused text. Then, a multimodal model can be used to extract features from the line drawing to be queried and the fused text to obtain multimodal features such as image features, text features, and image-text fusion features. Then, based on these multimodal features, several candidate line drawings can be obtained by querying the line drawing database, and the query result can be determined from the candidate line drawings.

[0115] Therefore, this manual supports multimodal query information when querying line drawings, eliminating the need for manual labeling and effectively avoiding inaccurate query results due to human error or omission. Furthermore, compared to query techniques based on RGB image features, this manual can accurately capture the geometric structure and design intent of line drawings through multimodal information, making the query results more accurate and more in line with expectations, thereby improving design efficiency.

[0116] The following sections will describe in detail the specific implementation process of this manual from three aspects: training of the multimodal model, construction and querying of the line drawing database, and determination of the target line drawing.

[0117] I. Training of Multimodal Models

[0118] In this embodiment, the multimodal model can be an image encoder-text encoder model. The image encoder can be a convolutional neural network or a Vision Transformer (ViT), and the text encoder can be a Transformer architecture.

[0119] Please refer to Figure 3 The training process of the multimodal model may include the following steps:

[0120] Step 302: Obtain the line drawing sample and the design text corresponding to the line drawing sample.

[0121] In this embodiment, before training the multimodal model, image-text pairs for training the multimodal model can be constructed first.

[0122] On the one hand, line drawing samples can be obtained from various sources. For example, these samples can be hand-drawn sketches submitted by designers, line drawings generated by CAD (Computer-Aided Design) software, or line drawings from the existing image libraries of relevant companies. Taking the apparel industry as an example, the obtained line drawing samples should cover as many apparel categories and styles as possible to ensure the diversity and representativeness of the samples. At the same time, high-resolution formats should be used as much as possible when collecting line drawing samples to preserve line details.

[0123] After obtaining the line art sample, in order to avoid the interference of background noise on model training, image processing tools can be used to denoise the line art sample, such as removing noise and uniformizing line thickness.

[0124] On the other hand, design text corresponding to the line drawing samples can be collected. This design text is textual data describing the key design points of the corresponding line drawing samples, and can come from process sheets, design specification documents, or interview records written by designers. After collecting the design text, it can be cleaned to remove redundant content and grammatical errors, and the expression style can be standardized. For example, "round neck short sleeve" and "short sleeve round neck" can be adjusted to a single expression to ensure the accuracy and consistency of the design text.

[0125] The design text may include multiple languages ​​such as Chinese and English to facilitate use by designers from various countries.

[0126] In this embodiment, each pair of "line drawing sample - design text" can be stored in a uniform format, such as a JSON or CSV file. The structured storage format can be a combination of the line drawing sample path and the design text description.

[0127] In this embodiment, after obtaining the "line drawing sample - design text", it can be expanded. For example, the line drawing sample can be expanded by operations such as rotation, flipping, and scaling; the design text can be expanded by methods such as synonym replacement to increase semantic diversity.

[0128] In this embodiment, after constructing the "line drawing sample - design text", the matching degree between the line drawing sample and the corresponding design text can be manually verified through visual random sampling to ensure the semantic consistency between the two. Simultaneously, the distribution of categories can be statistically analyzed to assess the balance and completeness of the samples. The processing and implementation of this part can be found in relevant technologies, and will not be elaborated upon here.

[0129] In this embodiment, after constructing the "line drawing sample - design text" for training the multimodal model, it can be divided into a training set, a validation set, and a test set, typically in a ratio of 70%:20%:10%. During the division process, it is usually necessary to evaluate the balanced distribution of style categories in different subsets to avoid situations where certain styles are not covered in the test set.

[0130] In this embodiment, before training the multimodal model using "line drawing samples - design text", the line drawing samples can be adjusted to a uniform resolution (e.g., 256×256 pixels) and converted into grayscale images to reduce the interference of color information on model training; the design text can be segmented and converted into an input format acceptable to the model (e.g., word embedding or sentence embedding).

[0131] Step 304: Generate descriptive text for the line drawing sample.

[0132] In this embodiment, for a line drawing sample, corresponding descriptive text can be generated. The process of generating the descriptive text can be referred to the above. Figure 2 The implementation of step 204 in the illustrated embodiment can be achieved, for example, by using a deep learning algorithm to generate the descriptive text, or by using a scheme based on a large visual model, etc., which will not be elaborated further in this specification.

[0133] Step 306: Merge the descriptive text of the line drawing sample and the design text to obtain the text sample corresponding to the line drawing sample.

[0134] In this embodiment, for each line drawing sample, a large language model can be used to fuse its descriptive text and corresponding design text to obtain the text sample corresponding to the line drawing sample.

[0135] During text fusion, operations such as redundancy removal and data cleaning can be performed, and effective prompts can be constructed to generate detailed descriptions. Taking a clothing line drawing as an example, text samples such as "A modern style dress, made of silk, dark blue in color, with floral patterns and lace trim on the cuffs" can be generated. See the previous section for details. Figure 2 Step 306 in the illustrated embodiment will not be repeated here.

[0136] Step 308: Input the line drawing sample and the text sample into the multimodal model. The image encoder in the multimodal model extracts the image features of the line drawing sample, and the text encoder extracts the text features of the text sample. The goal is to train the multimodal model to maximize the similarity of correct image-text pairings and minimize the similarity of incorrect image-text pairings.

[0137] In this embodiment, line drawing samples and corresponding text samples can be used as paired inputs to train the multimodal model. For example, the following contrastive learning loss function can be used:

[0138]

[0139] Among them, L Contrastive Let N represent the loss function, N represent the number of "line drawing sample - text sample" pairs, τ be a hyperparameter used to control the smoothness of the similarity distribution, and x be the loss function. i Let y represent the feature vector of the i-th line drawing sample. i Let sim(x) represent the feature vector of the i-th text sample. i ,y i ) represents x i and y i The similarity between them can be calculated using the cosine similarity algorithm as follows:

[0140]

[0141] The loss function optimizes the parameters of the multimodal model with the goal of maximizing the similarity of correct image-text pairings while minimizing the similarity of incorrect image-text pairings.

[0142] Optionally, since line art emphasizes design elements and primarily uses outlines, general optimization methods lack a specific understanding of outline information. To address this, contour loss can be introduced during multimodal model training. Contour damage, a type of edge loss, can better learn high-frequency structural information in the image, providing detailed representation of features.

[0143] For details, please refer to Figure 4 For line drawing samples, their contour features can be extracted as standard contour features. Then, a multimodal model is trained with the goal of maximizing the similarity of correct image-text pairings, minimizing the similarity of incorrect image-text pairings, and minimizing the contour feature loss.

[0144] The standard contour features can be extracted by edge detection (such as Canny, Sobel) or shape descriptors (such as Hu moments, Zernike moments), etc. The standard contour features may include global contour features or local contour features, and this specification does not impose any special restrictions on them.

[0145] The contour feature loss can be determined based on the standard contour features and the contour features of the line drawing samples extracted by the image encoder, and can be calculated using the following formula:

[0146]

[0147] Among them, Xi,edge It is the feature vector of the standard contour features of the line drawing sample, Y i,edge It is the feature vector of the contour features of the line drawing sample extracted by the image encoder in the multimodal model.

[0148] In this embodiment, combined with the contour loss, the damage function of the multimodal model can be adjusted as follows:

[0149] L=α×L Contrastive +(1-α)×L Edge

[0150] Where α is the preset weight.

[0151] II. Construction and Query of the Line Art Database

[0152] In this embodiment, please refer to Figure 5 ( Figure 5 (The language model of Zhongda is not shown), which can store historical line drawings and their design text (hereinafter referred to as historical design text) in the line drawing database for subsequent query and retrieval.

[0153] For example, for each historical line drawing, a historical description text corresponding to the historical line drawing can be generated. Then, the historical design text is fused to obtain the historical text corresponding to the historical line drawing. The historical line drawing and its corresponding historical text are then input into a trained multimodal model. The multimodal model outputs the corresponding historical image features, historical text features, and historical image-text fusion features, and these features are also stored in the line drawing database.

[0154] Therefore, for each historical line drawing, the line drawing database stores the historical design text, historical image features, historical text features, and historical image-text fusion features corresponding to that historical line drawing.

[0155] The generation method of the historical descriptive text and the fusion method of the historical text can be referred to the foregoing. Figure 2 or Figure 3 The embodiments shown are not described in detail here.

[0156] In this embodiment, the line drawing database can be a vector-indexed database, such as FAISS (Facebook AI Similarity Search), Annoy (Approximate Nearest Neighbors Oh Yeah), Milvus vector database, etc. These databases support efficient similarity search and nearest neighbor queries. During storage, image features, text features, and image-text fusion features can be indexed and mapped to corresponding historical line drawing images to quickly provide query results. Feature standardization and dimensionality reduction can also be performed during storage to optimize storage space and query efficiency.

[0157] In this embodiment, please continue to refer to Figure 5 When searching for line art, combine Figure 2 In the illustrated embodiment, after determining the image features, text features, and image-text fusion features corresponding to the line drawing and text to be queried, a query can be performed using the image features, text features, and image-text fusion features as query conditions, respectively.

[0158] In one example, queries can be performed based on features of the same type. For instance, when using image features as query criteria, queries can be performed based on historical image features stored in the line art database. This involves calculating the similarity between the query criteria and each historical image feature, thereby identifying several similar historical line art images as initial line art images.

[0159] Similarly, when using text features as query conditions, queries can be performed based on historical text features stored in the line art database; when using image-text fusion features as query conditions, queries can be performed based on historical image-text fusion features stored in the line art database.

[0160] In another example, in addition to querying based on the same type of features, cross-queries can be performed by combining features of different types, thereby improving the comprehensiveness and accuracy of the query results. For example, when using image features as query conditions, queries can be performed based on various features stored in the line drawing database. That is, the similarity between the query condition and each historical image feature is calculated to obtain a set of similar historical line drawing images as initial line drawing images; the similarity between the query condition and each historical text feature is calculated to obtain another set of similar historical line drawing images as initial line drawing images; and the similarity between the query condition and each historical image-text fusion feature is calculated to obtain yet another set of similar historical line drawing images as initial line drawing images. In this way, three sets of initial line drawing images can be obtained for image features.

[0161] Similarly, when using text features as query conditions, three sets of initial line art images can be obtained; when using image-text fusion features as query conditions, three sets of initial line art images can also be obtained. In this way, a total of nine sets of initial line art images can be obtained.

[0162] In this embodiment, after obtaining multiple sets of initial line drawings, these initial line drawings can be summarized as candidate line drawings.

[0163] During the query process, nearest neighbor search methods such as HNSW (Hierarchical Navigable Small World) and IVF-PQ (Inverted File with Product Quantization) can be used. Similarity calculation methods such as Euclidean distance and cosine similarity can be used during the search; this manual does not impose any special restrictions on these methods.

[0164] III. Determining the Target Line Drawing

[0165] In this embodiment, since line art has significant characteristics of sparse data representation, that is, the lines of line art are simple and the information density is low, in order to improve the accuracy of the query results, some candidate line art can be selected as target line art based on the similarity between the contour features of the candidate line art and the contour features of the effect image to be queried, so as to optimize the query results.

[0166] Please refer to Figure 6 The process of optimizing query results may include the following steps:

[0167] Step 602: Extract the outline features of the line drawing to be queried.

[0168] In this embodiment, the contour features of the line drawing to be queried can be extracted, referred to as the query contour features. The contour features can be extracted using methods such as edge detection and shape descriptors.

[0169] Step 604: For each candidate line drawing, extract the candidate contour features of the candidate line drawing.

[0170] In this embodiment, for each candidate line drawing, its contour features can be extracted, which are called candidate contour features.

[0171] Step 606: Determine the contour similarity between the candidate line drawing and the line drawing to be queried based on the contour features to be queried and the candidate contour features.

[0172] In this embodiment, for each candidate line drawing, the contour curve can be non-explicitly matched using shape matching based on Hausdorff distance, DTW (Dynamic Time Warping), or a graph matching method, such as a Graph Neural Network (GNN), to optimize the alignment of the topology. The similarity between the candidate contour features of the candidate line drawing and the query contour features of the line drawing to be queried is then calculated and referred to as contour similarity.

[0173] Step 608: Determine the target line drawing from the candidate line drawing based on the contour similarity.

[0174] Based on the aforementioned step 606, after calculating the contour similarity, a portion of the candidate line drawings can be determined as the target line drawing based on the contour similarity.

[0175] In one example, candidate line drawings whose contour similarity satisfies the contour constraint condition can be determined as the target line drawing. The contour constraint condition can be preset, for example, the contour constraint condition can be the four candidate line drawings with the highest contour similarity. Then, after calculating the contour similarity, the top four candidate line drawings can be selected as the target line drawing in descending order of contour similarity.

[0176] In another example, the contour similarity and the feature similarity calculated when querying the candidate line drawing can be combined to obtain the comprehensive similarity between the candidate line drawing and the line drawing to be queried. Then, the candidate line drawing that satisfies the comprehensive similarity constraint is determined as the target line drawing.

[0177] The feature similarity can be the similarity between the image features of the candidate line drawing and the image features of the line drawing to be queried, or the similarity between the text features corresponding to the candidate line drawing and the text features corresponding to the line drawing to be queried, or it can be a summary result of image feature similarity, text feature similarity, and image-text fusion feature similarity, etc. The specific result can be determined based on the query method of the candidate line drawing, and this specification does not impose any special restrictions on it.

[0178] In this example, weights can be assigned to contour similarity and feature similarity respectively, and then the overall similarity can be calculated by weighted summation. The overall constraint can be the top 5 candidate line drawings with the highest overall similarity. After calculating the overall similarity, the top 5 candidate line drawings can be selected as the target line drawing in descending order of overall similarity.

[0179] By using a weighted summation method to select target line art images, it can be ensured that the final target line art images are not only similar in the overall feature vector space, but also have a higher degree of matching in visual contour structure, thereby effectively improving the accuracy and robustness of line art image query.

[0180] Optionally, in another embodiment of this specification, the contour features may include global contour features and local contour features, that is, the contour features to be queried include global contour features to be queried and local contour features to be queried. Similarly, the candidate contour features may also include candidate global contour features and candidate local contour features.

[0181] Please refer to Figure 7 The process of optimizing query results may include the following steps:

[0182] Step 702: Determine the global contour similarity between each candidate line drawing and the line drawing to be queried based on the global contour features to be queried and the candidate global contour features.

[0183] In this embodiment, for each candidate line drawing, the global contour similarity between the candidate global contour features of the candidate line drawing and the query global contour features of the line drawing to be queried can be calculated.

[0184] Step 704: Select candidate line drawings whose global contour similarity satisfies the global contour constraint conditions as initial screening line drawings.

[0185] In this embodiment, the global contour constraint condition can also be preset. It can be that the global contour similarity is greater than a preset threshold, or it can be the 10 candidate line drawings with the highest global contour similarity, etc. This specification does not impose any special restrictions on this.

[0186] Assuming that the global contour constraint is that the global contour similarity is greater than 80%, then candidate line drawings with a global contour similarity greater than 80% can be selected as the initial screening line drawings obtained from the global contour coarse matching.

[0187] Step 706: For each preliminary screening line drawing, determine the local contour similarity between the preliminary screening line drawing and the line drawing to be queried based on the candidate local contour features of the preliminary screening line drawing and the query local contour features of the line drawing to be queried.

[0188] In this embodiment, after obtaining the preliminary line drawing images from the candidate line drawing images, for each preliminary line drawing image, the local contour similarity between the two can be calculated based on its local contour features (i.e., candidate local contour features) and the query local contour features of the line drawing image to be queried.

[0189] Step 708: Determine the target line drawing based on the local contour similarity in the initial screening line drawing.

[0190] As mentioned above Figure 6 Step 608 in the illustrated embodiment is similar; the initial screening line art images whose local contour similarity satisfies the local contour constraint condition can be determined as the target line art images. For example, the four initial screening line art images with the highest local contour similarity can be determined as the target line art images. Alternatively, local contour similarity and feature similarity can be combined to determine the target line art images.

[0191] In this embodiment, when performing contour similarity matching, a frequency domain analysis method based on Fourier descriptors can also be combined, and this specification does not impose any special limitations on this.

[0192] In this embodiment, by using multi-scale matching of contour features, a coarse global contour matching is first performed, and then a local contour detail matching is performed on the initial screening line drawing with high similarity to the global contour. This can enhance the adaptability to different deformation forms of line drawing and improve computational efficiency.

[0193] This demonstrates that after obtaining candidate line art images from the line art database, this manual can further filter these candidate line art images to determine the target line art image. Furthermore, it optimizes the query results by using key visual elements such as outlines as core related features, thereby improving the accuracy of the query results.

[0194] Figure 8 This is a schematic diagram illustrating an example of querying a line drawing, provided as an exemplary embodiment.

[0195] Please refer to Figure 8 Taking clothing line drawings as an example, designers can input the line drawing to be queried and the text to be queried shown on the left at the user interaction / user input terminal. Using the line drawing query optimization scheme provided in this manual, the four target line drawings shown on the right can be queried and output.

[0196] Figure 9 This is a schematic structural diagram of a device provided in an exemplary embodiment. Please refer to... Figure 9At the hardware level, the device includes a processor 902, an internal bus 904, a network interface 906, memory 908, and non-volatile memory 910, and may also include other hardware required for its functions. One or more embodiments of this specification can be implemented in software, for example, the processor 902 reads the corresponding computer program from the non-volatile memory 910 into memory 908 and then runs it. Of course, besides software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0197] Please refer to Figure 10 The line drawing query device 1000 can be applied to, for example... Figure 9 The device shown implements the technical solution of this specification. The line drawing query device 1000 may include:

[0198] Query and retrieve unit 1001 to retrieve the line drawing and text to be queried;

[0199] Text generation unit 1002 generates descriptive text for the line drawing to be queried;

[0200] Text fusion unit 1003 merges the query text and the description text to obtain fused text;

[0201] The feature extraction unit 1004 uses a multimodal model to extract features from the line drawing to be queried and the fused text to obtain image features, text features and image-text fusion features;

[0202] The candidate query unit 1005 queries the online drawing database based on the image features, the text features, and the image-text fusion features to obtain several candidate line drawings.

[0203] The target determination unit 1006 determines the target line drawing as the query result from the candidate line drawing diagrams.

[0204] Optionally, the process by which the target determination unit 1006 determines the target line drawing as the query result from the candidate line drawing diagrams includes:

[0205] Extract the outline features of the line drawing to be queried;

[0206] For each candidate line drawing, extract the candidate contour features of the candidate line drawing;

[0207] The contour similarity between the candidate line drawing and the line drawing to be queried is determined based on the contour features to be queried and the candidate contour features.

[0208] The target line drawing is determined from the candidate line drawing based on the contour similarity.

[0209] Optionally, the process of determining the target line drawing based on the contour similarity in the candidate line drawing includes:

[0210] For each candidate line drawing, a comprehensive similarity between the candidate line drawing and the line drawing to be queried is determined based on the feature similarity and the contour similarity between the candidate line drawing and the line drawing to be queried.

[0211] Candidate line drawings whose comprehensive similarity satisfies the comprehensive constraint conditions are determined as the target line drawings.

[0212] Optionally, the process by which the target determination unit 1006 determines the target line drawing as the query result from the candidate line drawing diagrams includes:

[0213] Candidate line drawings whose contour similarity satisfies the contour constraint conditions are determined as the target line drawings.

[0214] Optionally, the contour features include global contour features and local contour features;

[0215] The target determination unit 1006 determines the global contour similarity between each candidate line drawing and the line drawing to be queried based on the global contour features to be queried and the candidate global contour features.

[0216] Candidate line drawings whose global contour similarity satisfies the global contour constraint are selected as initial screening line drawings.

[0217] For each initial screening line drawing, the local contour similarity between the initial screening line drawing and the line drawing to be queried is determined based on the candidate local contour features of the initial screening line drawing and the query local contour features of the line drawing to be queried.

[0218] The target line drawing is determined from the initial screening line drawing based on the local contour similarity.

[0219] Optionally, the multimodal model includes an image encoder and a text encoder, and the training process includes:

[0220] Obtain the line drawing sample and the corresponding design text;

[0221] Generate descriptive text for the line drawing sample;

[0222] By combining the descriptive text of the line drawing sample and the design text, a text sample corresponding to the line drawing sample is obtained;

[0223] The line drawing samples and the text samples are input into a multimodal model. The image encoder in the multimodal model extracts the image features of the line drawing samples, and the text encoder extracts the text features of the text samples. The goal is to train the multimodal model to maximize the similarity of correct image-text pairings while minimizing the similarity of incorrect image-text pairings.

[0224] Optionally, the training process of the multimodal model further includes:

[0225] Extract the standard contour features of the line drawing sample;

[0226] The process of training a multimodal model with the goal of maximizing the similarity of correct image-text pairings while minimizing the similarity of incorrect image-text pairings includes:

[0227] The goal is to train a multimodal model that maximizes the similarity of correct image-text pairings, minimizes the similarity of incorrect image-text pairings, and minimizes the loss of contour features.

[0228] The contour feature loss is determined based on the standard contour features and the contour features of the line drawing sample extracted by the image encoder.

[0229] Optionally, the process by which the candidate query unit 1005 queries the online drawing database based on the image features, the text features, and the image-text fusion features to obtain several candidate line drawings includes:

[0230] Using the image features, text features, and image-text fusion features as query conditions, similar initial line drawings are searched in the line drawing database; the candidate line drawings are obtained by summarizing the initial line drawings found.

[0231] Optionally, the process by which the text fusion unit 1003 fuses the query text and the description text to obtain the fused text includes:

[0232] The query text and the description text are input into a large language model, and the query text and the description text are optimized and fused by the large language model to obtain the fused text; wherein, the large language model is obtained by fine-tuning a pre-trained language model in combination with a corpus and knowledge graph related to the domain to which the query line drawing belongs.

[0233] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0234] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0235] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

Claims

1. A method for querying line art diagrams, comprising: Retrieve the line drawing and text to be queried; Generate descriptive text for the line drawing to be queried; The queried text and the descriptive text are combined to obtain the fused text; A multimodal model is used to extract features from the line drawing to be queried and the fused text to obtain image features, text features and image-text fusion features; Based on the image features, text features, and image-text fusion features, several candidate line drawings were obtained by querying the online manuscript database; The target line drawing is selected from the candidate line drawing diagrams as the query result.

2. The method according to claim 1, wherein the process of determining the target line drawing as the query result in the candidate line drawing diagram includes: Extract the outline features of the line drawing to be queried; For each candidate line drawing, extract the candidate contour features of the candidate line drawing; The contour similarity between the candidate line drawing and the line drawing to be queried is determined based on the contour features to be queried and the candidate contour features. The target line drawing is determined from the candidate line drawing based on the contour similarity.

3. The method according to claim 2, wherein the process of determining the target line drawing based on the contour similarity in the candidate line drawing includes: For each candidate line drawing, a comprehensive similarity between the candidate line drawing and the line drawing to be queried is determined based on the feature similarity and the contour similarity between the candidate line drawing and the line drawing to be queried. Candidate line drawings whose comprehensive similarity satisfies the comprehensive constraint conditions are determined as the target line drawings.

4. The method according to claim 2, wherein the process of determining the target line drawing based on the contour similarity in the candidate line drawing includes: Candidate line drawings whose contour similarity satisfies the contour constraint conditions are determined as the target line drawings.

5. The method according to claim 2, wherein the contour features include global contour features and local contour features; The process of determining the contour similarity between the candidate line drawing and the line drawing to be queried based on the contour features to be queried and the candidate contour features, and determining the target line drawing from the candidate line drawing based on the contour similarity, includes: The global contour similarity between each candidate line drawing and the line drawing to be queried is determined based on the global contour features to be queried and the candidate global contour features. Candidate line drawings whose global contour similarity satisfies the global contour constraint are selected as initial screening line drawings. For each initial screening line drawing, the local contour similarity between the initial screening line drawing and the line drawing to be queried is determined based on the candidate local contour features of the initial screening line drawing and the query local contour features of the line drawing to be queried. The target line drawing is determined from the initial screening line drawing based on the local contour similarity.

6. The method according to claim 1, wherein the multimodal model comprises an image encoder and a text encoder, and the training process includes: Obtain the line drawing sample and the corresponding design text; Generate descriptive text for the line drawing sample; By combining the descriptive text of the line drawing sample and the design text, a text sample corresponding to the line drawing sample is obtained; The line drawing samples and the text samples are input into a multimodal model. The image encoder in the multimodal model extracts the image features of the line drawing samples, and the text encoder extracts the text features of the text samples. The goal is to train the multimodal model to maximize the similarity of correct image-text pairings while minimizing the similarity of incorrect image-text pairings.

7. The method according to claim 6, further comprising: Extract the standard contour features of the line drawing sample; The process of training a multimodal model with the goal of maximizing the similarity of correct image-text pairings while minimizing the similarity of incorrect image-text pairings includes: The goal is to train a multimodal model that maximizes the similarity of correct image-text pairings, minimizes the similarity of incorrect image-text pairings, and minimizes the loss of contour features. The contour feature loss is determined based on the standard contour features and the contour features of the line drawing sample extracted by the image encoder.

8. The method according to claim 1, wherein the process of querying a plurality of candidate line drawings in an online drawing database based on the image features, the text features, and the image-text fusion features includes: Using the image features, text features, and image-text fusion features as query conditions, similar initial line drawings are searched in the line drawing database. The candidate line drawing is obtained by summarizing the initial line drawing obtained from the query.

9. The method according to claim 1, wherein the process of fusing the query text and the description text to obtain the fused text comprises: The query text and the description text are input into a large language model, and the query text and the description text are optimized and fused by the large language model to obtain the fused text; The large language model is obtained by fine-tuning a pre-trained language model by combining a corpus and knowledge graph related to the domain to which the line drawing to be queried belongs.

10. A line drawing query device, comprising: The query retrieval unit retrieves the line drawing and text to be queried. The text generation unit generates descriptive text for the line drawing to be queried; The text fusion unit merges the query text and the description text to obtain fused text; The feature extraction unit uses a multimodal model to extract features from the line drawing to be queried and the fused text, obtaining image features, text features and image-text fusion features; The candidate query unit retrieves several candidate line drawings from the online drawing database based on the image features, the text features, and the image-text fusion features. The target determination unit determines the target line drawing as the query result from the candidate line drawing drawings.

11. An electronic device, comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1-9 by executing the executable instructions.

12. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-9.

13. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-9.

Images