Sketch retrieval method, device, electronic device and storage medium

Abstract

The present invention discloses a sketch retrieval method, device, electronic device and storage medium. The method comprises: obtaining a sketch image to be retrieved and a sketch perspective label of the sketch image to be retrieved, and obtaining the sketch image features of the sketch image to be retrieved based on a pre-trained sketch retrieval model and the sketch perspective label; obtaining a pre-extracted first real image feature; and determining the retrieval result of the sketch image to be retrieved based on the sketch image feature and the first real image feature. By running the technical solution provided by the embodiment of the present invention, the problem that the sketch retrieval model adopts a deep learning network with a complex structure and a high number of parameters, which requires high hardware computing power and storage space, and leads to high difficulty in implementing sketch retrieval can be solved, thereby achieving the beneficial effect of improving the accuracy and efficiency of sketch retrieval.

Description

Technical Field

[0001] The present invention relates to computer technology, and in particular to a sketch retrieval method, device, electronic equipment and storage medium. Background Art

[0002] With the development of computer technology, the requirements for image retrieval are getting higher and higher. Traditional image retrieval through text cannot meet the needs of image retrieval in some application scenarios. Therefore, image retrieval based on hand-drawn sketches came into being.

[0003] In the existing technology, sketch retrieval models use deep learning networks with complex structures and high parameters to improve retrieval accuracy. Therefore, these models have high requirements on hardware computing power and storage space, making it difficult to implement sketch retrieval. Summary of the Invention

[0004] The present invention provides a sketch retrieval method, device, electronic device and storage medium to improve the accuracy and efficiency of sketch retrieval.

[0005] According to one aspect of the present invention, a sketch retrieval method is provided, the method comprising:

[0006] Obtaining a sketch image to be retrieved and a sketch perspective label of the sketch image to be retrieved, and obtaining sketch image features of the sketch image to be retrieved based on a pre-trained sketch retrieval model and the sketch perspective label; wherein the sketch retrieval model includes a Swin Tiny model;

[0007] Obtaining a pre-extracted first real image feature;

[0008] A retrieval result of the sketch image to be retrieved is determined according to the sketch image feature and the first real image feature.

[0009] According to another aspect of the present invention, a sketch retrieval device is provided, the device comprising:

[0010] a sketch image feature acquisition module, configured to acquire a sketch image to be retrieved and a sketch perspective label of the sketch image to be retrieved, and acquire sketch image features of the sketch image to be retrieved based on a pre-trained sketch retrieval model and the sketch perspective label; wherein the sketch retrieval model includes a Swin Tiny model;

[0011] A first feature acquisition module, configured to acquire a pre-extracted first real image feature;

[0012] A retrieval result determination module is used to determine the retrieval result of the sketch image to be retrieved based on the sketch image features and the first real image features.

[0013] According to another aspect of the present invention, an electronic device is provided, comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the sketch retrieval method according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the sketch retrieval method according to any embodiment of the present invention when executed.

[0018] The technical solution of an embodiment of the present invention obtains a sketch image to be retrieved and a sketch perspective label for the sketch image to be retrieved, and obtains sketch image features of the sketch image to be retrieved based on a pre-trained sketch retrieval model and the sketch perspective label; obtains pre-extracted first real image features; and determines a retrieval result for the sketch image to be retrieved based on the sketch image features and the first real image features. This solves the problem that sketch retrieval models employ complex structures and high-parameter deep learning networks, which require high hardware computing power and storage space, making sketch retrieval more difficult. This solution achieves the beneficial effect of improving the accuracy and efficiency of sketch retrieval.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a sketch retrieval method provided in Example 1 of the present invention;

[0021] Figure 2 A training flow chart of a sketch retrieval model provided in the second embodiment of the present invention;

[0022] Figure 3 A schematic diagram of a sketch retrieval model training process provided in the second embodiment of the present invention;

[0023] Figure 4 A schematic structural diagram of a sketch retrieval device provided in a third embodiment of the present invention;

[0024] Figure 5FIG. 1 is a schematic structural diagram of an electronic device for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first," "second," "target," and the like in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this way are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0027] Example 1

[0028] Figure 1 This is a flowchart of a sketch retrieval method provided in the first embodiment of the present invention. This embodiment is applicable to the case of retrieving a real image corresponding to a sketch image. The method can be executed by the sketch retrieval device provided in the embodiment of the present invention, which can be implemented in software and / or hardware. Figure 1 The sketch retrieval method provided in this embodiment includes:

[0029] S110. Obtain a sketch image to be retrieved and a sketch perspective label of the sketch image to be retrieved, and obtain sketch image features of the sketch image to be retrieved based on a pre-trained sketch retrieval model and the sketch perspective label; wherein the sketch retrieval model includes a Swin Tiny model.

[0030] The sketch image to be retrieved can be a hand-drawn sketch, such as a hand-drawn sketch of an architectural image, and this embodiment does not limit this. The sketch perspective label is the perspective from which the content of the sketch image to be retrieved is depicted, such as a top view, bird's-eye view, plan view, interior view, or detail view. The sketch perspective label can be manually annotated, and this embodiment does not limit this.

[0031] The sketch image to be retrieved and the corresponding sketch perspective label are input into the pre-trained sketch retrieval model to obtain the sketch image features of the sketch image to be retrieved.

[0032] Among them, the Swin Tiny model is the version with the least number of parameters among the versions of the Swin Transformer model.

[0033] S120: Acquire a pre-extracted first real image feature.

[0034] The real image is an image actually taken, such as an image of a building, etc. The pre-extracted first real image feature is a feature obtained by performing a feature extraction operation on a pre-acquired real image set using a preset feature extraction model.

[0035] Among them, the first real image feature can be stored in a designated location after extraction. When the first real image feature needs to be obtained, it can be read from the designated location to the cache to further process the first real image feature in the cache, so as to improve the subsequent image retrieval efficiency.

[0036] In this embodiment, optionally, before obtaining the pre-extracted first real image feature, the method further includes:

[0037] Obtaining a real image feature file, and determining whether the first real image feature in the real image feature file is complete;

[0038] If not, the real image feature file is updated to obtain the first real image feature from the updated real image feature file.

[0039] The real image feature file may be obtained from a specified location such as a storage location of the real image feature, and includes the first real image feature read from the specified location.

[0040] Determining whether the first real image feature in the real image feature file is complete can be done by performing a hash search or other method to determine whether the feature is complete. For example, after obtaining the search result for the first sketch image to be retrieved, and before searching for the second sketch image to be retrieved, the historical first real image feature in the historical real image feature file obtained in the previous search process can be directly obtained. At this time, a hash search or other method is used to determine whether the historical first real image feature is all the features in the specified location. If not, it indicates that the first real image feature is incomplete. In this case, the unobtained first real image feature in the specified location is read into the real image feature file to update the real image feature file.

[0041] By judging whether the first real image feature in the real image feature file is complete; if not, updating the real image feature file, it is convenient to obtain the complete first real image feature from the real image feature file in the future, thereby improving the accuracy of sketch retrieval; at the same time, except for the first sketch retrieval, it is only updated when the first real image feature is not fully read, avoiding the need to read all the first real image features from the specified position every time the sketch is retrieved, thereby improving the acquisition speed of the first real image feature and improving the efficiency of sketch retrieval.

[0042] S130: Determine a retrieval result of the sketch image to be retrieved according to the sketch image feature and the first real image feature.

[0043] The distances between the sketch image features and all first real image features are calculated using methods such as the Euclidean formula, and the distances can be sorted in descending order. The search results can be one or more, determined based on different search requirements. For example, if the search requirement is to find the ten real images closest to the sketch image to be searched, the real images corresponding to the top ten distances can be sorted based on the distances and displayed as the search results.

[0044] The technical solution provided in this embodiment obtains the sketch image to be retrieved and the sketch perspective label of the sketch image to be retrieved, and obtains the sketch image features of the sketch image to be retrieved based on the pre-trained sketch retrieval model and the sketch perspective label, wherein the sketch retrieval model includes the Swin Tiny model; obtains the pre-extracted first real image features; and determines the retrieval result of the sketch image to be retrieved based on the sketch image features and the first real image features. The Swin Tiny model has a small number of parameters, which reduces the model's requirements for hardware computing power and storage space, and solves the problem that the deep learning network used has a complex structure, a high number of parameters, and high requirements for hardware computing power and storage space, which leads to high difficulty in implementing sketch retrieval, thereby improving the efficiency of sketch retrieval. In addition, the sketch perspective label obtains the sketch image features, which increases the information basis for comparing the sketch image features with the real image features, thereby improving the accuracy of sketch retrieval.

[0045] Example 2

[0046] Figure 2 This is a training flowchart for a sketch retrieval model provided in Example 2 of the present invention. This technical solution provides a supplementary explanation of the training process of the sketch retrieval model. Compared with the above solution, this solution is specifically optimized as follows: the training process of the sketch retrieval model includes:

[0047] Obtaining a sketch image sample, a real image positive sample, and a real image negative sample, and determining a ternary sample group according to the sketch image sample, the real image positive sample, and the real image negative sample;

[0048] The sketch retrieval model is obtained by training the ternary sample group and the preset training model; wherein the preset training model is a twin network model, and the sub-model of the twin network model is composed of the Swin Tiny model and the multi-layer perceptron. Specifically, the training flow chart of the sketch retrieval model is as follows Figure 2 As shown:

[0049] S210 , obtaining a sketch image sample, a real image positive sample, and a real image negative sample, and determining a ternary sample group according to the sketch image sample, the real image positive sample, and the real image negative sample.

[0050] The sketch image sample is a sketch image corresponding to the real image sample. The sketch image sample can be obtained by inputting the real image sample into the sketch generation model, which is not limited in this embodiment.

[0051] The sketch image samples constituting the ternary sample group are sketch image samples corresponding to the current real image positive samples. The current real image positive samples may be determined by randomly selecting from all real image samples, which is not limited in this embodiment.

[0052] The real image negative samples that make up the triplet sample group are real image samples that have a large difference in similarity with the current real image positive sample. Among them, one current real image positive sample can correspond to one or more real image negative samples.

[0053] A ternary sample group is determined based on the sketch image sample, the real image positive sample and the real image negative sample. For example, there is a sketch image sample a1, a real image positive sample b1, and real image negative samples c1 and c2, then the ternary sample group can be {a1, b1, c1} and {a1, b1, c2}.

[0054] In this embodiment, optionally, obtaining a sketch image sample, a real image positive sample, and a real image negative sample includes:

[0055] Determine a first target real image from the candidate real images according to a preset method, determine the first target real image as the real image positive sample, and obtain a second real image feature of the real image positive sample;

[0056] Determining a preset number of second target real images from the candidate real images according to the second real image features, and determining the second target real images as the real image negative samples;

[0057] A target sketch image corresponding to the first target real image is determined from candidate sketch images as the sketch image sample.

[0058] The first target real image is determined from the candidate real images according to a preset method, wherein the candidate real images can be all the acquired real images, and the preset method can be a random determination method or a sequential determination method according to the acquisition order. This embodiment does not limit this.

[0059] The determined real image is used as the first target real image, and the first target real image is determined as the real image positive sample x p , obtaining the second real image feature of the real image positive sample. The acquisition method can be to extract the directional gradient histogram feature of the real image positive sample by using the directional gradient histogram feature extraction method.

[0060] Determine a preset number of second target real images from the candidate real images based on the second real image features. This can be done by calculating the Euclidean distance between the directional gradient histogram features of the real image positive sample and the directional gradient histogram features of other real images in the candidate real images. The closer the distance, the more similar. Arrange all other real images according to their similarity, and randomly select a preset number of real images within a preset range, such as the top 30% of the dissimilarities, such as 40, as the second target real images, and determine the second target real images as the real image negative samples x. n .

[0061] The target sketch image corresponding to the first target ground-truth image is determined as the sketch image sample x from the candidate sketch images. a , where the candidate sketch images can be all sketch images.

[0062] Correspondingly, a real image positive sample x p It can correspond to a preset number of triplet sample groups {x a ,x p ,x n}.

[0063] By obtaining sketch image samples, real image positive samples and real image negative samples in the above manner, the accuracy of determining sketch image samples, real image positive samples and real image negative samples is improved, and since manual selection is avoided, the efficiency of determining sketch image samples, real image positive samples and real image negative samples is improved.

[0064] In this embodiment, optionally, the candidate sketch image is generated according to the candidate real image in a preset proportion through a DexiNed model or a Photo Sketching model.

[0065] The sketches generated by the DexiNed model are rich in detail and closer to edge-extracted images, while the hand-drawn images generated by the PhotoSketching model remove more detail and more closely resemble rough, real-world sketches. The sketch composition ratio in the candidate sketch image can be 90% generated by the DexiNed model and 10% generated by the PhotoSketching model, which is not limited in this embodiment. Incorporating rough sketches into the fine sketches avoids overfitting of the model to the fine sketches, thereby improving model generalization.

[0066] S220. Obtain the sketch retrieval model based on the ternary sample group and the preset training model; wherein the preset training model is a twin network model, and the sub-model of the twin network model consists of the Swin Tiny model and a multi-layer perceptron.

[0067] The ternary sample group is input into the preset training model, and the sketch retrieval model is obtained by training. The preset training model is a twin network model, and the sub-model of the twin network model is composed of a Swin Tiny model and a multi-layer perceptron. The sub-model of the twin network model is used to respectively obtain the sample features of each type of sample in the ternary sample group, and each sub-model shares weights. Exemplarily, the sketch image sample a1 in the ternary sample group is input into sub-model one of the twin network model, the real image positive sample b1 is input into sub-model two of the twin network model, and the real image negative sample c1 is input into sub-model three of the twin network model. Sub-models one, two and three together obtain a model calculation result.

[0068] In this embodiment, optionally, the sketch retrieval model is obtained by training according to the triple sample group and a preset training model, including:

[0069] Obtaining sample perspective labels of the real image positive samples and / or the real image negative samples in the ternary sample group according to a pre-trained perspective classification model;

[0070] Inputting the sketch image sample, the real image positive sample and the real image negative sample in the ternary sample group into the Swin Tiny model in the corresponding sub-model respectively, and obtaining the corresponding first-level sample features in each sub-model respectively;

[0071] Obtaining corresponding secondary sample features in each of the sub-models according to the primary sample features, the sample view labels, and the multi-layer perceptron, so as to calculate a ternary loss function according to the secondary sample features;

[0072] Determine whether the training process of the sketch retrieval model is completed according to the ternary loss function.

[0073] The sample perspective labels of the real image positive samples and / or real image negative samples in the ternary sample group are obtained according to the pre-trained perspective classification model, wherein the pre-trained perspective classification model can be a model trained to an accuracy rate of 95% based on a large number of pre-labeled real images.

[0074] Figure 3 This is a schematic diagram of a sketch retrieval model training process provided by the second embodiment of the present invention. Figure 3 Shown:

[0075] Input a real-world positive image sample and / or a real-world negative image sample into the pre-trained view classification model to obtain an output result. The output result may include multiple nodes, each representing a corresponding view type. For example, each node represents a view type of top view, bird's-eye view, plan view, indoor view, and detail view, respectively. The output is (0, 0, 0, 1, 0), indicating that the sample view label of the real-world positive image sample or the real-world negative image sample is indoor view.

[0076] The sketch image samples, real image positive samples and real image negative samples in the triple sample group are respectively input into the main model Swin Tiny model in the corresponding sub-model to obtain the corresponding first-level sample features in each sub-model.

[0077] The Swin Tiny model in each sub-model is connected to a multi-layer perceptron respectively, and the first-level sample features and the sample perspective label of the sample corresponding to the sub-model are jointly input into the multi-layer perceptron. Among them, the sample perspective label of the sketch image sample is the sample perspective label of the real image positive sample corresponding to the sketch image sample.

[0078] Secondary sample features are obtained through the multi-layer perceptron in each sub-model, and a ternary loss function is calculated using these secondary sample features. The training process of the sketch retrieval model is determined to be complete based on the ternary loss function. For example, if the training process converges to below 0.06, the training process of the sketch retrieval model is determined to be complete.

[0079] Existing sketch retrieval models based on convolutional neural networks excel at extracting local features but tend to overlook global features. They also perform poorly on datasets containing images with repeated local features. By combining a Siamese network model and a ternary loss function as the main framework and using the Swin Tiny model as the primary model for each Siamese network branch, the trained sketch retrieval model's ability to extract global image features is improved, resolving these issues and enhancing the practicality of the trained sketch model.

[0080] The embodiment of the present invention obtains sketch image samples, real image positive samples, and real image negative samples, and determines a ternary sample group based on the sketch image samples, real image positive samples, and real image negative samples; and obtains a sketch retrieval model by training the ternary sample group and a preset training model; wherein the preset training model is a twin network model, and the sub-model of the twin network model is composed of a Swin Tiny model and a multi-layer perceptron. Improvements are made to the twin network-based model, thereby improving the accuracy of the trained sketch retrieval model when performing sketch retrieval, while maintaining a simple model structure and a small number of parameters, thereby improving the practicality of the sketch retrieval model.

[0081] Example 3

[0082] Figure 4 This is a schematic diagram of the structure of a sketch retrieval device provided in the third embodiment of the present invention. The device can be implemented by hardware and / or software, can execute a sketch retrieval method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. Figure 4 As shown, the device includes:

[0083] A sketch image feature acquisition module 410 is configured to acquire a sketch image to be retrieved and a sketch perspective label of the sketch image to be retrieved, and to acquire sketch image features of the sketch image to be retrieved based on a pre-trained sketch retrieval model and the sketch perspective label; wherein the sketch retrieval model includes a Swin Tiny model;

[0084] A first feature acquisition module 420 is used to acquire a pre-extracted first real image feature;

[0085] The retrieval result determination module 430 is configured to determine a retrieval result of the sketch image to be retrieved based on the sketch image features and the first real image features.

[0086] Based on the above technical solutions, an optional sketch retrieval model training module includes:

[0087] a ternary sample group acquisition unit, configured to acquire a sketch image sample, a real image positive sample, and a real image negative sample, and determine a ternary sample group according to the sketch image sample, the real image positive sample, and the real image negative sample;

[0088] A sketch retrieval model training unit is used to obtain the sketch retrieval model based on the ternary sample group and the preset training model; wherein the preset training model is a twin network model, and the sub-model of the twin network model consists of the Swin Tiny model and a multi-layer perceptron.

[0089] Based on the above technical solutions, optionally, the ternary sample group acquisition unit includes:

[0090] a real image positive sample determination subunit, configured to determine a first target real image from the candidate real images according to a preset method, determine the first target real image as the real image positive sample, and obtain a second real image feature of the real image positive sample;

[0091] a real image negative sample determination subunit, configured to determine a preset number of second target real images from the candidate real images according to the second real image features, and determine the second target real images as the real image negative samples;

[0092] The sketch image sample determination subunit is configured to determine a target sketch image corresponding to the first target real image from candidate sketch images as the sketch image sample.

[0093] Based on the above technical solutions, optionally, the sketch retrieval model training unit includes:

[0094] a label obtaining subunit, configured to obtain, according to a pre-trained view classification model, sample view labels of the real image positive samples and / or the real image negative samples in the ternary sample group;

[0095] A first feature acquisition subunit is configured to input the sketch image sample, the real image positive sample, and the real image negative sample in the ternary sample group into the Swin Tiny model in the corresponding sub-model, respectively, to obtain the corresponding primary sample features in each sub-model;

[0096] A second feature acquisition subunit is configured to obtain corresponding secondary sample features in each of the sub-models based on the primary sample features, the sample view labels, and the multi-layer perceptron, so as to calculate a ternary loss function based on the secondary sample features;

[0097] The process end determination subunit is used to determine whether the training process of the sketch retrieval model is completed based on the ternary loss function.

[0098] On the basis of the above technical solutions, optionally, the candidate sketch image is generated according to the candidate real image in a preset proportion through a DexiNed model or a Photo Sketching model.

[0099] On the basis of the above technical solutions, optionally, the device further includes:

[0100] a feature integrity judgment module, configured to obtain a real image feature file before the first feature acquisition module, and to judge whether the first real image feature in the real image feature file is complete;

[0101] A file updating module is configured to update the real image feature file if the feature integrity judgment module determines that the real image feature is not present, so as to obtain the first real image feature from the updated real image feature file.

[0102] Example 4

[0103] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0104] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0105] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0106] The processor 11 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors for running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the sketch retrieval method.

[0107] In some embodiments, the sketch retrieval method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the sketch retrieval method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the sketch retrieval method in any other suitable manner (e.g., by means of firmware).

[0108] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0109] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0110] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0111] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0112] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0113] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0114] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0115] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A sketch retrieval method, characterized in that: include: Obtaining a sketch image to be retrieved and a sketch perspective label of the sketch image to be retrieved, and obtaining sketch image features of the sketch image to be retrieved based on a pre-trained sketch retrieval model and the sketch perspective label; wherein the sketch retrieval model includes a Swin Tiny model; the sketch perspective label is perspective description information of the sketch image to be retrieved, and the sketch perspective label includes: top view, bird's eye view, floor plan, interior view, and detail view; Obtaining a pre-extracted first real image feature; determining a retrieval result of the sketch image to be retrieved according to the sketch image feature and the first real image feature; The training process of the sketch retrieval model includes: Obtaining a sketch image sample, a real image positive sample, and a real image negative sample, and determining a ternary sample group according to the sketch image sample, the real image positive sample, and the real image negative sample; The sketch retrieval model is obtained by training according to the ternary sample group and a preset training model; wherein the preset training model is a twin network model, and a sub-model of the twin network model is composed of the Swin Tiny model and a multi-layer perceptron; The step of training the sketch retrieval model based on the triple sample group and a preset training model includes: Obtaining sample perspective labels of the real image positive samples and / or the real image negative samples in the ternary sample group according to a pre-trained perspective classification model; Inputting the sketch image sample, the real image positive sample and the real image negative sample in the ternary sample group into the Swin Tiny model in the corresponding sub-model respectively, and obtaining the corresponding first-level sample features in each sub-model respectively; Obtaining corresponding secondary sample features in each of the sub-models according to the primary sample features, the sample view labels, and the multi-layer perceptron, so as to calculate a ternary loss function according to the secondary sample features; Determine whether the training process of the sketch retrieval model is completed according to the ternary loss function.

2. The method according to claim 1, characterized in that Get sketch image samples, real image positive samples, and real image negative samples, including: Determine a first target real image from the candidate real images according to a preset method, determine the first target real image as the real image positive sample, and obtain a second real image feature of the real image positive sample; Determine a preset number of second target real images from the candidate real images according to the second real image features, and determine the second target real images as the real image negative samples; A target sketch image corresponding to the first target real image is determined from candidate sketch images as the sketch image sample.

3. The method according to claim 2, characterized in that The candidate sketch image is generated according to the candidate real image in a preset proportion through a DexiNed model or a Photo Sketching model.

4. The method according to claim 1, wherein Before obtaining the pre-extracted first real image feature, the method further includes: Obtaining a real image feature file, and determining whether the first real image feature in the real image feature file is complete; If not, the real image feature file is updated to obtain the first real image feature from the updated real image feature file.

5. A sketch retrieval device, characterized in that: include: A sketch image feature acquisition module is configured to acquire a sketch image to be retrieved and a sketch perspective label of the sketch image to be retrieved, and to acquire sketch image features of the sketch image to be retrieved based on a pre-trained sketch retrieval model and the sketch perspective label; wherein the sketch retrieval model includes a Swin Tiny model; the sketch perspective label is perspective description information of the sketch image to be retrieved, and the sketch perspective label includes: top view, bird's-eye view, floor plan, interior view, and detail view; A first feature acquisition module, configured to acquire a pre-extracted first real image feature; a retrieval result determination module, configured to determine a retrieval result of the sketch image to be retrieved based on the sketch image features and the first real image features; The sketch retrieval model training module includes: a ternary sample group acquisition unit, configured to acquire a sketch image sample, a real image positive sample, and a real image negative sample, and determine a ternary sample group according to the sketch image sample, the real image positive sample, and the real image negative sample; A sketch retrieval model training unit, configured to obtain the sketch retrieval model based on the ternary sample group and a preset training model; wherein the preset training model is a twin network model, and a sub-model of the twin network model is composed of the Swin Tiny model and a multi-layer perceptron; The sketch retrieval model training unit includes: a label obtaining subunit, configured to obtain, according to a pre-trained view classification model, sample view labels of the real image positive samples and / or the real image negative samples in the ternary sample group; A first feature acquisition subunit is configured to input the sketch image sample, the real image positive sample, and the real image negative sample in the ternary sample group into the Swin Tiny model in the corresponding sub-model, respectively, to obtain the corresponding primary sample features in each sub-model; A second feature acquisition subunit is configured to obtain corresponding secondary sample features in each of the sub-models based on the primary sample features, the sample view labels, and the multi-layer perceptron, so as to calculate a ternary loss function based on the secondary sample features; The process end determination subunit is used to determine whether the training process of the sketch retrieval model is completed based on the ternary loss function.

6. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the sketch retrieval method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the sketch retrieval method according to any one of claims 1 to 4 when executed.

Images