Method, computing device and storage medium for image search based on drawing
Patent Information
- Application Number
- CN202011023174.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-09-25
- Filing Date
- 2020-09-25
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-09-25
AI Technical Summary
然而,不管怎样,搜索结果对于用户来说可能并不令人满意
Smart Images

Figure CN112559790B_ABST
Abstract
Description
Technical Field
[0001] The techniques generally described herein relate to image searching. Background Art
[0002] Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.
[0003] Recently, image search services on user devices have been growing. In this image search, the query image may contain multiple objects, or the target of the image search may be in a portion of the query image. In order to specify the search target, the image search service generally provides an adjustable rectangle displayed on the query image, so that the user can adjust the rectangle to select an area covering the entire search target in the query image. After adjusting the rectangle, the user device uses the adjusted rectangle to crop the query image to obtain a cropped image for the image search. However, the cropped image may contain the background around the search target when the search target is not a rectangle, and therefore, the search results may be affected by the background. In addition, when the search target contains two or more objects such as a pair of shoes, the user may need to use a small rectangle to specify only one object among the multiple objects or apply a large rectangle covering all the multiple objects. However, in any case, the search results may not be satisfactory to the user. At the same time, the user device may neither distinguish between the foreground and the background nor determine the inclusion relationship in the adjusted rectangle. For example, when the rectangle is located on the upper body of a person wearing a cardigan over a T-shirt, it is difficult to use the rectangle-based method described above to determine whether the user wants to specify the cardigan or the T-shirt as the search target. Summary of the invention
[0004] The techniques generally described herein relate to image searching.
[0005] Various example methods using one or more computing devices are described. Example methods may include displaying an input image including at least one object using one or more computing devices; receiving, using the one or more computing devices, an operation to draw at least one object in the displayed input image to generate region data of at least one drawn region; preprocessing, using the one or more computing devices, the input image and the region data to produce preprocessed data; and searching an image database based, at least in part, on the preprocessed data using the one or more computing devices. In some instances, the receiving of the operation includes determining an end of the operation based, at least in part, on user input. In some instances, the preprocessing may include improving, using the one or more computing devices, the region data to produce improved region data. The preprocessing may further include at least one of: removing background from the improved region data using the one or more computing devices; extracting, using the one or more computing devices, at least one feature data from the input image based, at least in part, on the improved region data; and transforming, using the one or more computing devices, the at least one feature data to produce the preprocessed data.
[0006] In some examples, a computing device for image search is described. An example computing device may include at least one processor and at least one memory. The at least one memory may store instructions. The at least one processor executes the instructions to perform operations. The operations may include: obtaining an input image including at least one object that is a search target; displaying the input image on a display screen associated with the computing device; receiving an operation to draw at least one object in the displayed input image to generate region data of at least one drawn region; and displaying one or more search results on the display screen by an image search based at least in part on the input image and the region data. In some examples, the operations may further include preprocessing the input image and the region data to produce preprocessed data. In some examples, the preprocessing may include improving the region data to produce improved region data. The preprocessing may further include at least one of: eliminating background of the improved region data; extracting at least one feature data from the input image based at least in part on the improved region data; and transforming the at least one feature data to produce the preprocessed data.
[0007] In some examples, a non-transitory computer-readable storage medium is described. An example non-transitory computer-readable storage medium may store instructions that may be executed by a computing device to cause the computing device to perform operations. The operations may include: displaying an input image including at least one object that is a search target; receiving an operation to draw at least one object in the displayed input image to generate region data of at least one drawn region; preprocessing the input image and the region data to produce preprocessed data; and displaying one or more search results by an image search based at least in part on the preprocessed data. In some examples, the receiving of the operation includes determining the end of the operation based at least in part on user input. In some examples, the preprocessing may include improving the region data to produce improved region data. The preprocessing may further include: eliminating a background of the improved region data; extracting at least one feature data from the input image based at least in part on the improved region data; and transforming the at least one feature data to produce the preprocessed data.
[0008] The foregoing summary is illustrative only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] The foregoing and other features of the present disclosure will become more apparent from the following description and appended claims taken in conjunction with the accompanying drawings. It should be understood that these drawings depict only several embodiments in accordance with the present disclosure and are therefore not to be considered limiting of the scope of the present disclosure, which will be described with additional specificity and detail through the use of these drawings, in which:
[0010] Figure 1 A diagram showing an example image search system arranged in accordance with at least some embodiments described herein, wherein a user device is configured to communicate with a server via a network;
[0011] Figure 2 A block diagram illustrating an example user device arranged in accordance with at least some embodiments described herein;
[0012] Figure 3 shows a block diagram of an example server arranged in accordance with at least some embodiments described herein;
[0013] Figure 4 an example flow chart illustrating a process for image searching arranged in accordance with at least some embodiments described herein;
[0014] Figure 5A An example input image is shown, and Figure 5B and Figure 5C shows an illustrative example of drawing at least one region for image searching, arranged in accordance with at least some embodiments described herein;
[0015] Figure 6 shows an illustrative example of a user device configured to receive a drawing operation from a user, arranged in accordance with at least some embodiments described herein;
[0016] Figure 7 An example flow chart illustrating a process for pre-processing input images and region data arranged in accordance with at least some embodiments described herein;
[0017] Figure 8 an example flow chart illustrating a process for improving region data arranged in accordance with at least some embodiments described herein;
[0018] 9A to 9C Shown according to Figure 8 An example flowchart of an illustrative example of improving regional data;
[0019] Fig.10 An example flow chart illustrating a process for removing background from region data, arranged in accordance with at least some embodiments described herein;
[0020] Fig.11A and Fig. 11B Shown according to Fig.10 An illustrative example of removing background from a flowchart;
[0021] Fig.12 An example flow chart illustrating a process for transforming at least one feature data arranged in accordance with at least some embodiments described herein;
[0022] Fig.13 describing a computer program product that may be used to perform an image search, arranged in accordance with at least some embodiments described herein; and
[0023] Fig.14 Shown is a simplified block diagram of an example computing device that may be configured for image searching, arranged in accordance with at least some embodiments described herein. DETAILED DESCRIPTION
[0024] In the following detailed description, reference is made to the accompanying drawings, which form a part of this document. In the accompanying drawings, similar symbols generally identify similar components unless the context otherwise dictates. The illustrative embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein. The various aspects of the present disclosure as generally described herein and illustrated in the accompanying drawings may be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are expressly contemplated herein.
[0025] In general, the present disclosure relates to methods, devices, systems, apparatuses, and computer program products related to image searching, among other things.
[0026] Briefly, a technique for searching images using user input of a drawing operation received on a user device is generally described. In various instances, a user may want to use a user device to search for a certain object shown in an input image. The user device may display an input image containing at least one object that is a search target on a display screen. The input image may be captured by a camera associated with the user device, read from a memory of the user device, or retrieved via a network such as the Internet. The user may use an input member to draw a search target in the displayed input image. In an example, the user may use his / her finger to draw a search target on a display screen configured to receive touch input. In another example, the user may use a peripheral device such as a mouse and a stylus to draw the search target. In response to this operation of the user drawing the search target, the user device may receive the above operation and generate area data for at least one drawing area.
[0027] In some examples, the user device may perform at least one preprocessing on the input image and the region data to generate preprocessed data. The at least one preprocessing may in particular include extracting at least one feature data from the input image based on the region data. The user device may transmit the preprocessed data to the server via a network, and the server may search the image database for one or more images matching the preprocessed data. The server may transmit one or more search results obtained from the search of the image database to the user device, and the user device may receive the one or more search results and display the one or more search results on a display screen. Although the preprocessing and the search are described as being performed by the user device and the server, respectively, in the above, it is also possible that the preprocessing is performed by the server and / or the search is performed by the user device, depending on the desired configuration.
[0028] Figure 1A diagram of an example image search system arranged in accordance with at least some embodiments described herein is shown, wherein a user device is configured to communicate with a server via a network. As depicted, image search system 100 may include user device 110, network 120, and server 130. User device 110 may be any type of computing device such as user device 112, 114, ... or 116 that is configured to run, execute, or operate an application program containing instructions in accordance with the present disclosure and is configured to display an input image and one or more search results.
[0029] The user device 110 can receive data from the server 130 and / or other external devices (not shown) (e.g., such as Play) via the network 120. App The various types of user devices 110 may include, but are not limited to, desktop computers, mobile computers (e.g., laptops, ultrabooks), mobile phones, smart phones, tablet computers, smart TVs, smart watches, display screens, personal digital assistants (PDAs), smart furniture, smart home devices, smart vehicles, smart transportation devices, and / or smart accessories.
[0030] The user device 110 can access the server 130 via the network 120. The network 120 can be any wired network, wireless network or a combination thereof. In addition, the network 120 can include but is not limited to a personal area network, a local area network, a wide area network, an air broadcast network (e.g., for radio or television), a cable network, a satellite network, a cellular telephone network or a combination thereof. For example, the network 120 can be a publicly accessible network of linked networks that may be operated by different parties, such as the Internet. In some embodiments, the network 120 can be a private or semi-private network, such as an intranet of a company or university. The network 120 can include one or more wireless networks, such as a Global System for Mobile Communications (GSM) network, a code division multiple access (CDMA) network, a long-term evolution (LTE) network, an advanced long-term evolution (LTE-A) network, a microwave access interoperability (WiMax) network, a wireless fidelity (Wi-Fi) network and any other type of wireless network. For example, the protocols used by the network 120 may include Hypertext Transfer Protocol (HTTP), HTTP Secure (HTTPS), Message Queue Telemetry Transport (MQTT), Constrained Application Protocol (CoAP), etc. Protocols and components for communicating via the Internet or any other of the aforementioned types of communication networks are well known to those skilled in the art and are therefore not described in further detail herein.
[0031] The server 130 may be implemented as a single computing device, a server cluster including multiple servers, a distributed network, a cloud computing configuration, etc. The server 130 may include a memory and one or more processors.
[0032] In some embodiments, user device 110 may be configured to obtain an input image, which is a query image for image search. In some instances, user device 110 may capture the input image by a camera associated with user device 110. In some other instances, user device 110 may read the input image from a local memory of user device 110. In other cases, user device 110 may retrieve the input image via network 120 (e.g., the Internet). The input image may include at least one object that is a search target.
[0033] The user device 110 is configured to display an input image on a display screen associated with the user device 110. In addition, the user device 110 may be configured to receive an operation of drawing at least one object in the displayed input image. The user device 110 may receive the user's drawing operation via input means including, for example, but not limited to, a touch screen, a button, a keyboard, a mouse, a stylus, a touchpad, etc. When receiving the drawing operation, the user device 110 may determine the end of the drawing operation based on the user input. In some examples, the user may touch or click a button to stop the drawing operation and / or touch or click another button to start an image search, and the user device 110 may determine the end of the drawing operation when receiving this user input. However, the determination of the end of the drawing operation is not limited to this, and other examples for improving the user experience may also be used. For example, the user device 110 may use a timer, detect a predetermined user gesture, detect the end of a stroke of the user input, etc. to determine the end of the drawing operation. After determining the end of the drawing operation, the user device 110 may generate area data of at least one drawing area using the received drawing operation.
[0034] In some embodiments, the user device 110 may be configured to pre-process the input image and the region data to generate pre-processed data. The pre-processed data may include, for example, at least one of a vector, a graphic, and an image. In some instances, the user device 110 may be configured to extract at least one feature data from the input image based at least in part on the region data. The at least one feature data may include a set of decimal fractions extracted from the input image. In some instances, before extracting at least one feature data, the user device 110 may be configured to improve the region data to generate improved region data and / or eliminate the background of the improved region data. In some instances, after extracting at least one feature data, the user device 110 may be configured to transform at least one feature data to generate pre-processed data suitable for image search. In some instances, the user device 110 may be configured to transmit the pre-processed data to the server 130 via the network 120 for its image search. In some instances, the user device 110 may be configured to temporarily store at least one of the input image and the region data.
[0035] In some other embodiments, the user device 110 may transmit the region data to the server 130 without preprocessing. The server 130 may be configured to receive the region data and obtain the input image. In some instances where the input image is located in the user device 110, the user device 110 may transmit the input image and the region data to the server 130. In some other instances where the input image has been retrieved via the network 120, the server 130 may also retrieve the input image via the network 120. Then, the server 130 may preprocess the input image and the region data to generate preprocessed data.
[0036] In some embodiments, server 130 may be configured to search an image database based on pre-processed data obtained from the input image and the region data. The image database may be included in server 130 and / or in another computing device communicatively connected to server 130. Server 130 may then obtain one or more search results and transmit the one or more search results to user device 110. User device 110 may be configured to display the received one or more search results on a display screen associated with user device 110. In some instances, server 130 may be configured to store at least one of the input image, the region data, and the one or more search results.
[0037] In some other embodiments, the user device 110 may be configured to perform an image search. In these cases, the user device 110 may be equipped with its own image database or connected to another computing device that includes an image database, such as a server 130. In addition, the above-mentioned pre-processing may be performed in the user device 110 or in the server 130 depending on the desired configuration.
[0038] As compared to Figure 1 As described, a user can instinctively specify a search target by drawing the search target in a displayed input image, thereby allowing the user to experience a fast and convenient image search. In addition, the user is able to draw two or more separate objects for image search, and thus the user device or server can distinguish between a search target and an input image that includes multiple objects (e.g., a pair of shoes), and can also distinguish between a search target and an input image that overlaps another object (e.g., a cardigan over a T-shirt). At the same time, by drawing the search target, the separation between the foreground and the background for image search can be clearly achieved, and thus the result of the image search may be less affected by the background.
[0039] Figure 2 A block diagram of an example user device arranged in accordance with at least some embodiments described herein is shown. As illustrated, the user device 200 may include a display module 210, a rendering module 220, and a pre-processing module 230 that are operatively coupled to or otherwise in communication with each other. Additionally or optionally, the user device 200 may further include a search module 240. In some embodiments, the user device 200 may be any suitable type of computing device configured to run, execute, or operate an application program including instructions in accordance with the present disclosure. Figure 2 The configuration of the user device 200 described in Figure 1 Examples of user devices 112, 114, ... or 116 illustrated in FIG.
[0040] In some embodiments, user device 200 may obtain an input image including at least one object that is a target of an image search. In some examples, user device 200 may capture the input image using a camera (not shown) of user device 200; read the input image from a local memory (not shown) of user device 200; and / or retrieve the input image via a network (e.g., the Internet). Display module 210 may be configured to present the input image on at least one display screen operatively connected to user device 200.
[0041] The drawing module 220 may be configured to receive an operation to draw at least one object in the displayed input image. In some instances, the user may use an input member that can be operatively connected to the user device 200, including, for example, but not limited to, a touch screen, a button, a keyboard, a mouse, a touchpad, a stylus, etc., to perform the drawing operation. During the reception of the drawing operation, the drawing module 220 may determine the end of the drawing operation based on the user input. In some instances, the drawing module 220 may determine the end of the drawing operation when the user presses a button such as a "search" button or a "stop" button displayed on the display screen. In some other instances, the drawing module 220 may determine the end of the drawing operation when a predetermined time period has passed, when a predetermined user gesture is detected, when the end of a stroke of the user input is detected, etc.
[0042] The drawing module 220 may be configured to generate region data of at least one drawing region based on the drawing operation. In some instances, the region data may be, for example, a user drawing on a blank image having the same size as the input image. In some other instances, the region data may be represented by a pixel vector.
[0043] The preprocessing module 230 may be configured to preprocess the input image and the region data to generate preprocessed data. The preprocessed data may include, for example, at least one of a vector, a graph, and an image. The preprocessing module 230 may include at least one of a region improvement module 232, a background removal module 234, a feature extraction module 236, and a feature transformation module 238.
[0044] The region improvement module 232 may be configured to improve the region data to generate improved region data. In some examples, the region improvement module 232 may be configured to segment the input image into a plurality of segments. For example, the segmentation may be performed using at least one of the color, brightness, chroma, pattern and pixel values of the input image and / or using any other suitable algorithm (e.g., a graph cut algorithm). The region improvement module 232 may be configured to select one or more of the plurality of segments based on the region data to generate improved region data. For example, if the proportion of the segment drawn by the drawing operation exceeds a predetermined threshold, the region improvement module 232 may select the segment. In a similar manner, the region improvement module 232 may select one or more segments among the plurality of segments. Then, the region improvement module 232 may determine a portion of the input image corresponding to the selected one or more segments as improved region data.
[0045] The background removal module 234 may be configured to remove the background of the improved regional data. In some examples, the background removal module 234 may be configured to select a minimum area (e.g., a rectangle covering all drawn areas) and crop the image area corresponding to the selected minimum area. The size of the minimum area may be smaller than the image size of the input image and may be set to include possible margins therein to ensure that the minimum area covers the entire drawn area for image processing (e.g., feature extraction), which will be described below. Next, the background removal module 234 may be configured to mask pixels in at least one undrawn area of the cropped image area. In some examples, the background removal module 234 may mask pixels using random values.
[0046] The feature extraction module 236 may be configured to extract at least one feature data from the input image based at least in part on the region data. In the case where region improvement and background removal are performed as described above, the feature extraction module 236 may extract at least one feature data from the input image using the improved region data after region improvement and background removal. In some instances, the feature extraction module 236 may extract at least one feature data using a convolutional neural network (CNN). In addition, the at least one feature data may include a set of decimal fractions extracted from the input image. In some instances, the feature extraction module 236 may include multiple feature extraction modules. As an example and not limitation, each of the multiple feature extraction modules may extract feature data using its own feature extractor, and all extracted feature data may be cascaded in the feature extraction module 236. Alternatively or in addition, the feature extraction module 236 may extract at least one feature data hierarchically. As an example and not limitation, the feature extraction module 236 may include a first feature extractor module and a second feature extractor module, and the first feature extractor module may extract a raw feature vector and the second feature extractor module may extract a more complex vector using the raw feature vector.
[0047] The feature transformation module 238 may be configured to transform at least one feature data extracted by the feature extraction module 236 to produce preprocessed data suitable for image search. In some cases, the extracted feature data may be neither sufficient nor practical for image search, and therefore the feature transformation module 238 may need to use any suitable technique, such as encoding and pooling procedures, to transform the at least one feature data. In some instances, as an encoding procedure, the feature transformation module 238 may project the at least one feature data into a high-dimensional space to obtain at least one encoded feature data. An example of an encoding procedure is the Bug of Visual Words (BoVW) described in "Visual categorization with bags of keypoints by CSURKA, Gabriella et al. in the Workshop on statistical learning in computer vision, pp. 1-2, 2004 ECCV". In addition, the feature transformation module 238 may perform a pooling procedure on the at least one encoded feature data to obtain at least one global feature data. For example, the pooling procedure may include mean pooling that calculates the mean of the feature vectors and / or maximum pooling that selects the maximum value of each element of the vector. Additionally or optionally, when applying the pooling procedure, the feature transformation module 238 may add the position information of each feature vector using a procedure such as a spatial pyramid as described in "LAZEBNIK Svetlana, SCHMID Cordelia, PONCEJean Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories in 2006 IEEE Computer Society Conference on Computer Vision Processing and Pattern Recognition (CVPR'06), pp. 2169-2178.
[0048] As described above, the input image and region data may be pre-processed by the region improvement module 232, the background removal module 234, the feature extraction module 236, and the feature transformation module 238 to generate pre-processed data. In some embodiments, the pre-processing module 230 may not include one or more of the region improvement module 232, the background removal module 234, the feature extraction module 236, and the feature transformation module 238. For example, the pre-processing module 230 may only include the feature extraction module 236 to generate pre-processed data. In some other embodiments, the user device 200 may not include the pre-processing module 230. In such cases, the user device 200 may transmit the region data generated by the drawing module 220 to another computing device (e.g., Figure 1 The server 130 in the process is used to pre-process the server.
[0049] The generated pre-processed data may be used as a search query. In some embodiments, after generating the pre-processed data, the user device 200 may transmit the pre-processed data to a server (e.g., Figure 1 The server 130 in the user device 200 may use the pre-processed data as a search query to perform an image search to generate one or more search results, which are provided to the user device 200. In some other embodiments, the search module 240 in the user device 200 may perform an image search to generate one or more search results. The user device 200 may store at least one of the input image, the region data, and the one or more search results. The display module 210 may present the one or more search results to the user on a display screen operatively connected to the user device 200.
[0050] Figure 3 A block diagram of an example server arranged in accordance with at least some embodiments described herein is shown. As illustrated, server 300 may include a processor 310, a memory 320, and a network module 330 operatively coupled to or otherwise in communication with each other. Additionally, server 300 may include an image database 340. Server 300 may be implemented as a single computing device, a server cluster including multiple servers, a distributed network, a cloud computing configuration, etc.
[0051] Depending on the desired configuration, processor 310 may include one or more processors and memory 320 may include one or more memories. Processor 310 may be configured to execute instructions stored in memory 320 to perform operations according to the present disclosure, and memory 320 may be configured to store instructions for image search.
[0052] , the network module 330 may be configured to communicate via a network (eg, Figure 1 120) and a user device (e.g., Figure 1 The user device 110 or Figure 2 In some embodiments, the user device may display an input image including at least one object that is a search target; receive a user operation of drawing at least one object in the displayed input image; and generate area data of at least one drawing area based on the user operation. The network module 330 may be configured to receive area data from the user device. In addition, the network module 330 may be configured to obtain the input image. In some instances where the input image is located in the user device, the network module 330 may receive the input image from the user device. In some other instances where the user device retrieves the input image from the network, the network module 330 may also retrieve the input image from the network.
[0053] Processor 310 may be configured to pre-process the input image and region data to generate pre-processed data. In some examples, processor 310 may be configured to perform at least one of region improvement, background removal, feature extraction, and feature transformation. These one or more pre-processings performed by server 300 (i.e., processor 310) are similar to those described with respect to Figure 2 The preprocessing is described as being performed in the user device 200 (ie, the preprocessing module 230), and thus a detailed description thereof is omitted. The preprocessed data may include, for example, at least one of a vector, a graphic, and an image.
[0054] In some other embodiments, the network module 330 may be configured to receive pre-processed data from the user device. In such cases, the processor 310 does not perform one or more pre-processings, and the memory 320 may be configured to store the pre-processed data.
[0055] The processor 310 may be configured to search using the preprocessed data as a search query. In some embodiments, the server 300 may include an image database 340. The image database 340 may store a plurality of images or one or more feature data extracted from the plurality of images, one or more of which contain an object image that is the same or similar to the search target or one or more feature data extracted from the object image. Any conventional or available search algorithm may be used for image search, and therefore a detailed description thereof is omitted in the present disclosure. Based on the preprocessed data, the processor 310 may generate one or more search results selected from a plurality of images stored in the image database 340. In some instances, the memory 320 may be configured to store at least one of the input image, the regional data, and the one or more search results. The network module 330 may transmit one or more search results to a user device so that the user device may display the results on its display screen.
[0056] Figure 4An example flow diagram illustrating an image search process arranged in accordance with at least some embodiments described herein. Figure 4 The process 400 may use one or more computing devices (e.g., reference Figure 1 and Figure 2 The user device 110 or 200 described and / or referenced Figure 1 and Figure 3 In addition, in the following, reference is made to the server 130 or 300 shown in the illustrative example. FIG. 5A to FIG. 5C as well as Figure 6 Process 400 is described in further detail.
[0057] Process 400 may include one or more operations, actions or functions as illustrated by one or more of blocks S410, S420, S430 and / or S440. Although illustrated as discrete blocks, various blocks may be divided into additional blocks, combined into fewer blocks, supplemented with other blocks or eliminated, depending on the specific implementation. In some other examples, various of the blocks may be implemented as parallel processes rather than sequential processes, or may be implemented as a combination of parallel processes and sequential processes. Process 400 may begin at block S410, i.e., "display an input image containing at least one object."
[0058] At block S410, one or more computing devices may display an input image containing at least one object. The at least one object may be a search target. In some examples, the one or more computing devices may obtain the input image via various methods as described above. Then, the one or more computing devices may display the input image on a display screen associated with the one or more computing devices. Figure 5A An example input image is shown. Figure 5A In the example, the input image 510 may be displayed on a computer (e.g., Figure 1 or Figure 2 The input image 510 may include a jacket 512 that is a search target. Block S410 may be followed by block S420, i.e., “receiving an operation of drawing at least one object in the displayed input image to generate region data of at least one drawing region”.
[0059] At block S420, one or more computing devices may receive an operation from a user to draw at least one object in the displayed input image. The user may draw at least one object using an input member, and the one or more computing devices may receive the drawing operation. Figure 5B and Figure 5C An illustrative example of drawing at least one object for image search is shown. Figure 5B and Figure 5CAs depicted in , image 520 or 530 may be displayed on a display screen and may include jacket 522 or 532 that is a search target. Figure 5B In the embodiment of the present invention, the user may use any suitable input means to draw substantially the entire area 524 of the jacket 522. More typically, the user may draw a partial area 534 of the jacket 532, such as Figure 5C However, it should be understood by those skilled in the art that it is possible to specify the search target in a manner other than the described drawing object without departing from the spirit of the present disclosure.
[0060] Return to reference Figure 4 In block S420, upon receiving the drawing operation, one or more computing devices may determine the end of the drawing operation based on a predetermined condition. Figure 6 Non-limiting examples for determination are provided. Figure 6 An illustrative example of a user device as an example of one or more computing devices is shown, wherein the user device is configured to receive a drawing operation from a user. The user device 610 may include a display screen 612 having a touch screen. The user device 610 may display an input image including a jacket 620 that is a search target. The user may draw at least one area 630 on the jacket 620 displayed in the display screen 612. The display screen 612 may provide a "clear" button 640 to clear the user operation (e.g., drawing area 630), and may provide a "search" button 650 to stop the drawing operation. When the user presses the "search" button 650, the user device 610 may determine the end of the drawing operation. After determining the end of the drawing operation, the one or more computing devices may generate area data of at least one area drawn by the user. Box S420 may be followed by box S430, namely "pre-processing the input image and area data to generate pre-processed data".
[0061] At block S430, one or more computing devices may preprocess the input image and the region data to generate preprocessed data. The preprocessed data may include, for example, at least one of a vector, a graph, and an image. In some examples, the one or more computing devices may extract at least one feature data from the input image and the region data. In addition, the one or more computing devices may improve the region data to generate improved region data, eliminate background of the improved region data, and / or transform at least one feature data to generate preprocessed data. Block S430 may be followed by block S440, i.e., "Searching an image database based on the preprocessed data."
[0062] At box S440, one or more computing devices may use the preprocessed data as a search query to perform an image search on an image database. The image database may store multiple images or one or more feature data extracted from the multiple images, one or more of which contain an object image that is the same or similar to the search target or one or more feature data extracted from the object image. Any conventional or available search algorithm can be used for image search, and therefore its detailed description is omitted in this disclosure. One or more computing devices may obtain one or more search results through image search. In some instances, one or more computing devices may store at least one of the input image, the generated area data, and one or more search results. The obtained search results may be provided to the user on a display screen associated with one or more computing devices.
[0063] Figure 7 An example flow chart illustrating a process for pre-processing input images and region data, arranged in accordance with at least some embodiments described herein. Figure 7 The process 700 may use one or more computing devices (e.g., reference Figure 1 and Figure 2 The user device 110 or 200 described and / or referenced Figure 1 and Figure 3 Process 700 may be implemented as follows: Figure 4 An example of block S430 illustrated in FIG.
[0064] Process 700 may include one or more operations, actions, or functions as illustrated by one or more of blocks S710, S720, S730, and / or S740. Although illustrated as discrete blocks, various blocks may be divided into additional blocks, combined into fewer blocks, supplemented with other blocks, or eliminated, depending on the particular implementation. In some other examples, various of the blocks may be implemented as parallel processes rather than sequential processes, or may be implemented as a combination of parallel processes and sequential processes. Process 700 may begin at block S710, i.e., "Improve regional data to generate improved regional data."
[0065] At block S710, one or more computing devices may refine region data generated by a user operation of drawing at least one object in an input image displayed on the one or more computing devices. Figure 8 Describes a detailed example of improvements in interpreting regional data. Figure 8 An example flow chart illustrating improving region data arranged in accordance with at least some embodiments described herein. Figure 8 The process 800 may use one or more computing devices (e.g., reference Figure 1 and Figure 2The user device 110 or 200 described and / or referenced Figure 1 and Figure 3 The embodiment is implemented by the server 130 or 300 described above. Figure 8 Process 800 may begin at block S810, ie, "Segment an input image into a plurality of segments."
[0066] At block S810, one or more computing devices may segment the input image into a plurality of segments. For example, the segmentation may be performed using at least one of the color, brightness, chrominance, pattern, and pixel values of the input image and / or using any other suitable algorithm (e.g., a graph cut algorithm). As a non-limiting example, refer to 9A to 9C The illustrative examples describe the segmentation in more detail. Fig. 9A An illustrative example of a user drawing at least one area for image search is shown. Fig. 9A As depicted in , an image 910 displayed on a display screen may include a jacket 920 that is a search target. A user who wishes to search for one or more images corresponding to the jacket 920 may draw the jacket 920 on the displayed image 910 using any suitable input means, and one or more computing devices may generate at least one drawing area 930. For example, Fig. 9A As shown in , jacket 920 has stripes that form a plurality of segments in image 910 , and a user can draw a line passing through each of the plurality of segments. Fig. 9B An illustrative example of segmenting an input image is shown. Fig. 9B As depicted in FIG. 8 , the input image may be segmented into a plurality of segments 940, each of which is a small area shown in a different pattern. Thus, the input image 910 may be divided into a plurality of segments 940 for improving the regional data. Block S810 may be followed by block S820, i.e., “Selecting one or more of the plurality of segments based on the regional data to generate improved data”.
[0067] At block S820, one or more computing devices may select one or more of the plurality of segments based on the regional data to generate improved regional data. As an example and not limitation, if the proportion of a segment drawn by the user exceeds a predetermined threshold, then the segment may be selected by the one or more computing devices. The one or more computing devices may aggregate the selected segments to generate improved regional data. Fig. 9C The illustrative examples of describe the improved data in more detail. Fig. 9C An illustrative example of selecting one or more segments to generate improved regional data is shown. Fig. 9A At least one drawing area 930 and Fig. 9B Of the multiple segments 940 , one or more computing devices may select one or more segments 955 and generate improved regional data 950 .
[0068] Although the above has been described with reference to blocks S810 and S820 Figure 8 However, the embodiments described herein are not limited thereto, and one or more blocks may be omitted, modified, or switched without departing from the spirit of the present disclosure.
[0069] Block S710 may be followed by block S720, “Eliminate the background of the improved region data”. At block S720, one or more computing devices may eliminate the background of the improved region data. Fig.10 A detailed example is described to explain the background removal process. Fig.10 An example flow chart illustrating a process to remove background from region data, arranged in accordance with at least some embodiments described herein. Fig.10 The process 1000 may use one or more computing devices (e.g., reference Figure 1 and Figure 2 The user device 110 or 200 described and / or referenced Figure 1 and Figure 3 The embodiment is implemented by the server 130 or 300 described above. Fig.10 The process 1000 may start at block S1010, ie, “select a rectangle that includes all of at least one drawing area”.
[0070] At block S1010, one or more computing devices may select a rectangle that encompasses all drawing areas. The rectangle may be the smallest rectangle that covers all drawing areas. Depending on the desired implementation, another shape other than a rectangle may be used to select the drawing area. As a non-limiting example, refer to Fig.11A The illustrative example of describes the selection of the smallest rectangle in more detail. Fig.11A As depicted in FIG. 1 , improved region data 1110 may include at least one drawing region 1120 (e.g., one or more segments selected by the improvement process). Rectangle 1130 may be selected to include all of at least one drawing region 1120. Block S1010 may be followed by block S1020, “Crop the image region corresponding to the rectangle.”
[0071] At box S1020, one or more computing devices may crop an image area corresponding to a rectangle, specifically corresponding to a minimum rectangle. In some instances, one or more computing devices may crop an image corresponding to the improved region data having a minimum rectangle to generate a cropped image area. The cropped image area corresponding to the minimum rectangle has a smaller size than the improved region data and can be used for the next process. Box S1020 may be followed by box S1030, i.e., "masking pixels in at least one undrawn area of the cropped image area." At box S1030, one or more computing devices may mask pixels in at least one undrawn area of the cropped image area. In some instances, one or more computing devices may mask pixels using random values. As a non-limiting example, refer to Fig. 11B The illustrative examples of describe clipping and masking in more detail. Fig. 11B , one or more computing devices may crop an image for improved region data to generate a cropped image region 1140. The cropped image region 1140 may include at least one drawn region 1120 (e.g., one or more segments selected by the improvement process) and a region 1142 other than the at least one drawn region 1120. In some examples, the one or more computing devices may mask the region 1142 (i.e., an unpainted region) using a random value. In some other examples, the region 1142 may be masked using a predetermined value rather than a random value.
[0072] Although the above has been described with reference to blocks S1010, S1020, and S1030 Fig.10 The process 1000 of , but the embodiments described herein are not limited thereto, and one or more blocks may be omitted, modified, or switched without departing from the spirit of the present disclosure.
[0073] Return to reference Figure 7 , box S720 may be followed by box S730, namely, "Extracting at least one feature data from the input image based at least in part on the improved region data". At box S730, one or more computing devices may extract at least one feature data from the input image using the region data after preprocessing in boxes S710 and S720. In some instances, the one or more computing devices may extract at least one feature data using a convolutional neural network (CNN). In addition, the one or more computing devices may extract at least one feature data hierarchically. That is, the one or more computing devices may extract a raw feature vector and then use the raw feature vector to extract a more complex feature vector. In some instances, the at least one feature data may include a set of decimal fractions extracted from the input image. Box S730 may be followed by box S740, namely, "Transforming at least one feature data to produce preprocessed data".
[0074] At S740, one or more computing devices may transform the at least one feature data extracted at block S730 to generate pre-processed data. Fig.12 Detailed examples of explaining feature transformations arranged in accordance with at least some embodiments described herein are described. Fig.12 An example flow chart illustrating a process for transforming at least one feature data, arranged in accordance with at least some embodiments described herein. Fig.12 The process 1200 may use one or more computing devices (e.g., reference Figure 1 and Figure 2 The user device 110 or 200 described and / or referenced Figure 1 and Figure 3 The server 130 or 300 described above can be implemented. Fig.12 As depicted in , at least one extracted feature data may be transformed using an encoding and pooling procedure. Fig.12 The process 1200 may start at block S1210, namely, “projecting at least one feature data into a high-dimensional space to obtain at least one encoded feature data”.
[0075] At block S1210, one or more computing devices may perform an encoding procedure. One or more computing devices may project at least one feature data into a high-dimensional space to obtain at least one encoded feature data. Block S1210 may be followed by block S1220, i.e., "Perform a pooling process on at least one encoded feature data to obtain at least one global feature data." As an example and not a limitation, the pooling procedure may include average pooling that calculates the average value of the feature vector and / or maximum pooling that selects the maximum value of each element of the vector.
[0076] Although the above has been described with reference to blocks S1210 and S1220 Fig.12 The process 1200 of , but the embodiments described herein are not limited thereto, and one or more blocks may be omitted, modified, or switched without departing from the spirit of the present disclosure.
[0077] Fig.13 A computer program product 1300 that can be used to perform an image search in accordance with at least some embodiments described herein is illustrated. The computer program product 1300 can include a signal bearing medium 1302. The signal bearing medium 1302 can include one or more instructions 1304 that, in response to being executed by, for example, one or more processors, can provide the above-described Figures 1 to 12As an example, instructions 1304 may include at least one of: one or more instructions for displaying an input image containing at least one object that is a search target; one or more instructions for receiving an operation of drawing at least one object in the displayed input image to generate region data of at least one drawing region; one or more instructions for preprocessing the input image and the region data to generate preprocessed data; and one or more instructions for displaying one or more search results by performing an image search based at least in part on the preprocessed data. Thus, for example, with reference to Figures 1 to 3 , the user device 110 or 200 and / or the server 130 or 300 may respond to the instruction 1304 to perform Figure 4 , Figure 7 , Figure 8 , Fig.10 and Fig.12 One or more of the boxes shown in .
[0078] In some embodiments, the signal bearing medium 1302 may include a non-transitory computer-readable medium 1306, such as, but not limited to, a hard drive, a compact disc (CD), a digital video disc (DVD), a digital tape, a memory, etc. In some embodiments, the signal bearing medium 1302 may include a recordable medium 1308, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, etc. In some embodiments, the signal bearing medium 1302 may include a communication medium 1310, such as, but not limited to, a digital and / or analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communication link, a wireless communication link, etc.). Thus, for example, the program product 1300 may be transmitted to one or more modules of a user device, such as one or more modules of the user device 110 or 200, via the RF signal bearing medium 1302, wherein the signal bearing medium 1302 is transmitted via a wireless communication medium 1310 (e.g., a wireless communication medium that complies with the IEEE 802.11 standard).
[0079] Fig.14 A simplified block diagram of an example computing device that can be configured for image search is shown. The computing functions described herein (e.g., functions of receiving drawing operations and pre-processing region data based on the drawing operations) can be performed by a computing system. This computing system can be integrated into or in the form of a computing device, such as a mobile phone, tablet computer, laptop computer, server, cloud computing network, and / or programmable logic controller. For example, Fig.14 14 is a simplified block diagram illustrating some components of the example computing device 1400. Additionally, the example computing device 1400 may include a camera component 1424. The camera component 1424 may include a plurality of cameras, such as visible light cameras or other types of cameras.
[0080] By way of example and not limitation, computing device 1400 may be a cellular mobile telephone (e.g., a smart phone), a computer (e.g., a desktop, notebook, tablet, or handheld computer), a personal digital assistant (PDA), a home automation component, a digital television, a remote control, a wearable computing device, a robot, a drone, an autonomous vehicle, or some other type of device that may be equipped with an image display and / or user input receiving capabilities.
[0081] like Fig.14 As shown in , computing device 1400 may include a communication interface 1402 , a user interface 1404 , a processor 1406 , a data storage device 1408 , and a camera component 1424 , all of which may be linked together in a communicative manner via a system bus, network, or other connection mechanism 1410 .
[0082] The communication interface 1402 may be used to allow the computing device 1400 to communicate with other devices, access networks, and / or transport networks using analog or digital modulation of electrical, magnetic, electromagnetic, optical, or other signals. Thus, the communication interface 1402 may facilitate circuit switching and / or packet switching communications, such as plain old telephone service (POTS) communications and / or Internet Protocol (IP) or other packetized communications. For example, the communication interface 1402 may include a chipset and an antenna arranged for wireless communication with a wireless access network or access point. Furthermore, the communication interface 1402 may be in the form of or include a cable interface, such as an Ethernet, Universal Serial Bus (USB), or High Definition Multimedia Interface (HDMI) port. The communication interface 1402 may also be in the form of or include a wireless interface, such as WiFi, The communication interface 1402 may include a global positioning system (GPS) or a wide area wireless interface (e.g., WiMAX or 3GPP Long Term Evolution (LTE)). However, other forms of physical layer interfaces and other types of standard or dedicated communication protocols may be used on the communication interface 1402. In addition, the communication interface 1402 may include multiple physical communication interfaces (e.g., WiFi interface, interface and wide area wireless interface).
[0083] In some embodiments, the communication interface 1402 may be used to allow the computing device 1400 to communicate with other devices, remote servers, access networks, and / or transmission networks. For example, the communication interface 1402 may be used to access an input image and / or access an image database via communication with a remote server or other remote device or system to allow the computing device 1400 to display an input image, receive a drawing operation on a displayed input image to generate region data, pre-process the region data to generate pre-processed data, and / or perform a search of an image database. In addition, the computing device 1400 may include a camera or other image capture device configured to capture an image as an input image.
[0084] The user interface 1404 may be used to allow the computing device 1400 to interact with a user, such as receiving input from the user and / or providing output to the user. Thus, the user interface 1404 may include input components such as a keypad, a keyboard, a touch-sensitive or presence-sensitive panel, a computer mouse, a trackball, a joystick, etc. The user interface 1404 may also include one or more output components such as a display screen that may be combined with a presence-sensitive panel, for example. The display screen may be based on CRT, LCD, and / or LED technology or other currently known or later developed technology.
[0085] In some embodiments, the user interface 1404 may include a display screen that displays an input image and a user-drawn area on the input image. In addition, the user interface 1404 may include one or more buttons, switches, knobs, and / or dials that facilitate configuration of the display. It is possible that some or all of these buttons, switches, knobs, and / or dials are implemented as functions on a touch-sensitive or presence-sensitive panel. The user interface 1404 may be configured to permit a user to input an operation to draw at least one object in the displayed input image.
[0086] Processor 1406 may include one or more general purpose processors, such as microprocessors, and / or one or more special purpose processors, such as digital signal processors (DSPs), graphics processing units (GPUs), floating point units (FPUs), network processors, or application specific integrated circuits (ASICs). In some cases, the special purpose processors are capable of image processing, image alignment, and image merging, among other applications or functions. Data storage device 1408 may include one or more volatile and / or non-volatile storage components, such as magnetic, optical, flash, or organic storage devices, and may be integrated with processor 1406 in whole or in part. Data storage device 1408 may include removable and / or non-removable components.
[0087] The processor 1406 is capable of executing program instructions 1418 (e.g., compiled or uncompiled program logic and / or machine code) stored in the data storage device 1408 to perform various functions described herein. Thus, the data storage device 1408 may include a non-transitory computer-readable medium having program instructions stored thereon, which when executed by the computing device 1400 causes the computing device 1400 to perform any of the methods, processes, and functions disclosed in this specification and / or the accompanying drawings. Execution of the program instructions 1418 by the processor 1406 may cause the processor 1406 to use the data 1412.
[0088] As an example, program instructions 1418 may include an operating system 1422 (e.g., an operating system kernel, device drivers, and / or other modules) and one or more applications 1420 (e.g., a camera function, a contact list, email, web browsing, an e-commerce function, a social network, and / or a game application) installed on computing device 1400. Similarly, data 1412 may include operating system data 1416 and application data 1414. Operating system data 1416 may be primarily accessed by operating system 1422, and application data 1414 may be primarily accessed by one or more of application programs 1420. Application data 1414 may be arranged in a file system that is visible or hidden to a user of computing device 1400.
[0089] Applications 1420 may communicate with operating system 1422 via one or more application programming interfaces (APIs). These APIs may facilitate, for example, application 1420 reading and / or writing application data 1414, transmitting or receiving information via communication interface 1402, receiving and / or displaying information on user interface 1404, and the like.
[0090] The display of the input image, the reception of the operation of drawing at least one object in the displayed input image to generate region data, the pre-processing of the input image and the region data, and / or the search of the image database may be performed by one or more components of the processor 1406 (e.g., performed by one or more digital signal processors (DSPs), graphics processing units (GPUs), central processing units (CPUs), floating point units (FPUs), application specific integrated circuits (ASICs), some other type of computing components, and / or some combination of one or more types of computing components of the processor 1406). For example, some operations related to the reception of the operation may be performed by a first computing element or a first type of computing element (e.g., DSP, GPU) of the processor 1406 and some other operations related to the pre-processing may be performed by a second computing element or a second type of computing element (e.g., general-purpose CPU) of the processor 1406.
[0091] Applications 1420 may take the form of “apps” that may be downloaded to computing device 1400 via one or more online application stores or application marketplaces (e.g., via communication interface 1402). However, applications may also be installed on computing device 1400 in other ways, such as via a web browser or via a physical interface of computing device 1400 (e.g., a USB port).
[0092] The present disclosure is not limited to the specific embodiments described in this application, which are intended to be illustrations of various aspects. Without departing from the spirit and scope of the present disclosure, many modifications and changes may be made. In addition to the methods and devices listed herein, functionally equivalent methods and devices within the scope of the present disclosure are also possible according to the foregoing description. Such modifications and changes are intended to fall within the scope of the appended claims. The present disclosure is limited only by the terms of the appended claims and the full scope of equivalents authorized by the claims. The present disclosure is not limited to specific methods, reagents, compounds, compositions or biological systems that can be changed as a matter of course. The terms used herein are only for the purpose of describing specific embodiments and are not intended to be restrictive.
[0093] The subject matter described herein sometimes illustrates different components contained in or connected to different other components. Such depicted architectures are merely examples, and many other architectures that realize the same functionality can actually be implemented. In a conceptual sense, any arrangement of components that realize the same functionality is actually "associated" so that the desired functionality can be realized. Therefore, any two components that are combined to realize a specific functionality herein can be regarded as "associated" with each other so that the desired functionality can be realized, regardless of the architecture or intermediate components. Similarly, any two components so associated can also be regarded as "operationally connected" or "operationally coupled" to realize the desired functionality, and any two components that can be so associated can also be regarded as "operationally coupled" to realize the desired functionality. Specific instances that can be operably coupled include but are not limited to components that can physically cooperate and / or physically interact, and / or components that can wirelessly interact and / or wirelessly interact, and / or components that logically interact and / or can logically interact.
[0094] With respect to the use of substantially any plural and / or singular terms herein, those skilled in the art may translate from the plural to the singular and / or from the singular to the plural as appropriate depending on the context and / or application. For clarity, various singular / plural arrangements may be expressly set forth herein.
[0095] Those skilled in the art will understand that, in general, the terms used herein and particularly in the appended claims (e.g., the appended claim bodies) are generally intended to be "open" terms (e.g., the term "comprising" should be interpreted as "including but not limited to," the term "having" should be interpreted as "having at least," the term "including" should be interpreted as "including but not limited to," etc.). Those skilled in the art will further understand that if a specific number of an introduced claim recitation is intended, such intent will be expressly recited in the claim, and in the absence of such a recitation, no such intent is present. For example, to aid understanding, the appended claims may contain usage of the introductory phrases "at least one" and "one or more" to introduce claim recitations. However, the use of such phrases should not be interpreted as implying that a claim recitation introduced by the indefinite article "a" limits any particular claim containing such introductory claim recitation to embodiments containing only one such recitation, even when the same claim contains the introductory phrases "one or more" or "at least one" and an indefinite article such as "a" (e.g., "a" should be interpreted as meaning "at least one" or "one or more"); the use of definite articles used to introduce claim recitations also follows this principle. In addition, even if a specific number of introduced claim recitations is explicitly recited, one skilled in the art will recognize that such recitation should be interpreted as meaning at least the recited number (e.g., the explicit recitation "two recitations" without other modifiers means at least two recitations or two or more recitations). Furthermore, in those cases where a formula similar to "at least one of A, B, and C, etc." is used, generally speaking, such constructions are contemplated in the sense of the formula as would be understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" would include, but not be limited to, systems having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). In those cases where a formula similar to "at least one of A, B, and C, etc." is used, generally speaking, such constructions are contemplated in the sense of the formula as would be understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" would include, but not be limited to, systems having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). It would be further understood by a person skilled in the art that virtually any transitional word and / or phrase indicating two or more alternative terms, whether in the description, claims, or drawings, would be understood to encompass the possibility of including one of the terms, either of the terms, or both of the terms. For example, the phrase "A or B" will be understood to include the possibilities of "A" or "B" or "A and B."
[0096] In addition, where features or aspects of the disclosure are described in terms of Markush groups, those skilled in the art will recognize that the disclosure is also thereby described in terms of any individual member or subgroup of members of the Markush group.
[0097] Those skilled in the art will understand that, for any and all purposes, such as for providing a written description, all ranges disclosed herein also encompass any and all possible subranges and combinations of subranges thereof. Any enumerated range can be easily identified by sufficient description and can be decomposed into at least equal two, three, four, five, ten, etc., of the same range. As a non-limiting example, each range discussed herein can be easily decomposed into a lower third, a middle third, and an upper third, etc. Those skilled in the art will also understand that all language such as "at most," "at least," etc. includes the enumerated number and refers to a range that can be subsequently decomposed into subranges as discussed above. Finally, those skilled in the art will understand that a range includes each individual member.
[0098] According to the foregoing, various embodiments of the present disclosure have been described herein for illustrative purposes, and various modifications may be made without departing from the scope and spirit of the present disclosure. Therefore, the various embodiments disclosed herein are not intended to be limiting, and the true scope and spirit are indicated by the appended claims.
Claims
1. A method for image search based on drawing, comprising: displaying, using one or more computing devices, an input image containing at least one object; receiving, using the one or more computing devices, an operation of drawing the at least one object in the displayed input image to generate region data of at least one drawing region; preprocessing the input image and the region data using the one or more computing devices to produce preprocessed data; as well as performing a search of an image database based at least in part on the pre-processed data using the one or more computing devices, The pre-processing comprises: improving the region data using the one or more computing devices to produce improved region data; removing background from the improved region data using the one or more computing devices; extracting at least one feature data from the input image based at least in part on the improved region data using the one or more computing devices; as well as The at least one feature data is transformed using the one or more computing devices to produce the pre-processed data. 2 . The method of claim 1 , wherein the preprocessing comprises extracting at least one feature data from the input image based at least in part on the region data.
3. The method of claim 1, wherein the improving comprises: segmenting the input image into a plurality of segments using the one or more computing devices; and One or more of the plurality of segments are selected based on the regional data using the one or more computing devices to generate the improved regional data.
4. The method of claim 1, wherein said eliminating comprises: Selecting a minimum rectangle that contains all of the at least one drawing area; cropping the image area corresponding to the minimum rectangle; and Mask pixels in at least one unpainted region of the cropped image area.
5. The method according to claim 1, further comprising: At least one of the input image, the region data, and one or more search results is stored using the one or more computing devices. The method of claim 1 , wherein the receiving of the operation comprises determining an end of the operation based at least in part on user input.
7. The method of claim 1, wherein the pre-processed data comprises at least one of a vector, a graphic, and an image.
8. A computing device for performing image search based on drawing, comprising: at least one processor; and at least one memory for storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations including: obtaining an input image containing at least one object that is a search target; displaying the input image on a display screen associated with the computing device; receiving an operation of drawing the at least one object in the displayed input image to generate area data of at least one drawing area; preprocessing the input image and the region data to generate preprocessed data; as well as displaying one or more search results on the display screen from an image search based at least in part on the input image and the region data, The pre-processing comprises: improving the region data to produce improved region data; removing background from the improved area data; extracting at least one feature data from the input image based at least in part on the improved region data; as well as The at least one feature data is transformed to generate the pre-processed data.
9. The computing device of claim 8, wherein the preprocessing comprises: At least one feature data is extracted from the input image based at least in part on the region data.
10. The computing device of claim 8, wherein the operations further comprise: transmitting the pre-processed data to a second computing device; and The one or more search results obtained by the second computing device are received from the second computing device.
11. The computing device of claim 8, wherein the operations further comprise: transmitting the input image and the region image to a second computing device; and The one or more search results obtained by the second computing device are received from the second computing device.
12. The computing device of claim 8, wherein the receipt of the operation comprises determining an end of the operation based at least in part on user input.
13. A non-transitory computer-readable storage medium having stored therein instructions executable by a computing device to cause the computing device to perform operations comprising: displaying an input image containing at least one object that is a search target; receiving an operation of drawing the at least one object in the displayed input image to generate area data of at least one drawing area; preprocessing the input image and the region data to generate preprocessed data; as well as displaying one or more search results by performing an image search based at least in part on the pre-processed data, The pre-processing comprises: improving the region data to produce improved region data; removing background from the improved area data; extracting at least one feature data from the input image based at least in part on the improved region data; as well as The at least one feature data is transformed to generate the pre-processed data.
14. The non-transitory computer-readable storage medium of claim 13, wherein the receipt of the operation comprises determining an end of the operation based at least in part on user input.
15. The non-transitory computer-readable storage medium of claim 13, wherein the pre-processing comprises extracting at least one feature data from the input image based at least in part on the region data.
16. The non-transitory computer-readable storage medium of claim 13, wherein the operations further comprise: The image search is performed based at least in part on the pre-processed data to obtain the one or more search results.
