Search method, device and electronic device

By processing low-quality images to generate the corrected query images, the problems of cumbersome search steps and low efficiency caused by low-definition images in the prior art are solved, and more efficient and accurate item search is achieved.

CN112765471BActive Publication Date: 2024-11-05VIVO MOBILE COMM CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110105406.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-01-26
Publication Date
2024-11-05
Estimated Expiration
2041-01-26

AI Technical Summary

Technical Problem

In the prior art, when the image search function is passed, low definition or missing image quality leads to cumbersome search steps and low efficiency. Users need to upload high-quality images or manually enter keywords to re-search the item.

Method used

A search method and device are provided, which performs image processing through low-quality images input by a user, and generates a modified query image for improving search accuracy and efficiency and simplifying search steps. The method includes displaying an initial search image, processing the query image according to the target search image, generating a second query image, and searching with the modified image.

Benefits of technology

It simplifies the item search steps, improves search efficiency and accuracy, and makes the search results more in line with user needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112765471B_ABST
    Figure CN112765471B_ABST
Patent Text Reader

Abstract

The present application discloses a search method, device and electronic device, which belongs to the field of communication technology, and is used to solve the problem of cumbersome steps and low efficiency in the process of searching for items through image search function. The method includes: displaying M first search images according to N first query images input by the user; performing image processing on the target query image to obtain a second query image according to the target search image in the M first search images; displaying X second search images according to the second query image; wherein the N first query images all indicate the target query item, each first search image indicates a first search item, the target query image is at least one of the N first query images, each second search image indicates a second search item, and N, M and X are all positive integers. The present application is applied to the scenario of improving the efficiency of item search.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of communication technology, and specifically relates to a search method, device and electronic device. Background Art

[0002] With the development of electronic devices, shopping applications in the electronic devices have more and more functions. For example, the shopping applications have an image search function.

[0003] Currently, when a user searches for an item they want through an image search function, first, the user can upload an image containing the item they want to find through an electronic device, and then the electronic device can search for similar items based on the image and display links to the searched items.

[0004] However, if the image quality of the above-mentioned image is not high (such as low definition or missing image), it may be that the item that the user wants to find cannot be searched. In this case, the user is usually required to upload an image with high image quality or manually enter keywords to search for the item again. As a result, the process of searching for items through the image search function is cumbersome and inefficient. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a search method, device and electronic device that can solve the problems of complicated steps and low efficiency in the process of searching for objects through an image search function.

[0006] In order to solve the above technical problems, this application is implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides a search method, the method comprising: displaying M first search images based on N first query images input by a user; performing image processing on the target query image to obtain a second query image based on a target search image in the M first search images; and displaying X second search images based on the second query image; wherein the N first query images all indicate target query items, each first search image indicates a first search item, the target query image is at least one of the N first query images, each second search image indicates a second search item, and N, M and X are all positive integers.

[0008] In a second aspect, an embodiment of the present application provides a search device, comprising: a display module and an image processing module; the display module is used to display M first search images based on N first query images input by a user; the image processing module is used to perform image processing on the target query image to obtain a second query image based on the target search image in the M first search images displayed by the display module; the display module is used to display X second search images based on the second query image processed by the image processing module; wherein the N first query images all indicate target query items, each first search image indicates a first search item, the target query image is at least one of the N first query images, each second search image indicates a second search item, and N, M and X are all positive integers.

[0009] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0010] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0011] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0012] In an embodiment of the present application, first, the search device can display M first search images according to the N first query images input by the user. Then, the search device can perform image processing on the target query image according to the target search image in the M first search images to obtain a second query image. Finally, the search device can display X second search images according to the second query image. Among them, the N first query images all indicate the target query item, each first search image indicates a first search item, the target query image is at least one of the N first query images, and each second search image indicates a second search item. Through the above scheme, when the user wants to search for an item through the image search function of the application, compared with the scheme in the related art that requires the user to input an image with high image quality or manually input keywords to re-search the item, the search device in the present application can, when the image quality of the first query image input by the user is not high, perform image processing on the target query image through the target search image that has been searched to obtain the second query image, that is, to correct the target query image. Then, the search device can use the corrected second query image to search and obtain X second search images. In this way, not only can the steps of item search be simplified and efficiency improved, but also the accuracy of item search can be improved, making the searched items more in line with user needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A flowchart of a search method provided in an embodiment of the present application;

[0014] Figure 2 One of the interface schematic diagrams of a search method application provided in an embodiment of the present application;

[0015] Figure 3 A second schematic diagram of an interface of a search method application provided in an embodiment of the present application;

[0016] Figure 4 One of the schematic diagrams of a search method provided in an embodiment of the present application;

[0017] Figure 5 A second schematic diagram of a search method provided in an embodiment of the present application;

[0018] Figure 6 A third schematic diagram of an interface of a search method application provided in an embodiment of the present application;

[0019] Figure 7 A schematic diagram of the structure of a search device provided in an embodiment of the present application;

[0020] Figure 8 One of the structural schematic diagrams of an electronic device provided in an embodiment of the present application;

[0021] Fig. 9 The second structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here. And the objects distinguished by "first", "second", etc. are usually a class, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally represents that the objects associated before and after are in an "or" relationship.

[0024] The search method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0025] Figure 1 A schematic diagram of a search method flow provided in an embodiment of the present application includes steps 201 to 203:

[0026] Step 201: The search device displays M first search images according to N first query images input by a user.

[0027] The N first query images all indicate target query items, each first search image indicates a first search item, and N and M are positive integers.

[0028] In the embodiment of the present application, the N first query images mentioned above can be images collected by the search device, can be images stored in the search device system, can be images downloaded by the search device, etc., and the embodiment of the present application does not limit this. In the embodiment of the present application, each first query image can include an image of all or part of the target query object.

[0029] In the embodiments of the present application, the articles in the present application may be any articles. For example, the articles in the present application may be daily necessities (e.g., towels, quilts, etc.), electronic products (e.g., USB flash drives, cameras, etc.), or milk and alcoholic beverages (e.g., red wine, milk, etc.), which are not limited in the embodiments of the present invention.

[0030] In the embodiment of the present invention, the target query item may be one or more, which is not limited in the embodiment of the present invention.

[0031] It can be understood that the above-mentioned M first search images indicate M first search items.

[0032] In an example, the above step 201 may specifically include the following steps: the search device displays a first search interface according to the N first query images input by the user, wherein the first search interface includes the M first search images.

[0033] In addition, the first search interface may include, in addition to the M first search images, M first search links, wherein one first search link corresponds to one first search image.

[0034] Optionally, before the above step 201, the method may further include the following step A1:

[0035] Step A1: The search device receives the N first query images input by a user.

[0036] In one example, the user may input the N first query images through an image search control, wherein the image search control may be a newly added control or an existing control, which is not limited in the embodiment of the present application.

[0037] Exemplarily, the above-mentioned image search control may be a control in a target application. Exemplarily, the target application may be a shopping application or a search application, etc., which is not limited in the embodiments of the present application.

[0038] Optionally, in an embodiment of the present application, when the search device searches for multiple first search images based on N first query images, the search device may display the multiple first search images in a certain order, so that the user can view them conveniently.

[0039] Exemplarily, the displaying of M first search images in the above step 201 may specifically include the following step 201a:

[0040] Step 201a: The search device displays M first search images in a preset order.

[0041] Among them, the above-mentioned preset order can be a time order or an order of similar programs between images, etc., which is not limited in the embodiment of the present application.

[0042] In one example, when the above-mentioned preset order is the order of the similarity program between images, the order of the above-mentioned similarity program between images can be: the order of the similarity between each first search image and any first query image, or the order of the average similarity between each first search image and the above-mentioned N first query images.

[0043] Exemplarily, the search device may display the N first search images in descending order of similarity between images, or may display the N first search images in descending order of similarity between images, and the embodiments of the present application are not limited to this.

[0044] For example, the user inputs query image 1 and query image 2 through the mobile phone, and searches for search image 1 and search image 2 based on query image 1 and query image 2. At this time, the mobile phone can calculate that the similarity between search image 1 and query image 1 is 10, the similarity between search image 1 and query image 2 is 8, the similarity between search image 2 and query image 1 is 8, and the similarity between search image 2 and query image 2 is 9. Then, the mobile phone can determine that the average similarity between search image 1 and the two query images is 9, and the average similarity between search image 2 and the two query images is 8.5. Finally, the mobile phone can display search image 1 and search image 2 in descending order according to the average similarity.

[0045] In another example, when the above-mentioned preset order is a time order, the time order may be: the time order in which the search device searches for each first search image, or the time order in which each first search image is shot.

[0046] Exemplarily, the search device may display the N first search images in chronological order from front to back, or may display the N first search images in chronological order from back to front, which is not limited in the embodiment of the present application.

[0047] Step 202: The search device performs image processing on the target query image according to the target search image in the M first search images to obtain a second query image.

[0048] The target query image is at least one of the N first query images.

[0049] In the embodiment of the present application, the above-mentioned target search image can be one or more images, which is not limited in the embodiment of the present application. In the embodiment of the present application, the above-mentioned target search image can be a system default setting or a user-set one, which is not limited in the embodiment of the present application.

[0050] In an embodiment of the present application, the above-mentioned image processing of the target query image includes at least one of the following: replacing an area in the target query image with an area in the target search image, adjusting image parameters of the target query image, and splicing the target search image and the target query image. Exemplarily, the above-mentioned image parameters may include at least one of the following: image brightness, image color, and image exposure value. It should be noted that the above-mentioned image parameters include but are not limited to the above-mentioned three parameters, which can be specifically set according to actual needs, and the embodiment of the present application does not limit this.

[0051] It should be noted that the above-mentioned image processing on the target query image can correct the target query image, so that the corrected second query image better meets the needs of the user, thereby improving the accuracy of the item search.

[0052] It should be noted that, when N is equal to 1, the target query image is the first query image.

[0053] In one example, the target search item indicated by the target search image may belong to the same category as the target query item. For example, the target query item belongs to red wine, and the target search item also belongs to red wine.

[0054] Specifically, the search device can first determine the category to which the target query item belongs; then, the search device can determine the search items that belong to the same category as the target query item from the above-mentioned M first search items; then, the search device can determine the target search item from the search items that belong to the same category as the target query item, and the search image corresponding to the icon search item is the target search image.

[0055] In an example, the search device may perform the above step 202 after receiving the first input from the user.

[0056] Exemplarily, the first input mentioned above can be: a click input by the user on the screen of the search device, or a voice command input by the user, or a specific gesture input by the user. The specific input can be determined according to actual usage requirements, and the embodiments of the present application are not limited to this.

[0057] The specific gesture in the embodiment of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the click input in the embodiment of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input.

[0058] Step 203: The search device displays X second search images according to the second query image.

[0059] Each second search image indicates a second search item, and X is a positive integer.

[0060] In an embodiment of the present application, the above-mentioned display of X second search images may include the following steps: the search device displays X second search images in the above-mentioned preset order. For details, please refer to the description of displaying M first search images in a preset order in this application, which will not be repeated here.

[0061] In an example, the above step 202 may specifically include the following steps: the search device displays a second search interface according to the above second query image, wherein the second search interface includes X second search images.

[0062] In addition, the second search interface may include X second search links in addition to the X second search images, wherein one second search link corresponds to one second search image.

[0063] In the embodiment of the present application, after the above step 203, the method may further include the following steps: the search device may perform image processing on part or all of the second query images according to the third search image in the above X second search images to obtain a third query image. Then, the search device may display Y fourth search images according to the above third query image. And so on, until the user no longer searches. In this way, the accuracy of image search can be improved through multiple iterative searches, so that the searched items are more in line with user needs.

[0064] It should be noted that the process of performing image processing on part or all of the second query image according to the third search image can be specifically referred to the description of performing image processing on the target query image according to the target search image in the embodiment of the present application, and will not be repeated here.

[0065] The search method provided by the embodiment of the present application, first, the search device can display M first search images according to N first query images input by the user. Then, the search device can perform image processing on the target query image according to the target search image in the M first search images to obtain a second query image. Finally, the search device can display X second search images according to the second query image. Among them, the N first query images all indicate the target query item, each first search image indicates a first search item, the target query image is at least one of the above N first query images, and each second search image indicates a second search item. Through the above scheme, when the user wants to search for an item through the image search function of the application, compared with the scheme in the related art that requires the user to input an image with high image quality or manually input keywords to re-search the item, the search device in the present application can, when the image quality of the first query image input by the user is not high, perform image processing on the target query image through the target search image that has been searched to obtain the second query image, that is, to correct the target query image. Then, the search device can use the corrected second query image to search and obtain X second search images. In this way, not only can the steps of item search be simplified and efficiency improved, but also the accuracy of item search can be improved, making the searched items more in line with user needs.

[0066] Optionally, in an embodiment of the present application, the search device may determine the above-mentioned target search image in at least the following two possible ways.

[0067] The first possible implementation method (system default setting):

[0068] Exemplarily, before the above step 202, the method may further include the following steps 204a and 204b:

[0069] Step 204a: The search device determines the degree of similarity between each first search image and each first query image.

[0070] Illustratively, the similarity level in the present application can be determined according to a relevant similarity level evaluation algorithm, which will not be described in detail here.

[0071] In one example, the similarity in the present application can be represented by a feature distance, wherein a smaller feature distance represents a higher similarity, and vice versa. That is, the search device can determine the similarity between the first search image and the first query image by calculating the feature distance (e.g., Euclidean distance, Manhattan distance, Chebyshev distance, and angle cosine, etc.) between the first search image and the first query image.

[0072] Step 204b: The search device determines the first search image whose similarity is greater than or equal to the second preset threshold as the target search image.

[0073] Exemplarily, the second preset threshold mentioned above may be a system default setting or may be set by a user, which is not limited in the embodiments of the present application.

[0074] In one example, the second preset threshold value may be a fixed value or a non-fixed value, which is not limited in the present embodiment. For example, the second preset threshold value may be a fixed value of 90, or may be an average value of all determined similarities, or may be a maximum value of all determined similarities.

[0075] The second possible implementation method (user-defined):

[0076] Exemplarily, after the above step 201 and before the above step 202, the method may further include the following steps B1 and B2:

[0077] Step B1: the search device receives a second input from a user for at least one first search image among the M first search images.

[0078] Exemplarily, the second input may be: a click input of the user on at least one of the first search images, or a voice command input by the user, or a specific gesture input by the user. The specific input may be determined based on actual usage requirements, and the embodiments of the present application are not limited to this.

[0079] Step B2: In response to the second input, the search device determines that the at least one first search image is the target search image.

[0080] For example, Figure 2 As shown, the mobile phone displays an item search interface 31 of the shopping application 1, and the item search interface 31 displays a picture upload control 32 and a shooting control 33. When the user wants to search for red wine in the query image (i.e., the target query item mentioned above), the user can click the picture upload control 42 to find and click the query image. At this time, the mobile phone will use the query image to search. Then, as shown in FIG. Figure 3 As shown, the mobile phone can display a thumbnail 42 corresponding to the query image and thumbnails corresponding to three search images in the search interface 41, wherein the thumbnails corresponding to the three search images are respectively search image a thumbnail 43, search image b thumbnail 44, and search image c thumbnail 45. When the user clicks on the search image a thumbnail 43, the mobile phone can determine that the search image a is a replacement image, that is, an image used to correct the query image (that is, the target search image mentioned above).

[0081] The search method provided in the embodiment of the present application can be applied to the scenario of determining the target search image. In one example, the search device can directly determine the target search image from M first search images. In another example, the user can select the target search image according to needs. In this way, the process of determining the target search image is more flexible.

[0082] Optionally, in an embodiment of the present application, the search device may perform targeted image processing on the target query image according to different image problems existing in the target query image (for example, low image clarity or image missing) to correct the target query image, thereby improving the accuracy of image search.

[0083] Exemplarily, the above step 202 may specifically include the following step 202a:

[0084] Step 202a: The search device replaces the first area in the target query image with the second area in the target search image.

[0085] Exemplarily, the first region may be all or part of a target query image, and the second region may be all or part of a target search image, which is not limited in the present application. For example, the search device may divide the target query image into a plurality of regions, and the plurality of regions include the first region.

[0086] For example, the region in the present application may be a region of any shape, for example, a rectangular region, a triangular region, or a circular region, etc., which is not limited in the present embodiment.

[0087] Exemplarily, the search device may replace the first area with the second area in at least two possible implementation methods.

[0088] In the first possible implementation (smart correction):

[0089] Exemplarily, the search device may perform intelligent correction on the target query image, that is, the search device may automatically determine the first area and the second area, and replace the first area with the second area.

[0090] The following will take the first query image as one and the target search image as one as an example to illustrate the intelligent correction process.

[0091] Exemplarily, the search device may determine the first search image with the smallest feature distance (i.e., the highest similarity) to the first query image among the M first search images, and use the first search image as the target search image. Then, the search device may automatically detect the first area in the first query image and mark the first area, search for the second area corresponding to the first area in the target search image, and extract the second area. Finally, the search device may replace the first area in the first query image with the extracted second area. The specific process is as follows:

[0092] (1) The search device can use a deep learning classification model to preliminarily classify the first query image and the M first search images, and give the category attribute labels of the objects corresponding to all the images. For example, the category attribute label of the target query object corresponding to the first query image is red wine.

[0093] (2) The search device can select items with the same category attribute label as the target query item from the M first search items corresponding to the M first search images as candidate items. It should be noted that limiting the items to the same category attribute can minimize the problem of inaccurate matching of candidate items.

[0094] (3) The search device may scale the size of the first query image and the M first search images to H*W, and use a deep learning feature extractor to extract global features of the first query image and the M first search images, respectively. Wherein, H is the height of the query product image, and W is the width of the query product image, that is, H and W are the width and resolution of the image after uniform scaling.

[0095] (4) The search device may design an attention module in the deep learning feature extractor, and use the attention module to filter out unique detail area feature vectors that can identify objects from the global features of the first query image and each of the M first search images.

[0096] (5) The search device may expand the vector of the characteristic detail region feature with a dimension of H*W*C into a 1-dimensional column vector, where C is the number of channels of the output feature of the global feature after passing through the attention module.

[0097] (6) The search device may respectively calculate the feature distances between the one-dimensional column vector corresponding to the first query image and the one-dimensional column vector corresponding to each first search image, and select the first search image with the smallest feature distance as the target search image.

[0098] (7) The search device can locate the unique detail region in the first query image and locate the unique detail region in the target search image according to the unique detail region feature vector.

[0099] (8) The search device can use traditional image algorithms to extract feature points in the unique detail area of ​​the first query image and the unique detail area of ​​the target search image, respectively, and perform feature point matching. Then, the search device can delete mismatched feature points through geometric verification.

[0100] (9) The search device may extract a first region including all matching feature points from the first query image and a second region including all matching feature points from the target search image.

[0101] For example, combined with Figure 3 ,like Figure 4 As shown in (a) of FIG. 1 , a circumscribed rectangular frame 51 is displayed on the query image, and the area framed by the circumscribed rectangular frame 51 is the area of ​​the query image that needs to be replaced (i.e., the first area mentioned above). Among them, A(x1, y1) is the coordinate of the upper left corner of the area framed by the circumscribed rectangular frame 51, and B(x2, y2) is the coordinate of the lower right corner of the area framed by the circumscribed rectangular frame 51. Figure 3 ,like Figure 4 As shown in (b) of FIG. 1 , a circumscribed rectangular frame 52 is displayed on the search image a, and the area framed by the circumscribed rectangular frame 52 is the area for replacement of the search image a (i.e., the second area mentioned above). Wherein, C(p1, q1) is the coordinate of the upper left corner of the area framed by the circumscribed rectangular frame 52 in the search image a, and D(p2, q2) is the coordinate of the lower right corner of the area framed by the circumscribed rectangular frame 52 in the search image a.

[0102] (10) The search device can rotate and deform the target search image according to the geometric ordering of the feature points in the first area extracted from the first query image, so that the geometric ordering of a preset number (e.g., 80%) of feature points in the first query image and the target search image is the same.

[0103] (11) The search device can obtain the image integration mask M and the reverse mask For example, the search device can use the image integration mask M according to formula (1) and obtain the reverse mask according to formula (2): The specific formula is as follows:

[0104]

[0105]

[0106] Among them, x1 and y1 are the corresponding coordinate values ​​of coordinate point A, and x2 and y2 are the corresponding coordinate values ​​of coordinate point B.

[0107] (12) The search device may process the target search image so that the second area in the processed target search image has the same position as the first area. Exemplarily, taking the circumscribed rectangular area of ​​the target search image as a reference, wherein the upper left corner coordinates C(p1,q1) and the lower right corner coordinates D(p2,q2) of the circumscribed rectangular area, the search device may respectively expand outward to the left by p1-x1, expand outward to the top by q1-y1, expand outward to the right by p2+W-x2, and expand outward to the bottom by q2+H-y2, to obtain the processed target search image.

[0108] (13) The search device may replace the first area with the second area. For example, this may be achieved by the following formula (3):

[0109]

[0110] Among them, pic is the spliced ​​second query image, que is the first query image, and ask is the processed target search image.

[0111] For example, combining Figure 4 In (a) and (b), the mobile phone can replace the area framed by the circumscribed rectangular frame 51 in the query image with the area framed by the circumscribed rectangular frame 52 in the search image a, such as Figure 5 As shown, the mobile phone can finally obtain a corrected query image (ie, the second query image mentioned above). Then, the mobile phone can use the corrected query image to search again.

[0112] It should be noted that the search device may use an edge smoothing algorithm to reduce the color difference between the second region and other regions in the first query image to obtain an intelligently corrected second query image.

[0113] The search method provided in the embodiment of the present application can be applied to the scenario of intelligently correcting the first query image. The search device can automatically select the first area to be replaced in the first query image and the second area to be replaced in the target search image. In this way, not only can the process steps for correcting the first query image be simple and efficient, but also the accuracy of subsequent image searches can be improved.

[0114] In the second possible implementation (manual correction):

[0115] Exemplarily, the object search device may manually modify the target query image, that is, the user selects the first area and the second area, and then the object search device may replace the first area with the second area.

[0116] Exemplarily, before the above step 202a, the method may further include the following steps 202b and 202c:

[0117] Step 202b: The search device receives a third input from the user.

[0118] Exemplarily, the third input mentioned above can be: a click input of the user on the first query image, or a voice command input by the user, or a specific gesture input by the user. The specific input can be determined according to actual usage requirements, and the embodiment of the present application is not limited to this.

[0119] Step 202c: In response to the third input, determine the first area and the second area.

[0120] In an example, the search device may display the first query image and the target query image in a cutout interface.

[0121] For example, combining Figure 3 When the user wants to select search image a as a replacement image, he can click on the thumbnail 43 of the search image a. Figure 6 As shown, the mobile phone can display the cutout interface 61, and automatically display the query image and the rectangular frame 62 in the fixed area 61a on the left side of the cutout interface 61, and display the query image search image a and the rectangular frame 63 in the fixed area 61b on the right side of the cutout interface 61. Then, the user can manually adjust the rectangular frame 62 to frame the area to be replaced in the query image (i.e., the first area mentioned above), and manually adjust the rectangular frame 63 to frame the area to be replaced in the search image a (i.e., the second area mentioned above) as required. Finally, the mobile phone can replace the area framed by the rectangular frame 62 with the area framed by the rectangular frame 63.

[0122] The search method provided in the embodiment of the present application can be applied to the scenario of manually correcting the first query image. The user can select the first area to be replaced in the first query image and the second area to be replaced in the target search image according to needs. In this way, the replacement area can be more in line with user needs, thereby improving the accuracy of subsequent image searches.

[0123] Further optionally, in an embodiment of the present application, the search device may determine the degree of blurriness of the first area, and further determine whether area replacement is required, thereby reducing the workload of the search device.

[0124] Exemplarily, the above step 202a may specifically include the following step 202a1:

[0125] Step 202a1: when the blur degree of the first area is greater than or equal to a first preset threshold, the search device replaces the first area in the target query image with the second area in the target search image.

[0126] Exemplarily, the first preset threshold mentioned above may be a system default setting or may be set by a user, which is not limited in the embodiments of the present application.

[0127] Exemplarily, the search device may determine the blur degree of the first area by using a relevant blur detection algorithm, which will not be described in detail here.

[0128] It should be noted that, when the blur degree of the first area is less than the first preset threshold, the search device may not perform any processing on the target query image.

[0129] The search method provided in the embodiment of the present application can be applied to the scenario of determining whether to perform image correction on the first query image. The search device can determine whether to perform image correction by determining the blurriness of the first area. The first query image will be corrected only when the blurriness of the first area is greater than or equal to a first preset threshold. This not only makes the image correction process more flexible, but also reduces the workload of the search device.

[0130] Optionally, in an embodiment of the present application, when the search device obtains multiple query images containing the same object, the search device may search the multiple query images in parallel and process them jointly to improve the accuracy of the search.

[0131] In an example, when N is greater than 1, before displaying the M first search images in the above step 201, the method may further include the following steps C1 and C2:

[0132] Step C1: The search device determines target image feature information corresponding to the target query object based on the N first query images.

[0133] Step C2: The search device determines the M first search images according to the target image feature information.

[0134] Exemplarily, the search device may determine an image feature information group corresponding to the target query item in each first query image, the image feature information group including at least one image feature information, so as to obtain N image feature information groups corresponding to the N first query images. The above-mentioned target image feature information may include all or part of the image feature information in the N image feature information groups. For example, the target image feature information may include all repeated image feature information in the N image feature information groups.

[0135] In another example, when N is greater than 1, before displaying the M first search images in the above step 201, the method may further include the following steps D1 and D2:

[0136] Step D1: The search device determines the search image group corresponding to each first query image according to each first query image, and obtains N search image groups.

[0137] Step D2: The search device determines the M first search images from the N search image groups.

[0138] Exemplarily, the search device may perform image integration of the search images in the N search images. Exemplarily, the search device may scale the size of the first query image and each fifth search image in the group of N search images to (H, W). Wherein H is the height of the first query image, and W is the width of the first query image. Then, the search device may use a deep learning feature extraction model to extract the three-dimensional features H*W*C of the first query image and each fifth search image. Wherein, C is the number of feature channels corresponding to the global features of the model output image. Next, the search device may perform channel fusion of the three-dimensional feature vector to obtain a H*W*1 morphological vector. Finally, the search device may expand the feature vector of H*W*1 into a one-dimensional column vector, and calculate the feature distance between the column vector corresponding to the first query image and the column vector corresponding to each fifth search image in sequence.

[0139] Exemplarily, A(x1, y1) is the feature vector of the first query image, B(x2, y2) is the feature vector of any fifth search image in the N search image groups, and the calculation formula of the feature distance is as follows:

[0140]

[0141] Exemplarily, the search device may display the fifth search images in the N search image groups whose feature distance is less than or equal to the third preset threshold. Further, the search device may display the first preset number (e.g., first 30) of the fifth search images with smaller feature distances among all the fifth search images with feature distances less than or equal to the third preset threshold.

[0142] Exemplarily, the search device may count whether there are duplicate products among the search items corresponding to all fifth search images. If so, the search device may preferentially display the fifth search image corresponding to the duplicate product for the user to view. If not, the search device may display the fifth search image in ascending order of feature distance.

[0143] The search method provided in the embodiment of the present application can be applied to the scenario of searching for objects using multiple query images. The search device uses multiple images including the same query object to search for objects, which can improve the accuracy of the object search and make the object search process more flexible.

[0144] It should be noted that the search method provided in the embodiment of the present application can be executed by a search device or a control module in the search device for executing the search method. The search device provided in the embodiment of the present application is described by taking the search device executing the search method as an example.

[0145] Figure 7 A possible structural diagram of a search device provided in an embodiment of the present application is as follows: Figure 7 As shown, the search device 700 includes: a display module 701 and an image processing module 702, wherein: the display module 701 is used to display M first search images according to N first query images input by a user; the image processing module 702 is used to perform image processing on the target query image according to the target search image in the M first search images displayed by the display module 701 to obtain a second query image; the display module 701 is used to display X second search images according to the second query image processed by the image processing module 702; wherein the N first query images all indicate the target query item, each first search image indicates a first search item, the target query image is at least one of the N first query images, each second search image indicates a second search item, and N, M and X are all positive integers.

[0146] Optionally, the image processing module 702 is specifically configured to replace the first region in the target query image with the second region in the target search image.

[0147] Optionally, the image processing module 702 is specifically configured to replace the first area in the target query image with the second area in the target search image when the blur degree of the first area is greater than or equal to a first preset threshold.

[0148] Alternatively, if Figure 7 As shown, the search device 700 includes: a determination module 703; the determination module 703 is used to determine the similarity between each first search image and each first query image; and determine the first search image whose similarity is greater than or equal to the second preset threshold as the above-mentioned target search image.

[0149] Alternatively, if Figure 7 As shown, the search device 700 includes: a determination module 703; the determination module 703 is used to determine the target image feature information corresponding to the above-mentioned target query item based on the above-mentioned N first query images when N is greater than 1; and determine the above-mentioned M first search images based on the target image feature information.

[0150] Optionally, the display module 701 is specifically configured to display the M first search images in a preset order, where the preset order is a time order or an order of similarity between images.

[0151] It should be noted that if Figure 7 As shown, the modules that must be included in the search device 700 are indicated by solid-line boxes, such as the display module 701 ; the modules that may or may not be included in the search device 700 are indicated by dotted-line boxes, such as the determination module 703 .

[0152] The search device provided in the embodiment of the present application, first, the search device can display M first search images according to N first query images input by the user. Then, the search device can perform image processing on the target query image according to the target search image in the M first search images to obtain a second query image. Finally, the search device can display X second search images according to the second query image. Among them, the N first query images all indicate the target query item, each first search image indicates a first search item, the target query image is at least one of the above N first query images, and each second search image indicates a second search item. Through the above scheme, when the user wants to search for an item through the image search function of the application, compared with the scheme in the related art that requires the user to input an image with high image quality or manually input keywords to re-search the item, the search device in the present application can, when the image quality of the first query image input by the user is not high, perform image processing on the target query image through the target search image that has been searched to obtain the second query image, that is, to correct the target query image. Then, the search device can use the corrected second query image to search and obtain X second search images. In this way, not only can the steps of item search be simplified and efficiency improved, but also the accuracy of item search can be improved, making the searched items more in line with user needs.

[0153] The beneficial effects of various implementations in this embodiment can be specifically referred to the beneficial effects of the corresponding implementations in the above method embodiment. To avoid repetition, they will not be described again here.

[0154] The search device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0155] The search device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0156] The search device provided in the embodiment of the present application can achieve Figures 1 to 6 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0157] Optional, such as Figure 8 As shown, an embodiment of the present application also provides an electronic device 800, including a processor 801, a memory 802, and a program or instruction stored in the memory 802 and executable on the processor 801. When the program or instruction is executed by the processor 801, each process of the above-mentioned search method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0158] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0159] Fig. 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0160] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.

[0161] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0162] Among them, the display unit 106 is used to display M first search images according to N first query images input by the user; the processor 110 is used to perform image processing on the target query image to obtain a second query image according to the target search image in the M first search images displayed by the display unit 106; the display unit 106 is used to display X second search images according to the second query image processed by the processor 110; wherein the N first query images all indicate the target query item, each first search image indicates a first search item, the target query image is at least one of the N first query images, each second search image indicates a second search item, and N, M and X are all positive integers.

[0163] Optionally, the processor 110 is specifically configured to replace the first area in the target query image with the second area in the target search image.

[0164] Optionally, the processor 110 is specifically configured to replace the first area in the target query image with the second area in the target search image when the blur degree of the first area is greater than or equal to a first preset threshold.

[0165] Optionally, the processor 110 is further configured to determine a degree of similarity between each first search image and each first query image; and determine the first search image whose degree of similarity is greater than or equal to a second preset threshold as the target search image.

[0166] Optionally, the processor 110 is further used to determine the target image feature information corresponding to the target query item based on the N first query images when N is greater than 1; and determine the M first search images based on the target image feature information.

[0167] Optionally, the display unit 106 is specifically configured to display the M first search images in a preset order, where the preset order is a time order or an order of similarity between images.

[0168] The electronic device provided in the embodiment of the present application, first, the electronic device can display M first search images according to N first query images input by the user. Then, the electronic device can perform image processing on the target query image according to the target search image in the M first search images to obtain a second query image. Finally, the electronic device can display X second search images according to the second query image. Among them, the N first query images all indicate the target query item, each first search image indicates a first search item, the target query image is at least one of the above N first query images, and each second search image indicates a second search item. Through the above scheme, when the user wants to search for an item through the image search function of the application, compared with the scheme in which the user needs to input an image with high image quality or manually input keywords to re-search the item in the related art, the electronic device in the present application can, when the image quality of the first query image input by the user is not high, perform image processing on the target query image through the target search image that has been searched to obtain the second query image, that is, to correct the target query image. Then, the search device can use the corrected second query image to search and obtain X second search images. In this way, not only can the steps of item search be simplified and efficiency improved, but also the accuracy of item search can be improved, making the searched items more in line with user needs.

[0169] The beneficial effects of various implementations in this embodiment can be specifically referred to the beneficial effects of the corresponding implementations in the above method embodiment. To avoid repetition, they will not be described again here.

[0170] It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include but are not limited to a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here. The memory 109 can be used to store software programs and various data, including but not limited to applications and operating systems. The processor 110 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, the user interface, and the application program, etc., and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 110.

[0171] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned search method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0172] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0173] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned search method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0174] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0175] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0176] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0177] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A search method, It is characterized in that The method comprises: Displaying M first search images according to N first query images input by the user; According to the target search image in the M first search images, performing image processing on the target query image to obtain a second query image, wherein the image processing on the target query image includes at least one of the following: replacing an area in the target query image with an area in the target search image, adjusting an image parameter of the target query image, and splicing the target search image and the target query image; Based on the second query image, display X second search images; Among them, the N first query images all indicate target query items, each first search image indicates a first search item, the target query image is at least one of the N first query images, each second search image indicates a second search item, and N, M and X are all positive integers.

2. The method according to claim 1, It is characterized in that The step of performing image processing on the target query image to obtain a second query image according to the target search image in the M first search images includes: A first region in the target query image is replaced with a second region in the target search image.

3. The method according to claim 2, It is characterized in that The step of replacing the first area in the target query image with the second area in the target search image comprises: When the blur degree of the first area is greater than or equal to a first preset threshold, the first area in the target query image is replaced with the second area in the target search image.

4. The method according to any one of claims 1 to 3, It is characterized in that Before performing image processing on the target query image to obtain the second query image according to the target search image in the M first search images, the method further includes: determining a degree of similarity between each first search image and each first query image; The first search image whose similarity is greater than or equal to a second preset threshold is determined as the target search image.

5. The method according to claim 1, It is characterized in that When N is greater than 1, before displaying the M first search images, the method further includes: Determining target image feature information corresponding to the target query object according to the N first query images; The M first search images are determined according to the target image feature information.

6. The method according to claim 1, It is characterized in that The displaying of M first search images comprises: The M first search images are displayed in a preset order, where the preset order is a time order or an order of similarity between images.

7. A search device, It is characterized in that The search device comprises: a display module and an image processing module; The display module is used to display M first search images according to N first query images input by the user; The image processing module is used to perform image processing on the target query image to obtain a second query image according to the target search image in the M first search images displayed by the display module; The display module is used to display X second search images according to the second query image obtained by the image processing module, wherein the image processing of the target query image includes: replacing the area in the target query image with the area in the target search image, adjusting the image parameters of the target query image, and splicing the target search image and the target query image; Among them, the N first query images all indicate target query items, each first search image indicates a first search item, the target query image is at least one of the N first query images, each second search image indicates a second search item, and N, M and X are all positive integers.

8. The search device according to claim 7, It is characterized in that The image processing module is specifically used to replace the first area in the target query image with the second area in the target search image.

9. The search device according to claim 8, It is characterized in that The image processing module is specifically configured to replace the first area in the target query image with the second area in the target search image when the blur degree of the first area is greater than or equal to a first preset threshold.

10. The search device according to any one of claims 7 to 9, It is characterized in that The search device comprises: a determination module; The determination module is used to determine the similarity between each first search image and each first query image; and determine the first search image whose similarity is greater than or equal to a second preset threshold as the target search image.

11. The search device according to claim 7, It is characterized in that The search device comprises: a determination module; The determination module is used to determine the target image feature information corresponding to the target query item based on the N first query images when N is greater than 1; and determine the M first search images based on the target image feature information.

12. The search device according to claim 7, It is characterized in that The display module is specifically used to display M first search images in a preset order, where the preset order is a time order or an order of similarity between images.

13. An electronic device, It is characterized in that The method comprises a processor, a memory and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the search method as claimed in any one of claims 1 to 6.

14. A readable storage medium, It is characterized in that The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by the processor, the steps of the search method according to any one of claims 1 to 6 are implemented.