Image search method and apparatus

[0013] In order to make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of the embodiments of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

[0014] An image search method provided by an embodiment of the present invention, such as figure 1 As shown, the method includes:

[0015] Step 11. On the image search interface, receive the sketch of the image to be searched input by the user terminal, and obtain the spatial position relationship between the objects contained in the sketch;

[0016] Among them, the image search interface can be the interface presented by the gallery (or album, photo) after receiving the image search instruction sent by the user through the user terminal, or it can be directly presented when the user views the gallery (or album, photo) in the user terminal The user terminal can be a mobile phone, a tablet, a camera or a wearable smart device. As a more specific embodiment, an image search interface is presented after receiving an image search instruction sent by the user through the user terminal as an example. In the gallery (or An image search button is added to the album, photo) interface. If the user clicks or touches the image search button through the user terminal, it will be deemed to have received the image search instruction sent by the user terminal, and the gallery (or album, photo) will display the image search interface. The user terminal is prompted to input a sketch of the image to be searched.

[0017] After receiving the sketch of the image to be searched input by the user terminal, obtain the spatial position relationship between the objects contained in the sketch, specifically: obtaining the spatial position relationship between the objects contained in the sketch, including: receiving input from the user terminal The spatial position relationship between the objects included in the sketch; or the spatial position relationship between the objects included in the sketch is determined according to the sketch of the image to be searched input by the user terminal.

[0018] In specific implementation, the user enters the sketch of the image to be searched on the image search interface through the user terminal. If the sketch of the image to be searched is easy to draw, for example: the image to be searched contains fewer objects and the details of each object are less, the user can Only draw the sketch of the image to be searched on the image search interface. After receiving the sketch of the image to be searched by the user on the image search interface, the user terminal recognizes the objects contained in the sketch of the image to be searched according to the input sketch of the image to be searched. , And then determine the spatial position relationship between the objects included in the sketch; if the sketch of the image to be searched is difficult to draw or part of the sketch of the image to be searched is difficult to draw, the user draws the sketch of the image to be searched on the image search interface, and then You can use text to describe the parts that are difficult to draw in the sketch, for example: use text to describe the spatial position relationship between objects in the image to be searched, that is, the user terminal receives the sketch of the image to be searched input by the user on the image search interface After that, the spatial position relationship between the objects in the sketch input by the user on the image search interface is received. Of course, it is more preferable to input the sketch of the image to be searched on the image search interface, and then input the spatial position relationship between the objects in the sketch to improve the accuracy of the search.

[0019] It should be noted that if the user does not input the positional relationship between the objects in the sketch to be searched in the image search interface, the user terminal, after receiving the sketch of the image to be searched input by the user, according to the sketch of the image to be searched input by the user , To determine the spatial position relationship between the objects contained in the sketch, including: using a deep object detection network to detect each object included in the sketch; using a deep event detection network to detect the spatial position relationship between objects in the sketch.

[0020] In specific implementation, the deep object detection network is used to detect each object contained in the sketch of the image to be searched, and the deep event detection network is used to detect the spatial position relationship between the objects in the sketch of the image to be searched, which specifically includes the following steps:

[0021] (1) Predefine the types of objects to be detected, such as people, plants, animals, buildings, cars, etc., and train the detectors of each type of object through a deep object detection network (for example: Fast-RCNN);

[0022] (2) In the detection stage, the sketch of the image to be searched is first segmented, and bounding boxes are obtained for each block of the over-segmented image to obtain the detection candidate area;

[0023] (3) For each detection candidate area, input the detection candidate area into a trained object detection network (for example: Fast-RCNN) to obtain the object detection result in the detection candidate area;

[0024] (4) Establish the spatial position relationship between objects. Specifically, use a graph data structure to represent the detected objects. For example, if the detected objects in an image include: humans, animals, and plants, create a three vertices According to the position of the object in the image, establish the connecting edge between the two objects. The value on the side takes one of 0, 1, 2, and 3. Each value represents the positional relationship between up, down, left, and right, for example : 0 represents up, 1 represents down, 2 represents left, 3 represents right.

[0025] Step 13, According to the spatial position relationship between the objects contained in the sketch and the spatial position relationship between the objects contained in the image in the image database, the sketch is matched with the image in the image database to obtain the matching value of the image, The preset rule determines the target matching value in the matching value of the image;

[0026] In specific implementation, according to the spatial position relationship between the objects contained in the image, the sketch of the image to be searched is matched with the multiple images contained in the image database one by one to obtain the matching values ​​of multiple images, and preset rules Determine the target matching value from the calculated matching values ​​of multiple images. In order to facilitate the matching of the sketch of the image to be searched with the multiple images contained in the image database according to the spatial position relationship between the objects in the image, the image The database not only stores the image data of multiple images, but also stores the spatial position relationship between the objects contained in each image. The method for determining the spatial position relationship between the objects contained in each image in the image database , Is the same as the method for determining the spatial position relationship between the objects contained in the sketch of the image to be searched, that is, for each image in the image database, the deep object detection network is used to detect each object contained in the image, and the depth event The detection network detects the spatial position relationship between objects in the image, which will not be repeated here.

[0027] It is worth noting that according to the spatial position relationship between the objects contained in the image, the sketch of the image to be searched is matched with the multiple images contained in the image database one by one. In the specific implementation, the sketch of the image to be searched is included The spatial position relationship between the objects in the image database is matched with the spatial position relationship between the objects contained in each image in the image database, and the matching algorithm can use the algorithm in the prior art, which will not be repeated here. As a more specific embodiment, suppose that the sketch of the image to be searched input by the user terminal on the image search interface includes a person and a horse, and the spatial position relationship between the objects included in the sketch input by the user terminal is: the person is on the left of the horse , When matching, according to the spatial position relationship between the objects, that is, the person is on the left of the horse, the sketch of the image to be searched is matched with the multiple images contained in the image database one by one.

[0028] It is worth noting that after the matching values ​​of multiple images are obtained, the target matching value is determined from the matching values ​​of the multiple images according to preset rules, including: matching the matching values ​​of the multiple images that are greater than or equal to the preset threshold The value is determined as the target matching value, that is, the preset rule is to select a matching value greater than or equal to a preset threshold, where the preset threshold can be set freely, for example, the preset threshold is 85. Of course, in other embodiments of the present invention, the preset rule can also have other implementations, for example: selecting the maximum value among multiple matching values ​​as the target matching value, and selecting the larger N of the multiple matching values ​​as the target matching value. Value, N is a natural number.

[0029] Step 15: Determine the image corresponding to the target matching value as the searched image, wherein a plurality of images are stored in the image database, and the spatial position relationship between the objects contained in the image is correspondingly stored.

[0030] During specific implementation, the image corresponding to the target matching value is determined as the searched image. Specifically, the image for calculating the target matching value includes two images, that is, the sketch of the image to be searched input by the user through the user terminal and the image database Determining the image corresponding to the target matching value as the searched image includes: determining the image in the image database corresponding to the target matching value as the searched image.

[0031] As a more specific embodiment, it is assumed that the preset rule in determining the target matching value among the matching values ​​of multiple images is to select a matching value greater than or equal to a preset threshold, and the preset threshold takes a value of 90. Assuming that the matching values ​​of the sketch of the image to be searched and the image A, image B, image C, image D, and image E in the image database are 20, 70, 55, 90, 95, respectively, the target matching value determined by the preset rule Since the image in the image database corresponding to the target matching value of 90 is image D, and the image in the image database corresponding to the target matching value of 95 is image E, the images D and E are determined as the searched images.

[0032] As another more specific embodiment, it is assumed that the preset rule in determining the target matching value among the matching values ​​of multiple images is to select the maximum value, and it is still assumed that the sketch of the image to be searched and the image A, The matching values ​​of image B, image C, image D, and image E are respectively 20, 70, 55, 90, 95, and the target matching value determined by the preset rule is 95, because the target matching value of 95 corresponds to the image database The middle image is image E, and then image E is determined as the searched image.

[0033] As another more specific embodiment, it is assumed that a preset rule is used to determine the target matching value among the matching values ​​of multiple images. The preset rule is to select the larger three of the matching values ​​of the multiple images as the target matching value. Still suppose that the matching values ​​of the sketch of the image to be searched and the image A, image B, image C, image D, and image E in the image database are respectively 20, 70, 55, 90, 95, then the target determined by the preset rule matches The values ​​are 95, 90, 70. Since the image in the image database corresponding to the target matching value of 95 is image E, the image in the image database corresponding to the target matching value of 90 is image D, and the image in the image database corresponding to the target matching value of 70 If it is image B, then image B, image D, and image E are determined as the searched images.

[0034] In the method provided by the embodiment of the present invention, in the image search interface, the sketch of the image to be searched for input by the user terminal is received, and the spatial position relationship between the objects contained in the sketch is obtained, according to the spatial position of the objects contained in the sketch The relationship and the spatial position relationship between the objects contained in the image in the image database, the sketch is matched with the image in the image database to obtain the matching value of the image, and the target matching value is determined in the matching value of the image according to the preset rules. The image corresponding to the target matching value is determined as the searched image, that is, according to the spatial position relationship between the objects contained in the image, the image database containing specific content is searched for images without browsing the pictures one by one, thus improving the search The speed and search efficiency are compared with the time-consuming, labor-intensive, and low-efficiency search method of browsing pictures one by one in the prior art. The embodiment of the present invention improves the speed of searching images and the efficiency of searching images.

[0035] In a possible implementation manner, in the method provided by the embodiment of the present invention, acquiring the spatial position relationship between the objects included in the sketch includes: receiving the spatial position relationship between the objects included in the sketch input by the user terminal ; Or according to the sketch of the image to be searched input by the user terminal, the spatial position relationship between the objects contained in the sketch is determined.

[0036] In a possible implementation manner, in the method provided by the embodiment of the present invention, determining the spatial position relationship between the objects included in the sketch according to the sketch of the image to be searched input by the user terminal includes: adopting a deep object detection network Detect each object contained in the sketch; use a deep event detection network to detect the spatial position relationship between the objects in the sketch.

[0037] In a possible implementation manner, in the method provided by the embodiment of the present invention, the method further includes: presetting the image of the object included in the multiple images in the image database on the image search interface.

[0038] In specific implementation, when performing image search in the embodiment of the present invention, it is necessary to input a sketch of the image to be searched on the image search interface. By presetting the images of objects contained in multiple images in the image database on the image search interface, it is convenient for the user terminal Enter the sketch of the image to be searched in the search interface, so that the user terminal can select the preset image to complete the sketch drawing in the image search interface. At the same time, use the preset image to draw the sketch to improve the accuracy of drawing the sketch. Further, by searching in the image The interface presets the images of the objects contained in the multiple images in the image database, so that the user terminal selects the preset image to complete the sketch drawing on the image search interface, which can improve the recognition accuracy of the objects contained in the sketch and improve the accuracy of the sketch The accuracy of the recognition of the spatial position relationship between the objects included, thereby improving the accuracy of searching for images.

[0039] In a possible implementation manner, in the method provided in the embodiment of the present invention, determining the target matching value in the matching value of the image according to the preset rule includes: determining the matching value of the image that is greater than or equal to the preset threshold Determined as the target match value.

[0040] Combine below figure 2 The image search method provided by the embodiment of the present invention will be described in detail. Such as figure 2 As shown, the specific steps of the image search method provided by the embodiment of the present invention include:

[0041] Step 21: The user terminal turns on the image search function and prompts the user terminal to draw a sketch of the image to be searched. Specifically, if the user terminal needs to search for an image containing specific content in the gallery, then open the sketch search function in the gallery and follow the prompts on the drawing board Draw sketches. For parts that are difficult to draw, you can use vocabulary descriptions and input them to the search module;

[0042] Step 23: The user terminal preprocesses each image in the gallery through the built-in algorithm, and recognizes the object in each image and the spatial position relationship between the objects. Specifically, the deep learning method Position important objects, detect where the objects appear, and identify the spatial position relationship between objects, for example: a person is sitting on a sofa;

[0043] Step 25: According to the objects contained in the image and the spatial position relationship between the objects, the sketch of the image to be searched input by the user terminal is matched with each image in the gallery, and the target matching value is determined according to the preset rules, and the target matching is determined The image corresponding to the value, specifically, according to the object described in the sketch and the positional relationship between the object, the sketch of the image to be searched input by the user terminal is matched with each image in the gallery through a matching algorithm to obtain multiple The matching value of the image, and the target matching value is determined among the matching values ​​of multiple images according to preset rules, for example: selecting 3 or 5 images with a higher degree of matching among the matching values ​​of multiple images;

[0044] In step 27, the image corresponding to the target matching value is presented to the user through the image search interface. Specifically, the image corresponding to the target matching value is displayed on the image search interface for the user to make a selection and find the image to be searched.

[0045] An image search device provided by an embodiment of the present invention, such as image 3 As shown, the device includes: an acquiring unit 31, configured to receive a sketch of the image to be searched input by the user terminal on the image search interface, and to acquire the spatial position relationship between the objects contained in the sketch; the first processing unit 33 uses According to the spatial position relationship between the objects contained in the sketch and the spatial position relationship between the objects contained in the image in the image database, the sketch is matched with the image in the image database to obtain the matching value of the image, and preset The rule determines the target matching value in the matching value of the image; the second processing unit 35 is used to determine the image corresponding to the target matching value as the searched image, wherein a plurality of images are stored in the image database, and the corresponding stored image contains The spatial position relationship between the various objects.

[0046] In the device provided by the embodiment of the present invention, in the image search interface, the sketch of the image to be searched input by the user terminal is received, and the spatial position relationship between the objects contained in the sketch is obtained, according to the spatial position of the objects contained in the sketch The relationship and the spatial position relationship between the objects contained in the image in the image database, the sketch is matched with the image in the image database to obtain the matching value of the image, and the target matching value is determined in the matching value of the image according to the preset rules. The image corresponding to the target matching value is determined as the searched image, that is, according to the spatial position relationship between the objects contained in the image, the image database containing specific content is searched for images without browsing the pictures one by one, thus improving the search The speed and search efficiency are compared with the time-consuming, labor-intensive, and low-efficiency search method of browsing pictures one by one in the prior art. The embodiment of the present invention improves the speed of searching images and the efficiency of searching images.

[0047] In a possible implementation manner, in the apparatus provided by the embodiment of the present invention, the acquiring unit 31 acquires the spatial position relationship between the objects included in the sketch, which is specifically used to: receive one of the objects included in the sketch input by the user terminal Or determine the spatial position relationship between the objects contained in the sketch according to the sketch of the image to be searched input by the user terminal.

[0048] In a possible implementation manner, in the apparatus provided by the embodiment of the present invention, the acquiring unit 31 determines the spatial position relationship between the objects included in the sketch according to the sketch of the image to be searched input by the user terminal, which is specifically used for: The deep object detection network is used to detect the objects contained in the sketch; the deep event detection network is used to detect the spatial position relationship between the objects in the sketch.

[0049] In a possible implementation manner, in the device provided by the embodiment of the present invention, the first processing unit 33 is further configured to: preset images of objects contained in multiple images in the image database on the image search interface.

[0050] In a possible implementation manner, in the device provided by the embodiment of the present invention, the first processing unit 33 determines the target matching value in the matching value of the image according to a preset rule, which is specifically used to: set the matching value of the image to be greater than or The matching value equal to the preset threshold is determined as the target matching value.

[0051] The image search device provided by the embodiment of the present invention may be integrated in a user terminal, for example, integrated in a mobile phone, or integrated in image processing software, wherein the acquiring unit 31, the first processing unit 33, and the second processing unit 35 Both CPUs and other processors can be used.

[0052] An image search method and device provided by embodiments of the present invention receive a sketch of an image to be searched input by a user terminal on an image search interface, and obtain the spatial position relationship between objects included in the sketch, according to the objects contained in the sketch The spatial position relationship between and the spatial position relationship between the objects contained in the image in the image database, the sketch is matched with the image in the image database, and the matching value of the image is obtained, and the matching value of the image is set in the preset rule Determine the target matching value, and determine the image corresponding to the target matching value as the searched image, that is, according to the spatial position relationship between the objects contained in the image, search for images containing specific content in the image database without browsing the pictures one by one , Thereby improving the speed and efficiency of searching images.

[0053] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the objectives of the solutions of the embodiments. Those of ordinary skill in the art can understand and implement without creative work.

[0054] Through the description of the above implementation manners, those skilled in the art can clearly understand that each implementation manner can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, magnetic A disc, an optical disc, etc., include a number of instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiment.

[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions recorded in the foregoing embodiments are modified, or some of the technical features are equivalently replaced; these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.