A search method of an object instance, a computing device and a storage medium

Abstract

Embodiments of the present application provide a search method for object instances, a computing device and a storage medium. In the embodiments of the present application, an instance image of an object to be searched and a video to be searched are obtained, and the instance image and the video to be searched are input into a preset search model; a plurality of candidate images corresponding to the instance image are located from the video to be searched by a locating model in the search model, and the candidate images have one object instance; the same candidate image as the object to be searched in the instance image is recognized from the plurality of candidate images by a recognition model in the search model, and the candidate image is taken as a target image, and the target image is the instance image having the object to be searched. Since the image to be searched is not limited to images of some object types, the search model can meet the requirement of searching for a target image corresponding to an arbitrary type of object to be searched from the video to be searched, and the search is not limited to the object category.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method for searching object instances, a computing device, and a storage medium. Background Technology

[0002] Classic object recognition tasks, such as pedestrian and vehicle recognition, often involve identifying objects of fixed categories. For example, a vehicle recognition model only identifies vehicles, and a pedestrian recognition model only identifies pedestrians; thus, they identify objects of fixed types. If a user wants to use these models, such as pedestrian or vehicle recognition models, to search for other categories of objects, such as trash cans, buildings, or street scene merchandise, the search function cannot be achieved because these types are not in the recognition categories. Furthermore, while some image recognition methods can retrieve data across any category, they assume that the corresponding image has already been detected or located, or that the object occupies a dominant position in the image, which is unrealistic in practical scenarios. Summary of the Invention

[0003] This application provides a method, computing device, and storage medium for searching object instances, enabling the searching of instance images of any object.

[0004] This application provides a method for searching object instances, comprising: acquiring an instance image of the object to be searched and a video to be searched; inputting the instance image and the video to be searched into a preset search model; locating multiple candidate images corresponding to the instance image from the video to be searched using a localization model in the search model, wherein each candidate image contains an object instance; and identifying a candidate image identical to the object to be searched in the instance image from the multiple candidate images using a recognition model in the search model, and using the target image as the target image, wherein the target image is an instance image containing the object to be searched.

[0005] This application provides a product search method, comprising: acquiring an instance image of a product to be searched, and a video and / or product image of the product to be searched; inputting the instance image and the video and / or product image of the product to be searched into a preset search model; locating multiple candidate product images corresponding to the instance image from the video and / or product image of the product to be searched using a positioning model in the search model, wherein each candidate product image has a product instance; and identifying a candidate product image that is identical to the product to be searched in the instance image from the multiple candidate product images using a recognition model in the search model, and using the target product image as the target product image, wherein the target product image is an instance image containing the video and / or product to be searched.

[0006] This application embodiment also provides a computing device, including: a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program for: acquiring an instance image of a search object and a video of a search object, inputting the instance image and the video of a search object into a preset search model; locating multiple candidate images corresponding to the instance image from the video of a search object using a positioning model in the search model, wherein each candidate image has an object instance; and identifying a candidate image that is the same as the search object in the instance image from the multiple candidate images using a recognition model in the search model, and using it as a target image, wherein the target image is an instance image containing the search object.

[0007] This application also provides a computer-readable storage medium storing a computer program, which, when executed by one or more processors, causes the one or more processors to perform the steps in the above-described method.

[0008] In this embodiment, an instance image of the object to be searched and a video to be searched are obtained, and the instance image and the video to be searched are input into a preset search model. Using a localization model in the search model, multiple candidate images corresponding to the instance image are located from the video to be searched. Each candidate image contains an object instance. Using a recognition model in the search model, a candidate image identical to the object to be searched in the instance image is identified from the multiple candidate images and used as the target image. The target image is an instance image containing the object to be searched. This allows the localization model in the search model to directly locate candidate images from the video to be searched, and then the recognition model to find the target image corresponding to the instance image from the candidate images. Since the image to be searched is not limited to certain object types, the search model can find target images corresponding to any type of object to be searched from the video to be searched, without being limited to a specific object category. Attached Figure Description

[0009] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0010] Figure 1 A flowchart illustrating an exemplary embodiment of the method for searching object instances according to this application;

[0011] Figure 2 This is a schematic diagram illustrating model training as an exemplary embodiment of this application;

[0012] Figure 3 A schematic diagram illustrating a search service for object instances as an exemplary embodiment of this application;

[0013] Figure 4 This is a schematic diagram of a search page as an exemplary embodiment of this application;

[0014] Figure 5 This is a schematic diagram of the structure of a computing device provided for an exemplary embodiment of this application. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] As mentioned earlier, classic tasks such as face recognition, pedestrian re-identification, and vehicle re-identification often involve fixed-category object detection and recognition. Image retrieval problems, on the other hand, cannot provide bounding boxes and therefore cannot retrieve small objects within images. In practical applications, users may search for objects in video scenes that are not categorized, such as a backpack discarded on the ground or a logo on a vehicle. Because the categories are variable, a method for searching object instances is proposed, which can search for any object.

[0017] The following describes the object instance search process in detail with reference to the method implementation examples.

[0018] Figure 1 This is a flowchart illustrating a method for searching object instances, as exemplified by an embodiment of this application. The method 100 provided in this embodiment is executed by a computing device, such as a server. The method 100 includes the following steps:

[0019] 101: Obtain the instance image and the video of the object to be searched, and input the instance image and the video into the preset search model.

[0020] 102: Using the localization model in the search model, multiple candidate images corresponding to the instance image are located from the video to be searched.

[0021] Each candidate image contains one object instance.

[0022] 103: Using the recognition model in the search model, identify the candidate image that is the same as the object to be searched in the instance image from multiple candidate images, and use it as the target image.

[0023] The target image is an instance image containing the object to be searched.

[0024] It should be noted that the server can be either a physical server or a cloud server. For a physical server, the method can be deployed on a program stored in memory and run on the processor. For a cloud server, the method can be deployed on a virtual device, such as a virtual machine or a virtual container. Virtual devices are permitted through the physical devices on which the cloud server depends. Furthermore, this method 100 can also be implemented via a smart terminal, such as a computer.

[0025] The following is a detailed explanation of the above steps:

[0026] 101: Obtain the instance image and the video of the object to be searched, and input the instance image and the video into the preset search model.

[0027] In this context, an instance refers to a single, independent entity, while an object instance refers to a single, independent object, such as an umbrella, a person, a product, or a vehicle logo. An instance image is an image containing only one object instance, such as an image of an umbrella. This image can be a complete image, a cropped image from a separate image, or a cropped image from a video. Instance images can be of any category.

[0028] The video to be searched can be a pre-set video, which can be obtained from a camera. Cameras can be deployed in different locations, such as cameras in traffic areas, cameras in shopping malls, and other similar venues. They can also be cameras installed on different devices, such as mobile phone cameras, computer cameras, and dashcam cameras. Further details are omitted.

[0029] In addition to videos, the search term can also be images. These images can consist of one or more images, and one image can contain multiple instances of objects, such as people, trees, vehicles, and buildings.

[0030] A search model refers to a model used to search for instance images containing a given object. This search model can include a locator model, or a localizer, which provides bounding boxes. The localizer performs multi-scale sliding window scanning on the original image (which may originate from images in the search video mentioned earlier) to obtain the corresponding bounding boxes, thus exhibiting good robustness to scale changes. The search model may also include a recognition model, such as a ReID (Person Re-identification) model. Here, the model is designed for different objects, hence it can also be called ReID for everything. The localizer is used for localization and coarse ranking, while ReID is used for fine ranking. The process is as follows: the localizer obtains coarse localization results from a large number of search videos based on the query image (i.e., the instance image), and then the ReID model performs fine ranking to obtain the final instance search result.

[0031] The instance image and the video to be searched can be obtained by the server from other devices, such as other servers, or they can be uploaded by the user. For user uploads, the user can upload the instance image and the video to be searched via computer. The user can also upload only the instance image, while the video to be searched can be obtained by the server from other platforms or directly from the camera.

[0032] For example, as mentioned earlier, users can access the server's service interface through a web browser installed on their computer. This interface is used to search for instance images, essentially serving as a search interface. Users upload an instance image of an umbrella using the upload button provided on this interface. This instance image can be a cropped image from another image. Users can then upload videos to be searched, or directly select the corresponding video source on the interface, such as videos from a shopping mall or a street. The user then clicks the search button on their computer, sending the uploaded instance image and the corresponding video source to the server via a search request. Upon receiving the search request, the server retrieves the corresponding instance image and the video to be searched based on the instance image and video source carried in the request. The server then calls up the corresponding search model, inputting the umbrella instance image and the video to be searched into the search model to obtain the corresponding target image from the video to be searched.

[0033] The search model is trained as follows:

[0034] Specifically, the method 100 further includes: acquiring training data, which includes sample images and sample videos, and the training data does not have labeled data; inputting the training data into an initial model; inputting the training data into an initial localization model in the initial model to locate multiple candidate sample images corresponding to the sample images from the sample videos; inputting the multiple candidate sample images and the sample images into an initial recognition model in the initial model through the initial localization model, and using the initial recognition model to find candidate sample images with the same objects as the sample images from the candidate sample images; and training the initial localization model based on the found candidate sample images and the sample images to complete the training of the initial model.

[0035] The training data may include sample images and sample videos, but it does not contain labeled data. Examples include sample images of multiple object instances, such as a tree, a person, or a car, as well as multiple sample videos. Videos may contain multiple frames. It should be understood that sample images appear within sample videos.

[0036] The initial model is the initial model of the search model described above. It may include the following:

[0037] The initial localization model is the initial model of the localization model (e.g., the localizer) mentioned above, and its corresponding parameters are initial.

[0038] The initial recognition model is the initial model of the recognition model mentioned above (e.g., ReID), and its corresponding parameters are initial.

[0039] It should be understood that the initial model has the same function as the model described above, but the initial model needs to be trained.

[0040] For example, as described above, the server can train the initial model in the following way: The server can obtain multiple sample images and corresponding sample videos from other servers or platforms. Let's take one sample image and one corresponding sample video as an example: Figure 2As shown, a sample image 201 and a sample video 202 are input into the initial localization model 203 (e.g., localizer) of the initial model. The initial localization model 203 extracts the feature data of the sample image 201, such as feature vectors. Then, the initial localization model 203 performs multi-scale sliding window scanning on each frame image 207 in the sample video to determine the feature data of the image corresponding to the sliding window scan, such as feature vectors. Then, the two feature vectors are compared to determine the similarity. Based on the similarity threshold, it is determined whether it is a candidate sample image. For example, the image corresponding to the similarity is greater than the threshold is taken as a candidate sample image. Thus, the initial localization model 203 determines multiple detection boxes in the frame image 207. The image in the detection box is the candidate sample image, which is extracted from the frame image 207 and output from the initial localization model, and then input into the initial recognition model 204 (e.g., ReID). At the same time, the initial localization model also inputs the sample image 201 into the initial recognition model 204. After receiving these images, the initial recognition model 204 extracts feature data, such as feature vectors, from the sample image 201 and feature data, such as feature vectors, from each candidate sample image. It compares the feature vectors of the sample image 201 with the feature vectors of each candidate sample image to determine the similarity. Then, based on a similarity threshold, it identifies multiple candidate sample images 205 (images with similarity values ​​greater than the threshold). The initial recognition model then pairs these multiple candidate sample images 205 with the sample image 201 to form training sample pairs, which are fed back to the initial localization model via feedback path 206. The initial localization model then self-trains based on these training sample pairs to improve its accuracy. This iterative training process is repeated to improve the accuracy of the initial localization model, ultimately completing the training of the model.

[0041] This enables self-supervised training of the model.

[0042] To maximize the accuracy of the localization model and minimize training time, the initial localization and recognition models can be trained beforehand. Specifically, the initial localization and recognition models can be obtained by training on single-object tracking training data.

[0043] As mentioned above, the initial localization model and the initial recognition model can be trained using a wide range of single-object tracking training data, which may include labeled data. The specific training process is determined by existing technology and will not be elaborated here.

[0044] Therefore, the initial localization model and the initial recognition model have already achieved a certain level of model accuracy. Continuing the above training on this basis will allow for better and faster training, thus completing the training of the search model.

[0045] In the above process, the localization model, such as the localizer, and the recognition model, such as the data corresponding to ReID of everything, are noisy data. In order to make self-supervision more robust to noise, a sampling strategy guided by ReID and the multi-source clustering SimCLR method are proposed.

[0046] Specifically, the initial recognition model finds candidate sample images that have the same object as the sample image from the candidate sample images, including: determining the similarity between the sample image and the candidate sample images through the initial recognition model; and finding candidate sample images that have the same object as the sample image based on the similarity, so as to train the initial localization model based on the found candidate sample images and the sample image.

[0047] As already explained above, it will not be repeated here. Simply put: the initial recognition model selects candidate images larger than a threshold from the candidate sample images. These selected images are then paired with the original sample images to form training data pairs, optimizing the initial localization model and iterating accordingly. It should be understood that the same process applies to the iterated initial recognition model, allowing either the initial or iterated model to guide the sampling strategy. The threshold can also be adjusted as needed. Therefore, candidate images with higher thresholds can be used to optimize or iterate the initial localization model. Similarly, candidate images that are more similar to the original sample images can be used to optimize or iterate the initial localization model, improving model accuracy.

[0048] To improve the accuracy of both the recognition and localization models, the initial recognition model can be trained using the SimCLR (Multi-Source Clustering) method. This method involves learning representations by maximizing the protocol between different augmented views of the same data example through contrastive loss in the latent space.

[0049] Specifically, the method 100 further includes: during model training, locating multiple iterated candidate sample images corresponding to the sample images from the sample videos using the iterated initial localization model; inputting the multiple iterated candidate sample images into the initial recognition model using the iterated initial localization model, and finding iterated candidate sample images with the same objects as the sample images from the iterated candidate sample images; and performing object type aggregation training on the initial recognition model based on the found iterated candidate sample images to complete the training of the initial model.

[0050] As mentioned above, the initial localization model is optimized and iterated based on the training data. The iterated initial localization model is as follows: Figure 2As shown, more precise iterative candidate sample images can be determined from frame image 208 (the frame images are differentiated here to distinguish them from the previous training process). These iterative candidate sample images are then output and input into the initial recognition model 204. Following the method described above, iterative candidate sample images with the same objects as the sample images are found from the iterative candidate sample images. Then, the initial recognition model, according to the SimCLR method, can use the found iterative candidate sample images as a class for clustering training. Alternatively, it can use other types, i.e., those determined by similarity, for clustering training. This continuously improves the iterative or training initial recognition model, increasing its accuracy. Simultaneously, it improves the initial localization model after iteration.

[0051] This completes the model training. The trained search model can find the corresponding target image from the search video or image without being limited by the category of the object. Furthermore, the model training process does not require unlabeled training data, reducing manual costs and time. This search model does not require pre-detecting the object from the search image or video, or ensuring that the frame in the search image or video contains only one object (i.e., the object dominates the frame). This achieves instance-level search, where the search target is the same individual instance rather than the same category. It is not limited to videos from different sources and can achieve cross-camera or webcam search.

[0052] 102: Using the localization model in the search model, multiple candidate images corresponding to the instance image are located from the video to be searched.

[0053] Each candidate image contains one object instance.

[0054] For example, as described above, after the trained search model receives the image to be searched and the video to be searched (or multiple images), it inputs them into the localization model, such as the localizer. Through this localization model, such as the localizer, multiple candidate images, such as candidate umbrella images, are located corresponding to the instance image, such as the instance image of the umbrella.

[0055] 103: Using the recognition model in the search model, identify the candidate image that is the same as the object to be searched in the instance image from multiple candidate images, and use it as the target image.

[0056] The target image is an instance image containing the object to be searched.

[0057] In addition, the method also includes: inputting multiple candidate images and instance images into the recognition model in the search model through the localization model.

[0058] It should be noted that this process is similar to the training process described above, so it will not be repeated here. Simply put, the localization model locates candidate images from the video to be searched based on the image to be searched. Then, the located candidate images and the image to be searched are input into the recognition model. The recognition model then uses the image to be searched to retrieve the corresponding target image from the candidate images. Both the target image and the image to be searched contain the same object, such as an umbrella.

[0059] In addition, to improve search accuracy, user selections can be taken into account. Specifically, the method 100 also includes: providing at least one target image found in the search; receiving a target image selected by the user from the at least one target image, and searching for other target images from the video to be searched based on the selected target image, an instance image of the object to be searched, and a search model; and providing the other target images and at least one target image to the user.

[0060] The process of finding other target images from the video to be searched, based on the selected target image, the instance image of the object to be searched, and the search model, includes: obtaining feature data of the selected target image and feature data of the instance image of the object to be searched through the search model; obtaining corrected feature data based on the two feature data; and finding the corresponding target image based on the corrected feature data and the search model.

[0061] For example, as described above, the server can temporarily output the search model's results, i.e., the target image, and display it to the user's computer. The user can then select a target image that shares the same object as the image being searched. The computer can then return the selected target image to the server, or return the identifier of the selected target image, such as its name or ID. The server then determines the target image and obtains its feature vector and the feature vector of the image being searched using the search model. These two feature vectors are then added together and divided by 2 to obtain a corrected feature vector, which can be determined by the search model. The modified feature vector is then used to determine other target images based on the localization and recognition models within the search model. This will not be elaborated further. It is only noted that the corrected feature vector is used as a basis, or as the feature vector of a new image to be searched, for locating the video to be searched and selecting candidate images.

[0062] In addition, the method 100 further includes: providing at least one target image found; receiving a target image selected by the user from the at least one target image, and training the localization model and the recognition model in the search model based on the selected target image and the at least one target image, thereby updating the search model, wherein the selected target image has the same search object as the instance image.

[0063] As described above, the server can return the final target image to the user's computer for display. The user can then select the correct target image from the displayed images, or choose an incorrect one. The selected target image is then sent to the server via the user's browser. The server can then train the localization and recognition models within the search model based on the selected target image, thereby updating the models. The training method is similar to existing technologies and will not be elaborated further.

[0064] In addition, the method 100 further includes: acquiring at least one target image that has been finally searched, generating a corresponding video based on the at least one target image, and providing the video to the user so that the user can watch the at least one target image through the video.

[0065] For example, as described above, after the server finds at least one target image, it can generate a video from the target images, such as by sequentially creating video frames, which could be a short video. The server then sends the video to the user's computer for the user to view.

[0066] For the object instance search scenario: When searching for any object instance, for a user's search target Q, such as an instance image of an umbrella, the user can access the server's search page through a computer's web browser. Based on this search page, the user can upload the instance image, and can also upload the video to be searched, or select or specify the video source. Then, the user can click the search command or button on the computer interface to send the instance image and the video or video source to be searched to the server via a search request. After receiving the request, the server calls the search model, obtains the instance image and the video to be searched, and inputs the video and instance image into the search model to start the search task. Candidate images are obtained through the localization model in the search model, and the target image is identified through the recognition model in the search model. The entire search task takes N minutes to complete, and the complete search can obtain the target images of the correct results (p1, p2, p3, p4). Due to factors such as lighting occlusion, the target images of the correct answers (p5, p6) are missed. A progressive search approach can be adopted, which means that during the search process (e.g., at the first minute), the user is presented with some target images (e.g., p1, m1, m2) as search results, and these search results are sent to the user's computer (the model can continue searching). The user can select the target image p1 that they believe is correct from the search interface and add it to the search model. The user can also send the selected target image p1 to the server. After receiving it, the server merges the target Q and p1 together and searches simultaneously, obtaining the mean of the sum of the corresponding feature vectors. The search model can then use this mean to search for the corresponding results. Since the target image p1 can supplement information such as viewing angle and lighting, the previously missing target images (p5, p6) can eventually be found, improving search accuracy. The entire search process is dynamic rather than producing results immediately. The search results can be updated by displaying them, allowing the user to choose at any time to add the results they believe are correct to the search process, thus improving subsequent search accuracy. It should be noted that the video source can also be left unspecified, directly searching through the video sources stored on the server or preset, i.e., the default video source or default video resources.

[0067] The user interaction process described above did not involve a single complete search; information was added during the search process, resulting in only one search at the end. In contrast, other common graph search processes are based on vector computation, where the complete search ends quickly, and a second search is performed after the user adds information, resulting in two complete search rounds.

[0068] In addition to the search scenarios mentioned above, it can also be used for other application scenarios, such as searching for target individuals in security scenarios.

[0069] Specifically, the process involves acquiring an instance image of the person to be searched and a video of the person to be searched, and inputting the instance image and video into a pre-defined search model. Using the localization model within the search model, multiple candidate images corresponding to the instance image are located from the video of the person to be searched. Each candidate image contains a person instance. Using the recognition model within the search model, a candidate image identical to the person to be searched in the instance image is identified from the multiple candidate images and used as the target person image. The target person image is an instance image containing the person to be searched.

[0070] It should be noted that the specific implementation method is similar to that described above. The only difference is that when searching for a person, the person to be searched can be their face or their entire body, such as their back or side profile.

[0071] Other content not elaborated here can be found in Method 100 above, and will not be repeated here.

[0072] Based on the search scenarios described above, the scenario for searching for a target person here is similar and will not be repeated. Only the following explanation is provided: For a user's search for a specific image of a target person, the user can access the server's search page through a computer's web browser. Based on this search page, the user can upload the image, and can also upload the video to be searched, or select or specify the video source. The user can then click the search command or button on the computer interface to send the image and the video or video source to be searched to the server via a search request. Upon receiving the request, the server invokes the search model, obtains the image and the video (which can be obtained from a security camera), and inputs them into the search model to initiate the search task. The search model uses a localization model to obtain candidate images and a recognition model to identify the target image. Similarly, a progressive search can also be used, which will not be elaborated upon here.

[0073] Furthermore, this technology can also be applied to the retrieval of images of target individuals or related videos in entertainment scenarios. For example, users can access the server's search page through a computer's web browser or on a video website. Based on the search bar of this search page or video website, users can upload an image (only the image of the person) or the person's name. Then, users can click the search command or button on the computer interface to send the image or name to the server via a search request. After receiving the request, the server retrieves the corresponding video resources, such as movie videos, TV series, etc., from its database as the videos to be searched. It then calls the search model. If the server receives a person's name, it can find the corresponding image of that person using that name. After obtaining the image of the person and the video to be searched, the video and the example image are input into the search model to start the search task. The search model uses a localization model to obtain candidate images and a recognition model to identify the target person image. After identification, the source of the target person image can be determined, such as which video resource it comes from, such as which movie or TV series it belongs to. The identified video resources are then sent to the computer with identifiers such as names, allowing users to select and watch videos based on these names. Alternatively, the system can generate corresponding videos from found target images and return them to the computer. Or, after finding the target image, the corresponding image from the source video resource can be processed, such as by adding mosaics, overlaying images, or adding animations to mask it. For example, if a user only wants to watch videos of car races, the server can find the corresponding scene using the methods described above, then find the corresponding target image from the video resources based on that scene, and assemble the target image into a video to return to the user's computer for viewing. Similarly, a progressive search for images of people can also be used, which will not be elaborated further.

[0074] For other application scenarios: in education, such as searching for teaching materials, question content (homework or exam questions), handwritten content, and various cards, certificates, tickets, and receipts; and in the medical field, it can be applied to searching for patient medical images, medical records, and images related to a specific disease.

[0075] Specifically, the process involves acquiring instance images of the teaching materials or questions to be searched, as well as the video to be searched. These instance images and the video (and / or the image to be searched) are then input into a pre-defined search model. Using the localization model within the search model, multiple candidate images corresponding to the instance images are located from the video to be searched. These candidate images can be teaching materials or questions, each containing an instance, such as an instance of the teaching materials or questions. Using the recognition model within the search model, a candidate image identical to the teaching materials or questions to be searched in the instance images is identified from these multiple candidate images. This target image is then used as the target image, which is an instance image containing the teaching materials or questions to be searched.

[0076] The specific implementation method here is the same as that described above, so it will not be repeated here. It should be noted that, in addition to teaching materials or question content, the search can also include handwritten content, various cards, certificates, tickets, receipts, etc. Based on this, the object to be searched can also be handwritten content, various cards, certificates, tickets, receipts, etc., and its instance images can also be included.

[0077] For example, a user can access the server's search page through a computer's web browser. Based on this search page, they can upload an example image of the assignment question. The user can then click the search command or button on the computer interface to send the example image to the server via a search request. Upon receiving the request, the server invokes the search model and retrieves the example image and the video to be searched (which can be a pre-set video from the server's database, such as a problem-solving video or an image of the solution; one image can contain multiple solutions). The server then inputs the video and example image into the search model to initiate the search task. The search model uses a localization model to obtain candidate assignment question images and a recognition model to identify the target assignment question image. Finally, the corresponding video or image is sent to the user's computer for viewing. Similarly, a progressive search can be used, which will not be elaborated further.

[0078] Specifically, the process involves acquiring an instance image of the medical information to be searched and a video to be searched, and inputting the instance image and the video or image to be searched into a preset search model. Using the localization model within the search model, multiple candidate medical information images corresponding to the instance image are located from the video to be searched. Each candidate image contains a medical information instance. Using the recognition model within the search model, a candidate medical information image identical to the medical information to be searched in the instance image is identified from the multiple candidate medical information images. This candidate medical information image is then used as the target medical information image, which is the instance image containing the medical information to be searched.

[0079] The specific implementation method here is the same as that described above, so it will not be repeated. It should be noted that: medical information refers to information related to medical diagnosis and treatment, which may include, but is not limited to, medical images, medical records, and information related to a specific disease. Example images can be images corresponding to medical images, medical records, or information related to a specific disease. They can also be videos composed of these images.

[0080] For example, a user can access the server's search page through a computer's web browser. Based on this search page, they can upload an example image of the medical record. The user can then click the search command or button on the computer interface to send the example image to the server via a search request. Upon receiving the request, the server invokes the search model and retrieves the example image and the video or image to be searched (pre-set videos, such as medical record videos or images, can be obtained from the server's database; an image can contain multiple medical records). The server then inputs the video or image to be searched and the example image into the search model, enabling the search model to begin its search task. The search model uses a localization model to obtain candidate medical record images and a recognition model to identify the target medical record image. Finally, the corresponding target medical record image is sent to the user's computer for viewing. Similarly, a progressive search can be used, which will not be elaborated further.

[0081] For cloud service scenarios, such as Figure 3 As shown, the second device 302 provides external cloud services. Specifically, the second device 302 acquires an instance image of the object to be searched and a video to be searched, and inputs the instance image and the video to be searched into a preset search model; through the localization model in the search model, it locates multiple candidate images corresponding to the instance image from the video to be searched, each candidate image containing an object instance; through the recognition model in the search model, it identifies a candidate image that is the same as the object to be searched in the instance image from the multiple candidate images, and uses it as the target image, which is an instance image containing the object to be searched.

[0082] In response to a user's request, the first device 301 sends a search request to the second device 302. The second device 302 performs the above steps and then returns the target image as the search result to the first device 301.

[0083] The first device 301 can be a smart device, such as a computer. The second device 302 can be a server device, such as a cloud server.

[0084] Among them, the user can be a client user, such as a web browser user, who can directly trigger the search process through the browser.

[0085] That is, the second device 302 may receive a call request for the search service and use the processing resources corresponding to the search service to perform the following steps: obtain an instance image of the object to be searched in the call request; locate multiple candidate images corresponding to the instance image from the video to be searched through the positioning model in the search model, each candidate image containing an object instance; identify the candidate image that is the same as the object to be searched in the instance image from the multiple candidate images through the recognition model in the search model, and use it as the target image, the target image being an instance image containing the object to be searched, and return the target image as the search result.

[0086] The user who triggers this call request can be a developer user.

[0087] Other content not described herein can be found in the content of method 100 mentioned above, and will not be repeated here.

[0088] For client users, the specific implementation method can be:

[0089] As mentioned above, client users can access services provided by the cloud server through a first device (301), such as a web browser on a computer. This could be a search service, such as a search page on a web browser. Figure 4The search page 400 is shown. Users can upload the instance image based on this search page 400, and can also upload a video to be searched, or select or specify a video source. For example, a user can click the "Upload Image" button 402 on the computer's search page 400 to upload an instance image from their local machine. After uploading, the uploaded instance image can be displayed in the instance image display area 401 so the user can verify if an upload was made incorrectly. Secondly, users can also click the "Upload Video" button 404 on the computer's search page 400 to upload the corresponding video to be searched from their local machine. If no video is uploaded, the video can be found from the default video source on the server. The uploaded video's name can be displayed in the uploaded video list 403. Then, the user can click the search command or button 405 on the computer's search page 400 to send the instance image and the video or video source to be searched to a second device 302, such as a cloud server, via a search request or a request. After receiving the request, the cloud server invokes the search model, retrieves the instance image and the video to be searched, and inputs the video and instance image into the search model to initiate the search task. The search model uses a localization model to obtain candidate images and a recognition model to identify the target image. The entire search task takes N minutes to complete, and a full search yields the target images of the correct results (p1, p2, p3, p4). These target images are then returned to the search page 400 in the web browser for display. Similarly, a progressive search can be used, which will not be elaborated further.

[0090] For developers, the specific implementation method can be:

[0091] Developers can obtain information about the API call and the call address, thereby triggering the search service. In addition, the second device 302 can determine the request method and return type of the search service. It can also identify the corresponding request instance and provide this information, such as the request method, return type, and request instance, externally. This information corresponds to or is associated with the called service, i.e., the search service, allowing the location of the corresponding search service. The second device 302 can record the above information in the form of API documentation and provide it to the relevant developer users.

[0092] Based on the information above, the developer can send a request to the second device 302 via the first device 301, carrying an instance image. Upon receiving the request, the second device 302 invokes the search service according to the call address and interface specified in the request, thereby performing a search based on the instance image. This search service can be implemented using a corresponding search model. The corresponding search results are then obtained (please refer to the previous text for specific search implementation details, which will not be repeated here). The second device 302 then returns the search results—the searched image or its corresponding video frame—to the first device 301 for display.

[0093] In addition to the above-mentioned call addresses and API calls, the search service can also be invoked in the following ways: For example, as mentioned earlier, developers can use a web browser to input call information in the form of a request instance. This includes the name of the pre-defined API (Application Programming Interface) (e.g., the API for searching instance images), the call address (e.g., a URL (Uniform Resource Locator) address), the corresponding request method (e.g., POST), the return type (e.g., JSON), and the instance image. Then, the user triggers the call request, and the computer responds to the user's trigger operation by sending the above information to the cloud server via the call request. Upon receiving the request, the cloud server, based on the call address and the name of the pre-defined API in the call request, allows the use of the corresponding search service, invokes the corresponding search service, and inputs the request parameters, such as the instance image, into the search model of the corresponding search service (the specific search implementation is described above and will not be repeated here), obtaining the final search results. The search results are then returned to the user's computer browser in JSON format for display.

[0094] In one possible design, embodiments of this application propose a computing device, such as a server. Figure 5 As shown, the device 500 may include: a memory 501 and a processor 502;

[0095] Memory 501 is used to store computer programs.

[0096] The processor 502 is configured to execute a computer program for: acquiring an instance image of the object to be searched and a video to be searched; inputting the instance image and the video to be searched into a preset search model; locating multiple candidate images corresponding to the instance image from the video to be searched using a localization model in the search model, wherein each candidate image contains an object instance; and identifying, using a recognition model in the search model, a candidate image identical to the object to be searched in the instance image from the multiple candidate images, which is used as the target image, wherein the target image is an instance image containing the object to be searched.

[0097] In addition, the processor 502 is also used to: provide at least one target image found in the search; receive a target image selected by the user from at least one target image, and search for other target images from the video to be searched based on the selected target image, an instance image of the object to be searched, and a search model; and provide the other target images and at least one target image to the user.

[0098] In addition, the processor 502 is also used to: provide at least one target image found in the search; receive a target image selected by the user from at least one target image, and train the localization model and the recognition model in the search model based on the selected target image and at least one target image, thereby updating the search model, wherein the selected target image and the instance image have the same search object.

[0099] In addition, the processor 502 is also used to: acquire at least one target image that has been finally searched, generate a corresponding video based on the at least one target image, and provide the video to the user so that the user can watch the at least one target image through the video.

[0100] In addition, the processor 502 is also used to: input multiple candidate images and instance images into the recognition model in the search model through the localization model.

[0101] Specifically, processor 502 is used to: obtain feature data of the selected target image and feature data of the instance image of the object to be searched through a search model; obtain corrected feature data based on the two feature data, and find the corresponding target image according to the corrected feature data and the search model.

[0102] In addition, the processor 502 is also configured to: acquire training data, including sample images and sample videos, the training data having no labeled data, and input the training data into an initial model; input the training data into an initial localization model in the initial model to locate multiple candidate sample images corresponding to the sample images from the sample videos; input the multiple candidate sample images and the sample images into an initial recognition model in the initial model through the initial localization model, and find candidate sample images with the same objects as the sample images from the candidate sample images through the initial recognition model; and train the initial localization model based on the found candidate sample images and the sample images to complete the training of the initial model.

[0103] The initial localization model and the initial recognition model are obtained after training based on single-target tracking training data.

[0104] In addition, the processor 502 is also used to: during model training, locate multiple iterated candidate sample images corresponding to the sample images from the sample videos through the iterated initial localization model; input the multiple iterated candidate sample images into the initial recognition model through the iterated initial localization model, and find the iterated candidate sample images with the same objects as the sample images from the iterated candidate sample images; and perform object type aggregation training on the initial recognition model based on the found iterated candidate sample images to complete the training of the initial model.

[0105] Specifically, processor 502 is used to: determine the similarity between a sample image and a candidate sample image through an initial recognition model; and, based on the similarity, find candidate sample images that have the same object as the sample image, so as to train the initial localization model based on the found candidate sample images and the sample image.

[0106] In addition, embodiments of the present invention provide a computer storage medium in which a computer program, when executed by one or more processors, causes one or more processors to implement... Figures 1-2 The steps of a method for searching object instances are described in the method embodiment.

[0107] This application provides a schematic flowchart of a product search method according to an exemplary embodiment. The method 600 provided in this embodiment is executed by a computing device, such as a server. The method 600 includes the following steps:

[0108] 601: Obtain an instance image of the product to be searched, as well as a video and / or image of the product to be searched, and input the instance image and the video and / or image of the product to be searched into a preset search model.

[0109] 602: Using the localization model in the search model, locate multiple candidate product images corresponding to the instance image from the product video and / or product image to be searched.

[0110] Each candidate product image contains a single product instance.

[0111] 603: Using the recognition model in the search model, identify the candidate product image that is the same as the product to be searched in the instance image from multiple candidate product images, and use it as the target product image.

[0112] The target product image is an instance image containing the video and / or product to be searched.

[0113] It should be noted that the specific implementation method of this method 600 is similar to the implementation method described above, and will not be repeated here. Only it should be noted that the object to be searched is the product to be searched.

[0114] For example, a user can access the e-commerce purchase page on a server through a computer's web browser (or a mobile e-commerce application), and then upload a sample image of a product from the search box on that page. The user can then click the search command or button on the computer interface to send the sample image to the server as a search request. Upon receiving the request, the server invokes the search model, retrieves the sample image and the video or image to be searched, and inputs these into the search model to initiate the search task. The search model uses a localization model to obtain candidate product images and a recognition model to identify the target product image. Then, a short video or product footage set can be generated from the target product image and sent to the user's computer for viewing. Similarly, a progressive search can also be used, which will not be elaborated further.

[0115] In addition, the method 600 also includes: generating a corresponding set of product materials based on the target product image, and providing the set of product materials to the user so that the user can process the set of product materials.

[0116] Since similar implementation methods have already been described above, they will not be repeated here. It will only be noted that after receiving short videos or product material sets through a computer, users can view or process them, such as performing graphic design or editing on images or videos.

[0117] For any parts not described in the embodiments of this application, please refer to the method described in the preceding method 100, and will not be repeated here.

[0118] Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 101, 102, 103, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0119] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0120] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0121] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable multimedia data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable multimedia data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0122] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable multimedia data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0123] These computer program instructions may also be loaded onto a computer or other programmable multimedia data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0124] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0125] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0126] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for searching object instances, characterized in that, include: Obtain an instance image and a video of the object to be searched, and input the instance image and the video into a preset search model; Using the localization model in the search model, multiple candidate images corresponding to the instance image are located from the video to be searched, and each candidate image contains an object instance. The search model uses a recognition model to identify candidate images that are the same as the object to be searched in the instance images from the plurality of candidate images, and uses them as target images. The target images are instance images that contain the object to be searched. The training method of the search model includes: acquiring training data, which includes sample images and sample videos, without labeled data; inputting the training data into an initial model; inputting the training data into an initial localization model within the initial model to locate multiple candidate sample images corresponding to the sample image from the sample video; inputting the multiple candidate sample images and the sample image into an initial recognition model within the initial model through the initial localization model; determining the similarity between the sample image and the candidate sample images through the initial recognition model; finding candidate sample images with the same object as the sample image from the candidate sample images based on the similarity; training the initial localization model based on the found candidate sample images and the sample image to iterate the model and complete the training of the initial model. The training method further includes: during model training, locating multiple iterated candidate sample images corresponding to the sample image from the sample video using the iterated initial localization model; inputting the multiple iterated candidate sample images into the initial recognition model using the iterated initial localization model, and finding iterated candidate sample images with the same object as the sample image from the iterated candidate sample images; and performing object type aggregation training on the initial recognition model based on the found iterated candidate sample images to complete the training of the initial model.

2. The method according to claim 1, characterized in that, The method further includes: Provide at least one target image found in the search; The system receives a target image selected by the user from the at least one target image, and searches for other target images from the video to be searched based on the selected target image, an instance image of the object to be searched, and the search model. The selected target image has the same object to be searched as the instance image. The other target images and the at least one target image are provided to the user.

3. The method according to claim 1, characterized in that, The method further includes: Provide at least one target image found in the search; The system receives a target image selected by the user from the at least one target image, and trains the localization model and the recognition model in the search model based on the selected target image and the at least one target image, thereby updating the search model. The selected target image has the same search object as the instance image.

4. The method according to claim 1 or 2, characterized in that, The method further includes: Obtain at least one target image found in the final search, and generate a corresponding video based on the at least one target image; The video is provided to the user so that the user can view the at least one target image through the video.

5. The method according to claim 1, characterized in that, The method further includes: The location model is used to input the multiple candidate images and the instance images into the recognition model in the search model.

6. The method according to claim 2, characterized in that, The step of finding other target images from the video to be searched based on the selected target image, the instance image of the object to be searched, and the search model includes: The search model is used to obtain feature data of the selected target image and feature data of the instance image of the object to be searched. Based on the two feature data, the corrected feature data is obtained, and the corresponding target image is found according to the corrected feature data and the search model.

7. A method for searching for goods, characterized in that, include: Obtain an instance image of the product to be searched, as well as a video and / or image of the product to be searched, and input the instance image and the video and / or image of the product to be searched into a preset search model; Using the localization model in the search model, multiple candidate product images corresponding to the instance image are located from the product video and / or product image to be searched, and each candidate product image contains a product instance. The search model identifies candidate product images that are identical to the product to be searched in the instance images from the plurality of candidate product images, and uses these as target product images. The target product image is an instance image containing the video and / or product to be searched. The training method of the search model includes: acquiring training data, which includes sample images and sample videos, without labeled data; inputting the training data into an initial model; inputting the training data into an initial localization model within the initial model to locate multiple candidate sample images corresponding to the sample image from the sample video; inputting the multiple candidate sample images and the sample image into an initial recognition model within the initial model through the initial localization model; determining the similarity between the sample image and the candidate sample images through the initial recognition model; finding candidate sample images with the same object as the sample image from the candidate sample images based on the similarity; training the initial localization model based on the found candidate sample images and the sample image to iterate the model and complete the training of the initial model. The training method further includes: during model training, locating multiple iterated candidate sample images corresponding to the sample image from the sample video using the iterated initial localization model; inputting the multiple iterated candidate sample images into the initial recognition model using the iterated initial localization model, and finding iterated candidate sample images with the same object as the sample image from the iterated candidate sample images; and performing object type aggregation training on the initial recognition model based on the found iterated candidate sample images to complete the training of the initial model.

8. The method according to claim 7, characterized in that, The method further includes: Based on the target product image, a corresponding set of product materials is generated and provided to the user so that the user can process the set of product materials.

9. A computing device, comprising: Memory, processor; The memory is used to store computer programs; The processor is configured to execute the computer program for: Obtain an instance image and a video of the object to be searched, and input the instance image and the video into a preset search model; Using the localization model in the search model, multiple candidate images corresponding to the instance image are located from the video to be searched, and each candidate image contains an object instance. The search model uses a recognition model to identify candidate images that are the same as the object to be searched in the instance images from the plurality of candidate images, and uses them as target images. The target images are instance images that contain the object to be searched. The training method of the search model includes: acquiring training data, which includes sample images and sample videos, without labeled data; inputting the training data into an initial model; inputting the training data into an initial localization model within the initial model to locate multiple candidate sample images corresponding to the sample image from the sample video; inputting the multiple candidate sample images and the sample image into an initial recognition model within the initial model through the initial localization model; determining the similarity between the sample image and the candidate sample images through the initial recognition model; finding candidate sample images with the same object as the sample image from the candidate sample images based on the similarity; training the initial localization model based on the found candidate sample images and the sample image to iterate the model and complete the training of the initial model. The training method further includes: during model training, locating multiple iterated candidate sample images corresponding to the sample image from the sample video using the iterated initial localization model; inputting the multiple iterated candidate sample images into the initial recognition model using the iterated initial localization model, and finding iterated candidate sample images with the same object as the sample image from the iterated candidate sample images; and performing object type aggregation training on the initial recognition model based on the found iterated candidate sample images to complete the training of the initial model.

10. A computer-readable storage medium storing a computer program, characterized in that, When a computer program is executed by one or more processors, it causes the one or more processors to perform the steps of the method according to any one of claims 1-8.

Images