A method and device for optimizing effect of image-text retrieval

Abstract

Embodiments of the present application provide a kind of picture-text retrieval effect optimization method and device.Therein, the method comprises: in response to the input search information, first interface is shown by pre-training model based on the search information retrieved first search result;In the case where the optimization mode is opened, at least one first image in the first interface shown by the user input label operation for the first image is received;In response to the labeling operation, the first label result is configured for the at least one first image and the labeling result is shown;The first label result is used to indicate that the first image matches or does not match the search information;In response to fine-tuning operation, the pre-training model is trained according to the at least one first image to obtain target model;Second search result retrieved by the target model based on the search information is shown by second interface.Can improve picture-text retrieval effect.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and apparatus for optimizing image and text retrieval performance. Background Technology

[0002] Current text retrieval solutions have been widely applied in video target retrieval scenarios, including community security, campus management, hospital monitoring, and shopping mall surveillance, improving the efficiency of quickly querying targets such as people and vehicles, and providing efficient retrieval tools for maintenance personnel and related users.

[0003] However, existing text retrieval solutions still have significant shortcomings in semantic fine-grained differentiation, leading to poor accuracy in retrieval results for certain specific terms. Specifically, semantically similar target descriptions with clear differences in actions or states are prone to mismatches, causing the search results to deviate from the user's actual needs. For example, when a user enters the precise description "person holding a child," the search results often include unexpected targets such as "person leading a child" or "person pushing a stroller." Similarly, when searching for "construction worker wearing a safety helmet," "passerby wearing a baseball cap" may be mistakenly identified as the target result. Such mismatch problems not only increase the time cost for users to filter valid information but may also affect the rapid location of targets in scenarios with high requirements for search accuracy, such as security tracking and event tracing. Therefore, there is an urgent need to provide a technical solution that can improve the accuracy of user search results. Summary of the Invention

[0004] The purpose of this application is to provide a method and apparatus for optimizing image and text retrieval performance, thereby providing users with a convenient way to optimize models and enable them to fine-tune image and text retrieval performance according to their own needs. The specific technical solution is as follows:

[0005] In a first aspect of this application, a method for optimizing image and text retrieval performance is provided, the method comprising:

[0006] In response to the input search information, a first search result retrieved by the pre-trained model based on the search information is displayed through a first interface; wherein, the first search result includes at least one first image;

[0007] When the optimization mode is enabled, the annotation operation for at least one of the first images displayed in the first interface is received from the user.

[0008] In response to the annotation operation, a first annotation result is configured for the at least one first image and the annotation result is displayed; the first annotation result is used to indicate whether the first image matches or does not match the search information.

[0009] In response to the fine-tuning operation, the pre-trained model is trained based on the at least one first image to obtain the target model;

[0010] The second interface displays the second search results retrieved by the target model based on the search information; wherein the second search results include at least one second image.

[0011] In one possible embodiment, in response to a fine-tuning management instruction, at least one of the following is displayed through a third interface: search information, the progress of the fine-tuning, the number of positive samples, the number of negative samples, all or part of the content of the positive samples, and all or part of the content of the negative samples; wherein, the number of positive samples is the number of images in the at least one first search result whose labeled results represent images that match the search information, and the number of negative samples is the number of images in the at least one first search result whose labeled results represent images that do not match the search information.

[0012] In one possible embodiment, after training the pre-trained model to obtain the target model based on the at least one first image in response to the fine-tuning operation, the method further includes:

[0013] In response to a retrieval command, determine the state of the target model;

[0014] If the target model is set to enabled, then the retrieval is performed using the target model;

[0015] If the target model is set to disabled, the retrieval is performed using the pre-trained model.

[0016] In one possible embodiment, the method further includes:

[0017] In response to the adjustment instruction for the search information, the search information is adjusted to obtain new search information, and the process returns to execute the search information in response to the input, displaying the steps of the pre-trained model retrieving the first search result based on the search information through the first interface.

[0018] In one possible embodiment, the method further includes:

[0019] In response to the model export command, display all or part of the historically input retrieval information;

[0020] In response to a selection instruction for any of the search information, the target model file corresponding to the selected search information and the first search result are exported.

[0021] In one possible embodiment, the method further includes:

[0022] In response to a filtering result instruction, at least one third image is displayed through a fourth interface, wherein the third image includes: the images remaining in the first search result other than the second image.

[0023] In one possible embodiment, the method further includes:

[0024] In response to an editing instruction for any of the stated annotation results, the annotation results are edited.

[0025] In one possible embodiment, the method further includes:

[0026] In response to the uploaded fourth image; the fourth image is labeled with a second annotation result, which indicates whether the fourth image matches or does not match the search information;

[0027] Using the first image and the fourth image as new first images, return to the step of training the pre-trained model based on the at least one first image to obtain the target model.

[0028] In a second aspect of this application, an image and text retrieval performance optimization device is provided, the device comprising:

[0029] The retrieval module is used to respond to input retrieval information and display a first retrieval result retrieved by a pre-trained model based on the retrieval information through a first interface; wherein the first retrieval result includes at least one first image;

[0030] The display module is used to receive user input for annotation operations on at least one of the first images displayed on the first interface when the optimization mode is enabled.

[0031] A labeling module is configured to, in response to the labeling operation, configure a first labeling result for the at least one first image and display the labeling result; the first labeling result is used to indicate whether the first image matches or does not match the search information.

[0032] A fine-tuning module is used to train the pre-trained model to obtain a target model based on the at least one first image in response to a fine-tuning operation.

[0033] The re-retrieval module is used to display the second retrieval results obtained by the target model based on the retrieval information through a second interface; wherein the second retrieval results include at least one second image.

[0034] In one possible embodiment, the device further includes a management module, configured to, in response to a fine-tuning management command, display at least one of the following via a third interface: search information, the progress of the fine-tuning, the number of positive samples, the number of negative samples, all or part of the content of the positive samples, and all or part of the content of the negative samples; wherein the number of positive samples is the number of images in the at least one first search result whose labeled results indicate a match with the search information, and the number of negative samples is the number of images in the at least one first search result whose labeled results indicate a mismatch with the search information.

[0035] In one possible embodiment, the retrieval module is further configured to determine the state of the target model in response to a retrieval instruction after training the pre-trained model based on the at least one first image to obtain the target model in response to a fine-tuning operation.

[0036] If the target model is set to enabled, then the retrieval is performed using the target model;

[0037] If the target model is set to disabled, the retrieval is performed using the pre-trained model.

[0038] In one possible embodiment, the retrieval module is further configured to adjust the retrieval information in response to an adjustment instruction for the retrieval information, obtain new retrieval information, and return to execute the steps of retrieving the first retrieval result based on the retrieval information by the pre-trained model through a first interface.

[0039] In one possible embodiment, the apparatus further includes an export module for displaying all or part of the historically input retrieval information in response to a model export instruction.

[0040] In response to a selection instruction for any of the search information, the target model file corresponding to the selected search information and the first search result are exported.

[0041] In one possible embodiment, the display module is further configured to display at least one third image through a fourth interface in response to a filter result instruction, wherein the third image includes: the remaining images in the first search result other than the second image.

[0042] In one possible embodiment, the annotation module is further configured to edit the annotation result in response to an editing instruction for any of the annotation results.

[0043] In one possible embodiment, the optimization module is further configured to respond to an uploaded fourth image; the fourth image is labeled with a second annotation result, the second annotation result being used to indicate whether the fourth image matches or does not match the search information;

[0044] Using the first image and the fourth image as new first images, return to the step of training the pre-trained model based on the at least one first image to obtain the target model.

[0045] This application also provides a computer program product containing instructions that, when run on a computer, causes the computer to execute any of the above-described image and text retrieval performance optimization methods.

[0046] Beneficial effects of the embodiments in this application:

[0047] This application provides a method and apparatus for optimizing image and text search performance. In response to input search information, it displays a first search result retrieved by a pre-trained model based on the search information through a first interface; wherein the first search result includes at least one first image; when optimization mode is enabled, it receives user input for annotation operations on at least one of the first images displayed on the first interface; in response to the annotation operation, it configures a first annotation result for the at least one first image and displays the annotation result; the first annotation result indicates whether the first image matches or does not match the search information; in response to a fine-tuning operation, it trains the pre-trained model based on the at least one first image to obtain a target model; and displays a second search result retrieved by the target model based on the search information through a second interface; wherein the second search result includes at least one second image. This allows users to annotate search results and fine-tune the model used for retrieval based on the annotated search results, thereby optimizing the search performance.

[0048] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0050] Figure 1 A flowchart illustrating a method for optimizing the performance of image and text retrieval provided in this application;

[0051] Figure 2aA schematic diagram of a first interface provided for this application;

[0052] Figure 2b A schematic diagram illustrating another style of the first interface provided in this application;

[0053] Figure 2c A schematic diagram illustrating yet another style of the first interface provided in this application;

[0054] Figure 3a A schematic diagram of a third interface style provided for this application;

[0055] Figure 3b A schematic diagram illustrating another style of third interface provided in this application;

[0056] Figure 4 A schematic diagram illustrating yet another type of third interface provided in this application;

[0057] Figure 5 A schematic diagram illustrating the style of a switchable second and fourth interface provided in this application;

[0058] Figure 6 An interactive schematic diagram illustrating a method for optimizing the image and text retrieval effect provided in this application;

[0059] Figure 7 An interactive diagram illustrating the process of importing a target model file, as provided in this application;

[0060] Figure 8 A schematic diagram of a device for optimizing image and text retrieval performance provided in this application;

[0061] Figure 9 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0062] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0063] The image and text retrieval performance optimization method provided in this application, such as Figure 1As shown, the method includes: Step S101, in response to the input search information, displaying the first search result retrieved by the pre-trained model based on the search information through a first interface; Step S102, when the tuning mode is enabled, receiving user input for annotation operations on at least one first image displayed on the first interface; Step S103, in response to the annotation operation, configuring first annotation results for at least one first image and displaying the annotation results; Step S104, in response to the fine-tuning operation, training the pre-trained model based on at least one first image to obtain a target model; Step S105, displaying the target model based on the retrieved second search result through a second interface. This embodiment allows users to annotate search results and fine-tune the model used for retrieval based on the annotated search results, thereby optimizing the search effect.

[0064] In step S101 of this embodiment, the first search result includes at least one first image. The first image can be an independent image, a video frame in a video, or a part of a video frame. For example, if the search information entered by the user is "a person wearing a black top", then the first image can be the images of all scenes containing a person wearing a black top retrieved from the preset image library. Alternatively, the first image can be the video frames of all scenes containing a person wearing a black top retrieved from the video file or video stream specified by the user. Furthermore, the first image can be a sub-image of the area where the person wearing a black top is located, extracted from the aforementioned video frames of the person wearing a black top.

[0065] The search results can be plain text or text plus an image. For example, the search results could be the text "a person wearing a black shirt" or the text "a person wearing a shirt with the pattern shown in the image" plus an image of that pattern. Understandably, some features are difficult to describe using only natural language, such as complex patterns or rare colors. Therefore, combining text with images helps users accurately describe the target they wish to search for (hereinafter referred to as the target).

[0066] In this article, the target can refer to a living being or an object, or it can refer to an event. For example, when the search query is "a person wearing a black shirt", the user wants to search for a person wearing a black shirt. When the search query is "a scene of someone falling down", the user wants to search for the event of falling down.

[0067] In step S102 of this embodiment, when the optimization mode is enabled, the first interface can be as follows: Figure 2aAs shown, users can perform annotation operations by clicking the "Error" or "Correct" buttons below each first image. Users can view these first images. When a first image contains the target, the user should click the "Correct" button to label the first image as a positive sample. When a first image does not contain the target, the user should click the "Error" button to label the first image as a negative sample. For example, if the search information is "a person wearing a black shirt," and one of the first images in the search results does not contain a person wearing a black shirt, then the user should click the "Error" button below that first image. Figure 2a The diagram shown is merely a schematic representation of one possible layout for the first interface. In other possible embodiments, the first interface may also have other layout styles. Figure 2a There are no restrictions on this.

[0068] Furthermore, to facilitate users switching the optimization mode on and off, it is also possible to... Figure 2b As shown, a toggle control 21 is set in the first interface. Users can turn the optimization mode on / off by clicking this toggle control 21. Furthermore, in other possible embodiments, the toggle control can also have other styles or be located in other positions on the first interface. Figure 2b No restrictions are imposed on this.

[0069] In step S103 of this embodiment, the fine-tuning operation is input by the user after completing the annotation of the first image, for example, as shown below. Figure 2c As shown, a fine-tuning control 22 is provided in the first interface. After the user completes the annotation of the first image, clicking the fine-tuning control 22 instructs the execution subject to perform step S104. This clicking operation of the fine-tuning control 22 is the fine-tuning operation. Furthermore, in other possible embodiments, the fine-tuning control can also be of other styles or located in other positions within the first interface. Figure 2c No restrictions are imposed on this.

[0070] In this article, completing the annotation of the first image can mean either completing the annotation of all first images or that the annotated first images can meet the user's needs. For example, suppose a total of 50 first images are retrieved, and the user believes that annotating 20 of them is sufficient to train the pre-trained model to be accurate enough. Then the user can input fine-tuning operations after annotating 20 first images.

[0071] Furthermore, in one possible embodiment, to allow users to know the number of positive and negative samples that have been labeled, thereby providing a more intuitive understanding of the labeling progress and enabling users to determine whether labeling needs to continue, such as... Figure 2c As shown, the first interface displays the number of labeled positive and negative samples. Figure 2cIn the example shown, “correct (15)” means that the number of labeled positive samples is 15, and “incorrect (21)” means that the number of labeled negative samples is 21.

[0072] In step S104 of this embodiment, the pre-trained model can be any model capable of image and text retrieval, including but not limited to multimodal large models, large language models, algorithm models, etc., or it can be a hybrid model formed by mixing the aforementioned multiple models. For example, the pre-trained model can be a multimodal pre-trained neural network (CLIP, Contrastive Language-Image Pre-training), or a hybrid model of a large language model and an algorithm model.

[0073] The sample data used in training the target model can include only the first labeled image, or it can include other samples, such as samples uploaded by the user.

[0074] In step S105 of this embodiment, the second search result includes at least one second image. The second interface can be an interface independent of the first interface, or it can be a sub-interface of the first interface. By displaying the second search result in the second interface, users can easily understand the performance of the trained target model, so that they can decide whether further fine-tuning is needed. Further fine-tuning can be performed by using the target model as a new pre-trained model, using the second search result as a new first search result, and performing fine-tuning according to the aforementioned steps S102 to S104.

[0075] In one possible embodiment, in Figure 1 Based on the example shown, the image and text retrieval performance optimization method provided in this application further includes: responding to a fine-tuning management instruction, displaying at least one of the following on a third interface: retrieval information, fine-tuning precision, number of positive samples, number of negative samples, all or part of the content of positive samples, and all or part of the content of negative samples; wherein, the number of positive samples is the number of images in at least one first retrieval result whose annotation indicates a match with the retrieval information, and the number of negative samples is the number of images in at least one first retrieval result whose annotation indicates a mismatch with the retrieval information. Using this embodiment allows users to easily understand the fine-tuning situation, thereby enabling them to more rationally plan subsequent retrieval operations.

[0076] The following will explain each piece of information displayed on the third interface:

[0077] Regarding search information: Displaying search information allows users to know which search terms have been fine-tuned, enabling them to have reasonable expectations about the accuracy of the search results. For example, if a user finds that the displayed search information includes "a person wearing a black top," it means that optimization has been performed on "a person wearing a black top." In this case, if the user re-enters the search information "a person wearing a black top" for a text and image search, the accuracy of the search results obtained will be relatively reliable.

[0078] Regarding the progress of fine-tuning: In this article, the progress of fine-tuning refers to the progress of training the pre-trained model. Training the pre-trained model takes time, and before training is complete, users can only use the pre-trained model for image and text retrieval. Therefore, the reliability of search results obtained before training is complete is low. By observing the progress of fine-tuning, users can understand the model currently being used for retrieval, thus having a reasonable expectation of the reliability of the search results.

[0079] Regarding the number of positive samples, the number of negative samples, all or part of the content of positive samples, and all or part of the content of negative samples: In this article, "all or part of the content of positive samples" refers to all or part of the positive samples, and "all or part of the content of negative samples" refers to all or part of the negative samples. By showing users all or part of the positive and negative samples, as well as the number of positive and negative samples, users can have reasonable expectations about the effect of fine-tuning, and thus have reasonable expectations about the accuracy of the search results based on the fine-tuning.

[0080] Figure 3a The image shows a possible third interface. Figure 3a The example shown not only corresponds to the search information (i.e. Figure 3a The table displays the search term content, the number of positive samples (i.e., the correct samples), the number of negative samples (i.e., the incorrect samples), and the progress of fine-tuning (i.e., the task status). It also shows the editing time for each fine-tuning task.

[0081] In one possible embodiment, Figure 1 The method shown further includes, after step S104: in response to a retrieval command, determining the state of the target model; if the target model is set to an enabled state, performing a retrieval using the target model; if the target model is set to an disabled state, performing a retrieval using a pre-trained model. By using this embodiment, users can enable or disable the target model according to their actual needs, thereby retrieving search results that meet their specific requirements.

[0082] For example, such as Figure 3bAs shown, the state of each target model can be switched using the model state control 23 in the third interface. Users can click the model state control 23 to switch the target model's state to enabled or disabled. Furthermore, it can be understood that the model state control 23 in the "woman in red" row controls the target model corresponding to "woman in red," and the target model corresponding to "woman in red" refers to the target model trained according to the aforementioned steps S101 to S104 when the search information is "woman in red." The same applies to the model state controls 23 in other rows. It can be understood that in other possible embodiments, the model state controls can also be set in other interfaces, and the model state controls can also be of other styles. Figure 3b No restrictions are imposed on this.

[0083] In one possible embodiment, Figure 1 The method further includes: adjusting the search information in response to an adjustment instruction for the search information to obtain new search information, and returning to execute the aforementioned step S101. By using this embodiment, users can perform further searches as needed to obtain more refined search results, and fine-tune the model based on these more refined search results, thereby further improving the fine-tuning effect.

[0084] For example, after a user enters the search query "people wearing black shirts" and performs a search, they find that a large number of people wearing brown short-sleeved shirts appear in the search results. At this point, the user can further adjust the search query to "people wearing black short-sleeved shirts" and perform a search to focus on training the pre-trained model's ability to distinguish between black and brown short-sleeved shirts, thereby optimizing the model in a more targeted manner.

[0085] In one possible embodiment, Figure 1 The method also includes: in response to a model export command, displaying all or part of the historically input search information; and in response to a selection command for any search information, exporting the target model file corresponding to the selected search information and the first search result. Using this embodiment, users can export the target model file to deploy the target model on other platforms, and also export the first error detection result to reproduce the training of the target model, or to enrich the training of other models with similar functions.

[0086] For example, such as Figure 4As shown, users can select at least one search item and click the export button to export the target model file and the first search result corresponding to this search item. The target model file corresponding to the search item refers to the model file of the target model trained using steps S101 to S104 with the search item. The first search result corresponding to the search item refers to the first search result obtained using step S101 with the search item. All first images in the first search result can be exported, or only a portion of the first images in the first search result can be exported. For example, only the first images annotated by the user in steps S102 to S103 can be exported. These first images can be regarded as the sample images on which the target model file is based when it is trained.

[0087] Furthermore, in some embodiments, in order to prevent irrelevant users from exporting the target model file and the first search results, identity authentication (such as account password authentication, facial recognition, etc.) can be performed before exporting, and the target model file and the first search results are only exported after the authentication is successful.

[0088] In one possible embodiment, Figure 1 The method also includes: in response to a filtering result instruction, displaying at least one third image through a fourth interface, wherein the third image includes: the remaining images in the first search results, excluding the second image. Using this embodiment allows users to more intuitively view the performance of the target model, helping them determine whether further fine-tuning is needed.

[0089] It is understandable that the second image is the image in the second search result obtained by the target model, while the first image is the image in the first search result obtained by the pre-trained model. The third image is in the first search result but not in the second search result, so it can be considered as an image that has been excluded after fine-tuning. By browsing whether these images should be excluded (i.e., these images do not contain the target), users can know the performance of the target model and thus decide whether further fine-tuning is needed.

[0090] For example, suppose the search query is "people wearing black shirts," and all or most of the third images do not contain people wearing black shirts. The user can then know that before fine-tuning, the pre-trained model incorrectly identified objects in these third images as people wearing black shirts. After fine-tuning, the target model can now identify these objects as people not wearing black shirts, demonstrating significant improvement and requiring no further fine-tuning. However, if most of the third images contain people wearing black shirts, the user can then know that before fine-tuning, the pre-trained model could accurately identify these objects as people wearing black shirts. After fine-tuning, the target model incorrectly identifies these objects as people not wearing black shirts, indicating that the fine-tuning effect is clearly unsatisfactory and requires further fine-tuning.

[0091] The fourth interface and the aforementioned second interface displaying the second search results can be two sub-interfaces under the same interface, for example, such as... Figure 5 As shown, if the user clicks the "Fine-tuned Results" button, the second interface will be displayed; if the user clicks the "Filtered Results" button, the fourth interface will be displayed.

[0092] In one possible embodiment, the aforementioned Figure 1 The method also includes editing the annotation result in response to an editing instruction for any annotation result. This embodiment allows users to adjust the annotation results to improve their accuracy.

[0093] Users can input editing commands during the execution of steps S102 to S103, or they can input editing commands to edit the annotation results if they are not satisfied with the fine-tuning results after fine-tuning, and then fine-tune again based on the edited annotation results.

[0094] In one possible embodiment, the aforementioned Figure 1 The method further includes: responding to the uploaded fourth image; the fourth image is annotated with a second annotation result, the second annotation result being used to indicate whether the fourth image matches or does not match the search information; using the first image and the fourth image as a new first image, returning to execute the aforementioned step S104. This embodiment allows users to upload images according to their own needs to fine-tune the pre-trained model, thereby enriching the sample data during fine-tuning and further improving the fine-tuning effect.

[0095] The following will provide an example of the image and text retrieval performance optimization method provided in this application from the perspective of human-computer interaction, such as... Figure 6As shown, the process includes: Step S61, the user labels positive and negative samples through a web interface or graphical user interface (GUI); Step S62, the user issues a fine-tuning task to the text search image application through a web interface or GUI; Step S63, the text search image application submits the fine-tuning task to the intelligent management terminal; Step S64, the intelligent management terminal accesses the data storage terminal to query CLIP modeling; Step S65, the data storage terminal returns the CLIP model to the intelligent management terminal; Step S66, the intelligent management terminal submits a model fine-tuning task to the general-purpose GPU engine; Step S67, the general-purpose GPU engine performs model fine-tuning, that is, fine-tuning the pre-trained model; Step S68, the general-purpose GPU engine returns the fine-tuning result to the alarm management terminal, and the fine-tuning result is the target model; S69, the alarm management terminal returns the fine-tuning results to the text search application; Step 610, the text search application processes the fine-tuning results to determine whether fine-tuning has been performed on the current search information. If fine-tuning has been performed, steps S611-S612 are executed; if fine-tuning has not been performed, steps S613-S614 are executed; Step S611, the text search application updates the model file stored on the data storage terminal; Step S612, after the update, the data storage terminal notifies the text search application that the update was successful; Step S613, the text search application persists the model file to the data storage terminal; Step S614, after successful persistence, the storage terminal notifies the text search application that persistence was successful.

[0096] After exporting the target model file and the first search result, users can import the exported target model file and the first search result into other platforms. The human-computer interaction process during the import is as follows: Figure 7 As shown, the process includes: Step S71, the user imports the upgrade package, i.e., imports the target model file and the first search result; Step S72, the WEB / GUI reads the header information of the upgrade package; Step S73, the WEB / GUI displays the contents of the upgrade package in the form of a text list, which can be the aforementioned... Figure 4 The search information is displayed in the form of a list; Step S74, the user selects the content to be imported and sends it to the WEB / GUI; Step S75, the WEB / GUI selects the content to be imported; Step S76, the WEB / GUI imports the selected content into the text search image application; Step S77, the text search image application returns the import progress, and steps S75 to S77 are executed repeatedly until all data is imported; Step S78, the text search image application imports the target model file and sample image into the data storage terminal. The sample image is the first image labeled by the user in the aforementioned first search result; Step S79, the text search image application adds the imported content to the fine-tuning training queue so that it can be used for subsequent training. Figure 3a or Figure 3bThe interface shown presents fine-tuning information to the user; in step S710, the WEB / GUI displays the import result to the user after the import is completed.

[0097] Corresponding to the aforementioned method, this application also provides a device for optimizing image and text retrieval performance, such as... Figure 8 As shown, the device includes:

[0098] The retrieval module 801 is used to respond to the input retrieval information and display the first retrieval result retrieved by the pre-trained model based on the retrieval information through a first interface; wherein the first retrieval result includes at least one first image;

[0099] Display module 802 is configured to receive user input of annotation operations for at least one of the first images displayed on the first interface when the optimization mode is enabled.

[0100] The annotation module 803 is configured to, in response to the annotation operation, configure a first annotation result for the at least one first image and display the annotation result; the first annotation result is used to indicate whether the first image matches or does not match the search information.

[0101] Fine-tuning module 804 is used to train the pre-trained model to obtain a target model based on the at least one first image in response to a fine-tuning operation.

[0102] The re-retrieval module 805 is used to display the second retrieval results obtained by the target model based on the retrieval information through a second interface; wherein the second retrieval results include at least one second image.

[0103] In one possible embodiment, the device further includes a management module, configured to, in response to a fine-tuning management command, display at least one of the following via a third interface: search information, the progress of the fine-tuning, the number of positive samples, the number of negative samples, all or part of the content of the positive samples, and all or part of the content of the negative samples; wherein the number of positive samples is the number of images in the at least one first search result whose labeled results indicate a match with the search information, and the number of negative samples is the number of images in the at least one first search result whose labeled results indicate a mismatch with the search information.

[0104] In one possible embodiment, the retrieval module is further configured to determine the state of the target model in response to a retrieval instruction after training the pre-trained model based on the at least one first image to obtain the target model in response to a fine-tuning operation.

[0105] If the target model is set to enabled, then the retrieval is performed using the target model;

[0106] If the target model is set to disabled, the retrieval is performed using the pre-trained model.

[0107] In one possible embodiment, the retrieval module is further configured to adjust the retrieval information in response to an adjustment instruction for the retrieval information, obtain new retrieval information, and return to execute the steps of retrieving the first retrieval result based on the retrieval information by the pre-trained model through a first interface.

[0108] In one possible embodiment, the apparatus further includes an export module for displaying all or part of the historically input retrieval information in response to a model export instruction.

[0109] In response to a selection instruction for any of the search information, the target model file corresponding to the selected search information and the first search result are exported.

[0110] In one possible embodiment, the display module is further configured to display at least one third image through a fourth interface in response to a filter result instruction, wherein the third image includes: the remaining images in the first search result other than the second image.

[0111] In one possible embodiment, the annotation module is further configured to edit the annotation result in response to an editing instruction for any of the annotation results.

[0112] In one possible embodiment, the optimization module is further configured to respond to an uploaded fourth image; the fourth image is labeled with a second annotation result, the second annotation result being used to indicate whether the fourth image matches or does not match the search information;

[0113] Using the first image and the fourth image as new first images, return to the step of training the pre-trained model based on the at least one first image to obtain the target model.

[0114] This application also provides an electronic device, such as... Figure 9 As shown, it includes:

[0115] Memory 901 is used to store computer programs;

[0116] When processor 902 executes a program stored in memory 901, it performs the following steps:

[0117] In response to the input search information, a first search result retrieved by the pre-trained model based on the search information is displayed through a first interface; wherein, the first search result includes at least one first image;

[0118] When the optimization mode is enabled, the annotation operation for at least one of the first images displayed in the first interface is received from the user.

[0119] In response to the annotation operation, a first annotation result is configured for the at least one first image and the annotation result is displayed; the first annotation result is used to indicate whether the first image matches or does not match the search information.

[0120] In response to the fine-tuning operation, the pre-trained model is trained based on the at least one first image to obtain the target model;

[0121] The second interface displays the second search results retrieved by the target model based on the search information; wherein, the second search results include at least one second image.

[0122] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 902, communication interface, and memory 901 communicating with each other via the communication bus.

[0123] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0124] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0125] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0126] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0127] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described image and text retrieval effect optimization methods.

[0128] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the image and text retrieval performance optimization methods described above.

[0129] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0130] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0131] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, embodiments of devices, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0132] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for fine-tuning of text retrieval effectiveness, characterized in that, The method includes: In response to the input search information, a first search result retrieved by the pre-trained model based on the search information is displayed through a first interface; wherein, the first search result includes at least one first image; When the optimization mode is enabled, the annotation operation for at least one of the first images displayed in the first interface is received from the user. In response to the annotation operation, a first annotation result is configured for the at least one first image and the annotation result is displayed; the first annotation result is used to indicate whether the first image matches or does not match the search information. In response to the fine-tuning operation, the pre-trained model is trained based on the at least one first image to obtain the target model; The second interface displays the second search results retrieved by the target model based on the search information; wherein the second search results include at least one second image.

2. The method according to claim 1, characterized in that, The method further includes: In response to the fine-tuning management command, at least one of the following is displayed on the third interface: search information, the progress of the fine-tuning, the number of positive samples, the number of negative samples, all or part of the content of the positive samples, and all or part of the content of the negative samples; wherein, the number of positive samples is the number of images in the at least one first search result that are labeled as matching the search information, and the number of negative samples is the number of images in the at least one first search result that are labeled as not matching the search information.

3. The method according to claim 1, characterized in that, After training the pre-trained model to obtain the target model based on the at least one first image in response to the fine-tuning operation, the method further includes: In response to a retrieval command, determine the state of the target model; If the target model is set to enabled, then the retrieval is performed using the target model; If the target model is set to disabled, the retrieval is performed using the pre-trained model.

4. The method according to claim 1, characterized in that, The method further includes: In response to the adjustment instruction for the search information, the search information is adjusted to obtain new search information and the process of returning to execute the search information in response to the input is performed. The first interface displays the steps of the pre-trained model retrieving the first search result based on the search information.

5. The method according to claim 1, characterized in that, The method further includes: In response to the model export command, display all or part of the historically input retrieval information; In response to a selection instruction for any of the search information, the target model file corresponding to the selected search information and the first search result are exported.

6. The method according to claim 1, characterized in that, The method further includes: In response to a filtering result instruction, at least one third image is displayed through a fourth interface, wherein the third image includes: the images remaining in the first search result other than the second image.

7. The method according to claim 1, characterized in that, The method further includes: In response to an editing instruction for any of the stated annotation results, the annotation results are edited.

8. The method according to claim 1, characterized in that, The method further includes: In response to the uploaded fourth image; the fourth image is labeled with a second annotation result, which indicates whether the fourth image matches or does not match the search information; Using the first image and the fourth image as new first images, return to the step of training the pre-trained model based on the at least one first image to obtain the target model.

9. A device for optimizing image and text retrieval performance, characterized in that, The device includes: The retrieval module is used to respond to input retrieval information and display a first retrieval result retrieved by a pre-trained model based on the retrieval information through a first interface; wherein the first retrieval result includes at least one first image; The display module is used to receive user input for annotation operations on at least one of the first images displayed on the first interface when the optimization mode is enabled. A labeling module is configured to, in response to the labeling operation, configure a first labeling result for the at least one first image and display the labeling result; the first labeling result is used to indicate whether the first image matches or does not match the search information. A fine-tuning module is used to train the pre-trained model to obtain a target model based on the at least one first image in response to a fine-tuning operation. The re-retrieval module is used to display the second retrieval results obtained by the target model based on the retrieval information through a second interface; wherein the second retrieval results include at least one second image.

10. The apparatus according to claim 9, characterized in that, The device further includes a management module, configured to, in response to a fine-tuning management command, display at least one of the following via a third interface: search information, the progress of the fine-tuning, the number of positive samples, the number of negative samples, all or part of the content of the positive samples, and all or part of the content of the negative samples; wherein, the number of positive samples is the number of images in the at least one first search result whose labeled results indicate a match with the search information, and the number of negative samples is the number of images in the at least one first search result whose labeled results indicate a mismatch with the search information; and / or, The retrieval module is further configured to determine the state of the target model in response to a retrieval command after training the pre-trained model based on the at least one first image to obtain the target model in response to a fine-tuning operation. If the target model is set to enabled, then the retrieval is performed using the target model; If the target model is set to disabled, retrieval is performed using the pre-trained model; and / or, The retrieval module is further configured to, in response to an adjustment instruction for the retrieval information, adjust the retrieval information to obtain new retrieval information, and return to execute the steps of retrieving a first retrieval result based on the retrieval information via a first interface, as described in the input retrieval information; and / or, The device also includes an export module, which, in response to a model export command, displays all or part of the historically input retrieval information. In response to a selection instruction for any of the search information, export the target model file corresponding to the selected search information and the first search result; and / or, The display module is further configured to, in response to a filtering result instruction, display at least one third image through a fourth interface, wherein the third image includes: images remaining in the first search result other than the second image; and / or, The annotation module is further configured to edit the annotation result in response to an editing instruction for any of the annotation results; and / or, The optimization module is also used to respond to the uploaded fourth image; the fourth image is labeled with a second annotation result, which indicates whether the fourth image matches or does not match the search information; Using the first image and the fourth image as new first images, return to the step of training the pre-trained model based on the at least one first image to obtain the target model.

Images