Method for searching images, related device and computer program product

Abstract

The invention provides an image searching method, a related device and a computer program product, and the method comprises the steps: determining a target video based on the first content of a sample image used for searching a semantically similar image, and enabling the semantic similarity between the second content of the target video and the first content to be greater than or equal to a first similarity threshold value; in response to the fact that the target image can be determined from the video frame of the target video, the target image serves as a semantic similar image search result for the sample image, and the semantic similarity between the third content of the sample image and the first content is larger than or equal to a second similarity threshold value; the value of the second similarity threshold value is greater than or equal to the first similarity threshold value. Therefore, the search capability and recall capability for similar images can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for searching images, an electronic device, a computer-readable medium, and a computer program product. Background Art

[0002] With the development of society, computer technology is also improving day by day. At the same time, image search technology is an important branch of computer technology in the field of computer vision. Image search technology can identify, match and retrieve related and similar images by analyzing image content.

[0003] In image search technology, after a user inputs an image as a "reference", other images that meet the requirements of similarity with the "reference image" can be provided and matched as search results. Therefore, how to provide search services to users more efficiently and save computing resources while ensuring search quality is worthy of attention and urgent. Summary of the invention

[0004] Multiple aspects of the present application provide a method, device, electronic device, computer-readable storage medium, and computer program product for searching images, which can determine videos whose content meets the semantic difference between the content included and the content included in the sample image based on semantic matching of the content. Then, based on the method of "splitting" the video, using the video as a source, search results of semantically similar images that can serve as sample images are provided and mined. In this way, videos can be used to enrich the source of image materials, improve the search and recall capabilities for similar images, and provide search services to users more efficiently and save computing resources while ensuring the quality of the search.

[0005] In one aspect of the present application, a method for searching images is provided, comprising: determining a target video based on a first content of a sample image used to search for semantically similar images, wherein a semantic similarity between a second content of the target video and the first content is greater than or equal to a first similarity threshold; in response to being able to determine a target image from a video frame of the target video, using the target image as a semantically similar image search result for the sample image, wherein a semantic similarity between a third content of the sample image and the first content is greater than or equal to a second similarity threshold, and a value of the second similarity threshold is greater than or equal to the first similarity threshold.

[0006] In another aspect of the present application, a device for searching images is provided, comprising: a target video determination module, configured to determine a target video based on a first content of a sample image used to search for semantically similar images, wherein a semantic similarity between a second content of the target video and the first content is greater than or equal to a first similarity threshold; a search result determination module, configured to, in response to being able to determine a target image from a video frame of the target video, use the target image as a semantically similar image search result for the sample image, wherein a semantic similarity between a third content of the sample image and the first content is greater than or equal to a second similarity threshold, and a value of the second similarity threshold is greater than or equal to the first similarity threshold.

[0007] Another aspect of the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method for searching images provided above.

[0008] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method for searching images provided above.

[0009] In another aspect of the present application, a computer program product includes a computer program having computer program instructions stored thereon. When the computer program is executed by a processor, the method for searching images as provided above can be implemented.

[0010] In the scheme provided by the embodiment of the present application, based on the first content of the sample image for searching for semantically similar images, the target video is determined, wherein the semantic similarity between the second content of the target video and the first content is greater than or equal to the first similarity threshold; in response to being able to determine the target image from the video frame of the target video, the target image is used as the search result of the semantically similar image for the sample image, wherein the semantic similarity between the third content of the sample image and the first content is greater than or equal to the second similarity threshold, and the value of the second similarity threshold is greater than or equal to the first similarity threshold. This method can determine the video whose semantic difference between the content included and the content included in the sample image meets the requirements based on the semantic matching of the content. Then, based on the method of "splitting" the video, the video is used as a source to provide and mine the search results of semantically similar images that can be used as sample images. As a result, the video can be used to enrich the source of image materials, improve the search ability and recall ability for similar images, so as to provide search services to users more efficiently and save computing resources under the premise of ensuring the search quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0012] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0013] Figure 1 A schematic diagram of a process of searching for an image provided by an embodiment of the present application;

[0014] Figure 2 A schematic diagram of an architecture of a process that can be used to search for images in an application scenario provided by an embodiment of the present application;

[0015] Figure 3 A schematic diagram of the structure of a device for searching images provided in an embodiment of the present application;

[0016] Figure 4 The present invention is a schematic diagram of the structure of an electronic device suitable for implementing the solution in the embodiment of the present application.

[0017] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0020] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0021] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0022] As discussed above, how to provide search services to users more efficiently and with less computing resources while ensuring search quality is worthy of attention and an urgent need.

[0023] In some solutions, it is chosen to configure as many candidate images as possible in, for example, an image database or an image data set, and to add more dimensional annotation information and auxiliary information to the candidate images in the image database to expand the candidate images and enhance the quality of use of the candidate images, thereby hoping to provide a search service with better search quality.

[0024] However, in this way, due to the limited source of images and the increase and expansion of the number of images in the image database, it is difficult to maintain annotation information and auxiliary information for each image in advance. Therefore, the number of images that can be maintained in some scenes may not meet the demand, and it is impossible to provide users with similar pictures in "sufficient quantities".

[0025] For example, in the scenario of training an image classification model, a large number of “same type” images may be required for training the image classification model. To this end, it may be desirable to use, for example, a “basic image” to search and recall a large number of images similar to the “basic image” to construct a sample image set as a model input for training the image classification model. However, in the strategy provided by the above-described method, it may be difficult to maintain a large number of images of the same type in advance in the image database or image data set, resulting in difficulty in providing enough images to construct a training sample set, which in turn affects the training effect of the classification model due to lack of samples or difficulty in obtaining training samples during the training process of the image classification model.

[0026] In this regard, an embodiment of the present application provides a method for searching images, which determines a target video based on the first content of a sample image for searching for semantically similar images, wherein the semantic similarity between the second content of the target video and the first content is greater than or equal to a first similarity threshold; in response to being able to determine the target image from the video frame of the target video, the target image is used as a semantically similar image search result for the sample image, wherein the semantic similarity between the third content of the sample image and the first content is greater than or equal to a second similarity threshold, and the value of the second similarity threshold is greater than or equal to the first similarity threshold. This method can determine a video whose semantic difference between the content included and the content included in the sample image meets the requirements based on the semantic matching of the content. Then, based on the method of "splitting" the video, the video is used as a source to provide and mine search results for semantically similar images that can be used as sample images. As a result, the video can be used to enrich the source of image materials, improve the search capability and recall capability for similar images, so as to provide search services to users more efficiently and save computing resources on the premise of ensuring the search quality.

[0027] In actual scenarios, the execution subject of the method can be a user device, or a device formed by integrating a user device and a network device through a network, or an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing (Cloud Computing), where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0028] When the execution subject is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here.

[0029] Figure 1 A process 100 for searching an image provided by an embodiment of the present application is shown. The process 100 includes at least the following processing steps:

[0030] (Step) S101, determining a target video based on a first content of a sample image for searching for a semantically similar image.

[0031] In the embodiments of the present application, in combination with different application scenarios, the execution subject may obtain sample images used as "benchmarks" for searching semantically similar images in different ways. For example, in the case where the execution subject is a server of a platform capable of providing image search services, the sample image may be communicated to the execution subject by the user through the terminal device used by the user (for example, the user requests the execution subject to search and provide other images that are "similar in content" to the sample image).

[0032] For another example, in the scenario described above where it is desired to construct an input image sample set for training an image classification model, the execution subject may also determine, based on the training purpose, images that can be used to train the image classification model from, for example, a locally maintained (sample) image database or image dataset, as “benchmark” sample images for expanding the images in the image database or image dataset. For example, these may include “images” that have been clearly identified and whose classification labels meet the training requirements.

[0033] Accordingly, the execution subject can search and determine the target video based on the content of the sample image (described as the first content for ease of understanding). The semantic similarity between the content of the target video (described as the second content for ease of understanding) and the first content is greater than a (preset) first similarity threshold. In some embodiments, the "content" of the image or video can be the content of the image or video represented by natural language.

[0034] In some embodiments, the value of the first similarity threshold can be determined based on the criterion of considering that two contents simultaneously include the "same main content" (i.e., based on the value corresponding to the minimum semantic similarity allowed when it can be reliably considered that two contents simultaneously point to the "same content" in terms of main content).

[0035] For example, if the content of the sample image indicated by the first content is "Subject A did XXX", and if the video content indicated by the second content corresponding to the I video is "The video tells the story of Subject A doing XXX in the past", the execution subject can determine that the semantic similarity between the two contents meets the requirements based on the semantic comparison of the two contents (for example, it may be because both "Subject A did XXX" is included, and it is believed that the first content and the second content at least point to the "same content"). Accordingly, in such a case, the semantic difference between the two contents (or between the two contents) can meet the requirements of the above-mentioned first similarity threshold, that is, the semantic similarity between the two contents is greater than or equal to the first similarity threshold, and the execution subject can determine that the I video is the "target video".

[0036] Usually, both the first content and the second content can be in "text form". Therefore, it is convenient to directly identify the semantics based on language processing and text processing, complete the semantic comparison, and determine the semantic similarity between the two.

[0037] In some embodiments, the first content may be generated by the execution subject locally or using other devices in real time after acquiring the sample image (for example, the sample image is generated by processing the sample image using an image content generation model deployed locally or on other devices, for example, the image content generation model may determine the content included in the image based on semantic analysis of the image). For example, the sample image is processed using a deployed convolutional neural network, recurrent neural network, generative adversarial network, etc. to obtain its "content".

[0038] In some embodiments, the first content may also be obtained in advance by, for example, other devices processing the sample image in a similar manner. Thus, after the execution subject determines and obtains the sample image, the first content may be directly read (for example, the first content may be communicated to the execution subject together with the sample image).

[0039] In some embodiments, the second content can be generated by processing the title information and content introduction information of the target video by a content generation model. For example, the content generation model can be a recursive neural network (RNN), a convolutional neural network, or other models.

[0040] The content generation model can process the title information and content introduction information of the target video (for example, after combining the title information and content introduction information of the target video to obtain the combined result, the "semantics" of the "combined result" is parsed by the content generation model, and the text form of the "semantics" is used as the second content indicating the content of the target video). As a result, the content of the video can be simply and efficiently "understood" and the second content corresponding to the content of the video can be generated by using the "title information and content introduction information of the target video" as a reference.

[0041] Similarly, in some embodiments, if the first content needs to be generated by an execution entity, the execution entity may also similarly use such a content generation model to generate the first content, which will not be repeated here.

[0042] In some embodiments, in order to more accurately determine the second content of the (target) video, it is also possible to select a combination result based on the fourth content of each video frame included in the video, that is, the image combination content of the target video, to determine the second content corresponding to the video. Thus, the second content of the video is determined with higher quality by actually "analyzing" the content of each video frame in the video (or, in other words, each video image constituting the video).

[0043] Specifically, the content generation model can be configured to first process each video frame in the video and identify the fourth content of each video frame (for example, the fourth content can also be in text form). Then, the fourth contents are combined to obtain the image combination content for the target video. For example, the fourth contents in text form are spliced ​​to obtain the image combination content expressed in text form.

[0044] Finally, similarly, by processing the semantics of the image combination (for example, analyzing the text content of the image combination content to determine the "semantics" of the text form of the "image combination content"), the "semantics" is used as the "second content" of the video. Thus, the execution subject can use the content generation model to determine the second content of the video from the perspective of "the content included in each video frame in the video" to improve the quality of the second content.

[0045] In some embodiments, considering that a video may actually be composed of modal data of multiple different modalities such as text, audio, and images, in order to further and more accurately determine the "content" included in the video and determine a higher quality "second content", in the process of generating the second content, it is possible to simultaneously refer to the modal data under each modality to determine the "second content".

[0046] Specifically, first, the target video can be split into a set of unimodal data, each unimodal data in a set of modal data corresponds to only one data modality. For example, a set of modal data can be composed of unimodal data Q, unimodal data W, and unimodal data E. The unimodal data Q can correspond to the content in the text mode of the video, the unimodal data W can correspond to the content in the audio mode of the video, and the unimodal data E can correspond to the content in the image word mode of the video.

[0047] Taking into account the differences in recognition capabilities and performances of different modalities, in this method, you can choose to use "semantics" as a substitute for "content" for combination. In this way, it is possible to avoid interference with the combination result due to differences in the way the "content" is determined. Accordingly, you can choose to generate target modal data for each unimodal data under the target modality, or in other words, "unify each unimodal data into the target modality." In this way, subsequent analysis of modal data and determination of semantics can be performed in the modal space of the same "target modality" to avoid processing differences caused by different modal spaces affecting the "semantic" recognition quality.

[0048] For example, in the case of "text mode as the target mode", the target modal data corresponding to the unimodal data W and the unimodal data E in the "text mode" can be determined respectively. For example, after converting image, audio and other data into Embedding vectors, based on the modality alignment method, the Embedding vectors can be mapped from the original modal space to the target modal space of the text mode to determine the target modal data corresponding to the unimodal data W and the unimodal data E in the "text mode".

[0049] Furthermore, the execution subject may similarly process the target modal data to obtain the corresponding target modal data semantics, and then combine the target modal data semantics of each target modal data to obtain a target modal data semantic combination result (for example, the target modal data semantics and the target modal data semantic combination result may also be in text form). Then, the execution subject uses the content generation model to process the target modal data semantic combination result to generate the second content (for example, the textual target modal data semantic combination result is subjected to "semantic recognition" again to finally determine the second content corresponding to the video). Thus, in the process of generating the second content, the modal data under each modality can be "fully" referenced, so that the "content" of the video can be more accurately "analyzed and determined", thereby accurately generating the "second content".

[0050] Next, when the execution subject determines the target video, the execution subject may similarly detect whether the video frame of the target video includes a target image whose semantic similarity with the sample image is greater than or equal to a second similarity threshold by comparing semantic similarities.

[0051] That is, the execution subject determines whether there is a "target image" in the video frame of the target video whose image semantic similarity difference with the sample image satisfies the requirement.

[0052] In the embodiment of the present disclosure, the value of the second similarity threshold is greater than or equal to the first similarity threshold. Thus, at least based on the screening criteria for the target video (for example, the values ​​of the two similarity thresholds are equal), the "target image" is found by a "fine screening" that is more stringent than the "coarse screening" for the target video (for example, the value of the second similarity threshold is greater than or equal to the first similarity threshold).

[0053] Therefore, after determining a target video whose semantic similarity between the second content and the first content is greater than or equal to the first similarity threshold, the determined "target video" can be used as a carrier to "mine" a target image from the video frame of the target video that meets the requirement that the semantic similarity between the second content and the first content is greater than or equal to the second similarity threshold.

[0054] Accordingly, this method not only allows "unsplit videos" to be directly used as the source of materials for image search and matching services, enriching the source of "candidate images", but also eliminates the need to pre-split the video, which can save the "waste of resources" that exists in some solutions that pre-splitting videos to build image data sets (for example, pre-splitting videos that may not be needed into multiple images). Moreover, this method does not require or require a large amount of image annotation content (for example, during preliminary screening, the content of the video can be referenced to "limit" the image range, and then the image can be annotated and searched based on the "limited" range, without the need to annotate all images in the image database), which can further save computing and storage resources.

[0055] S102: In response to being able to determine a target image from a video frame of a target video, using the target image as a semantically similar image search result for the sample image.

[0056] In the embodiment of the present application, based on the above S101, if the target image can be determined from the video frame of the target video, the execution subject responds to this and uses the target image as the search result of the semantically similar image for the sample image. Thus, the search and mining process of semantically similar images for the sample image based on the "video" is completed.

[0057] Then, the method for searching images provided by the present application determines the target video based on the first content of the sample image used to search for semantically similar images, wherein the semantic similarity between the second content of the target video and the first content is greater than or equal to the first similarity threshold; in response to being able to determine the target image from the video frame of the target video, the target image is used as the search result of the semantically similar image for the sample image, wherein the semantic similarity between the third content of the sample image and the first content is greater than or equal to the second similarity threshold, and the value of the second similarity threshold is greater than or equal to the first similarity threshold. This method can determine a video that meets the semantic similarity requirements with the sample image based on the semantic matching method. Then, based on the method of "splitting" the video, semantically similar images of the sample image are provided and mined. As a result, the video can be used to enrich the source of image materials, improve the search capability and recall capability for similar images, so as to provide users with search services more efficiently and save computing resources under the premise of ensuring the search quality.

[0058] In some embodiments, images and videos may be further classified based on the perspective of "type". Therefore, it is convenient to search for images of the same type or different types that have semantic similarity with the sample image that meets the requirements based on different usage requirements. For example, in the scenario of training an image classification model as described above, if the image classification model is expected to have the classification ability for images of the same type, then the search target for sample images may be "same type", while if the image classification model is expected to have the classification ability for images of multiple scenes and different types, then the search target for sample images may be "different type".

[0059] The “type” of images and videos can be exemplarily classified into “real scene”, “cartoon scene”, etc. Therefore, different search needs can be further met by refining the “type”. For example, when the above search target is “different type”, or in other words, when the user expects to search “across types” (for example, hoping to search for the visual effect of the image content of the sample image of “real scene” under “cartoon scene”), the user can indicate that the search target for the sample image is a semantically similar image search result of a different image type from the sample image when providing a sample image.

[0060] In some embodiments, the "type" of an image or video can be represented by a type label, and the type label for a sample image can be uploaded together with the sample image, or generated locally by the executing entity based on the user's instructions and search target (for example, based on an image category recognition model, the sample image is processed to determine its type label, such as "real scene" or "cartoon scene").

[0061] Then, the execution subject determines a set of candidate videos based on the search target of the semantically similar image search results of different image types from the sample image and the type label of the sample image. The type label associated with each candidate video in the set of candidate videos is different from the type label associated with the sample image. For example, in the case where the sample image is a "real scene", each candidate video in the set of candidate videos can be a video of a "cartoon scene". As a result, it is possible to find a "target video" that meets the requirements in each "cartoon scene" video.

[0062] For example, the execution subject may select, as an alternative or in lieu thereof, the first content based on the sample image during the execution of the above S101 to determine the target video from a group of candidate videos. Thus, in the scenario of, for example, the above-mentioned training image classification model, "cross-type" images may be provided and mined as training samples based on the "semantic similarity" method, thereby enriching the training sample set and improving the classification effect of the training image classification model.

[0063] In some embodiments, the target video that can determine the search results can also be provided to the user according to the user's needs for reference and decision-making assistance. For example, the user can communicate with the execution subject through the terminal device used by the user to request to obtain the target video by sending a video provision request for the target video to the execution subject.

[0064] Accordingly, the execution subject can respond to a video provision request sent by a terminal device used by a user of the target device, for example, and provide the target video to the target device based on the communication path between the terminal device and the target device, thereby providing the target video to the user.

[0065] On the basis of any of the above embodiments, in order to improve the content generation capability and efficiency, a large language model may be used as a content generation model to generate the first content and / or the second content.

[0066] Large Language Model (LLM) is an artificial intelligence model that aims to understand and generate human language. Moreover, the generative language model can perform processing operations accordingly based on the content it understands to obtain corresponding processing results.

[0067] LLMs are characterized by their large size and can often include a large number of parameters to help them learn complex patterns in language data. These models are often based on deep learning architectures such as transformers, which helps them provide better processing performance on various NLP tasks.

[0068] As discussed above, in the embodiments of the present application, the execution subject may also choose to use LLM to improve processing performance. For example, in the above example, in the process of generating the second content of the video, LLM may actually be used to process the title information and content introduction information of the video to generate the second content. Alternatively, LLM may be used to generate the second content based on the modal data under multiple modes of the video.

[0069] Accordingly, the execution subject can instruct LLM to generate the second content of the video based on the title information and content introduction information of the video based on pre-configured guide words, guide tags, etc. For example, in this process, LLM can be based on, for example, "Please summarize and conclude the content of the video based on the title information and content introduction information of the video, and use the results of the summary as the second content of the video." Or, for the scenario of generating the second content with reference to the video content in different modes, the guide words can be, for example, "Please refer to the text, audio and image content of the video to generate the text summaries corresponding to the text content, audio and image content of the video, and then summarize and conclude the content of the video based on the combination of the three summaries, and use the results of the summary as the second content of the video" and so on.

[0070] Similarly, for LLM, it can be configured by default to omit the "guide word". For example, for the purpose of generating the second content of a video based on the title information and content introduction information of the video, LLM can understand the need to generate the second content of the video based on the title information and content introduction information of the video based on the default configuration. Thus, by the default configuration, the generative language model can stably and directionally determine that the second content of the video needs to be generated based on the title information and content introduction information of the video.

[0071] Therefore, LLM can be used to generate the second content more efficiently and with higher quality.

[0072] Similarly, in S102 discussed in the above process, LLM can also be used to find the target image from the video frame of the target video. For example, the prompt word "from the video frame of the target video, find whether there is a target image whose semantic similarity with the sample image is greater than or equal to the second similarity threshold" can be constructed to instruct LLM to find the target image from the video frame of the target video, which will not be repeated here. Therefore, LLM can be directly selected to fully implement the complete process of generating content, finding images and videos, and finding target images, simplifying the overall configuration.

[0073] In order to deepen the understanding, this application also combines a specific application scenario and exemplarily gives a specific implementation solution, please refer to Figure 2 The architecture 200 that can be used for a process of searching for images is shown by way of example.

[0074] Exemplarily, in the architecture, for video 220 and video 230, the execution entity can use LLM 240 to process video 220 and video 230 to obtain content 225 corresponding to video 220 (used to indicate the content of video 220) and content 235 corresponding to video 230 (used to indicate the content of video 230).

[0075] Then, after determining the sample image 210 (for example, as discussed above, it may be an image provided by a user and expected to be a "benchmark" for searching for semantically similar images), the execution subject uses the content 215 of the sample image 210 to determine whether the "target image" discussed above exists in the video 220 and the video 230. That is, the execution subject can determine the semantic difference between the content 215 and the content 225, and the semantic difference between the content 215 and the content 235, respectively.

[0076] Exemplarily, the semantic similarity between content 215 and content 225 can meet the requirements of the first similarity threshold, and the semantic similarity is greater than or equal to the first similarity threshold. In this case, the execution subject searches for a "target image" from the video frames of video 220 (e.g., video frames 220-1, 220-2, ... 220-N, where N is a positive integer) whose content has a semantic similarity greater than or equal to the second similarity threshold with the content of sample image 210.

[0077] Exemplarily, if the semantic similarity between the contents of video frame 220 - 1 and sample image 210 is greater than or equal to a second similarity threshold, the execution entity may determine that video frame 220 - 1 is a “target image” and use video frame 220 - 1 as a semantically similar image search result for sample image 210 .

[0078] The present application also provides a device for searching images, the structure of which is as follows: Figure 3 The device 300 shown. The device 300 includes: a target video determination module 310, configured to determine a target video based on a first content of a sample image used to search for semantically similar images, wherein the semantic similarity between the second content of the target video and the first content is greater than or equal to a first similarity threshold; a search result determination module 320, configured to respond to being able to determine the target image from a video frame of the target video, use the target image as a semantically similar image search result for the sample image, wherein the semantic similarity between the third content of the sample image and the first content is greater than or equal to a second similarity threshold, and the value of the second similarity threshold is greater than or equal to the first similarity threshold.

[0079] In some embodiments, the second content is generated by a content generation model by processing the title information and content introduction information of the target video.

[0080] In some embodiments, the second content is generated by processing an image combination content of a target video by a content generation model, and the image combination content is obtained by combining fourth contents of video frames of the target video.

[0081] In some embodiments, the second content is generated based on the following method: splitting the target video into a group of unimodal data, wherein each unimodal data in a group of modal data corresponds to only one data modality; generating target modal data of each unimodal data in the target modality; combining the target modal data semantics of each target modal data to obtain a target modal data semantic combination result; and using a content generation model to process the target modal data semantic combination result to generate the second content.

[0082] In some embodiments, the content generation model includes a large language model.

[0083] In some embodiments, in response to the search results for semantically similar images whose search target for the sample image is a different image type from the sample image, the device 300 also includes: a candidate video determination module, configured to determine a group of candidate videos based on the type label of the sample image, wherein the type label associated with each candidate video in the group of candidate videos is different from the type label associated with the sample image; and a target video determination module 310 is further configured to determine a target video from a group of candidate videos based on the first content of the sample image.

[0084] In some embodiments, the apparatus 300 further includes: a target video providing module configured to provide the target video to the target device based on a communication path with the target device in response to receiving a video providing request for the target video sent by the target device.

[0085] Based on the same inventive concept, an electronic device, a readable storage medium and a computer program product are also provided in an embodiment of the present application. The method corresponding to the electronic device may be the method for searching images in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0086] The electronic device may be a user device, or a device formed by integrating a user device and a network device through a network, or an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0087] Figure 4 The structure of an electronic device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown, and the electronic device 400 includes a central processing unit (CPU, Central Processing Unit) 401, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 402 or the program loaded from the storage part 408 to the random access memory (RAM, Random Access Memory) 403. In RAM 403, various programs and data required for system operation are also stored. CPU 401, ROM 402 and RAM 403 are connected to each other through bus 405. Input / output (I / O, Input / Output) interface 404 is also connected to bus 405.

[0088] The following components are connected to the I / O interface 404: an input section 406 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, etc., and a speaker, etc.; a storage section 408 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet.

[0089] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 401, the above functions defined in the method of the present application are executed.

[0090] Another embodiment of the present application further provides a computer-readable storage medium and a computer program product, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application.

[0091] Specifically, the present embodiment can adopt any combination of one or more computer-readable media. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, systems, devices or devices including but not limited to electricity, magnetism, light, electromagnetic, infrared, or semiconductors, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0092] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0093] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0094] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0095] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0096] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0097] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules and units is only a logical function division. There may be other division methods in actual implementation. For example, with units as an example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0098] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0099] In addition, each functional module or unit in each embodiment of the present application may be integrated into one processing module or unit, or each module or unit may exist physically separately, or two or more units may be integrated into one module or unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules or units.

[0100] The above-mentioned integrated modules and units implemented in the form of software function modules and units can be stored in a computer-readable storage medium. The above-mentioned software function modules and units are stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform some steps of the methods of each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0102] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A method for searching an image, characterized in that: include: Determine a target video based on a first content of a sample image used to search for semantically similar images, wherein a semantic similarity between a second content of the target video and the first content is greater than or equal to a first similarity threshold; In response to being able to determine a target image from a video frame of the target video, the target image is used as a semantically similar image search result for the sample image, wherein a semantic similarity between a third content of the sample image and the first content is greater than or equal to a second similarity threshold, and a value of the second similarity threshold is greater than or equal to the first similarity threshold.

2. The method according to claim 1, characterized in that The second content is generated by a content generation model by processing the title information and content introduction information of the target video.

3. The method according to claim 1, characterized in that The second content is generated by processing the image combination content of the target video by a content generation model, and the image combination content is obtained by combining the fourth content of the video frames of the target video.

4. The method according to claim 1, characterized in that: The second content is generated based on the following method: Splitting the target video into a set of unimodal data, wherein each unimodal data in the set of modal data corresponds to only one data modality; Generating target modality data of each of the single modality data in the target modality; Combining the target modality data semantics of each of the target modality data to obtain a target modality data semantic combination result; The content generation model is used to process the semantic combination result of the target modal data to generate the second content.

5. The method according to any one of claims 2 to 4, characterized in that: The content generation model includes a large language model.

6. The method according to claim 1, characterized in that In response to the search target for the sample image being a search result of a semantically similar image of a different image type from the sample image, the method further includes: Determining a group of candidate videos based on the type label of the sample image, wherein the type label associated with each candidate video in the group of candidate videos is different from the type label associated with the sample image; and The step of determining a target video based on the first content of the sample image includes: Based on the first content of the sample image, a target video is determined from the set of candidate videos.

7. The method according to claim 1, characterized in that Also includes: In response to receiving a video providing request sent by a target device for the target video, the target video is provided to the target device based on a communication path with the target device.

8. A device for searching images, characterized in that: include: A target video determination module is configured to determine a target video based on a first content of a sample image used to search for semantically similar images, wherein a semantic similarity between a second content of the target video and the first content is greater than or equal to a first similarity threshold; A search result determination module is configured to, in response to being able to determine a target image from a video frame of the target video, use the target image as a semantically similar image search result for the sample image, wherein the semantic similarity between the third content of the sample image and the first content is greater than or equal to a second similarity threshold, and the value of the second similarity threshold is greater than or equal to the first similarity threshold.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer-readable medium, characterized in that Computer program instructions are stored thereon, and the computer program instructions can be executed by a processor to implement the method as claimed in any one of claims 1 to 7.

11. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images