Semantic retrieval method and device of image, equipment and medium

Abstract

The invention discloses an image semantic retrieval method and device, equipment and a medium, and belongs to the technical field of Internet. The method comprises the following steps: acquiring a monitoring image, performing target detection on the monitoring image, and intercepting a target image; obtaining an original vector of the monitoring image and a target vector of the target image; fusing the original vector and the target vector to obtain a fused vector, and storing the fused vector in a vector retrieval database; receiving a retrieval text, and extracting a text vector of the retrieval text; identifying a fusion vector matched with the text vector in a vector retrieval database; and obtaining a corresponding monitoring image and a target image according to the fusion vector, and restoring a coordinate frame of the target image into the monitoring image to generate a retrieval result. According to the technical scheme, the target information and the background information of the picture can be fully utilized through vector fusion, so that the retrieval result obtained through vector matching better conforms to the intention of a user, and the accuracy of matching of the retrieval image and the retrieval text of the user is improved.

Description

Technical Field

[0001] The present application belongs to the field of Internet technology, and specifically relates to a semantic retrieval method, device, equipment and medium for images. Background Art

[0002] Image semantic retrieval is a technology that searches for relevant images in an image database based on the semantic content of the image, such as concepts such as objects, scenes, and behaviors, rather than simple low-level visual features such as color and texture.

[0003] Traditional image retrieval methods rely on keyword matching mechanisms to extract target entities and their attribute features in images. However, this method has limitations in capturing the comprehensiveness and accuracy of image information, and video structuring algorithms often have difficulty acquiring complex details and contextual relationships in images. At the same time, traditional keyword retrieval models are limited by the ambiguity of language and the diversity of user intent, making it difficult to accurately capture and reflect the real needs of users, affecting the efficiency and accuracy of retrieval.

[0004] Therefore, how to better understand the user's query intention and obtain more accurate search results is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0005] The embodiments of the present application provide a semantic retrieval method, apparatus, device and medium for an image, the purpose of which is to fully utilize the target information and background information of the image through vector fusion to obtain a feature vector with richer representation, so that the retrieval results obtained by vector matching are more in line with the user's intention, which is conducive to improving the accuracy of matching the retrieved image with the user's retrieval text.

[0006] In a first aspect, this embodiment provides a semantic retrieval method for an image, the method comprising:

[0007] Acquire a surveillance image to be processed from a surveillance video, perform target detection on the surveillance image, and capture a target image according to the target detection result;

[0008] Acquire the original vector of the monitoring image through a pre-built retrieval model, and acquire the target vector of the target image;

[0009] Fusing the original vector and the target vector to obtain a fused vector, and storing the fused vector in a vector retrieval database;

[0010] Receiving a search text input by a user, and extracting a text vector of the search text through a pre-built search model;

[0011] Identifying in the vector search database a fusion vector that matches the text vector;

[0012] The corresponding monitoring image and target image are acquired according to the fusion vector, and the coordinate frame of the target image is restored to the monitoring image to generate a retrieval result.

[0013] Furthermore, capturing a target image according to the target detection result includes:

[0014] Acquire a coordinate range of a target in the target detection result in the monitoring image, and expand the coordinate range based on a preset expansion coefficient to obtain a capture range;

[0015] The monitoring image is intercepted according to the interception range, and the intercepted portion is used as the target image.

[0016] Furthermore, the interception range is calculated using the formula:

[0017]

[0018] Among them, (x1, y1) and (x2, y2) are the position coordinates of the target, (x1′, y1′) and (x2′, y2′) are the position coordinates of the interception range, a is the expansion coefficient, w is the width of the detected target, and h is the height of the detected target.

[0019] Furthermore, the original vector and the target vector are fused to obtain a fused vector, including:

[0020] A preset fusion factor is obtained, and the original vector and the target vector are fused based on the fusion factor. Specifically, the fusion vector can be obtained by the following formula:

[0021] v ij ′=αv i +(1-α)v ij ;

[0022] Among them, v ij ′ is the fusion vector, α is the fusion factor, v i is the ith original vector, v ij is the jth target vector on the i-th original image.

[0023] Further, identifying a fusion vector matching the text vector in the vector search database includes:

[0024] Calculating the similarity between the text vector and the fused vector in the vector search database, and sorting the fused vector according to the similarity;

[0025] A preset number of fused vectors are obtained in the sorting, or fused vectors satisfying a preset similarity condition are obtained in the sorting as output results.

[0026] Furthermore, after generating the search results, the method further includes:

[0027] In the case that the search result is inaccurate, obtaining positive sample images with semantics close to the search text and negative sample images with semantics opposite to the search text;

[0028] Calculating the image vectors of the positive sample image and the negative sample image, calculating the classification loss according to the image vector and the text vector, and performing back propagation based on the classification loss to fine-tune the parameters of the large retrieval model;

[0029] The fusion vectors in the original search results are reordered based on the fine-tuned large search model, and the secondary search results are determined according to the reordered results.

[0030] Furthermore, re-ranking the fusion vectors in the original search results based on the fine-tuned large search model includes:

[0031] The large retrieval model uses the fine-tuned parameters to vectorize the retrieval text to obtain a secondary retrieval text vector, and substitutes the secondary retrieval text vector, the fusion vector and its original similarity into the following formula to calculate the secondary retrieval similarity:

[0032]

[0033] Among them, S′ is the secondary retrieval similarity, t′ is the secondary retrieval text vector, and v i is the i-th fused vector, and S is the original similarity of the i-th fused vector.

[0034] The fusion vectors are reordered according to the secondary retrieval similarity.

[0035] In a second aspect, this embodiment provides a semantic retrieval device for an image, the device comprising:

[0036] An image acquisition module is used to acquire a surveillance image to be processed from a surveillance video, perform target detection on the surveillance image, and capture a target image according to the target detection result;

[0037] A vector acquisition module, used to acquire the original vector of the monitoring image and the target vector of the target image through a pre-built retrieval model;

[0038] A fusion vector acquisition module is used to fuse the original vector and the target vector to obtain a fusion vector, and store it in a vector retrieval database;

[0039] A text vector acquisition module, used to receive a search text input by a user, and extract a text vector of the search text through a pre-built search model;

[0040] A fusion vector matching module, used to identify a fusion vector matching the text vector in the vector search database;

[0041] The retrieval result generating module is used to obtain the corresponding monitoring image and target image according to the fusion vector, and restore the coordinate frame of the target image to the monitoring image to generate the retrieval result.

[0042] Furthermore, the image acquisition module is specifically used for:

[0043] Acquire a coordinate range of a target in the target detection result in the monitoring image, and expand the coordinate range based on a preset expansion coefficient to obtain a capture range;

[0044] The monitoring image is intercepted according to the interception range, and the intercepted portion is used as the target image.

[0045] Furthermore, the interception range is calculated using the formula:

[0046]

[0047] Among them, (x1, y1) and (x2, y2) are the position coordinates of the target, (x1′, y1′) and (x2′, y2′) are the position coordinates of the interception range, a is the expansion coefficient, w is the width of the detected target, and h is the height of the detected target.

[0048] Furthermore, the fusion vector acquisition module is specifically used for:

[0049] A preset fusion factor is obtained, and the original vector and the target vector are fused based on the fusion factor. Specifically, the fusion vector can be obtained by the following formula:

[0050] v ij ′=αv i +(1-α)v ij ;

[0051] Among them, v ij ′ is the fusion vector, α is the fusion factor, v i is the ith original vector, v ij is the jth target vector on the i-th original image.

[0052] Furthermore, the fusion vector matching module is specifically used for:

[0053] Calculating the similarity between the text vector and the fused vector in the vector search database, and sorting the fused vector according to the similarity;

[0054] A preset number of fused vectors are obtained in the sorting, or fused vectors satisfying a preset similarity condition are obtained in the sorting as output results.

[0055] Furthermore, the device also includes:

[0056] A fine-tuning module, used for obtaining positive sample images with similar semantics to the search text and negative sample images with opposite semantics to the search text when the search result is inaccurate;

[0057] Calculating the image vectors of the positive sample image and the negative sample image, calculating the classification loss according to the image vector and the text vector, and performing back propagation based on the classification loss to fine-tune the parameters of the large retrieval model;

[0058] The re-ranking module is used to re-rank the fusion vectors in the original retrieval results based on the fine-tuned retrieval model, and determine the secondary retrieval results according to the re-ranking results.

[0059] Furthermore, the rearrangement module is specifically used for:

[0060] The large retrieval model uses the fine-tuned parameters to vectorize the retrieval text to obtain a secondary retrieval text vector, and substitutes the secondary retrieval text vector, the fusion vector and its original similarity into the following formula to calculate the secondary retrieval similarity:

[0061]

[0062] Among them, S′ is the secondary retrieval similarity, t′ is the secondary retrieval text vector, and v i is the i-th fused vector, and S is the original similarity of the i-th fused vector.

[0063] The fusion vectors are reordered according to the secondary retrieval similarity.

[0064] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0065] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0066] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0067] In an embodiment of the present application, a surveillance image to be processed is obtained from a surveillance video, and target detection is performed on the surveillance image, and a target image is captured according to the target detection result; the original vector of the surveillance image is obtained through a pre-constructed retrieval model, and the target vector of the target image is obtained; the original vector and the target vector are fused to obtain a fused vector, which is stored in a vector retrieval database; a search text input by a user is received, and a text vector of the search text is extracted through a pre-constructed retrieval model; a fused vector matching the text vector is identified in the vector retrieval database; the corresponding surveillance image and target image are obtained according to the fused vector, and the coordinate frame of the target image is restored to the surveillance image to generate a retrieval result. Through the above-mentioned method of semantic retrieval of an image, the target information and background information of the image can be fully utilized through vector fusion to obtain a feature vector with richer representation, so that the retrieval results obtained by vector matching are more in line with the user's intention, which is conducive to improving the accuracy of matching the search image with the user's search text. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a flowchart of a semantic retrieval method for an image provided in Example 1 of the present application;

[0069] Figure 2 It is a flowchart of a semantic retrieval method for an image provided in the second embodiment of the present application;

[0070] Figure 3 It is a flowchart of a semantic retrieval method for an image provided in Embodiment 3 of the present application;

[0071] Figure 4 It is a flowchart of the image secondary retrieval method provided in Example 3 of the present application;

[0072] Figure 5 It is a structural schematic diagram of a semantic retrieval device for an image provided in Embodiment 4 of the present application;

[0073] Figure 6 It is a schematic diagram of the structure of an electronic device provided in Example 5 of the present application. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0075] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0076] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0077] In conjunction with the accompanying drawings, a semantic retrieval method, device, equipment and medium for an image provided by an embodiment of the present application are described in detail below through specific embodiments and their application scenarios.

[0078] Embodiment 1

[0079] Figure 1 It is a flowchart of a semantic retrieval method for an image provided in Example 1 of the present application.

[0080] like Figure 1 As shown, the specific steps include:

[0081] S101, obtaining a surveillance image to be processed from a surveillance video, performing target detection on the surveillance image, and capturing a target image according to the target detection result;

[0082] The application scenario of this solution can be a scenario of retrieving images in surveillance videos.

[0083] Based on the above scenario, it can be understood that the executor of this scenario can be an intelligent retrieval system that screens images that match the search text, such as a computer or software component that provides services in a network environment, etc., and no excessive limitations are made here.

[0084] The surveillance video may be a video collected by surveillance equipment installed in public places such as train stations and subways.

[0085] The surveillance image may be an image frame extracted from a surveillance video.

[0086] The target image may be an image containing only the detection target captured by removing redundant image background from the surveillance image. For example, in the surveillance image of a subway station, a target image of a fight between multiple people may be captured.

[0087] The method of obtaining the surveillance image to be processed from the surveillance video can be, after obtaining the surveillance video, dividing the surveillance video into frames of images, comparing the similarity of adjacent images, deleting images with greater similarity, and using the remaining images as surveillance images.

[0088] The method of performing target detection on the monitoring image can be to segment the image based on the color, texture, size and other features of the monitoring image, then merge similar areas to generate candidate areas, then extract features from each candidate area, and then use a classifier to classify it to determine whether it contains the target, intercept the area containing the target, and deduplicate the intercepted image again to obtain the corresponding target image. It can be understood that for target detection in images, detection methods such as sliding windows and deep learning can also be used, and no excessive restrictions are made here.

[0089] S102, obtaining the original vector of the monitoring image and the target vector of the target image through a pre-built retrieval model;

[0090] The original vector may be an image feature vector extracted from a surveillance image.

[0091] The target vector may be an image feature vector extracted from the target image.

[0092] The original vector and the target vector can be obtained by extracting the original vector of the monitoring image and the target vector of the target image through a convolutional neural network. Specifically, in the convolution layer, the convolution kernel slides on the monitoring image and the target image to extract local features, that is, to capture low-level features such as edges, lines and textures in the image. As the number of network layers increases, the convolution layer can gradually extract higher-level semantic features, such as parts and the whole of an object. In the pooling layer, the size of the feature map is reduced by downsampling operations to retain important information. Nonlinearity is introduced through activation functions to enable the convolutional neural network to learn more complex features. The corresponding original vector and target vector are generated according to the learned image features.

[0093] In this solution, specifically, the image encoder model can be used to convert the input image data into a low-dimensional vector representation through a series of neural network layers and a specific architecture design. This vector can capture the semantic information and features of the image to a certain extent. Its core principle is to use the characteristics of architectures such as convolutional neural networks (CNN) or visual transformers (ViT) to automatically extract key features from the image and map these features into a fixed-length vector.

[0094] In encoders based on convolutional neural networks (CNNs), such as the ResNet architecture, the image first passes through multiple convolutional layers, which extract local features of the image, such as edges and textures, through convolution kernels of different sizes. As the network goes deeper, the size of the feature map gradually decreases, but the number of channels increases to capture higher-level and more abstract semantic information. Finally, the feature map is converted into a fixed-length vector through operations such as global average pooling or fully connected layers. For example, the last global average pooling layer of ResNet-50 converts the input image into a 2048-dimensional vector. Encoders based on the Visual Transformer (ViT) can divide the image into a series of fixed-size image patches and linearly project each image patch into a vector as the input token of the Transformer. These tokens pass through the encoder layer of the Transformer, including the Multi-Head Self-Attention mechanism and the Feed Forward Neural Network, to continuously learn and integrate the relationship between image patches, thereby obtaining a feature vector containing the global information of the image. Taking ViT-Base as an example, after patch embedding, the input image obtains a series of vectors with a length of 768. After being processed by a 12-layer Transformer encoder, it finally outputs an image feature vector with the same length of 768.

[0095] Image data usually has a high dimension, and directly processing this high-dimensional data will bring huge computational costs and storage overhead. After the image is vectorized, the image encoder can compress a large amount of image information into a relatively low-dimensional vector space, while retaining the key features of the image, which is convenient for subsequent model processing and analysis. In a multimodal large model, the image vector generated by the image encoder can be fused with the feature vectors of other modalities such as text, so that the model can simultaneously process and understand information from multiple modalities, thereby achieving more powerful language-image interaction and understanding capabilities, such as generating image descriptions, answering image-related questions, etc., and the image features learned by the image encoder pre-trained with large-scale data have certain universality and generalization capabilities. These pre-trained models can be fine-tuned or directly applied in different downstream tasks, reducing the cost and data requirements of retraining models for specific tasks, and improving the development efficiency and performance of the model.

[0096] S103, fusing the original vector and the target vector to obtain a fused vector, and storing the fused vector in a vector retrieval database;

[0097] The vector retrieval database may be a database that stores all fused vectors in a specified location according to a specified format to facilitate calling or retrieving the fused vectors.

[0098] The fusion vector can be obtained by performing a fusion operation according to a predetermined fusion factor. For example, corresponding fusion weights can be set for the original vector and the target vector, and the sum of the products of the original vector and the target vector with their respective fusion weights can be calculated to obtain the fusion vector.

[0099] S104, receiving a search text input by a user, and extracting a text vector of the search text through a pre-built search macro model;

[0100] The retrieval text may be the semantic information of the image that the user wants to obtain.

[0101] The pre-built large retrieval model can be a learning model that can extract text vectors from retrieval texts. A large number of retrieval texts with known text vectors are obtained, and the text vectors are used as labels and the corresponding retrieval texts are used as input variables for supervised training to learn the relationship between the retrieval texts and the text vectors, thereby building a large retrieval model.

[0102] The method of receiving the search text input by the user can be that the user enters the search text in the search input box of the interface provided by the intelligent search system through the display device, and the display device uploads the user's input content to the intelligent search system for processing.

[0103] The text vector can be extracted by using a neural network model, such as a recurrent neural network, a long short-term memory network or a convolutional neural network, to learn and encode the search text, extract features such as semantic representation information and grammatical structure in the search text, and determine the rules for mapping language units to the vector space based on the extracted features, so that the text vector corresponding to the search text can be obtained according to the rules.

[0104] In this solution, specifically, the text encoder model of the retrieval model can be used to vectorize the text. The retrieval model is a transformer structure. Among them, the text encoder in the retrieval model based on the Transformer structure mainly learns the long-term dependencies in the text through the self-attention mechanism, and then converts the text into a vector representation. It first performs preprocessing operations such as word segmentation on the input text, converts the text into a series of tokens, and then generates a corresponding vector for each token. These vectors will enter the encoder layer of the Transformer as input. The self-attention mechanism in the encoder layer will calculate the degree of association between each token and other tokens, thereby capturing the semantic information and grammatical structure of the text. Through the stacking of multiple layers of encoders, this information is continuously refined and integrated, and finally a fixed-length vector that can comprehensively represent the entire text is obtained. The self-attention mechanism of the Transformer can effectively capture the long-distance dependencies in the text, so that the model can better understand the semantics and contextual information of the text. Compared with traditional methods based on convolutional neural networks (CNN) or recurrent neural networks (RNN), it has obvious advantages in processing long texts and complex semantic structures. In addition, the Transformer architecture supports parallel computing and can handle multiple tasks at the same time, greatly improving the training and reasoning speed of the model. This makes it possible to train on large-scale text data and vectorize large amounts of text, improving the efficiency and scalability of the system. It can first be pre-trained on large-scale unsupervised data to learn general language knowledge and semantic representations, and then fine-tuned on specific retrieval tasks. This pre-training-fine-tuning paradigm can make full use of large amounts of unsupervised data, improve the performance of the model on specific tasks, and reduce the need for labeled data for specific tasks.

[0105] S105, identifying a fusion vector matching the text vector in the vector search database;

[0106] The method of identifying the fusion vector that matches the text vector can be to identify the correlation between all fusion vectors and the text vector in the vector retrieval database, output the similarity between each fusion vector and the current text vector according to the correlation, and use the fusion vector corresponding to the similarity greater than a preset similarity threshold value or a preset number as the fusion vector that matches the text vector. The preset similarity threshold value or the preset number can be pre-set by the staff, for example, setting the fusion vector with a similarity greater than 80% as the fusion vector that matches the text vector, or setting the top 10 fusion vectors with the largest similarity as the fusion vector that matches the text vector.

[0107] S106 , acquiring a corresponding monitoring image and a target image according to the fusion vector, and restoring the coordinate frame of the target image to the monitoring image to generate a retrieval result.

[0108] The coordinate frame can be a rectangular frame that selects the edge of the target image. The purpose of the coordinate frame is to allow users to find the location of the target in the surveillance image more quickly and accurately. Therefore, the coordinate frame can be made more eye-catching in the surveillance image by using a bold frame or a red frame. There are no restrictions on the display form of the coordinate frame.

[0109] The method for generating the retrieval result can be, after obtaining the fusion vector matching the text vector, determining the corresponding monitoring image and target image according to the original vector and the target vector for generating the fusion vector, obtaining the location of the target image in the monitoring image, and using a coordinate box to select the location of the target image, thereby obtaining the monitoring image marked with the coordinate box of the target image, and taking the marked monitoring images corresponding to each fusion vector matching the text vector as the retrieval result.

[0110] On the basis of the above solution, optionally, the original vector and the target vector are fused to obtain a fused vector, including:

[0111] A preset fusion factor is obtained, and the original vector and the target vector are fused based on the fusion factor. Specifically, the fusion vector can be obtained by the following formula:

[0112] v ij ′=αv i +(1-α)v ij ;

[0113] Among them, v ij ′ is the fusion vector, α is the fusion factor, v i is the ith original vector, v ij is the jth target vector on the i-th original image.

[0114] The fusion factor can be obtained by performing a fusion operation according to a predetermined fusion factor. For example, corresponding fusion weights can be set for the original vector and the target vector, and the sum of the products of the original vector and the target vector with their respective fusion weights can be calculated to obtain a fusion vector.

[0115] The advantage of this setting of the scheme is that when retrieving images that match the text, the impact of the background and target of the image on the relevance of the results can be considered at the same time, so that the retrieval results are more in line with the user's intentions and the user experience is improved.

[0116] Based on the above solution, optionally, identifying a fusion vector matching the text vector in the vector search database includes:

[0117] Calculating the similarity between the text vector and the fused vector in the vector search database, and sorting the fused vector according to the similarity;

[0118] A preset number of fused vectors are obtained in the sorting, or fused vectors satisfying a preset similarity condition are obtained in the sorting as output results.

[0119] The similarity of the fused vector can be calculated by taking the vector dot product of the fused vector and the text vector in the vector retrieval database as the similarity between the fused vector and the text vector. For example, the similarity can be calculated by the following formula:

[0120] S i =t·v i ;

[0121] Among them, S i is the similarity between the text vector and the i-th vector in the vector retrieval database, t is the text vector, v i is the i-th vector in the vector retrieval database.

[0122] The output result can be obtained by calculating the similarity between all fused vectors and text vectors in the vector retrieval database according to a preset formula, sorting all fused vectors according to the similarity from large to small, and obtaining the fused vector corresponding to the maximum similarity greater than the preset similarity, or the maximum similarity before the preset number, as the output result.

[0123] The benefit of this arrangement of the scheme is that the output result can be determined by the similarity between the fusion vector and the text vector, which is beneficial to improve the accuracy of the retrieval results and increase the matching degree between the retrieved image and the retrieval text input by the user.

[0124] Optionally, this solution provides a program code for obtaining search results based on the search text for reference:

[0125]

[0126]

[0127]

[0128] In an embodiment of the present application, a surveillance image to be processed is obtained from a surveillance video, and target detection is performed on the surveillance image, and a target image is captured according to the target detection result; the original vector of the surveillance image is obtained through a pre-constructed retrieval model, and the target vector of the target image is obtained; the original vector and the target vector are fused to obtain a fused vector, which is stored in a vector retrieval database; a search text input by a user is received, and a text vector of the search text is extracted through a pre-constructed retrieval model; a fused vector matching the text vector is identified in the vector retrieval database; the corresponding surveillance image and target image are obtained according to the fused vector, and the coordinate frame of the target image is restored to the surveillance image to generate a retrieval result. Through the above-mentioned method of semantic retrieval of an image, the target information and background information of the image can be fully utilized through vector fusion to obtain a feature vector with richer representation, so that the retrieval results obtained by vector matching are more in line with the user's intention, which is conducive to improving the accuracy of matching the search image with the user's search text.

[0129] Embodiment 2

[0130] Figure 2 This is a flow chart of a semantic retrieval method for an image provided in Example 2 of the present application. This solution makes a better improvement on the above-mentioned embodiment, and the specific improvement is: intercepting the target image according to the target detection result, including: obtaining the coordinate range of the target in the target detection result in the monitoring image, expanding the coordinate range based on a preset expansion coefficient to obtain the interception range; intercepting the monitoring image according to the interception range, and using the intercepted part as the target image.

[0131] like Figure 2 As shown, specifically including the following:

[0132] S201, obtaining a surveillance image to be processed from a surveillance video, and performing target detection on the surveillance image;

[0133] S202, obtaining a coordinate range of the target in the target detection result in the monitoring image, and expanding the coordinate range based on a preset expansion coefficient to obtain a capture range;

[0134] The preset magnification factor may be a parameter value that can determine the magnification factor of the coordinate range.

[0135] The coordinate range can be obtained by extracting the boundary coordinates of the rectangular box where the target in the target detection result is located, and determining the coordinate range according to the boundary coordinates. For example, if the top corner coordinates and the bottom right corner coordinates of the rectangular box of the target detection result in the monitoring image are (23, 36) and (45, 50) respectively, then the coordinate range is the intersection of the area with abscissas from 23 to 45 and the area with ordinates from 36 to 50.

[0136] The interception range may be obtained by respectively expanding the length and width of the acquired coordinate range by corresponding multiples according to a preset expansion coefficient, and the obtained expanded range is used as the interception range.

[0137] S203, performing a capture operation on the monitoring image according to the capture range, and taking the captured portion as a target image;

[0138] The target image may be obtained by cutting and deleting the image portion outside the interception range in the monitoring image, and taking the remaining portion within the interception range as the target image.

[0139] S204, obtaining the original vector of the monitoring image and the target vector of the target image through a pre-built retrieval model;

[0140] S205, fusing the original vector and the target vector to obtain a fused vector, and storing the fused vector in a vector retrieval database;

[0141] S206, receiving a search text input by a user, and extracting a text vector of the search text by using a pre-built search macro model;

[0142] S207, identifying a fusion vector matching the text vector in the vector search database;

[0143] S208 . Acquire a corresponding monitoring image and a target image according to the fusion vector, and restore the coordinate frame of the target image to the monitoring image to generate a retrieval result.

[0144] The advantage of this setting of the scheme is that, based on target detection, the detection frame can be expanded to obtain more comprehensive image features, thereby supporting multi-subject retrieval text in subsequent retrieval and improving the image retrieval capability of the intelligent retrieval system.

[0145] Based on the above solution, optionally, the interception range is calculated using the formula:

[0146]

[0147] Among them, (x1, y1) and (x2, y2) are the position coordinates of the target, (x1′, y1′) and (x2′, y2′) are the position coordinates of the interception range, a is the expansion coefficient, w is the width of the detected target, and h is the height of the detected target.

[0148] The coordinate position of the target may be the upper left vertex and the lower right vertex in the detection frame, that is, the width range and the height range of the detection frame may be included, thereby achieving accurate acquisition of the range where the target is located.

[0149] The expansion coefficient can be preset by the staff. In the above formula, the expansion coefficient can represent the expansion multiple of the coordinate range. For example, when the expansion coefficient is 0.1, the maximum and minimum values ​​of the horizontal coordinate of the coordinate range are expanded by 0.1 times, and the maximum and minimum values ​​of the vertical coordinate are expanded by 0.1 times.

[0150] The advantage of this setting of the scheme is that the coordinate range can be accurately expanded through the set formula, which is conducive to improving the efficiency of determining the screenshot range, shortening the retrieval time, and improving the user experience.

[0151] Embodiment 3

[0152] Figure 3 It is a flowchart of a semantic retrieval method for an image provided in Example 3 of the present application. This solution makes a better improvement on the above-mentioned embodiment, and the specific improvement is: identifying a fusion vector matching the text vector in the vector retrieval database, including: calculating the similarity between the text vector and the fusion vector in the vector retrieval database, and sorting the fusion vector according to the similarity; obtaining a preset number of fusion vectors in the sorting, or obtaining a fusion vector that meets the preset similarity condition as an output result in the sorting.

[0153] like Figure 3 As shown, specifically including the following:

[0154] S301, obtaining a surveillance image to be processed from a surveillance video, performing target detection on the surveillance image, and capturing a target image according to the target detection result;

[0155] S302, obtaining the original vector of the monitoring image and the target vector of the target image through a pre-built retrieval model;

[0156] S303, fusing the original vector and the target vector to obtain a fused vector, and storing the fused vector in a vector retrieval database;

[0157] S304, receiving a search text input by a user, and extracting a text vector of the search text through a pre-built search macro model;

[0158] S305, identifying a fusion vector matching the text vector in the vector search database;

[0159] S306, acquiring a corresponding monitoring image and a target image according to the fusion vector, and restoring the coordinate frame of the target image to the monitoring image to generate a retrieval result;

[0160] S307: when the search result is inaccurate, obtain positive sample images with semantics similar to the search text and negative sample images with semantics opposite to the search text;

[0161] Positive sample images can be images that are semantically consistent with the search text in the surveillance images. For example, if the search text is "couples walking side by side", the positive sample images are all images containing boys and girls walking together.

[0162] Negative sample images can be images that are opposite to the semantics of the search text in the monitoring images. For example, if the search text is a couple walking side by side, the negative sample images are all images that do not contain boys and girls walking together.

[0163] S308, calculating the image vectors of the positive sample image and the negative sample image, calculating the classification loss according to the image vector and the text vector, and performing back propagation based on the classification loss to fine-tune the parameters of the large retrieval model;

[0164] The way to calculate the classification loss is to obtain the image vectors of the positive sample image and the negative sample image through the image encoder of the semantic retrieval model, and then substitute the image vector and the text vector into the following formula to obtain the corresponding classification loss:

[0165]

[0166] Among them, L is the classification loss; batch is the batch size; y i represents the label, the positive sample label is 1, and the negative sample label is 0; t represents the retrieved text vector; v i Represents the i-th image vector.

[0167] The back propagation method can be to calculate the gradient of the loss function with respect to the parameters of the retrieval model through the chain rule, and then obtain the updated parameters of the retrieval model through optimization algorithms such as gradient descent. Among them, gradient descent is the most basic optimization algorithm, which is based on the fact that the gradient direction of the function at a certain point is the direction in which the function value rises fastest, and the negative gradient direction is the direction in which the function value drops fastest. That is, with continuous updates, the parameters will gradually tend to the value that minimizes the loss function.

[0168] S309: reorder the fusion vectors in the original search results based on the fine-tuned large search model, and determine the secondary search results according to the reordering results.

[0169] The fusion vectors may be reordered by recalculating the text vectors of the retrieved texts through the fine-tuned large retrieval model, and then obtaining new similarities with the fusion vectors based on the newly generated text vectors, and reordering the fusion vectors from large to small according to the similarities.

[0170] The method for determining the secondary search result can be to obtain a preset number of fusion vectors before, or to obtain a fusion vector that meets a preset similarity condition in the sorting, obtain the monitoring image and the target image corresponding to the fusion vector, restore the coordinate frame of the target image to the monitoring image, and generate a secondary search result.

[0171] For example, Figure 4 is a flow chart of the image secondary retrieval method provided in the third embodiment of the present application. Figure 4 As shown in the figure, during the target storage process, the monitoring image is detected, the coordinate range of the target in the image is obtained, the coordinate range is expanded and the NMS (Non-Maximum Suppression) deduplication operation is performed to obtain the interception range, the monitoring image is intercepted according to the interception range, and then the corresponding target image is obtained, the vectors of the monitoring image and the target image are extracted and fused to obtain the fusion vector, and stored in the vector retrieval database. After the vector retrieval database is built, the target retrieval process can be entered. The user enters the retrieval text, the retrieval model vectorizes the retrieval text, and the fusion vector matching the text vector is retrieved from the vector retrieval database as the retrieval result. If the retrieval result meets the expectation, the corresponding monitoring image with the target box mark is obtained; if the retrieval result does not meet the expectation, the semantic retrieval model is fine-tuned, and the fine-tuned model is used for secondary retrieval, and the original retrieval results are rearranged to obtain the retrieval results of the secondary retrieval, and the monitoring image with the target box mark corresponding to the retrieval result of the secondary retrieval is obtained. Among them, non-maximum suppression is a post-processing technology widely used in computer vision tasks such as target detection. Its main purpose is to eliminate redundant detection frames and retain the detection frames that are most likely to be the target objects, thereby improving the accuracy and efficiency of detection.

[0172] The benefit of this arrangement of the scheme is that undesirable search results can be reordered through secondary search, so that the images in the search results are more in line with the user's intention, which is conducive to improving the accuracy of the search results.

[0173] Based on the above solution, optionally, re-ranking the fusion vectors in the original search results based on the fine-tuned large search model includes:

[0174] The large retrieval model uses the fine-tuned parameters to vectorize the retrieval text to obtain a secondary retrieval text vector, and substitutes the secondary retrieval text vector, the fusion vector and its original similarity into the following formula to calculate the secondary retrieval similarity:

[0175]

[0176] Among them, S′ is the secondary retrieval similarity, t′ is the secondary retrieval text vector, and v i is the i-th fused vector, and S is the original similarity of the i-th fused vector.

[0177] The fusion vectors are reordered according to the secondary retrieval similarity.

[0178] The method of calculating the secondary retrieval similarity is to update the parameters of the text encoder in the retrieval model after fine-tuning, so as to obtain a text vector with clearer semantics, namely, the secondary retrieval text vector. In addition, the calculation of the secondary retrieval similarity is also different from the calculation of the similarity in the first retrieval. In the above formula, the original similarity obtained from the first retrieval is adjusted by the secondary retrieval text vector, so that the obtained secondary retrieval similarity can better reflect the correlation between the fusion vector and the text vector. Among them, the text encoder is a tool that converts text data into a digital representation that can be understood and processed by a computer, namely, a vector representation.

[0179] The advantage of this arrangement of the scheme is that by adjusting the text vector and the similarity calculation formula, the images in the retrieval results can be made more consistent with the semantics of the input text, thereby improving the accuracy of the secondary retrieval results.

[0180] Embodiment 4

[0181] Figure 5 It is a structural diagram of the semantic retrieval device for images provided in Example 4 of the present application.

[0182] like Figure 5 As shown, specifically including the following:

[0183] The image acquisition module 501 is used to acquire the surveillance image to be processed from the surveillance video, perform target detection on the surveillance image, and capture the target image according to the target detection result;

[0184] A vector acquisition module 502, used to acquire the original vector of the monitoring image and the target vector of the target image through a pre-built retrieval model;

[0185] A fusion vector acquisition module 503 is used to fuse the original vector and the target vector to obtain a fusion vector, and store it in a vector retrieval database;

[0186] The text vector acquisition module 504 is used to receive the search text input by the user and extract the text vector of the search text through the pre-built search model;

[0187] A fusion vector matching module 505, used to identify a fusion vector matching the text vector in the vector search database;

[0188] The retrieval result generating module 506 is used to obtain the corresponding monitoring image and target image according to the fusion vector, and restore the coordinate frame of the target image to the monitoring image to generate a retrieval result.

[0189] Optionally, the image acquisition module is specifically used to:

[0190] Acquire a coordinate range of a target in the target detection result in the monitoring image, and expand the coordinate range based on a preset expansion coefficient to obtain a capture range;

[0191] The monitoring image is intercepted according to the interception range, and the intercepted portion is used as the target image.

[0192] Optionally, calculate the intercept range using the formula:

[0193]

[0194] Among them, (x1, y1) and (x2, y2) are the position coordinates of the target, (x1′, y1′) and (x2′, y2′) are the position coordinates of the interception range, a is the expansion coefficient, w is the width of the detected target, and h is the height of the detected target.

[0195] Optionally, the fusion vector acquisition module is specifically used to:

[0196] A preset fusion factor is obtained, and the original vector and the target vector are fused based on the fusion factor. Specifically, the fusion vector can be obtained by the following formula:

[0197] v ij ′=αv i +(1-α)v ij ;

[0198] Among them, v ij ′ is the fusion vector, α is the fusion factor, v i is the ith original vector, v ijis the jth target vector on the i-th original image.

[0199] Optionally, the fusion vector matching module is specifically used to:

[0200] Calculating the similarity between the text vector and the fused vector in the vector search database, and sorting the fused vector according to the similarity;

[0201] A preset number of fused vectors are obtained in the sorting, or fused vectors satisfying a preset similarity condition are obtained in the sorting as output results.

[0202] Optionally, the device further comprises:

[0203] A fine-tuning module, used for obtaining positive sample images with similar semantics to the search text and negative sample images with opposite semantics to the search text when the search result is inaccurate;

[0204] Calculating the image vectors of the positive sample image and the negative sample image, calculating the classification loss according to the image vector and the text vector, and performing back propagation based on the classification loss to fine-tune the parameters of the large retrieval model;

[0205] The re-ranking module is used to re-rank the fusion vectors in the original retrieval results based on the fine-tuned retrieval model, and determine the secondary retrieval results according to the re-ranking results.

[0206] Optionally, the rearrangement module is specifically used for:

[0207] The large retrieval model uses the fine-tuned parameters to vectorize the retrieval text to obtain a secondary retrieval text vector, and substitutes the secondary retrieval text vector, the fusion vector and its original similarity into the following formula to calculate the secondary retrieval similarity:

[0208]

[0209] Among them, S′ is the secondary retrieval similarity, t′ is the secondary retrieval text vector, and v i is the i-th fused vector, and S is the original similarity of the i-th fused vector.

[0210] The fusion vectors are reordered according to the secondary retrieval similarity.

[0211] In the embodiment of the present application, the image acquisition module is used to acquire the surveillance image to be processed from the surveillance video, and perform target detection on the surveillance image, and capture the target image according to the target detection result; the vector acquisition module is used to acquire the original vector of the surveillance image through a pre-built retrieval model, and acquire the target vector of the target image; the fusion vector acquisition module is used to fuse the original vector and the target vector to obtain a fusion vector, and store it in a vector retrieval database; the text vector acquisition module is used to receive the retrieval text input by the user, and extract the text vector of the retrieval text through a pre-built retrieval model; the fusion vector matching module is used to identify the fusion vector that matches the text vector in the vector retrieval database; the retrieval result generation module is used to acquire the corresponding surveillance image and target image according to the fusion vector, and restore the coordinate frame of the target image to the surveillance image to generate a retrieval result. Through the above-mentioned device for semantic retrieval of an image, the target information and background information of the image can be fully utilized through vector fusion to obtain a feature vector with richer representation, so that the retrieval result obtained by vector matching is more in line with the user's intention, which is conducive to improving the accuracy of matching the retrieval image with the user's retrieval text.

[0212] In the embodiments of the present application, a semantic retrieval device for an image may be a device, or a component, an integrated circuit, or a chip in a terminal. The device may be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc., and the non-mobile electronic device may be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc., which is not specifically limited in the embodiments of the present application.

[0213] In the embodiment of the present application, a device for semantic retrieval of an image may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0214] An apparatus for semantic retrieval of an image provided in an embodiment of the present application is capable of implementing each process implemented in the above-mentioned method embodiments, and will not be described again here to avoid repetition.

[0215] Embodiment 5

[0216] like Figure 6 As shown, an embodiment of the present application also provides an electronic device 600, including a processor 601, a memory 602, and a program or instruction stored in the memory 602 and executable on the processor 601. When the program or instruction is executed by the processor 601, each process of the above-mentioned method embodiment for semantic retrieval of an image is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0217] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0218] Embodiment 6

[0219] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned method embodiment for semantic retrieval of an image is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0220] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0221] Embodiment 7

[0222] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiment for semantic retrieval of an image, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0223] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0224] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0225] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0226] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0227] The above are only preferred embodiments of the present application and the technical principles used. The present application is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not deviate from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A semantic retrieval method for an image, characterized in that: The method comprises: Acquire a surveillance image to be processed from a surveillance video, perform target detection on the surveillance image, and capture a target image according to the target detection result; Acquire the original vector of the monitoring image through a pre-built retrieval model, and acquire the target vector of the target image; Fusing the original vector and the target vector to obtain a fused vector, and storing the fused vector in a vector retrieval database; Receiving a search text input by a user, and extracting a text vector of the search text through a pre-built search model; Identifying in the vector search database a fusion vector that matches the text vector; The corresponding monitoring image and target image are acquired according to the fusion vector, and the coordinate frame of the target image is restored to the monitoring image to generate a retrieval result.

2. The image semantic retrieval method according to claim 1, characterized in that: Capture the target image based on the target detection result, including: Acquire a coordinate range of a target in the target detection result in the monitoring image, and expand the coordinate range based on a preset expansion coefficient to obtain a capture range; The monitoring image is intercepted according to the interception range, and the intercepted portion is used as the target image.

3. The image semantic retrieval method according to claim 2, characterized in that: The intercept range is calculated using the formula: Among them, (x1, y1) and (x2, y2) are the position coordinates of the target, (x1′, y1′) and (x2′, y2′) are the position coordinates of the interception range, a is the expansion coefficient, w is the width of the detected target, and h is the height of the detected target.

4. The image semantic retrieval method according to claim 1, characterized in that: The original vector and the target vector are fused to obtain a fused vector, including: A preset fusion factor is obtained, and the original vector and the target vector are fused based on the fusion factor. Specifically, the fusion vector can be obtained by the following formula: v ij ′=αv i +(1-a)v ij ; Among them, v ij ′ is the fusion vector, α is the fusion factor, v i is the ith original vector, v ij is the jth target vector on the i-th original image.

5. The image semantic retrieval method according to claim 1, characterized in that: Identifying a fusion vector matching the text vector in the vector search database includes: Calculating the similarity between the text vector and the fused vector in the vector search database, and sorting the fused vector according to the similarity; A preset number of fused vectors are obtained in the sorting, or fused vectors satisfying a preset similarity condition are obtained in the sorting as output results.

6. The image semantic retrieval method according to claim 1, characterized in that: After generating the search results, the method further includes: In the case that the search result is inaccurate, obtaining positive sample images with semantics close to the search text and negative sample images with semantics opposite to the search text; Calculating the image vectors of the positive sample image and the negative sample image, calculating the classification loss according to the image vector and the text vector, and performing back propagation based on the classification loss to fine-tune the parameters of the large retrieval model; The fusion vectors in the original search results are reordered based on the fine-tuned large search model, and the secondary search results are determined according to the reordered results.

7. The image semantic retrieval method according to claim 6, characterized in that: Re-ranking the fusion vectors in the original search results based on the fine-tuned large search model includes: The large retrieval model uses the fine-tuned parameters to vectorize the retrieval text to obtain a secondary retrieval text vector, and substitutes the secondary retrieval text vector, the fusion vector and its original similarity into the following formula to calculate the secondary retrieval similarity: Among them, S′ is the secondary retrieval similarity, t′ is the secondary retrieval text vector, and v i is the i-th fused vector, S is the original similarity of the i-th fused vector; The fusion vectors are reordered according to the secondary retrieval similarity.

8. A semantic retrieval device for an image, characterized in that: The device comprises: An image acquisition module is used to acquire a surveillance image to be processed from a surveillance video, perform target detection on the surveillance image, and capture a target image according to the target detection result; A vector acquisition module, used to acquire the original vector of the monitoring image and the target vector of the target image through a pre-built retrieval model; A fusion vector acquisition module is used to fuse the original vector and the target vector to obtain a fusion vector, and store it in a vector retrieval database; A text vector acquisition module, used to receive a search text input by a user, and extract a text vector of the search text through a pre-built search model; A fusion vector matching module, used to identify a fusion vector matching the text vector in the vector search database; The retrieval result generating module is used to obtain the corresponding monitoring image and target image according to the fusion vector, and restore the coordinate frame of the target image to the monitoring image to generate the retrieval result.

9. An electronic device, characterized in that: The method comprises a processor, a memory and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of a semantic retrieval method for an image as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of a semantic retrieval method for an image as described in any one of claims 1-7 are implemented.

Images