A target retrieval method and related apparatus

Abstract

This application relates to the field of computer technology, and in particular to a target retrieval method and related apparatus for improving retrieval efficiency and accuracy. The method comprises: acquiring an image to be retrieved containing the target retrieval object; inputting the image to be retrieved into a target classification model, which includes a feature extraction layer and an output layer; obtaining visual features from the feature extraction layer and semantic features from the output layer; then fusing the semantic features and visual features to obtain the target fusion features corresponding to the image to be retrieved; and finally, based on the target fusion features, determining the target image that matches the image to be retrieved from each candidate image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a target retrieval method and related apparatus. Background Technology

[0002] With the continuous development of computer technology, visual target retrieval is being applied in more and more scenarios. Visual target retrieval compares the feature vector of the image to be detected with the feature vectors of massive images in the database to identify target images similar to the image to be detected.

[0003] In related technologies, machine learning models are typically used to obtain feature vectors of the image to be detected. However, the interpretability of feature vectors obtained through machine learning models is poor, and the feature dimension of the feature vector depends on the model output dimension, making it difficult to adapt well to feature types outside the training set. For example, a classification model trained using images of cats is not suitable for classifying and retrieving dogs, exhibiting poor versatility. To retrieve other categories of objects, retraining is required, which reduces retrieval efficiency. Summary of the Invention

[0004] This application provides a target retrieval method and related apparatus to improve the robustness and scene adaptability of features, and to improve retrieval efficiency and accuracy.

[0005] The specific technical solutions provided in this application are as follows:

[0006] Firstly, a target retrieval method includes:

[0007] Retrieve the image to be retrieved that contains the target object;

[0008] The image to be retrieved is input into a target classification model that includes a feature extraction layer and an output layer to obtain the visual features output by the feature extraction layer and the semantic features output by the output layer. The semantic features are used to characterize the classification result of the target detection object.

[0009] The semantic features and the visual features are fused to obtain the target fusion features corresponding to the image to be retrieved.

[0010] Based on the target fusion features corresponding to each candidate image and the image to be retrieved, at least one target image that matches the image to be retrieved is determined from the candidate images.

[0011] Optionally, the step of fusing the semantic features and the visual features to obtain the target fusion features corresponding to the image to be retrieved includes:

[0012] The semantic features and the visual features are concatenated to obtain the initial fusion features corresponding to the image to be retrieved, and the initial fusion features are directly used as the target fusion features; or,

[0013] The semantic features and the visual features are concatenated to obtain the initial fusion features corresponding to the image to be retrieved, and the initial fusion features corresponding to each image to be fused associated with the image to be retrieved are obtained. Based on the initial fusion features corresponding to the image to be retrieved and the initial fusion features corresponding to each image to be fused, the target fusion features are obtained.

[0014] Optionally, the step of concatenating the semantic features and the visual features to obtain the initial fusion features corresponding to the image to be retrieved includes:

[0015] The semantic features and the visual features are concatenated according to the specified feature concatenation order;

[0016] Based on the preset weight coefficients corresponding to the semantic features and the visual features respectively, the spliced ​​features are weighted to obtain the initial fusion features corresponding to the image to be retrieved.

[0017] Optionally, obtaining the target fusion feature based on the initial fusion feature corresponding to the image to be retrieved and the initial fusion feature corresponding to each image to be fused includes:

[0018] Based on the weight coefficients corresponding to the image to be retrieved and each image to be fused, a weighted sum is performed on the initial fusion features corresponding to the image to be retrieved and the initial fusion features corresponding to each image to be fused.

[0019] Based on the number of each image to be fused, the weighted summation of the fusion features is averaged to obtain the target fusion features.

[0020] Optionally, the semantic features include classification confidence and normalization information; the concatenation of the semantic features and the visual features to obtain the initial fusion features corresponding to the image to be retrieved includes:

[0021] Based on the normalization information contained in the semantic features, the classification confidence scores and visual features contained in the semantic features are normalized to obtain classification confidence scores and visual features that conform to a preset value range.

[0022] The confidence scores and visual features of each category that meet the preset value range are stitched together to obtain the initial fusion features corresponding to the image to be retrieved.

[0023] Optionally, the confidence scores for each category include confidence scores for each category and / or confidence scores for each attribute.

[0024] Optionally, the image to be retrieved is a video frame in a video, and the video may also contain other video frames;

[0025] Before obtaining the initial fusion features corresponding to each image to be fused that is associated with the image to be retrieved, the method further includes:

[0026] Other video frames that contain the target retrieval object and whose playback time is earlier than the video frame will be used as images to be fused and associated with the image to be retrieved.

[0027] For any one of the images to be merged, perform the following operations:

[0028] The image to be fused is input into the target classification model to obtain the visual features and semantic features corresponding to the image to be fused, wherein the semantic features of the image to be fused are used to characterize the classification result of the image to be fused.

[0029] Based on the visual and semantic features corresponding to any one of the images to be fused, the initial fusion features corresponding to any one of the images to be fused are obtained.

[0030] Optionally, before determining at least one target image matching the image to be retrieved from the candidate images based on the target fusion features corresponding to each candidate image and the image to be retrieved, the method further includes:

[0031] From each candidate image, at least one candidate region is determined, and each candidate region contains a retrieval object of a retrieval type.

[0032] For any one of the identified candidate regions, perform the following operations:

[0033] The candidate region is input into the target classification model to obtain the visual and semantic features corresponding to the candidate region, and the target fusion feature corresponding to the candidate region is obtained based on the visual and semantic features corresponding to the candidate region.

[0034] Record the mapping relationship between the target fusion features corresponding to any candidate region and the corresponding candidate image.

[0035] Optionally, determining at least one target image matching the image to be retrieved from the candidate images based on the target fusion features corresponding to each candidate image and the image to be retrieved includes:

[0036] Calculate the similarity between the target fusion features corresponding to each candidate image and the target fusion features corresponding to the image to be retrieved;

[0037] Based on the calculated similarity scores, at least one target image matching the image to be retrieved is determined from candidate images whose similarity to the image to be retrieved is greater than a preset threshold.

[0038] Secondly, a target retrieval device includes:

[0039] The acquisition unit is used to acquire the image to be retrieved that contains the target retrieval object;

[0040] The output unit is used to input the image to be retrieved into a target classification model that includes a feature extraction layer and an output layer, to obtain the visual features output by the feature extraction layer and the semantic features output by the output layer. The semantic features are used to characterize the classification result of the target detection object.

[0041] A fusion unit is used to fuse the semantic features and the visual features to obtain the target fusion features corresponding to the image to be retrieved.

[0042] The matching unit is used to determine at least one target image that matches the image to be retrieved from the candidate images based on the target fusion features corresponding to each candidate image and the image to be retrieved.

[0043] Optionally, when fusing the semantic features and the visual features to obtain the target fused features corresponding to the image to be retrieved, the fusion unit is specifically used for:

[0044] The semantic features and the visual features are concatenated to obtain the initial fusion features corresponding to the image to be retrieved, and the initial fusion features are directly used as the target fusion features; or,

[0045] The semantic features and the visual features are concatenated to obtain the initial fusion features corresponding to the image to be retrieved, and the initial fusion features corresponding to each image to be fused associated with the image to be retrieved are obtained. Based on the initial fusion features corresponding to the image to be retrieved and the initial fusion features corresponding to each image to be fused, the target fusion features are obtained.

[0046] Optionally, when concatenating the semantic features and the visual features to obtain the initial fusion features corresponding to the image to be retrieved, the fusion unit is specifically used for:

[0047] The semantic features and the visual features are concatenated according to the specified feature concatenation order;

[0048] Based on the preset weight coefficients corresponding to the semantic features and the visual features respectively, the spliced ​​features are weighted to obtain the initial fusion features corresponding to the image to be retrieved.

[0049] Optionally, when obtaining the target fusion feature based on the initial fusion feature corresponding to the image to be retrieved and the initial fusion feature corresponding to each image to be fused, the fusion unit is specifically used for:

[0050] Based on the weight coefficients corresponding to the image to be retrieved and each image to be fused, a weighted sum is performed on the initial fusion features corresponding to the image to be retrieved and the initial fusion features corresponding to each image to be fused.

[0051] Based on the number of each image to be fused, the weighted summation of the fusion features is averaged to obtain the target fusion features.

[0052] Optionally, the semantic features include classification confidence and normalization information; when concatenating the semantic features and the visual features to obtain the initial fusion features corresponding to the image to be retrieved, the fusion unit is specifically used for:

[0053] Based on the normalization information contained in the semantic features, the classification confidence scores and visual features contained in the semantic features are normalized to obtain classification confidence scores and visual features that conform to a preset value range.

[0054] The confidence scores and visual features of each category that meet the preset value range are stitched together to obtain the initial fusion features corresponding to the image to be retrieved.

[0055] Optionally, the confidence scores for each category include confidence scores for each category and / or confidence scores for each attribute.

[0056] Optionally, the image to be retrieved is a video frame in a video, and the video may also contain other video frames;

[0057] Before acquiring the initial fusion features corresponding to each image to be fused associated with the image to be retrieved, the fusion unit is further configured to:

[0058] Other video frames that contain the target retrieval object and whose playback time is earlier than the video frame will be used as images to be fused and associated with the image to be retrieved.

[0059] For any one of the images to be merged, perform the following operations:

[0060] The image to be fused is input into the target classification model to obtain the visual features and semantic features corresponding to the image to be fused, wherein the semantic features of the image to be fused are used to characterize the classification result of the image to be fused.

[0061] Based on the visual and semantic features corresponding to any one of the images to be fused, the initial fusion features corresponding to any one of the images to be fused are obtained.

[0062] Optionally, before determining at least one target image matching the image to be retrieved from the candidate images based on the target fusion features corresponding to each candidate image and the image to be retrieved, the fusion unit is further configured to:

[0063] From each candidate image, at least one candidate region is determined, and each candidate region contains a retrieval object of a retrieval type.

[0064] For any one of the identified candidate regions, perform the following operations:

[0065] The candidate region is input into the target classification model to obtain the visual and semantic features corresponding to the candidate region, and the target fusion feature corresponding to the candidate region is obtained based on the visual and semantic features corresponding to the candidate region.

[0066] Record the mapping relationship between the target fusion features corresponding to any candidate region and the corresponding candidate image.

[0067] Optionally, when determining at least one target image matching the image to be retrieved from the candidate images based on the target fusion features corresponding to each candidate image and the image to be retrieved, the matching unit is specifically used for:

[0068] Calculate the similarity between the target fusion features corresponding to each candidate image and the target fusion features corresponding to the image to be retrieved;

[0069] Based on the calculated similarity scores, at least one target image matching the image to be retrieved is determined from candidate images whose similarity to the image to be retrieved is greater than a preset threshold.

[0070] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the above-described target retrieval method.

[0071] Fourthly, embodiments of this application provide a computer-readable storage medium including a computer program, which, when run on an electronic device, causes the electronic device to perform the steps of the target retrieval method described above.

[0072] Fifthly, embodiments of this application provide a computer program product, the program product including a computer program stored in a computer-readable storage medium, wherein a processor of an electronic device reads from and executes the computer program from the computer-readable storage medium, causing the electronic device to perform the steps of the above-described target retrieval method.

[0073] In summary, in this embodiment, after obtaining the image to be retrieved containing the target object, the image is input into a target classification model. This model includes a feature extraction layer and an output layer. Visual features are obtained from the feature extraction layer, and semantic features are obtained from the output layer. Then, the semantic and visual features are fused to obtain the target fusion feature corresponding to the image to be retrieved. Based on the target fusion feature, the target image matching the image to be retrieved is determined from the candidate images. Since the target fusion feature contains both visual and semantic features, and visual features have good interpretability and can adapt well to feature types outside the training set, while semantic features have good accuracy and robustness, the target fusion feature has better robustness and wider scene adaptability, thereby improving the accuracy and detection efficiency of image retrieval. Attached Figure Description

[0074] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 This is a schematic diagram of an application scenario provided in the embodiments of this application;

[0076] Figure 2 This is a flowchart illustrating a target retrieval method provided in an embodiment of this application;

[0077] Figure 3 This is a logical diagram illustrating a feature splicing method provided in an embodiment of this application;

[0078] Figure 4 This is a logical schematic diagram of determining an image to be fused, provided in an embodiment of this application;

[0079] Figure 5This is a logical schematic diagram illustrating the determination of target fusion features provided in an embodiment of this application;

[0080] Figure 6 This is a logical diagram illustrating a target retrieval process provided in an embodiment of this application;

[0081] Figure 7 This is a schematic diagram of the structure of a target retrieval device provided in an embodiment of this application;

[0082] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0083] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0084] It should be noted that in this paper, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0085] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0086] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0087] refer to Figure 1 The diagram illustrates a possible application scenario provided by an embodiment of this application. This application scenario includes at least a terminal device 110 and a server 120. The number of terminal devices 110 can be one or more, and the number of servers 120 can also be one or more. This application does not specifically limit the number of terminal devices 110 and servers 120.

[0088] The terminal device 110 is equipped with an application that has information processing functions such as information verification and information retrieval. This application can be a client application, a web application, a mini-program application, etc. The terminal device 110 can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, smart home appliance, vehicle terminal, aircraft, etc., but is not limited to these.

[0089] Server 120 can be a backend server for an application, providing corresponding information verification services for the application. Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0090] Terminal device 110 and server 120 can be directly or indirectly connected via wired or wireless communication and communicate and transmit data through a network; this application does not impose any restrictions on this.

[0091] The target retrieval method mentioned in this application can be applied to a terminal device or a server alone, or it can be executed by both a terminal device and a server.

[0092] For example, after the terminal device obtains the image to be retrieved containing the target object, it inputs the image to be retrieved into the target classification model to obtain the visual features output by the feature extraction layer and the semantic features output by the output layer. Then, the semantic features and visual features are fused to obtain the target fusion features corresponding to the image to be retrieved. Based on the target fusion features, the target image that matches the image to be retrieved is determined from each candidate image.

[0093] For example, after the server obtains the image to be retrieved containing the target object, it inputs the image to be retrieved into the target classification model to obtain the visual features output by the feature extraction layer and the semantic features output by the output layer. Then, the semantic features and visual features are fused to obtain the target fusion features corresponding to the image to be retrieved. Based on the target fusion features, the target image that matches the image to be retrieved is determined from each candidate image.

[0094] For example, after the terminal device acquires an image containing the target object, it sends the image to the server. Upon receiving the image, the server inputs it into a target classification model to obtain the visual features output by the feature extraction layer and the semantic features output by the output layer. The server then fuses the semantic and visual features to obtain the target fusion feature corresponding to the image, and sends this feature to the terminal device. Based on the target fusion feature, the terminal device determines the target image that matches the image to be retrieved from among the candidate images.

[0095] See Figure 2 The diagram shown is a flowchart of a target retrieval method provided in an embodiment of this application. This process can be applied to electronic devices, which can be servers or terminal devices. The following description uses a terminal device as an example. The specific process is as follows:

[0096] S201. The terminal device acquires the image to be retrieved, which contains the target object.

[0097] S202. The terminal device inputs the image to be retrieved into the target classification model, which includes a feature extraction layer and an output layer, to obtain the visual features output by the feature extraction layer and the semantic features output by the output layer. The semantic features are used to characterize the classification result of the target detection object.

[0098] It should be noted that, in the embodiments of this application, the visual features include Speeded UpRobust Feature (SURF) features, Histogram of Oriented Gradient (HOG) features, and Haar features, but are not limited to these.

[0099] Semantic features may include confidence scores for each category, which includes confidence scores for each class and / or confidence scores for each attribute. Categories include, but are not limited to, pedestrians, motor vehicles, non-motor vehicles, ships, and airplanes. Attributes include, but are not limited to, color, vehicle type, and whether or not glasses are worn. Semantic features may also include normalization information, including normalization width and normalization height.

[0100] Semantic features are represented by feature vector a1. Assuming that the semantic features output by the target classification model have n1 categories, n2 attributes, l normalization width, and h normalization height, the category confidence, attribute confidence, normalization width, and normalization height are encoded into feature vector a1 in a specified order. Feature vector a1 can be represented as [category 1 confidence, category 2 confidence, ..., category n1 confidence, attribute 1 confidence, attribute 2 confidence, ..., attribute n2 confidence, l, h]. The length of feature vector a1 is n1 + n2 + 2.

[0101] Where l = lt / lp, h = ht / hp, lp is the width of the original image, hp is the height of the original image, lt is the width of the target image, and ht is the height of the target image.

[0102] For example, the categories include pedestrians and vehicles, and the attributes include red, orange, yellow, and green. The feature vector a1 can be represented as [pedestrian confidence, vehicle confidence, red confidence, orange confidence, yellow confidence, green confidence, l, h].

[0103] The visual features are represented by feature vector a2. Assuming that the width, height and number of channels in the feature size output by the feature extraction layer are w, z and c respectively, then the length of feature vector a2 is w*z*c.

[0104] S203. The terminal device performs feature fusion on semantic features and visual features to obtain the target fusion features corresponding to the image to be retrieved.

[0105] In S203, the embodiments of this application can directly fuse semantic features and visual features, or can fuse semantic features and visual features by combining images associated with the image to be retrieved. The two feature fusion methods are described below.

[0106] Method 1: The terminal device concatenates semantic features and visual features to obtain the initial fusion features corresponding to the image to be retrieved, and uses the initial fusion features directly as the target fusion features.

[0107] In the process of concatenating semantic features and visual features to obtain the initial fusion features corresponding to the image to be retrieved, the terminal device can concatenate semantic features and visual features according to the specified feature concatenation order, and then weight the concatenated features based on the preset weight coefficients corresponding to the semantic features and visual features respectively to obtain the initial fusion features corresponding to the image to be retrieved.

[0108] In this paper, feature vector a1 represents semantic features, feature vector a2 represents visual features, and feature vector a represents the initial fused features. Preset weight coefficients are used to represent the weight percentage of each feature; these preset weight coefficients can be set according to specific application scenarios.

[0109] It should be noted that in the embodiments of this application, the feature extraction layer can also be called the network feature layer, and the output layer can also be called the network output layer.

[0110] See Figure 3 As shown, assuming the feature concatenation order is semantic features and visual features joined end-to-end, then the feature vector a can be represented as {p1*a1, p2*a2}, where p1 represents the weight coefficient corresponding to the semantic feature, p2 represents the weight coefficient corresponding to the visual feature, and the length of the fused feature vector a is the sum of the lengths of feature vectors a1 and a2. For example, p1 = 0.5, p2 = 0.5.

[0111] In some embodiments, if the semantic features include classification confidence and normalization information, the terminal device concatenates the semantic features and visual features to obtain the initial fusion features corresponding to the image to be retrieved, including:

[0112] Based on the normalization information contained in the semantic features, the classification confidence and visual features contained in the semantic features are normalized to obtain the classification confidence and visual features that conform to the preset value range.

[0113] The confidence scores and visual features of each category that meet the preset value range are spliced ​​together to obtain the initial fusion features corresponding to the image to be retrieved.

[0114] For example, the preset value range is 0 to 1. The terminal device normalizes the classification confidence and visual features contained in the semantic features based on the normalization information contained in the semantic features. After normalization, the values ​​of each component in the classification confidence and visual features are all between 0 and 1.

[0115] It should be noted that, in this embodiment of the application, the terminal device can also perform normalization processing in the feature extraction layer to obtain the confidence scores of each category that conform to the preset value range. Correspondingly, in the process of splicing semantic features and visual features, the visual features are normalized based on the normalization information contained in the semantic features to obtain the visual features that conform to the preset value range.

[0116] Method 2: The terminal device concatenates semantic features and visual features to obtain the initial fusion features corresponding to the image to be retrieved, and obtains the initial fusion features corresponding to each image to be fused associated with the image to be retrieved. Based on the initial fusion features corresponding to the image to be retrieved and the initial fusion features corresponding to each image to be fused, the target fusion features are obtained.

[0117] Since the process of the terminal device in Method 2 splicing semantic features and visual features to obtain the initial fusion features corresponding to the image to be retrieved is the same as the process of the terminal device in Method 1 splicing semantic features and visual features to obtain the initial fusion features corresponding to the image to be retrieved, it will not be described again here.

[0118] In this embodiment, if the image to be retrieved is a video frame in a video, and the video contains at least one other video frame besides the retrieved image, the terminal device can use other video frames that contain the target retrieval object and whose playback time is earlier than the retrieved image as the images to be fused associated with the retrieved image. That is, the terminal device can use historical video frames containing the target retrieval object as the images to be fused associated with the retrieved image. Then, for any one of the images to be fused, the terminal device performs the following operations:

[0119] Input any image to be fused into the target classification model to obtain the visual features and semantic features corresponding to any image to be fused. The semantic features of any image to be fused are used to characterize the classification result of any image to be fused.

[0120] Based on the visual and semantic features of any image to be fused, the initial fusion features of any image to be fused are obtained.

[0121] It should be noted that, in the embodiments of this application, the image to be retrieved as a video frame can refer to a specific video frame within the video, or it can refer to a portion of the image contained within a specific video frame. Since the generation process of the initial fusion features corresponding to each image to be fused is the same as the generation process of the initial fusion features corresponding to the image to be retrieved, it will not be described again here.

[0122] Suppose that the video contains m video frames, namely video frame 1, video frame 2, ..., video frame i, ..., video frame m, and the playback times of the m video frames are as follows: video frame 1, video frame 2, ..., video frame i, ..., video frame m. The image to be retrieved is video frame i.

[0123] If among m video frames, there are n video frames containing the target retrieval object: video frame k1, video frame k2, ..., video frame ki, ..., video frame kn, and other video frames containing the target retrieval object and whose playback time is earlier than the image to be retrieved: video frame k1, video frame k2, ..., video frame ki-1, then obviously, the image to be fused includes: video frame k1, video frame k2, ..., video frame ki-1.

[0124] The terminal device generates corresponding initial fusion features for video frames k1, k2, ..., ki-1 respectively. The initial fusion features corresponding to video frames k1, k2, ..., ki-1 are represented as a(k1), a(k2), ..., a(ki-1) respectively, and the initial fusion feature corresponding to the image to be retrieved is represented as a(ki).

[0125] For example, see Figure 4 As shown, the video contains 10 video frames, including video frame 1, video frame 2, ..., video frame 10. Assume that the image to be retrieved is video frame 5, and the target retrieval object is the pedestrian in video frame 5. Among the 10 video frames, the other video frames that contain the target retrieval object and whose playback time is earlier than video frame 5 are: video frame 1, video frame 2, video frame 3, and video frame 4. The terminal device uses video frames 1, 2, 3, and 4 as the images to be fused.

[0126] See Figure 5 As shown, the terminal device obtains the initial fusion features corresponding to video frames 1, 2, 3, and 4 respectively. The initial fusion feature corresponding to video frame 1 is a(k1), the initial fusion feature corresponding to video frame 2 is a(k2), the initial fusion feature corresponding to video frame 3 is a(k3), the initial fusion feature corresponding to video frame 4 is a(k4), and the initial fusion feature corresponding to video frame 5 is a(k5).

[0127] Specifically, based on the initial fusion features corresponding to the image to be retrieved and the initial fusion features corresponding to each image to be fused, the terminal device can obtain the target fusion features in the following ways:

[0128] A. The terminal device performs a weighted summation of the initial fusion features corresponding to the image to be retrieved and the initial fusion features corresponding to each image to be fused, based on the weight coefficients corresponding to the image to be retrieved and each image to be fused.

[0129] It should be noted that, in the embodiments of this application, the weight coefficients corresponding to the image to be retrieved and each image to be fused can be determined according to the number of images to be fused, or they can be preset, and there is no limitation on this.

[0130] Suppose that the weight coefficients of each image to be fused are q1, q2, ..., qi-1, and the weight coefficient of the image to be retrieved is qi. The fusion feature obtained after weighted summation can be expressed as q1*a(k1)+q2*a(k2)+...++qi-1*a(ki-1)+qi*a(ki).

[0131] Taking video frame 5 as an example, we assume that the weight coefficients of the image to be retrieved and each image to be fused are all 1. The fusion feature obtained after weighted summation is a(k1)+a(k2)+a(k3)+a(k4)+a(k5).

[0132] B. The terminal device averages the weighted summation of the fusion features based on the number of images to be fused, and obtains the target fusion features.

[0133] Assume that the weight coefficients for the image to be retrieved and each image to be fused are both 1. The target fusion feature is represented by the following formula, which can be calculated using the following formula:

[0134]

[0135] Taking video frame 5 as the image to be retrieved as an example, the number of images to be fused is 4. The fusion features obtained after weighted summation are averaged to obtain the target fusion features, which are represented as {a(k1)+a(k2)+a(k3)+a(k4)+a(k5)} / 5.

[0136] It should be noted that, in the embodiments of this application, the image to be fused may also be other video frames in the video that contain the target retrieval object within a specified duration and whose playback time is earlier than the video frame.

[0137] S204. The terminal device determines at least one target image that matches the image to be retrieved from each candidate image based on the target fusion features corresponding to each candidate image and the image to be retrieved.

[0138] Specifically, when executing S204, the following methods may be used, but are not limited to:

[0139] The terminal device calculates the target fusion features corresponding to each candidate image and the similarity between them and the target fusion features corresponding to the image to be retrieved. Then, based on the calculated similarity, it determines at least one target image that matches the image to be retrieved from the candidate images whose similarity to the image to be retrieved is greater than a preset threshold.

[0140] For example, but not limited to, cosine similarity can be used as the similarity between the target fusion feature corresponding to the image to be retrieved and the target fusion feature corresponding to any candidate image. The formula for calculating cosine similarity is as follows:

[0141]

[0142] Among them, s x Indicates similarity. b represents the target fusion feature corresponding to the image to be retrieved. x This represents the target fusion feature corresponding to any candidate image.

[0143] In this embodiment of the application, when the semantic features and visual features are normalized to 0 to 1, due to the feature vector With b x Each component ranges from 0 to 1, therefore the calculated s x The value range of is also from 0 to 1, and s x The larger the value of , the better. With b x The higher the similarity between them, the better. x The smaller the value of , the better. With b x The lower the similarity between them.

[0144] Based on the calculated similarity scores, in the process of determining the target image matching the image to be retrieved from candidate images whose similarity to the image to be retrieved is greater than a preset threshold, the terminal device filters the candidate images using the preset threshold to obtain a set of candidate images with similarity greater than the preset threshold. Then, according to the order of similarity values ​​from largest to smallest, a set number of target images are selected sequentially from the candidate image set. Furthermore, the terminal device can also display the determined target images on the operation interface.

[0145] In some embodiments, before determining at least one target image matching the image to be retrieved from each candidate image based on the target fusion features corresponding to each candidate image and the image to be retrieved, the terminal device may also determine at least one candidate region contained in each candidate image, wherein each candidate region contains a retrieval object of a retrieval type.

[0146] The terminal device can perform the following operations on any one of the identified candidate regions:

[0147] Input any candidate region into the target classification model to obtain the visual and semantic features corresponding to the candidate region, and obtain the target fusion features corresponding to the candidate region based on the visual and semantic features corresponding to the candidate region.

[0148] Record the mapping relationship between the target fusion features corresponding to any candidate region and the corresponding candidate image.

[0149] Since the process of obtaining the target fusion features corresponding to any candidate region is the same as the process of obtaining the target fusion features corresponding to the image to be retrieved mentioned above, it will not be repeated here.

[0150] In this embodiment of the application, the candidate image can be a video frame or an image.

[0151] For example, suppose candidate image 1 contains a pedestrian and a car. The terminal device determines candidate region 1 and candidate region 2 from candidate image 1. Candidate region 1 contains a pedestrian, and candidate region 2 contains a car. Then, for candidate region 1, the terminal device inputs candidate region 1 into the target classification model to obtain the visual and semantic features corresponding to candidate region 1. Based on the visual and semantic features corresponding to candidate region 1, the terminal device obtains the target fusion features corresponding to candidate region 1 and records the mapping relationship between the target fusion features corresponding to candidate region 1 and candidate image 1. Similarly, the terminal device can obtain the target fusion features corresponding to candidate region 2 and record the mapping relationship between the target fusion features corresponding to candidate region 1 and candidate image 1.

[0152] The present application will now be described with reference to a specific embodiment.

[0153] See Figure 6 As shown, in a video surveillance scenario, video frame 4 in the surveillance video is taken as the image to be retrieved. The target object in the image to be retrieved is a vehicle. After the terminal device obtains video frame 4, it obtains the initial fusion feature corresponding to video frame 4. Assuming that the images to be fused associated with video frame 4 are video frames 1, 2, and 3, the terminal device obtains the target fusion feature corresponding to the image to be retrieved based on the initial fusion features corresponding to video frames 1, 2, and 3, and based on the initial fusion feature corresponding to video frame 4.

[0154] Then, the terminal device matches the target fusion features corresponding to the image to be retrieved with the target fusion features corresponding to candidate image 1. The target fusion features corresponding to candidate image 1 include the target fusion features corresponding to candidate region 1 and the target fusion features corresponding to candidate region 2. Based on the target fusion features corresponding to the candidate image and the target fusion features corresponding to the image to be retrieved, it is determined that candidate image 1 matches the image to be retrieved. Specifically, it is determined that candidate region 1 in candidate image 1 matches the image to be retrieved.

[0155] Based on the same inventive concept, see [reference] Figure 7As shown, this application embodiment provides a target retrieval device, the target retrieval device 700 including at least:

[0156] The acquisition unit 701 is used to acquire the image to be retrieved containing the target retrieval object;

[0157] The output unit 702 is used to input the image to be retrieved into a target classification model containing a feature extraction layer and an output layer to obtain the visual features output by the feature extraction layer and the semantic features output by the output layer. The semantic features are used to characterize the classification result of the target detection object.

[0158] The fusion unit 703 is used to perform feature fusion on the semantic features and the visual features to obtain the target fusion features corresponding to the image to be retrieved;

[0159] The matching unit 704 is used to determine at least one target image that matches the image to be retrieved from the candidate images based on the target fusion features corresponding to each candidate image and the image to be retrieved.

[0160] Optionally, when fusing the semantic features and the visual features to obtain the target fused features corresponding to the image to be retrieved, the fusion unit 703 is specifically used for:

[0161] The semantic features and the visual features are concatenated to obtain the initial fusion features corresponding to the image to be retrieved, and the initial fusion features are directly used as the target fusion features; or,

[0162] The semantic features and the visual features are concatenated to obtain the initial fusion features corresponding to the image to be retrieved, and the initial fusion features corresponding to each image to be fused associated with the image to be retrieved are obtained. Based on the initial fusion features corresponding to the image to be retrieved and the initial fusion features corresponding to each image to be fused, the target fusion features are obtained.

[0163] Optionally, when concatenating the semantic features and the visual features to obtain the initial fusion features corresponding to the image to be retrieved, the fusion unit 703 is specifically used for:

[0164] The semantic features and the visual features are concatenated according to the specified feature concatenation order;

[0165] Based on the preset weight coefficients corresponding to the semantic features and the visual features respectively, the spliced ​​features are weighted to obtain the initial fusion features corresponding to the image to be retrieved.

[0166] Optionally, when obtaining the target fusion feature based on the initial fusion feature corresponding to the image to be retrieved and the initial fusion feature corresponding to each image to be fused, the fusion unit 703 is specifically used for:

[0167] Based on the weight coefficients corresponding to the image to be retrieved and each image to be fused, a weighted sum is performed on the initial fusion features corresponding to the image to be retrieved and the initial fusion features corresponding to each image to be fused.

[0168] Based on the number of each image to be fused, the weighted summation of the fusion features is averaged to obtain the target fusion features.

[0169] Optionally, the semantic features include classification confidence and normalization information; when concatenating the semantic features and the visual features to obtain the initial fusion features corresponding to the image to be retrieved, the fusion unit 703 is specifically used for:

[0170] Based on the normalization information contained in the semantic features, the classification confidence scores and visual features contained in the semantic features are normalized to obtain classification confidence scores and visual features that conform to a preset value range.

[0171] The confidence scores and visual features of each category that meet the preset value range are stitched together to obtain the initial fusion features corresponding to the image to be retrieved.

[0172] Optionally, the confidence scores for each category include confidence scores for each category and / or confidence scores for each attribute.

[0173] Optionally, the image to be retrieved is a video frame in a video, and the video may also contain other video frames;

[0174] Before acquiring the initial fusion features corresponding to each image to be fused associated with the image to be retrieved, the fusion unit 703 is further configured to:

[0175] Other video frames that contain the target retrieval object and whose playback time is earlier than the video frame will be used as images to be fused and associated with the image to be retrieved.

[0176] For any one of the images to be merged, perform the following operations:

[0177] The image to be fused is input into the target classification model to obtain the visual features and semantic features corresponding to the image to be fused, wherein the semantic features of the image to be fused are used to characterize the classification result of the image to be fused.

[0178] Based on the visual and semantic features corresponding to any one of the images to be fused, the initial fusion features corresponding to any one of the images to be fused are obtained.

[0179] Optionally, before determining at least one target image matching the image to be retrieved from the candidate images based on the target fusion features corresponding to each candidate image and the image to be retrieved, the fusion unit 703 is further configured to:

[0180] From each candidate image, at least one candidate region is determined, and each candidate region contains a retrieval object of a retrieval type.

[0181] For any one of the identified candidate regions, perform the following operations:

[0182] The candidate region is input into the target classification model to obtain the visual and semantic features corresponding to the candidate region, and the target fusion feature corresponding to the candidate region is obtained based on the visual and semantic features corresponding to the candidate region.

[0183] Record the mapping relationship between the target fusion features corresponding to any candidate region and the corresponding candidate image.

[0184] Optionally, when determining at least one target image matching the image to be retrieved from the candidate images based on the target fusion features corresponding to each candidate image and the image to be retrieved, the matching unit 704 is specifically used for:

[0185] Calculate the similarity between the target fusion features corresponding to each candidate image and the target fusion features corresponding to the image to be retrieved;

[0186] Based on the calculated similarity scores, at least one target image matching the image to be retrieved is determined from candidate images whose similarity to the image to be retrieved is greater than a preset threshold.

[0187] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0188] Regarding the apparatus in the above embodiments, the specific manner in which each unit executes the request has been described in detail in the embodiments related to the method, and will not be elaborated here.

[0189] In this embodiment, after obtaining the image to be retrieved containing the target object, the image is input into a target classification model. The target classification model includes a feature extraction layer and an output layer. Visual features are obtained from the feature extraction layer, and semantic features are obtained from the output layer. Then, the semantic and visual features are fused to obtain the target fusion feature corresponding to the image to be retrieved. Based on the target fusion feature, the target image matching the image to be retrieved is determined from the candidate images. Since the target fusion feature contains both visual and semantic features, and visual features have good interpretability and can adapt well to feature types outside the training set, while semantic features have good accuracy and robustness, the target fusion feature has better robustness and wider scene adaptability, thereby improving the accuracy and detection efficiency of image retrieval.

[0190] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0191] Based on the same inventive concept, embodiments of this application also provide an electronic device. In one embodiment, the electronic device can be a server or a terminal device. See also... Figure 8 As shown, it is a schematic diagram of the structure of a possible electronic device provided in an embodiment of this application. Figure 8 In the electronic device 800, there are: processor 810 and memory 820.

[0192] The memory 820 stores a computer program that can be executed by the processor 810. The processor 810 can execute the steps of the target retrieval method described above by executing the instructions stored in the memory 820.

[0193] Memory 820 may be volatile memory, such as random-access memory (RAM); memory 820 may also be non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 820 may be any other medium capable of carrying or storing desired program code having the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 820 may also be a combination of the above-mentioned memories.

[0194] The processor 810 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 810 implements the aforementioned target retrieval method when executing a computer program stored in the memory 820.

[0195] In some embodiments, the processor 810 and the memory 820 may be implemented on the same chip, while in other embodiments they may be implemented on separate chips.

[0196] This application embodiment does not limit the specific connection medium between the processor 810 and the memory 820. This application embodiment takes the connection between the processor 810 and the memory 820 via a bus as an example. Figure 8 The diagram uses thick lines to describe the connections between other components; these are merely illustrative and not intended to be limiting. Buses can be categorized as address buses, data buses, control buses, etc. For ease of description, Figure 8 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0197] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium including a computer program. When the computer program is run on an electronic device, it causes the electronic device to perform the steps of the aforementioned target retrieval method. In some possible implementations, various aspects of the target retrieval method provided in this application can also be implemented as a program product including a computer program. When the program product is run on an electronic device, the computer program causes the electronic device to perform the steps in the aforementioned target retrieval method. For example, the electronic device can perform actions such as... Figure 2 The steps are shown in the figure.

[0198] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0199] The program product of the embodiments of this application may be a CD-ROM and include a computer program, and may run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a computer program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0200] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a computer program for use by or in conjunction with a command execution system, apparatus, or device.

[0201] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0202] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0203] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0204] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0205] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0206] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A target retrieval method, characterized in that, include: Retrieve the image to be retrieved that contains the target object; The image to be retrieved is input into a target classification model that includes a feature extraction layer and an output layer to obtain the visual features output by the feature extraction layer and the semantic features output by the output layer. The semantic features are used to characterize the classification result of the target retrieval object. The visual features include at least one of acceleration robust features, histogram of oriented gradients features, and Haar features. The semantic features and the visual features are fused to obtain the target fusion features corresponding to the image to be retrieved. From each candidate image, at least one candidate region is determined, and each candidate region contains a retrieval object of a retrieval type. For any candidate region among the identified candidate regions, perform the following operations: input the candidate region into the target classification model to obtain the visual and semantic features corresponding to the candidate region, and obtain the target fusion feature corresponding to the candidate region based on the visual and semantic features corresponding to the candidate region. Record the mapping relationship between the target fusion features corresponding to any candidate region and the corresponding candidate image; Based on the target fusion features corresponding to each candidate image and the image to be retrieved, at least one target image matching the image to be retrieved is determined from the candidate images. The target fusion features corresponding to each candidate image include the target fusion features corresponding to at least one candidate region contained in the candidate image.

2. The method as described in claim 1, characterized in that, The step of fusing the semantic features and the visual features to obtain the target fusion features corresponding to the image to be retrieved includes: The semantic features and the visual features are concatenated to obtain the initial fusion features corresponding to the image to be retrieved, and the initial fusion features are directly used as the target fusion features; or, The semantic features and the visual features are concatenated to obtain the initial fusion features corresponding to the image to be retrieved, and the initial fusion features corresponding to each image to be fused associated with the image to be retrieved are obtained. Based on the initial fusion features corresponding to the image to be retrieved and the initial fusion features corresponding to each image to be fused, the target fusion features are obtained.

3. The method as described in claim 2, characterized in that, The step of concatenating the semantic features and the visual features to obtain the initial fusion features corresponding to the image to be retrieved includes: The semantic features and the visual features are concatenated according to the specified feature concatenation order; Based on the preset weight coefficients corresponding to the semantic features and the visual features respectively, the spliced ​​features are weighted to obtain the initial fusion features corresponding to the image to be retrieved.

4. The method as described in claim 2, characterized in that, The process of obtaining the target fusion feature based on the initial fusion feature corresponding to the image to be retrieved and the initial fusion feature corresponding to each image to be fused includes: Based on the weight coefficients corresponding to the image to be retrieved and each image to be fused, a weighted sum is performed on the initial fusion features corresponding to the image to be retrieved and the initial fusion features corresponding to each image to be fused. Based on the number of each image to be fused, the weighted summation of the fusion features is averaged to obtain the target fusion features.

5. The method as described in claim 2, characterized in that, The semantic features include confidence scores and normalization information for each category; The step of concatenating the semantic features and the visual features to obtain the initial fusion features corresponding to the image to be retrieved includes: Based on the normalization information contained in the semantic features, the classification confidence scores and visual features contained in the semantic features are normalized to obtain classification confidence scores and visual features that conform to a preset value range. The confidence scores and visual features of each category that meet the preset value range are stitched together to obtain the initial fusion features corresponding to the image to be retrieved.

6. The method as described in claim 5, characterized in that, The confidence scores for each category include confidence scores for each category and / or confidence scores for each attribute.

7. The method as described in claim 2, characterized in that, The image to be retrieved is a video frame in a video, and the video also contains other video frames; Before obtaining the initial fusion features corresponding to each image to be fused that is associated with the image to be retrieved, the method further includes: Other video frames that contain the target retrieval object and whose playback time is earlier than the video frame will be used as images to be fused and associated with the image to be retrieved. For any one of the images to be merged, perform the following operations: The image to be fused is input into the target classification model to obtain the visual features and semantic features corresponding to the image to be fused, wherein the semantic features of the image to be fused are used to characterize the classification result of the image to be fused. Based on the visual and semantic features corresponding to any one of the images to be fused, the initial fusion features corresponding to any one of the images to be fused are obtained.

8. The method according to any one of claims 1-7, characterized in that, The step of determining at least one target image matching the target image from the candidate images based on the target fusion features corresponding to each candidate image and the image to be retrieved includes: Calculate the similarity between the target fusion features corresponding to each candidate image and the target fusion features corresponding to the image to be retrieved; Based on the calculated similarity scores, at least one target image matching the image to be retrieved is determined from candidate images whose similarity to the image to be retrieved is greater than a preset threshold.

9. A target retrieval device, characterized in that, include: The acquisition unit is used to acquire the image to be retrieved that contains the target retrieval object; The output unit is used to input the image to be retrieved into a target classification model that includes a feature extraction layer and an output layer, to obtain the visual features output by the feature extraction layer and the semantic features output by the output layer. The semantic features are used to characterize the classification result of the target retrieval object. The visual features include at least one of acceleration robust features, histogram of oriented gradients features, and Haar features. A fusion unit is used to fuse the semantic features and the visual features to obtain the target fusion features corresponding to the image to be retrieved. From each candidate image, at least one candidate region is determined, and each candidate region contains a retrieval object of a retrieval type. For any candidate region among the determined candidate regions, the following operations are performed: input the any candidate region into the target classification model to obtain the visual features and semantic features corresponding to the any candidate region, and obtain the target fusion features corresponding to the any candidate region based on the visual features and semantic features corresponding to the any candidate region. Record the mapping relationship between the target fusion features corresponding to any candidate region and the corresponding candidate image; The matching unit is used to determine at least one target image that matches the image to be retrieved from the candidate images based on the target fusion features corresponding to each candidate image and the image to be retrieved. The target fusion features corresponding to each candidate image include the target fusion features corresponding to at least one candidate region contained in the candidate image.

10. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of any of the methods described in claims 1-8.

11. A computer-readable storage medium, characterized in that, It includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of any of the methods described in claims 1-8.

Images