Instance retrieval method, device, equipment and storage medium

Abstract

The present invention discloses an instance retrieval method, apparatus, device and storage medium. The method comprises the following steps: obtaining the shot type in a video and extracting the key frames of each shot in the video; detecting the person and location in each key frame of the target shot based on the key frames to determine the person prediction vector and location prediction vector of the target shot; establishing a person semantic visual similarity matrix and a location semantic visual similarity matrix based on the person prediction vector and location prediction vector of the target shot; encoding the position of each shot in the video; inputting the shot type, person semantic visual similarity matrix, location semantic visual similarity matrix and the position encoding of each shot into a spatiotemporal attention model, and finally obtaining a person-location semantic retrieval result with semantic relevance. The present application can improve the accuracy of compound semantic instance retrieval and effectively avoid the problem of mutually exclusive person and location semantic retrieval scores.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an instance retrieval method, apparatus, device and storage medium. Background Art

[0002] Instance retrieval technology meticulously locates specific individuals within a certain category of objects in a video. For example, in video investigations, investigators typically focus not on pedestrians or vehicles in the general sense, but rather on specific suspects closely related to the case. Similarly, when watching a film or television drama, the audience's focus is typically not on the general characters, but rather on the male and female protagonists. Compared to concept retrieval, the goal of instance retrieval is no longer to find a specific category of objects, but to locate specific individuals within that category. This change in retrieval granularity makes multimedia instance retrieval a more challenging problem than previous category retrieval.

[0003] Thanks to the rapid development of deep learning technology, video instance retrieval technology for single semantics has made gratifying progress. Now it is possible to retrieve specific characters from film and television videos, accurately retrieve specific pedestrians or vehicles from surveillance videos, and accurately locate specific star players from sports videos.

[0004] There are two existing methods for "person-place" compound semantic retrieval:

[0005] The first method doesn't consider the semantic associations between characters and places in film and television dramas. Instead, it simply superimposes the search results for "character" and "place" to generate "character-place" binary semantic search results. This method doesn't consider complex semantic associations, so it's not ideal when faced with mutually exclusive character-place semantic search scores.

[0006] The second method uses a fixed range (eight shots, four before and four after) and a fixed correlation (using a Gaussian distribution with decaying values based on shot spacing) to estimate the semantic retrieval score for the current shot. However, this approach has two significant drawbacks: First, the optimal "fixed range" varies significantly across videos, and even within the same video, different scenes can vary widely. Therefore, using a fixed range of shots to reconstruct scores is not very generalizable.

[0007] Therefore, how to improve the accuracy of compound semantic retrieval results is a technical problem that needs to be solved urgently. Summary of the Invention

[0008] The main purpose of the present invention is to provide an instance retrieval method, apparatus, device and storage medium, which can improve the accuracy of compound semantic instance retrieval and effectively avoid the problem of mutually exclusive semantic retrieval scores of people and places.

[0009] In a first aspect, the present application provides an example retrieval method, the method comprising the steps of:

[0010] Obtaining shot types in the video and extracting key frames of each shot in the video, and detecting people and places in each key frame of the target shot based on the key frames to determine a person prediction vector and a place prediction vector of the target shot;

[0011] Establishing a person semantic visual similarity matrix and a place semantic visual similarity matrix based on the person prediction vector and the place prediction vector of the target shot;

[0012] Encode the position of each shot in the video;

[0013] Input the shot type, the character semantic visual similarity matrix, the location semantic visual similarity matrix, and the position code of each shot into the spatiotemporal attention model to calculate the character semantic correlation coefficient and the location semantic correlation coefficient;

[0014] Based on the relationship between the person semantic correlation coefficient and the person prediction vector and the relationship between the place semantic correlation coefficient and the place prediction vector, a revised person prediction retrieval score and a revised place prediction retrieval score are determined, and the highest prediction retrieval score is selected as the final person-place association retrieval result.

[0015] In one possible implementation, a video is divided into shot sets, and a shot type detection algorithm is used to detect the shot type, wherein the shot type includes: shot scale and shot motion, wherein the shot scale includes: close shot, medium shot, and long shot, and the shot motion includes: push shot, pull shot, and still shot;

[0016] Extract the key frames of each shot in the video through average sampling, and determine the key frames of each person and location in the target shot;

[0017] Generate a frame-level character prediction vector and a frame-level location prediction vector based on each character key frame and location key frame of the target shot;

[0018] Average pooling is performed on the frame-level person prediction vector and the frame-level location prediction vector to obtain a person prediction vector and a location prediction vector of the target shot.

[0019] In a possible implementation, under a set scene, a visual similarity matrix of character semantics and a visual similarity matrix of location semantics between different shots are established;

[0020] The visual similarity matrix of the character semantics is in Indicates the visual similarity between the i-th shot and the j-th shot in terms of character semantics;

[0021] The visual similarity matrix of the place semantics is in It represents the visual similarity between the i-th shot and the j-th shot in terms of location semantics.

[0022] In one possible implementation, according to the formula Calculate the character semantic relevance coefficient, where stack() represents stacking three vectors of equal length to obtain a 3×N-dimensional input matrix, v is the lens type vector, w p W p is the projection parameter matrix of the self-attention model, h n is the position vector of the lens, is the visual similarity of the nth shot with other shots in the scene in terms of character semantics;

[0023] According to the formula Calculate the location semantic relevance coefficient, w l W l is the projection parameter matrix of the self-attention model is the visual similarity of the nth shot with other shots in the scene in terms of location semantics.

[0024] In a possible implementation, according to the product of the character semantic correlation coefficient and the character prediction vector, the formula Calculate the modified person prediction retrieval score, where is the character prediction vector, is the character semantic correlation coefficient;

[0025] According to the product of the location semantic relevance coefficient and the location prediction vector, the formula Calculate the modified location prediction retrieval score, where is the location prediction vector, is the location semantic relevance coefficient.

[0026] In one possible implementation, the revised person prediction score is multiplied by the unrevised location prediction score, and the revised location prediction score is multiplied by the unrevised person prediction score, and the maximum of the two is taken as the final person-location association retrieval score;

[0027] According to the formula Calculate the highest predicted retrieval score, where is the corrected character prediction score, is the uncorrected location prediction score, For the uncorrected character prediction score, Score for the revised location prediction.

[0028]

[0029] Contains predictions for the query person p and the query location l, For the corresponding answer.

[0030] In a second aspect, the present application provides an example retrieval device, the device comprising:

[0031] an acquisition module for acquiring the shot type in the video and extracting the key frames of each shot in the video, and detecting the person and location in each key frame of the target shot based on the key frames to determine the person prediction vector and location prediction vector of the target shot;

[0032] An establishment module, which is used to establish a character semantic visual similarity matrix and a location semantic visual similarity matrix based on the character prediction vector and the location prediction vector of the target shot;

[0033] An encoding module, which is used to encode the position of each shot in the video;

[0034] a calculation module, configured to input the shot type, the character semantic visual similarity matrix, the location semantic visual similarity matrix, and the position code of each shot into a spatiotemporal attention model, and calculate a character semantic correlation coefficient and a location semantic correlation coefficient;

[0035] A determination module is used to determine a modified person prediction retrieval score and a modified place prediction retrieval score based on the relationship between the person semantic correlation coefficient and the person prediction vector and the relationship between the place semantic correlation coefficient and the place prediction vector, and select the highest prediction retrieval score as the final person-place association retrieval result.

[0036] In a third aspect, the present application further provides an electronic device comprising: a processor; and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the method described in any one of the first aspects is implemented.

[0037] In a fourth aspect, the present application further provides a computer-readable storage medium storing computer program instructions, which, when executed by a computer, enables the computer to execute any one of the methods described in the first aspect.

[0038] The present application provides an instance retrieval method, apparatus, device and storage medium, the method comprising the steps of: obtaining the shot type in a video, and extracting the key frames of each shot in the video, detecting the person and place in each key frame of the target shot based on the key frames to determine the person prediction vector and place prediction vector of the target shot; establishing a person semantic visual similarity matrix and a place semantic visual similarity matrix based on the person prediction vector and place prediction vector of the target shot; encoding the position of each shot in the video; inputting the shot type, the person semantic visual similarity matrix, the place semantic visual similarity matrix and the position encoding of each shot into a spatiotemporal attention model to calculate the person semantic correlation coefficient and the place semantic correlation coefficient; determining a revised person prediction retrieval score and a revised place prediction retrieval score based on the relationship between the person semantic correlation coefficient and the person prediction vector and the relationship between the place semantic correlation coefficient and the place prediction vector, and selecting the highest prediction retrieval score as the final person-place association retrieval result. The present application can improve the accuracy of compound semantic instance retrieval and effectively avoid the problem of mutual exclusion of person and place semantic retrieval scores.

[0039] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0041] Figure 1 This is a flow chart of an example retrieval method provided in this embodiment;

[0042] Figure 2 A schematic diagram of an example retrieval device provided in this embodiment;

[0043] Figure 3 This is a structural diagram of the shot type-guided spatiotemporal attention model provided in this embodiment;

[0044] Figure 4 A schematic diagram of an electronic device provided in an embodiment of the present application;

[0045] Figure 5 This is a schematic diagram of a computer-readable program medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0047] Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Some of the blocks shown in the drawings are functional entities that do not necessarily correspond to physically or logically separate entities.

[0048] Reference Figure 1 , Figure 1 The flowchart of an example retrieval method provided by the present invention is shown as follows: Figure 1 As shown, the method includes the steps of:

[0049] Step S101: Obtain the shot type in the video and extract the key frames of each shot in the video. Based on the key frames, detect the person and location in each key frame of the target shot to determine the person prediction vector and location prediction vector of the target shot.

[0050] Specifically, a video structuring tool is used to divide film and television drama videos into shot sets. A video scene segmentation algorithm is then used to combine each shot into scene units, ensuring that the average number of shots per scene is high. A shot type detection algorithm is used to detect shot types, which include shot scale and shot motion. Shot scales include close-up, medium, and long shots, and shot motion includes push-in, pull-out, and static shots. A person retrieval algorithm and a location retrieval algorithm are then used to determine the person and location retrieval scores for each shot.

[0051] In practice, we extract key frames from each shot using average sampling. The sampling interval is freely selectable and can be increased if storage space is limited. For each key frame of the target shot, we detect people and locations, generating frame-level person and location prediction vectors. We then apply average pooling to all frame-level prediction vectors for the target shot to obtain the shot's person and location prediction vectors.

[0052] Optionally, the video is divided into shot sets, and the shot type is detected by a shot type detection algorithm, where the shot type includes: shot scale and lens motion, wherein the shot scale includes: close shot, medium shot, and long shot, and the lens motion includes: push shot, pull shot, and static shot. The key frames of each shot in the video are extracted by average sampling, and each character key frame and location key frame of the target shot are determined. Based on each character key frame and location key frame of the target shot, a frame-level character prediction vector and a frame-level location prediction vector are generated, and the frame-level character prediction vector and the frame-level location prediction vector are average pooled to obtain the character prediction vector and location prediction vector of the target shot.

[0053] Optionally, use a video structuring tool to divide a film or TV series video into a set of shots, use a video scene segmentation algorithm to merge the shots into scene units for each shot, use a shot type detection algorithm to detect the shot scale type, and use a character retrieval algorithm and a location retrieval algorithm to obtain the character retrieval score and location retrieval score of the shot.

[0054] Step S102: establishing a person semantic visual similarity matrix and a place semantic visual similarity matrix according to the person prediction vector and the place prediction vector of the target shot.

[0055] Specifically, in a set scene, according to the character prediction vector and location prediction vector of the target shot, a visual similarity matrix of character semantics and a visual similarity matrix of location semantics between different shots are established. The visual similarity matrix of character semantics is in It represents the visual similarity between the i-th shot and the j-th shot in terms of character semantics. The visual similarity matrix of location semantics is in It represents the visual similarity between the i-th shot and the j-th shot in terms of location semantics.

[0056] For example, given a video scene containing N shots, the shot type vector can be expressed as in, It can be understood as a scene with N shots. The corresponding shot type vector is v, which has a total of N elements. The element subscripts i to N correspond to the shot type of the i-th shot. Their retrieval score vectors for the query person p and the query location l are respectively and in It can be understood as a scene with N shots. Each of the corresponding N shots has a character search result. Sp has a total of N elements. The element subscripts i to N correspond to the character search results of the i-th shot. Based on the shot-level semantic recognition results, we can build a visual similarity matrix of the semantics of the characters in different shots in the scene. Its nth row It represents the visual similarity between the n-th shot and other shots in the scene in terms of character semantics. Similar symbols and definitions also apply to the visual similarity matrix of location semantics in the scene. and its row vector

[0057] It should be noted that the visual similarity matrix is an N*N dimensional matrix. The element ei,jP in the i-th row and j-th column represents the visual similarity of the character semantics between the i-th shot and the j-th shot. The n-th row of this matrix is the visual similarity of the character semantics between the n-th shot and other shots in the scene. The visual similarity of the character semantics between two shots is obtained based on the character semantic retrieval results of the two shots. The character semantic retrieval result of a shot is a matrix, in which each element represents the probability of a specific character appearing in the shot. The dot product of the two matrices can be used as the similarity between the shots.

[0058] Step S103: Encode the position of each shot in the video.

[0059] Specifically, assuming that the semantic score of the n-th shot in the scene is to be predicted, its relative position in the scene can be encoded as a vector where h i =|in| reflects the time interval between the n-th shot and the i-th shot. It should be noted that if a scene has N shots, the n-th shot will have a position vector h n , this vector is N in length, and the element hi represents the absolute value of the difference between the subscripts of the nth lens and the i-th lens. For example, an h5 may be [2,1,0,1,2], which means this is the third lens.

[0060] Step S104: input the shot type, the character semantic visual similarity matrix, the location semantic visual similarity matrix, and the position code of each shot into the spatiotemporal attention model to calculate the character semantic correlation coefficient and the location semantic correlation coefficient.

[0061] According to the above steps, the shot type, character semantic visual similarity matrix, location semantic visual similarity matrix and the position code of each shot are input into the spatiotemporal attention model S 2 T 2 A(Spatial*2 Temporal*2Attention) is used to calculate the semantic relevance coefficient of people and the semantic relevance coefficient of places.

[0062] In one embodiment, according to the formula Calculate the character semantic relevance coefficient, where stack() represents stacking three vectors of equal length to obtain a 3×N-dimensional input matrix, v is the lens type vector, w p W p is the projection parameter matrix of the self-attention model, h n is the position vector of the lens, is the visual similarity of the character semantics between the nth shot and other shots in the scene;

[0063] According to the formula Calculate the location semantic relevance coefficient, w l W l is the projection parameter matrix of the self-attention model is the visual similarity of the nth shot with other shots in the scene in terms of location semantics.

[0064] It is understandable that the shot type guided spatiotemporal attention model S 2 T 2 A, based on the scene shot type v, the shot interval h in the time dimension n and visual similarity in spatial dimensions To jointly estimate the semantic relevance of other shots in the scene with the nth shot:

[0065]

[0066] Where stack(·,·,·) means stacking three vectors of equal length to obtain a 3×N-dimensional input matrix. and are two projection parameter matrices that follow the self-attention model and are responsible for generating correlation coefficients consistent with the number of scene shots. The i-th dimension Represents the correlation between the i-th shot and the n-th shot in terms of the character semantics P. The higher the correlation, the more consistent the retrieval scores of the i-th shot and the n-th shot in terms of the character semantics. Similar symbols and definitions also apply to the correlation coefficient of location semantics.

[0067] It should be noted that stack(·,·,·) can be understood as three vectors that need to be stacked into a matrix before they can be input into the model W P and w P These are the two parameter matrices in the model, which need to be obtained through training. There are N elements in total, where the i-th dimension Represents the correlation between the i-th shot and the n-th shot in the character semantics P.

[0068] Step S105: Determine the revised person prediction retrieval score and the revised place prediction retrieval score based on the relationship between the person semantic correlation coefficient and the person prediction vector and the relationship between the place semantic correlation coefficient and the place prediction vector, and select the highest prediction retrieval score as the final person-place association retrieval result.

[0069] Specifically, the maximum value of the corrected person prediction score multiplied by the uncorrected place prediction score and the corrected place prediction score multiplied by the uncorrected person prediction score is taken as the final person-place association retrieval score. Calculate the highest predicted retrieval score, where is the corrected character prediction score, is the uncorrected location prediction score, For the uncorrected character prediction score, Score for the revised location prediction.

[0070] For example, based on the i-th shot in the scene, the retrieval score of the n-th shot for the query character p can be written as To overcome the fluctuation of inference scores caused by changes in shot content, we traverse all shots in the scene using the consistency competition method and use the largest inference score as the predicted retrieval score for the query character p in the nth shot (that is, the character prediction score vector without the model is multiplied by the correlation coefficient vector output by the model to obtain the model-corrected character prediction score vector):

[0071]

[0072] Based on a similar approach, we can get the predicted retrieval score of the nth shot about the query location l (That is, the location prediction score vector that has not been tested by the model is multiplied by the correlation coefficient vector output by the model to obtain the location prediction score vector after model correction).

[0073] Multiply the corrected person prediction score by the uncorrected location prediction score, and vice versa. Considering the mutual exclusivity of person and location scores, the highest score is finally selected from the two solutions of person and location prediction as the associated retrieval score for the n-th shot regarding the binary semantic query (p, l) (i.e., the maximum value of the corrected person prediction score multiplied by the uncorrected location prediction score and the corrected location prediction score multiplied by the uncorrected person prediction score is taken as the binary semantic retrieval score):

[0074]

[0075] In one embodiment, the object of the person-place association retrieval model training is the spatiotemporal attention model parameter W guided by the shot type. P 、w P 、W L and w L The above model parameters are trained by constructing the mean square error loss for the associated retrieval results:

[0076] The prediction for query location l is It is the corresponding answer.

[0077] In one embodiment, according to the formula Train the parameters in the spatiotemporal attention model, where It is a prediction of whether the nth shot contains the query person p and the query location l. For the corresponding answer.

[0078] To facilitate understanding of this application, an example is given below:

[0079] Use video structuring tools to divide film and television drama videos into shot sets, use video scene segmentation algorithms to merge shots into scene units for each shot, use shot type detection algorithms to detect shot scale types, and use character retrieval algorithms and location retrieval algorithms to obtain the character retrieval score and location retrieval score of the shot.

[0080] Given a video scene containing N shots, the shot type vector can be expressed as Their retrieval score vectors for the query person p and the query location l are Know Based on the shot-level semantic recognition results, a visual similarity matrix of the character semantics between different shots in the scene can be established. Its nth line It represents the visual similarity between the n-th shot and other shots in the scene in terms of character semantics. Similar symbols and definitions also apply to the visual similarity matrix E of the location semantics in the scene. L and its row vector

[0081] It should be noted that the visual similarity matrix is an N*N dimensional matrix. The element ei,jP in the i-th row and j-th column represents the visual similarity of the character semantics between the i-th shot and the j-th shot. The n-th row of this matrix is the visual similarity of the character semantics between the n-th shot and other shots in the scene. The visual similarity of the character semantics between two shots is obtained based on the character semantic retrieval results of the two shots. The character semantic retrieval result of a shot is a matrix, in which each element represents the probability of a specific character appearing in the shot, so the dot product of the two matrices can be used as the similarity between the shots.

[0082] Suppose we want to predict the semantic score of the n-th shot in the above scene, its relative position in the scene can be encoded as a vector where h i =|in| reflects the time interval between the n-th shot and the i-th shot. It should be noted that if a scene has N shots, the n-th shot will have a position vector h n , this vector is N in length, and the element hi represents the absolute value of the difference between the subscripts of the nth lens and the i-th lens. For example, an h5 may be [2, 1, 0, 1, 2], which means this is the third lens.

[0083] Shot type guided spatiotemporal attention model S 2 T 2 A(Spatial*2 Temporal *2 Attention), based on the scene shot type v, the shot interval h in the time dimension n and visual similarity in spatial dimensions To jointly estimate the semantic relevance of other shots in the scene with the nth shot:

[0084]

[0085] Where stack(·,·,·) means stacking three vectors of equal length to obtain a 3×N-dimensional input matrix. Know are two projection parameter matrices that follow the self-attention model and are responsible for generating correlation coefficients consistent with the number of scene shots. The i-th dimension Represents the correlation between the i-th shot and the n-th shot in terms of the character semantics P. The higher the correlation, the more consistent the retrieval scores of the i-th shot and the n-th shot in terms of the character semantics. Similar symbols and definitions also apply to the correlation coefficient of location semantics. It should be noted that stack(·,·,·) is three vectors that need to be stacked into a matrix before they can be input into the model.

[0086] W P and w P These are the two parameter matrices in the model, which need to be obtained through training. There are N elements in total, where the i-th dimension Represents the correlation between the i-th shot and the n-th shot in the character semantics P.

[0087] On this basis, the retrieval score of the nth shot about the query character p can be inferred based on the i-th shot in the scene and written as To overcome the fluctuation of the inference score caused by the change of the shot content, we traverse all the shots in the scene by imitating the consistency competition method, and use the largest inference score as the predicted retrieval score of the query character p in the nth shot:

[0088]

[0089] Based on a similar approach, we can get the predicted retrieval score of the nth shot about the query location l It should be noted that the character prediction score vector after model correction is obtained by multiplying the character prediction score vector that has not been processed by the model by the correlation coefficient vector output by the model.

[0090] Considering the mutual exclusivity of person-place scores, the solution with the highest score in both the person-predicted and place-predicted solutions is selected as the associated retrieval score of the n-th shot for the binary semantic query (p, l):

[0091]

[0092] It can be understood that the associated retrieval score is the maximum value of the corrected person prediction score multiplied by the uncorrected location prediction score, and the corrected location prediction score multiplied by the uncorrected person prediction score as the binary semantic retrieval score.

[0093] Reference Figure 2 , Figure 2 FIG. 1 is a schematic diagram of an example retrieval device provided by the present invention, as shown in FIG. Figure 2 As shown, the device includes:

[0094] An acquisition module 201 is configured to acquire shot types in a video and extract key frames for each shot in the video. Based on the key frames, the module detects the person and location in each key frame of a target shot to determine a person prediction vector and a location prediction vector for the target shot.

[0095] Establishing module 202, which is used to establish a character semantic visual similarity matrix and a location semantic visual similarity matrix based on the character prediction vector and the location prediction vector of the target shot;

[0096] An encoding module 203, which is used to encode the position of each shot in the video;

[0097] A calculation module 204 is configured to input the shot type, the character semantic visual similarity matrix, the location semantic visual similarity matrix, and the position code of each shot into a spatiotemporal attention model to calculate a character semantic correlation coefficient and a location semantic correlation coefficient;

[0098] Determination module 205, which is used to determine the modified person prediction retrieval score and the modified place prediction retrieval score based on the relationship between the person semantic correlation coefficient and the person prediction vector and the relationship between the place semantic correlation coefficient and the place prediction vector, and select the highest prediction retrieval score as the final person-place association retrieval result.

[0099] Furthermore, in a possible implementation, the acquisition module 201 is further configured to divide the video into shot sets and detect shot types using a shot type detection algorithm, wherein the shot types include shot scale and shot motion, wherein the shot scale includes close shot, medium shot, and long shot, and the shot motion includes push shot, pull shot, and still shot.

[0100] Extract the key frames of each shot in the video through average sampling, and determine the key frames of each person and location in the target shot;

[0101] Generate a frame-level character prediction vector and a frame-level location prediction vector based on each character key frame and location key frame of the target shot;

[0102] Average pooling is performed on the frame-level person prediction vector and the frame-level location prediction vector to obtain a person prediction vector and a location prediction vector of the target shot.

[0103] Furthermore, in a possible implementation, the establishing module 202 is further configured to establish a visual similarity matrix of character semantics and a visual similarity matrix of location semantics between different shots in a set scene;

[0104] The visual similarity matrix of the character semantics is in Indicates the visual similarity between the i-th shot and the j-th shot in terms of character semantics;

[0105] The visual similarity matrix of the place semantics is in It represents the visual similarity between the i-th shot and the j-th shot in terms of location semantics.

[0106] Furthermore, in a possible implementation manner, the calculation module 204 is further configured to, according to the formula Calculate the character semantic relevance coefficient, where stack() represents stacking three vectors of equal length to obtain a 3×N-dimensional input matrix, v is the lens type vector, w p W p is the projection parameter matrix of the self-attention model, h n is the position vector of the lens, is the visual similarity of the character semantics between the nth shot and other shots in the scene;

[0107] According to the formula Calculate the location semantic relevance coefficient, w l W l is the projection parameter matrix of the self-attention model is the visual similarity of the nth shot with other shots in the scene in terms of location semantics.

[0108] Furthermore, in a possible implementation, the determination module 205 is further configured to, according to the product of the character semantic relevance coefficient and the character prediction vector, calculate the character by the formula Calculate the modified person prediction retrieval score, where is the character prediction vector, is the character semantic correlation coefficient;

[0109] According to the product of the location semantic relevance coefficient and the location prediction vector, the formula Calculate the modified location prediction retrieval score, where is the location prediction vector, is the location semantic relevance coefficient.

[0110] Furthermore, in one possible implementation, the determination module 205 is further configured to multiply the revised person prediction score by the unrevised location prediction score, and multiply the revised location prediction score by the unrevised person prediction score, and take the maximum value of the two as the final person-location association retrieval score;

[0111] According to the formula Calculate the highest predicted retrieval score, where is the corrected character prediction score, is the uncorrected location prediction score, For the uncorrected character prediction score, Score for the revised location prediction.

[0112] Furthermore, a possible implementation further includes a training module for Train the parameters in the spatiotemporal attention model, where It is a prediction of whether the nth shot contains the query person p and the query location l. For the corresponding answer.

[0113] Reference Figure 3 , Figure 3 The structure diagram of the spatiotemporal attention model guided by the shot type provided by the present invention is shown as follows: Figure 3 As shown:

[0114] s p is the character prediction vector, s l is the location prediction vector, is the semantic similarity of characters, is the location semantic similarity, v is the shot type, h n Predict the semantic score for the nth shot, the encoding vector of its relative position in the scene, For the uncorrected character prediction score, is the uncorrected location prediction score, is the corrected character prediction score, Score for the revised location prediction.

[0115] It should be noted that the uncorrected person prediction score and the uncorrected location prediction score are obtained by the face detection algorithm and the location detection algorithm.

[0116] It is understandable that the shot type guided spatiotemporal attention model S 2 T 2 A, based on the scene shot type v, the shot interval h in the time dimension n and visual similarity in spatial dimensions To jointly estimate the semantic relevance of other shots in the scene with the nth shot

[0117] Similarly, the spatiotemporal attention model S guided by shot type 2 T 2 A, based on the scene shot type v, the shot interval h in the time dimension n and visual similarity in spatial dimensions To jointly estimate the semantic relevance of other shots in the scene with the nth shot in terms of location

[0118] The retrieval score of the nth shot about the query character p based on the i-th shot in the scene can be written as To overcome the fluctuation of the inference score caused by the change of the shot content, we traverse all the shots in the scene by imitating the consistency competition method, and use the largest inference score as the predicted retrieval score of the query character p in the nth shot:

[0119]

[0120] Based on a similar approach, we can get the predicted retrieval score of the nth shot about the query location l

[0121] Considering the mutual exclusivity of person-place scores, the solution with the highest score in both the person-predicted and place-predicted solutions is selected as the associated retrieval score of the n-th shot for the binary semantic query (p, l):

[0122]

[0123] It can be understood that the video structuring module divides the input film and television video to obtain a video with shots as units, detects the scale type of the shots and merges the shots into units with scenes as units, and encodes the positions of the shots in the scenes.

[0124] The unary semantic retrieval module detects the action and location of each shot in the scene and generates a person prediction vector and a location prediction vector, and generates a person visual similarity matrix and a location visual similarity matrix based on the person prediction vectors and location prediction vectors of other shots in the scene.

[0125] The "person-place" association retrieval module inputs the position code of each shot in the scene, the shot type, the person visual similarity matrix and the place visual similarity matrix into the shot type-guided spatiotemporal attention model to obtain the person speculation retrieval score and the place speculation retrieval score.

[0126] The result output module, taking into account the mutual exclusivity of the "person-place" scores, selects the highest score among the two results of inferred person and inferred place as the "person-place" association retrieval score of the shot.

[0127] Refer to the following Figure 4 An electronic device 400 according to this embodiment of the present invention will be described. Figure 4 The electronic device 400 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0128] like Figure 4 As shown, electronic device 400 is implemented as a general-purpose computing device. Components of electronic device 400 may include, but are not limited to, at least one processing unit 410, at least one storage unit 420, and a bus 430 connecting various system components (including storage unit 420 and processing unit 410).

[0129] The storage unit stores program codes, which can be executed by the processing unit 410, so that the processing unit 410 performs the steps according to various exemplary embodiments of the present invention described in the above “Example Method” section of this specification.

[0130] The storage unit 420 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 421 and / or a cache memory unit 422 , and may further include a read-only memory unit (ROM) 423 .

[0131] The storage unit 420 may also include a program / utility 424 having a set (at least one) of program modules 425, such program modules 425 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0132] Bus 430 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0133] The electronic device 400 may also communicate with one or more external devices (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 400, and / or any device that enables the electronic device 400 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication may occur via an input / output (I / O) interface 450. Furthermore, the electronic device 400 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 460. As shown, the network adapter 460 communicates with other modules of the electronic device 400 via a bus 430. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 400, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0134] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0135] According to the solution of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above-mentioned method of this specification is stored. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product, which includes program code. When the program product is executed on a terminal device, the program code is used to cause the terminal device to perform the steps according to various exemplary embodiments of the present invention described in the "Exemplary Methods" section of this specification.

[0136] refer to Figure 5 As shown, a program product 500 for implementing the above method according to an embodiment of the present invention is described. The program product 500 may be a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0137] The program product may be implemented in any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0138] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0139] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0140] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, and the like, as well as conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0141] Furthermore, the above-described figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes illustrated in the above-described figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0142] In summary, the present application provides an instance retrieval method, apparatus, device and storage medium, which includes the following steps: obtaining the shot type in the video, and extracting the key frames of each shot in the video, detecting the person and place in each key frame of the target shot based on the key frames to determine the person prediction vector and place prediction vector of the target shot; establishing a person semantic visual similarity matrix and a place semantic visual similarity matrix based on the person prediction vector and place prediction vector of the target shot; encoding the position of each shot in the video; inputting the shot type, person semantic visual similarity matrix, place semantic visual similarity matrix and the position encoding of each shot into the spatiotemporal attention model, calculating the person semantic correlation coefficient and the place semantic correlation coefficient; determining the revised person prediction retrieval score and the revised place prediction retrieval score based on the relationship between the person semantic correlation coefficient and the person prediction vector and the relationship between the place semantic correlation coefficient and the place prediction vector, and selecting the highest prediction retrieval score as the final person-place association retrieval result. The present application can improve the accuracy of compound semantic instance retrieval and effectively avoid the problem of mutual exclusivity of person and place semantic retrieval scores.

[0143] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the broadest scope consistent with the principles and novel features of the present application.

[0144] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

Claims

1. An instance retrieval method, characterized in that: include: Obtaining shot types in the video and extracting key frames of each shot in the video, and detecting people and places in each key frame of the target shot based on the key frames to determine a person prediction vector and a place prediction vector of the target shot; Establishing a person semantic visual similarity matrix and a place semantic visual similarity matrix based on the person prediction vector and the place prediction vector of the target shot; Encode the position of each shot in the video; Input the shot type, the character semantic visual similarity matrix, the location semantic visual similarity matrix, and the position code of each shot into the spatiotemporal attention model to calculate the character semantic correlation coefficient and the location semantic correlation coefficient; Determining a revised person prediction retrieval score and a revised place prediction retrieval score based on a relationship between a person semantic correlation coefficient and the person prediction vector and a relationship between a place semantic correlation coefficient and the place prediction vector, and selecting the highest prediction retrieval score as a final person-place association retrieval result; According to the product of the character semantic correlation coefficient and the character prediction vector, the formula Calculate the modified person prediction retrieval score, where is the character prediction vector, is the character semantic correlation coefficient; According to the product of the location semantic relevance coefficient and the location prediction vector, the formula Calculate the modified location prediction retrieval score, where is the location prediction vector, is the location semantic relevance coefficient; Multiply the corrected person prediction score by the uncorrected place prediction score, and the corrected place prediction score by the uncorrected person prediction score, and take the maximum of the two as the final person-place association retrieval score; According to the formula Calculate the highest predicted retrieval score, where is the corrected character prediction score, is the uncorrected location prediction score, For the uncorrected character prediction score, Score for the revised location prediction.

2. The method according to claim 1, characterized in that The step of obtaining a shot type in a video, extracting a key frame of each shot in the video, and detecting a person and a location in each key frame of a target shot based on the key frame to determine a person prediction vector and a location prediction vector of the target shot includes: Divide the video into shot sets and detect the shot type using a shot type detection algorithm. The shot type includes: shot scale and shot motion. Shot scale includes: close shot, medium shot, long shot; and shot motion includes: push shot, pull shot, and still shot. Extract the key frames of each shot in the video through average sampling, and determine the key frames of each person and location in the target shot; Generate a frame-level character prediction vector and a frame-level location prediction vector based on each character key frame and location key frame of the target shot; Average pooling is performed on the frame-level person prediction vector and the frame-level location prediction vector to obtain a person prediction vector and a location prediction vector of the target shot.

3. The method according to claim 1, characterized in that According to the character prediction vector and the location prediction vector of the target shot, a character semantic visual similarity matrix and a location semantic visual similarity matrix are established, including: Under a set scenario, establish a visual similarity matrix of character semantics and a visual similarity matrix of location semantics between different shots; The visual similarity matrix of the character semantics is in Indicates the visual similarity between the i-th shot and the j-th shot in terms of character semantics; The visual similarity matrix of the place semantics is in It represents the visual similarity between the i-th shot and the j-th shot in terms of location semantics.

4. The method according to claim 1, wherein The calculating of the person semantic correlation coefficient and the location semantic correlation coefficient includes: According to the formula Calculate the character semantic relevance coefficient, where stack() represents stacking three vectors of equal length to obtain a 3×N-dimensional input matrix, v is the lens type vector, w p W p is the projection parameter matrix of the self-attention model, h n is the position vector of the lens, is the visual similarity of the nth shot with other shots in the scene in terms of character semantics; According to the formula Calculate the location semantic relevance coefficient, w l W l is the projection parameter matrix of the self-attention model is the visual similarity of the nth shot with other shots in the scene in terms of location semantics.

5. The method according to claim 1, wherein: According to the formula Train the parameters in the spatiotemporal attention model, where It is a prediction of whether the nth shot contains the query person p and the query location l. For the corresponding answer.

6. An example retrieval device, characterized in that: include: an acquisition module for acquiring the shot type in the video and extracting the key frames of each shot in the video, and detecting the person and location in each key frame of the target shot based on the key frames to determine the person prediction vector and location prediction vector of the target shot; An establishment module, which is used to establish a character semantic visual similarity matrix and a location semantic visual similarity matrix based on the character prediction vector and the location prediction vector of the target shot; An encoding module, which is used to encode the position of each shot in the video; a calculation module, configured to input the shot type, the character semantic visual similarity matrix, the location semantic visual similarity matrix, and the position code of each shot into a spatiotemporal attention model, and calculate a character semantic correlation coefficient and a location semantic correlation coefficient; a determination module configured to determine a revised person prediction retrieval score and a revised place prediction retrieval score based on a relationship between the person semantic correlation coefficient and the person prediction vector and a relationship between the place semantic correlation coefficient and the place prediction vector, and select the highest prediction retrieval score as a final person-place association retrieval result; According to the product of the character semantic relevance coefficient and the character prediction vector, the formula Calculate the modified person prediction retrieval score, where is the character prediction vector, is the character semantic correlation coefficient; According to the product of the location semantic relevance coefficient and the location prediction vector, the formula Calculate the modified location prediction retrieval score, where is the location prediction vector, is the location semantic relevance coefficient; Multiply the corrected person prediction score by the uncorrected place prediction score, and the corrected place prediction score by the uncorrected person prediction score, and take the maximum of the two as the final person-place association retrieval score; According to the formula Calculate the highest predicted retrieval score, where is the corrected character prediction score, is the uncorrected location prediction score, For the uncorrected character prediction score, Score for the revised location prediction.

7. An electronic device, characterized in that: The electronic device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that The computer program instructions are stored therein, and when the computer program instructions are executed by a computer, the computer is caused to execute the method according to any one of claims 1 to 5.

Images