Method, apparatus, and computer-readable storage medium for determining the number of matching feature vectors

Abstract

The present application discloses a method, apparatus, and computer-readable storage medium for determining the number of matching feature vectors. Among them, the method includes: obtaining a first image and a second image in an image sequence, where the first image is the previous frame image of the second image; extracting the image features of the first image and the image features in the second image through a multi-layer convolutional network, and respectively generating a first feature matrix corresponding to the first image and a second feature matrix corresponding to the second image according to the extracted image features; dividing the first feature matrix into multiple matrix blocks, where each matrix block corresponds to a contrast; determining a target value corresponding to each matrix block according to the contrast corresponding to each matrix block, where the target value is used to characterize the number of matching feature vectors. The present application solves the technical problem of poor flexibility in setting the number of matching feature vectors during the process of optical flow estimation for images.

Description

Technical Field

[0001] The present application relates to the field of deep learning, and more particularly, to a method, apparatus, and computer-readable storage medium for determining the number of matching feature vectors. Background Art

[0002] In the prior art, when obtaining the optical flow field of an image through an optical flow estimation network, it is first necessary to obtain the feature matrix corresponding to the image and the feature matrix corresponding to the next frame of the image, then construct a similarity matrix based on the matching feature vectors between the two feature matrices, and finally obtain the optical flow field of the image according to the similarity matrix. Among them, when calculating the similarity matrix, a specific area needs to be allocated on the video memory to store all the matching feature vectors between the two feature matrices.

[0003] It is easy to understand that the more the number of matching feature vectors, the more matching information between the two feature matrices can be provided, and the higher the accuracy of optical flow estimation. However, the corresponding video memory expenditure is also higher. On this basis, in the prior art, the number of matching feature vectors is usually set to a fixed value. If the fixed value is set too high, it will cause waste of video memory storage resources in the high-contrast area of the image, and if the fixed value is set too low, it will cause poor optical flow estimation accuracy in the low-contrast area of the image.

[0004] In view of the above problems, no effective solution has been proposed yet. Summary of the Invention

[0005] Embodiments of the present application provide a method, apparatus, and computer-readable storage medium for determining the number of matching feature vectors, so as to at least solve the technical problem of poor flexibility in setting the number of matching feature vectors during the process of optical flow estimation of an image.

[0006] According to one aspect of the embodiments of the present application, a method for determining the number of matching feature vectors is provided, including: obtaining a first image and a second image in an image sequence, where the first image is the previous frame image of the second image; extracting the image features of the first image and the image features in the second image through a multi-layer convolutional network, and respectively generating a first feature matrix corresponding to the first image and a second feature matrix corresponding to the second image according to the extracted image features, where the image features are used to characterize the image structure information of the first image or the image structure information of the second image; dividing the first feature matrix into a plurality of matrix blocks, where each matrix block corresponds to a contrast ratio, and the contrast ratio is used to characterize the standard deviation of the image pixel intensities in the matrix block; determining the target value corresponding to each matrix block according to the contrast ratio corresponding to each matrix block, where the target value is used to characterize the number of matching feature vectors, and the matching feature vectors are the feature vectors in the first feature matrix that are matched in the second feature matrix based on vector similarity, and there is a negative correlation between the contrast ratio and the target value.

[0007] Further, the method for determining the number of matching feature vectors further includes: dividing the first feature matrix into a plurality of consecutive matrix blocks through a batch processing mechanism, where the area of each matrix block is a preset area, and the difference value between the contrast ratios of all the feature vectors in each matrix block is less than a preset threshold.

[0008] Further, the method for determining the number of matching feature vectors further includes: after dividing the first feature matrix into a plurality of matrix blocks, obtaining the size information, dimension, and the brightness value of each feature vector in each matrix block corresponding to each matrix block, where the size information includes the length and width of the matrix block; calculating the average value of the brightness values of all the feature vectors in each matrix block to obtain the average brightness corresponding to each matrix block; determining the contrast ratio of the matrix block according to the average brightness, size information, dimension, and the brightness value of each feature vector in the matrix block corresponding to each matrix block.

[0009] Further, the method for determining the number of matching feature vectors further includes: inputting the average brightness, size information, dimension, and the brightness value of each feature vector in each matrix block corresponding to each matrix block into a preset function to obtain the contrast ratio of the matrix block, where the preset function is as follows:

[0010]

[0011] In the preset function, F represents the unique identifier of each matrix block, N represents the length of the matrix block, M represents the width of the matrix block, D represents the dimension of the matrix block, F ijk represents the brightness value of the feature vector k at the (i, j) position in the matrix block, avg(F) k represents the average brightness corresponding to the matrix block, and CMS represents the contrast ratio of the matrix block.

[0012] Furthermore, the method for determining the number of matching feature vectors further includes: normalizing the contrast corresponding to each matrix block according to the hyperbolic tangent function to obtain the normalized value corresponding to each matrix block, where there is a positive correlation between the contrast and the normalized value; obtaining the value range of the target value, where the value range includes the minimum value and the maximum value of the target value; determining the target value corresponding to each matrix block according to the maximum value, the minimum value, and the normalized value corresponding to each matrix block, where there is a negative correlation between the normalized value and the target value.

[0013] Furthermore, the method for determining the number of matching feature vectors further includes: after determining the target value corresponding to each matrix block according to the contrast corresponding to each matrix block, performing the following steps:

[0014] Step 1, arbitrarily select a feature vector from the first feature matrix as the target feature vector, and detect the vector similarity between each feature vector in the second feature matrix and the target feature vector;

[0015] Step 2, sort all the feature vectors in the second feature matrix according to the vector similarity to obtain a sorting result;

[0016] Step 3, select N feature vectors from the second feature matrix as the matching feature vectors corresponding to the target feature vector according to the sorting result, where N is the target value;

[0017] Step 4, repeat the process of Step 1 to Step 3 until the matching feature vectors corresponding to each feature vector in the first feature matrix are obtained, and obtain the position information of each matching feature vector in the second feature matrix.

[0018] Furthermore, the method for determining the number of matching feature vectors further includes: after obtaining the matching feature vectors corresponding to each feature vector in the first feature matrix and obtaining the position information of each matching feature vector in the second feature matrix, storing each matching feature vector and the position information of each matching feature vector in the second feature matrix into the target video memory; constructing a similarity matrix according to the matching feature vectors and position information stored in the target video memory, where the similarity matrix represents the matching information between the first feature matrix and the second feature matrix; inputting the similarity matrix and the image features of the first image into the optical flow estimation network to obtain the optical flow field corresponding to the first image.

[0019] According to another aspect of the embodiments of the present application, there is also provided an apparatus for determining the number of matching feature vectors, including: an acquisition module, configured to acquire a first image and a second image in an image sequence, where the first image is the previous frame image of the second image; a feature extraction module, configured to extract the image features of the first image and the image features in the second image through a multi-layer convolutional network, and respectively generate a first feature matrix corresponding to the first image and a second feature matrix corresponding to the second image according to the extracted image features; a division module, configured to divide the first feature matrix into a plurality of matrix blocks, where each matrix block corresponds to a contrast, and the contrast is used to characterize the standard deviation of the image pixel intensities in the matrix block; a determination module, configured to determine the target value corresponding to each matrix block according to the contrast corresponding to each matrix block, where the target value is used to characterize the number of matching feature vectors, and the matching feature vectors are the feature vectors in the first feature matrix that are matched in the second feature matrix according to the vector similarity, and there is a negative correlation between the contrast and the target value, where the image features are used to characterize the image structure information of the first image or the image structure information of the second image.

[0020] According to another aspect of the embodiments of the present application, there is also provided a computer-readable storage medium, in which a computer program is stored, where the computer program is configured to execute the above-mentioned method for determining the number of matching feature vectors when running.

[0021] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including one or more processors and a memory, where the memory is configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the above-mentioned method for determining the number of matching feature vectors.

[0022] In the present application, by using the method of determining the target value corresponding to each matrix block according to the contrast corresponding to each matrix block, first, a first image and a second image in the image sequence are acquired, then the image features of the first image and the image features in the second image are extracted through a multi-layer convolutional network, and a first feature matrix corresponding to the first image and a second feature matrix corresponding to the second image are respectively generated according to the extracted image features. Subsequently, the first feature matrix is divided into a plurality of matrix blocks, and the target value corresponding to each matrix block is determined according to the contrast corresponding to each matrix block. Wherein, the first image is the previous frame image of the second image, each matrix block corresponds to a contrast, and the contrast is used to characterize the standard deviation of the image pixel intensities in the matrix block; the target value is used to characterize the number of matching feature vectors, and the matching feature vectors are the feature vectors in the first feature matrix that are matched in the second feature matrix according to the vector similarity, and there is a negative correlation between the contrast and the target value.

[0023] As can be seen from the above, in this application, the first feature matrix corresponding to the first image is divided into multiple matrix blocks, and the target value corresponding to each matrix block is determined according to the contrast corresponding to each matrix block. Since the target value is used to represent the number of matching feature vectors, and there is a negative correlation between the contrast and the target value, the greater the contrast corresponding to a matrix block, the smaller the target value corresponding to the matrix block. Therefore, this application achieves the purpose of setting a higher target value in the image area with high contrast and a lower target value in the image area with low contrast, thereby saving video memory resources and ensuring the accuracy of the first image during optical flow estimation, and further achieving the technical effect of flexibly setting the number of matching feature vectors.

[0024] Thus, through the technical solution of this application, the purpose of configuring the corresponding number of matching feature vectors for each matrix block based on the contrast of each matrix block is achieved, thereby achieving the technical effect of flexibly adjusting the number of matching feature vectors corresponding to each image area in the first image, and further solving the technical problem of poor flexibility in setting the number of matching feature vectors during the optical flow estimation of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of this application, and constitute a part of this application. The illustrative embodiments of this application and their descriptions are used to explain this application, and do not constitute an improper limitation of this application. In the drawings:

[0026] Figure 1 is a flowchart of a method for determining the number of matching feature vectors according to an embodiment of this application;

[0027] Figure 2 is a schematic diagram of an optional deep learning optical flow network framework according to an embodiment of this application;

[0028] Figure 3 is a schematic diagram of the division of a first feature matrix according to an embodiment of this application;

[0029] Figure 4 is a schematic diagram of a device for determining the number of matching feature vectors according to an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] In order to enable those skilled in the art to better understand the solution of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0031] It should be noted that the terms "first", "second", etc. in the description, claims and above-mentioned drawings of this application are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of this application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0032] Embodiment 1

[0033] According to an embodiment of this application, an embodiment of a method for determining the number of matching feature vectors is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0034] Figure 1 is a flowchart of a method for determining the number of matching feature vectors according to an embodiment of this application, as Figure 1 shown, the method includes the following steps:

[0035] Step S101, obtain a first image and a second image in the image sequence.

[0036] In step S101, the first image is the previous frame image of the second image. Specifically, the image sequence can be a video stream to be subjected to optical flow estimation. Among them, optical flow estimation is an important direction in computer vision research. Optical flow estimation provides a motion vector for each pixel in the image, and the matrix composed of these motion vectors is the optical flow field. The optical flow field characterizes the motion field of the entire image and has very rich semantic information. For the optical flow field obtained by optical flow estimation, there are very broad application scenarios, such as being used in technical fields such as moving target detection, image registration, and video super-resolution.

[0037] In the embodiment of this application, the first image is an image to be subjected to optical flow estimation. In addition, in the embodiment of this application, an optical flow estimation system can be used as the execution subject of the method for determining the number of matching feature vectors in the embodiment of this application. Among them, in addition to being able to determine the matching feature vectors, the optical flow estimation system can also perform optical flow estimation on the first image.

[0038] Step S102: Extract the image features of the first image and the second image through a multi-layer convolutional network, and generate a first feature matrix corresponding to the first image and a second feature matrix corresponding to the second image respectively according to the extracted image features.

[0039] In step S102, the image features are used to characterize the image structure information of the first image or the second image.

[0040] Specifically, the optical flow estimation system is a system equipped with a recurrent neural network. The structure of the optical flow estimation system can be divided into three parts, namely, a feature extraction part, a feature matching part, and an iterative calculation part. Among them, Figure 2 FIG. shows a schematic diagram of an optional deep learning optical flow network framework according to an embodiment of the present application.

[0041] As Figure 2 shown, in the feature extraction part, the optical flow estimation system can extract image features from two input frames of images (such as the first image Image1 and the second image Image2) through a multi-layer convolutional network. Generally, the optical flow estimation system will perform several downsamplings on each image, and then generate two feature matrices respectively based on the extracted image features, namely, the first feature matrix F1 corresponding to the first image and the second feature matrix F2 corresponding to the second image.

[0042] In addition, as Figure 2 shown, in the feature matching part, like the prior art, the optical flow estimation system will also construct a similarity matrix based on the first feature matrix and the second feature matrix. The similarity matrix characterizes the matching information between the first feature matrix F1 and the second feature matrix F2. Finally, in the iterative calculation part, the optical flow estimation system will also splice the image features of the first image and the similarity matrix into a new matrix to be input, and then input the spliced matrix to be input into the optical flow estimation network, and after several iterations, obtain the result of optical flow estimation, that is, the optical flow field of the first image.

[0043] It should be noted that when calculating the similarity matrix, a region needs to be allocated on the video memory to store the top_k feature vectors with the highest similarity on the second feature matrix for each feature vector in the first feature matrix and their position information. Among them, the top_k feature vectors with the highest similarity are the matching feature vectors. Obviously, the larger the top_k, the more video memory is consumed. Generally speaking, for a feature matrix with a size of 200*200 and a depth of 256 for one frame, when top_k takes 4, when calculating the similarity matrix. At least 150MB of video memory space needs to be occupied to store the matching feature vectors and the position information of the matching feature vectors. However, it should also be noted that the larger the top_k, the more matching information can be provided, and the higher the accuracy of optical flow estimation.

[0044] In the prior art, top_k in the recurrent neural network optical flow estimation algorithm is usually set to a fixed value. On this basis, if top_k is set too large, it will cause waste of video memory resources when estimating the optical flow of high-contrast image regions. If top_k is set too small, it will lead to inaccurate accuracy when estimating the optical flow of low-contrast image regions.

[0045] To solve the above problems, after obtaining the first feature matrix, the optical flow estimation system performs the following steps S103 and S104.

[0046] Specifically, in step S103, the first feature matrix is divided into multiple matrix blocks.

[0047] In step S103, each matrix block corresponds to a contrast ratio, and the contrast ratio is used to characterize the standard deviation of the image pixel intensities in the matrix block.

[0048] Specifically, for the first feature matrix, the optical flow estimation system divides the first feature matrix into multiple consecutive matrix blocks, and then the optical flow estimation system calculates the contrast ratio corresponding to each matrix block according to the brightness values of each feature vector in each matrix block.

[0049] Step S104, determine the target value corresponding to each matrix block according to the contrast ratio corresponding to each matrix block.

[0050] In step S104, the target value is used to characterize the number of matching feature vectors. The matching feature vector is the feature vector matched by each feature vector in the first feature matrix in the second feature matrix based on vector similarity. There is a negative correlation between the contrast ratio and the target value. Among them, the target value is the above-mentioned top_k, and the negative correlation between the contrast ratio and the target value means that the greater the contrast ratio corresponding to a matrix block, the smaller the target value corresponding to the matrix block. For example, the contrast ratio corresponding to matrix block 1 is A, and the contrast ratio corresponding to matrix block 2 is B. Among them, assuming A > B, the target value corresponding to matrix block 1 is less than the target value corresponding to matrix block 2.

[0051] Specifically, for the region with high contrast ratio in the first feature matrix, each feature vector may have only a very small number of feature vectors with high similarity to it in the second feature matrix. Then, for the region with high contrast ratio, the target value top_k can take a smaller value; similarly, for the region with low contrast ratio in the first feature matrix, the target value top_k should take a larger value.

[0052] In other words, through the technical solution of the present application, the optical flow estimation system divides the first feature matrix into several adjacent matrix blocks. By calculating the contrast of each matrix block, for the matrix blocks with high contrast, the optical flow estimation system sets a lower target value top_k for them, and for the matrix blocks with low contrast, the optical flow estimation system sets a higher target value top_k for them. Finally, by dynamically adjusting the size of the target value top_k corresponding to the matrix block according to the contrast of the matrix block, the present application achieves the purpose of taking into account both the consumption of video memory and the accuracy of optical flow estimation.

[0053] Based on the content of the above steps S101 to S104, it can be seen that in the present application, the method of determining the target value corresponding to each matrix block according to the contrast corresponding to each matrix block is adopted. First, the first image and the second image in the image sequence are obtained, and then the image features of the first image and the image features in the second image are extracted through a multi-layer convolutional network, and the first feature matrix corresponding to the first image and the second feature matrix corresponding to the second image are respectively generated according to the extracted image features. Subsequently, the first feature matrix is divided into multiple matrix blocks, and the target value corresponding to each matrix block is determined according to the contrast corresponding to each matrix block. Among them, the first image is the previous frame image of the second image, each matrix block corresponds to a contrast, and the contrast is used to characterize the standard deviation of the image pixel intensity in the matrix block; the target value is used to characterize the number of matching feature vectors, and the matching feature vector is the feature vector in the first feature matrix that is matched in the second feature matrix according to the vector similarity. There is a negative correlation between the contrast and the target value.

[0054] From the above content, it can be seen that the present application divides the first feature matrix corresponding to the first image into multiple matrix blocks, and determines the target value corresponding to each matrix block according to the contrast corresponding to each matrix block. Since the target value is used to characterize the number of matching feature vectors, and there is a negative correlation between the contrast and the target value, the greater the contrast corresponding to a matrix block, the smaller the target value corresponding to the matrix block. Thus, the present application achieves the purpose of setting a higher target value in the image area with high contrast and a lower target value in the image area with low contrast, thereby saving video memory resources and ensuring the accuracy of the first image during optical flow estimation, and further achieving the technical effect of flexibly setting the number of matching feature vectors.

[0055] It can be seen from this that through the technical solution of the present application, the purpose of configuring the corresponding number of matching feature vectors for each matrix block based on the contrast of each matrix block is achieved, thereby achieving the technical effect of flexibly adjusting the number of matching feature vectors corresponding to each image area in the first image, and further solving the technical problem of poor flexibility in setting the number of matching feature vectors during the optical flow estimation of the image.

[0056] In an alternative embodiment, the optical flow estimation system divides the first feature matrix into a plurality of consecutive matrix blocks through a batch processing mechanism, where the area of each matrix block is a preset area, and the difference value between the contrasts of all the feature vectors in each matrix block is less than a preset threshold.

[0057] Optionally, considering the characteristic that pixels with low contrast often aggregate in a certain area of the image in practice, using this characteristic, the optical flow estimation system introduces a batch processing mechanism to divide the first feature matrix F1 into a number of consecutive blocks, as Figure 3 shown. It should be noted that as long as the preset area of each matrix block is not particularly large, the contrasts of all the feature vectors within a matrix block should be close. Therefore, a same target value top_k can be set for all the feature vectors within a matrix block. By dividing the first feature matrix into a plurality of consecutive matrix blocks, the computational amount can be reduced and the speed of subsequent optical flow estimation by the optical flow estimation network can be improved.

[0058] In an alternative embodiment, after dividing the first feature matrix into a plurality of matrix blocks, in order to calculate the contrast corresponding to each matrix block, the optical flow estimation system acquires the size information, dimension corresponding to each matrix block, and the brightness value of each feature vector in each matrix block, where the size information includes the length and width of the matrix block. Then, the optical flow estimation system calculates the average value of the brightness values of all the feature vectors in each matrix block to obtain the average brightness corresponding to each matrix block. Finally, the optical flow estimation system determines the contrast of the matrix block according to the average brightness, size information, dimension corresponding to each matrix block, and the brightness value of each feature vector in the matrix block.

[0059] Specifically, the optical flow estimation system can input the average brightness, size information, dimension corresponding to each matrix block, and the brightness value of each feature vector in the matrix block into a preset function to obtain the contrast of the matrix block, where the preset function is as follows:

[0060]

[0061] In the preset function, F represents the unique identifier of each matrix block, N represents the length of the matrix block, M represents the width of the matrix block, D represents the dimension of the matrix block, F ijk represents the brightness value of the feature vector k at the (i, j) position in the matrix block, avg(F) k represents the average brightness corresponding to the matrix block, and CMS represents the contrast of the matrix block.

[0062] It should be noted that when quantifying the image contrast, the RMS contrast (i.e., the standard deviation of the image pixel intensity) is commonly used for calculation, as shown in the following formula (1):

[0063]

[0064] Among them, in formula (1), I ij represents the luminance value at (i, j) of an image with a size of M*N, and avg(I) is the average luminance of all pixels in the image.

[0065] Referring to the calculation method of RMS, the present application defines the calculation formula of the contrast CMS (standard deviation of characteristic matrix) of the feature matrix as the above preset function.

[0066] In an alternative embodiment, after obtaining the contrast corresponding to each matrix block, in order to determine the target value corresponding to each matrix block according to the contrast corresponding to each matrix block, the optical flow estimation system normalizes the contrast corresponding to each matrix block based on the hyperbolic tangent function to obtain the normalized value corresponding to each matrix block. Then, the optical flow estimation system obtains the value range of the target value, and determines the target value corresponding to each matrix block according to the maximum value, the minimum value, and the normalized value corresponding to each matrix block, where the value range includes the minimum value and the maximum value of the target value, and there is a positive correlation between the contrast and the normalized value, and a negative correlation between the normalized value and the target value.

[0067] Specifically, in order to map the contrast corresponding to each matrix block to the target value top_k, the optical flow estimation system first refers to the hyperbolic tangent function to normalize the contrast value CMS to obtain the normalized value corresponding to the contrast Among them, the normalization process can be completed by the following formula (2):

[0068]

[0069] Assume that the value range of the target value top_k is [top_k min , top_k max , where top_k min is the minimum value of the target value top_k, and top_k max is the maximum value of the target value top_k. The target value top_k corresponding to each matrix block is calculated by the following formula (3).

[0070]

[0071] As can be seen from the above, the target values top_k corresponding to different matrix blocks of the first feature matrix F1 are not exactly the same. The target value top_k corresponding to the matrix block with a higher contrast is smaller, and the target value top_k corresponding to the matrix block with a lower contrast is larger.

[0072] In an alternative embodiment, after determining the target value corresponding to each matrix block according to the contrast corresponding to each matrix block, the optical flow estimation system further obtains all the matching feature vectors between the first feature matrix and the second feature matrix according to the target value corresponding to each matrix block. Specifically, it includes the following steps:

[0073] Step 1, arbitrarily select a feature vector from the first feature matrix as the target feature vector, and detect the vector similarity between each feature vector in the second feature matrix and the target feature vector;

[0074] Step 2, sort all the feature vectors in the second feature matrix according to the vector similarity to obtain a sorting result;

[0075] Step 3, select N feature vectors from the second feature matrix as the matching feature vectors corresponding to the target feature vector according to the sorting result, where N is the target value;

[0076] Step 4, repeat the process of Step 1 to Step 3 until the matching feature vectors corresponding to each feature vector in the first feature matrix are obtained, and obtain the position information of each matching feature vector in the second feature matrix.

[0077] Specifically, assume that the first feature matrix contains 15 feature vectors. By dividing the first feature matrix, two matrix blocks are obtained, namely matrix block 1 and matrix block 2. Among them, matrix block 1 contains 5 feature vectors, and matrix block 2 contains 10 feature vectors. In addition, assume that by calculating the contrast A of matrix block 1 and the contrast B of matrix block 2, it is found that the contrast A is greater than the contrast 2. Therefore, set the target value T for matrix block 1 and the target value W for matrix block 2, where the target value T is less than the target value W.

[0078] Assume that the target value T is 2 and the target value W is 3. On this basis, for each feature vector in matrix block 1, the optical flow estimation system will select the 2 (target value T) feature vectors with the highest similarity to this feature vector from the second feature matrix as the matching feature vectors of this feature vector. Since there are 5 feature vectors in matrix block 1 in total, matrix block 1 corresponds to 10 matching feature vectors in total. Similarly, for each feature vector in matrix block 2, the optical flow estimation system will select the 3 (target value W) feature vectors with the highest similarity to this feature vector from the second feature matrix as the matching feature vectors of this feature vector. Since there are 10 feature vectors in matrix block 2 in total, matrix block 2 corresponds to 30 matching feature vectors in total.

[0079] In an alternative embodiment, after obtaining the matching feature vectors corresponding to each feature vector in the first feature matrix and acquiring the position information of each matching feature vector in the second feature matrix, the optical flow estimation system further stores each matching feature vector and the position information of each matching feature vector in the second feature matrix in the target video memory, and then constructs a similarity matrix based on the matching feature vectors and position information stored in the target video memory, where the similarity matrix represents the matching information between the first feature matrix and the second feature matrix. Subsequently, the optical flow estimation system inputs the similarity matrix and the image features of the first image into the optical flow estimation network to obtain the optical flow field corresponding to the first image, where the optical flow field is used to represent the motion field of the first image.

[0080] Specifically, in order to perform optical flow estimation on the first image, the optical flow estimation system reads all the matching feature vectors related to the first feature matrix and the position information of each matching feature vector from the target video memory. Then, the optical flow estimation system constructs a similarity matrix based on each matching feature vector and the position information of each matching feature vector, splices the image features of the first image and the similarity matrix into a new matrix to be input, and then inputs the spliced matrix to be input into the optical flow estimation network. After several iterations, the result of the optical flow estimation, that is, the optical flow field of the first image, is obtained.

[0081] Among them, the specific process of constructing the similarity matrix is as follows: Let the calculation of the matching process be C, then the feature matching process is shown in the following formula (4):

[0082]

[0083] Among them, cor is the similarity matrix, n is the number of downsampling times, H and W are the height and width of the image respectively (the sizes of the input first image and the second image are the same), the matching calculation C generally uses the Euclidean distance between two feature vectors or the inner product of two feature vectors, D represents the number of channels of the first feature matrix or the second feature matrix, that is, the depth of the first feature matrix or the second feature matrix. Among them, the number of channels of the first feature matrix is the same as that of the second feature matrix, and this number of channels is the dimension of each matrix block corresponding to the first feature matrix. After the calculation is completed, the optical flow estimation system outputs the top_k feature vectors with the highest similarity on the second feature matrix for each feature vector in the first feature matrix and their position information. Finally, the optical flow estimation system uses the obtained cor to perform linear interpolation on the content of the last two dimensions to obtain a multi-level resolution similarity matrix C i , where R represents a real number matrix.

[0084] The technical solution of this application can be verified on the validation set of the open-source dataset FlyingChairs. Among them, the CPU of the device used for verification is an i5-8600K with a main frequency of 3.6 GHZ, and the GPU is an NVIDIA GeForce GTX 1080Ti with a video memory of 11GB.

[0085] Assume the maximum value of the target value is set to 8, and the minimum value top_k of the target value min is set to 2. The target value top_k is set to fixed values 3 and 4 respectively, and the target value top_k is set to a dynamic value (the result is averaged) by adopting the technical solution of this application, and the experimental results shown in Table 1 are obtained.

[0086] Table 1

[0087]

[0088]

[0089] As can be seen from Table 1, by adopting the strategy of dynamically determining the target value top_k of this application, both the consumption of video memory and the accuracy of optical flow estimation are taken into account, thereby optimizing the overall performance of the network. Since there is a linear relationship between the size of the target value top_k and the usage of video memory, compared with the prior art, the technical solution of this application can save more than 20% of video memory resources when the optical flow estimation accuracy is comparable.

[0090] Embodiment 2

[0091] This application embodiment also provides a device for determining the number of matching feature vectors. It should be noted that the device for determining the number of matching feature vectors in this application embodiment can be used to execute the method for determining the number of matching feature vectors provided in Embodiment 1 of this application. The following introduces the device for determining the number of matching feature vectors provided in this application embodiment.

[0092] Figure 4 is a schematic diagram of the device for determining the number of matching feature vectors according to the embodiment of this application. As Figure 4As shown in the figure, the device includes: an acquisition module 401, configured to acquire a first image and a second image in an image sequence, where the first image is the previous frame image of the second image; a feature extraction module 402, configured to extract the image features of the first image and the image features in the second image through a multi-layer convolutional network, and respectively generate a first feature matrix corresponding to the first image and a second feature matrix corresponding to the second image according to the extracted image features, where the image features are used to characterize the image structure information of the first image or the image structure information of the second image; a partitioning module 403, configured to partition the first feature matrix into a plurality of matrix blocks, where each matrix block corresponds to a contrast ratio, and the contrast ratio is used to characterize the standard deviation of the image pixel intensities in the matrix block; a determination module 404, configured to determine the target value corresponding to each matrix block according to the contrast ratio corresponding to each matrix block, where the target value is used to characterize the number of matching feature vectors, and the matching feature vectors are the feature vectors in the first feature matrix that are matched in the second feature matrix according to the vector similarity, and there is a negative correlation between the contrast ratio and the target value.

[0093] Optionally, the partitioning module further includes: a partitioning unit, configured to partition the first feature matrix into a plurality of consecutive matrix blocks through a batch processing mechanism, where the area of each matrix block is a preset area, and the difference value between the contrast ratios of all the feature vectors in each matrix block is less than a preset threshold.

[0094] Optionally, the device for determining the number of matching feature vectors further includes: a first acquisition module, a first calculation module, and a first determination module. The first acquisition module is configured to acquire the size information, dimension, and the brightness value of each feature vector in each matrix block corresponding to each matrix block, where the size information includes the length and width of the matrix block; the first calculation module is configured to calculate the average value of the brightness values of all the feature vectors in each matrix block to obtain the average brightness corresponding to each matrix block; the first determination module is configured to determine the contrast ratio of the matrix block according to the average brightness, size information, dimension, and the brightness value of each feature vector in the matrix block corresponding to each matrix block.

[0095] Optionally, the first determination module further includes: an input unit, configured to input the average brightness, size information, dimension, and the brightness value of each feature vector in the matrix block corresponding to each matrix block into a preset function to obtain the contrast ratio of the matrix block, where the preset function is as follows:

[0096]

[0097] In the preset function, F represents the unique identifier of each matrix block, N represents the length of the matrix block, M represents the width of the matrix block, D represents the dimension of the matrix block, F ijkThe luminance value of the eigenvector k at the position (i, j) in the matrix block, avg(F) k Characterize the average luminance corresponding to the matrix block, and CMS characterize the contrast of the matrix block.

[0098] Optionally, the determination module further includes: a normalization processing unit, a first acquisition unit, and a first determination unit. Among them, the normalization processing unit is used to normalize the contrast corresponding to each matrix block according to the hyperbolic tangent function to obtain the normalized value corresponding to each matrix block, where there is a positive correlation between the contrast and the normalized value; the first acquisition unit is used to acquire the value range of the target value, where the value range includes the minimum value and the maximum value of the target value; the first determination unit is used to determine the target value corresponding to each matrix block according to the maximum value, the minimum value, and the normalized value corresponding to each matrix block, where there is a negative correlation between the normalized value and the target value.

[0099] Optionally, the device for determining the number of matching eigenvectors further includes: a first execution module, a second execution module, a third execution module, and a fourth execution module. Among them, the first execution module is used to execute step 1, arbitrarily select an eigenvector from the first eigenmatrix as the target eigenvector, and detect the vector similarity between each eigenvector in the second eigenmatrix and the target eigenvector; the second execution module is used to execute step 2, sort all the eigenvectors in the second eigenmatrix according to the vector similarity to obtain a sorting result; the third execution module is used to execute step 3, select N eigenvectors from the second eigenmatrix as the matching eigenvectors corresponding to the target eigenvector according to the sorting result, where N is the target value; the fourth execution module is used to execute step 4, repeat the process of steps 1 to 3 until the matching eigenvectors corresponding to each eigenvector in the first eigenmatrix are obtained, and obtain the position information of each matching eigenvector in the second eigenmatrix.

[0100] Optionally, the device for determining the number of matching eigenvectors further includes: a storage module, a construction module, and a first input module. Among them, the storage module is used to store each matching eigenvector and the position information of each matching eigenvector in the second eigenmatrix in the target video memory; the construction module is used to construct a similarity matrix according to the matching eigenvectors and position information stored in the target video memory, where the similarity matrix characterizes the matching information between the first eigenmatrix and the second eigenmatrix;; the first input module is used to input the similarity matrix and the image features of the first image into the optical flow estimation network to obtain the optical flow field corresponding to the first image.

[0101] Embodiment 3

[0102] According to another aspect of the embodiments of the present application, there is also provided a computer-readable storage medium storing a computer program, where the computer program is configured to execute the method for determining the number of matching feature vectors in Embodiment 1 above when running.

[0103] Embodiment 4

[0104] According to another aspect of the embodiments of the present application, there is also provided an electronic device including one or more processors and a memory for storing one or more programs, where when the one or more programs are executed by the one or more processors, the one or more processors implement the method for determining the number of matching feature vectors in Embodiment 1 above.

[0105] The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0106] In the above embodiments of the present application, the descriptions of the respective embodiments have their own emphases. For parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0107] In several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of units can be a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some interfaces. The indirect coupling or communication connection of units or modules can be in an electrical or other form.

[0108] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0109] In addition, the functional units in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0110] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in various embodiments of this application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), mobile hard disks, magnetic disks, or optical discs.

[0111] The above are only the preferred embodiments of this application. It should be noted that for those of ordinary skill in the art, without departing from the principle of this application, several improvements and refinements can still be made, and these improvements and refinements should also be regarded as the protection scope of this application.

Claims

1. A method for determining the number of matching feature vectors, characterized in that, Including: Obtain a first image and a second image in an image sequence, where the first image is the previous frame image of the second image, and the image sequence is a video stream to be subjected to optical flow estimation; Extract the image features of the first image and the image features in the second image through a multi-layer convolutional network, and respectively generate a first feature matrix corresponding to the first image and a second feature matrix corresponding to the second image according to the extracted image features, where the image features are used to represent the image structure information of the first image or the image structure information of the second image; Divide the first feature matrix into multiple matrix blocks, where each matrix block corresponds to a contrast, and the contrast is used to represent the standard deviation of the image pixel intensities in the matrix block; Determine the target value corresponding to each matrix block according to the contrast corresponding to each matrix block, where the target value is used to represent the number of matching feature vectors, and the matching feature vectors are the feature vectors in the second feature matrix matched by each feature vector in the first feature matrix according to vector similarity, and there is a negative correlation between the contrast and the target value; Wherein, after dividing the first feature matrix into multiple matrix blocks, the method further includes: Obtain the size information, dimension, and the brightness value of each feature vector in each matrix block corresponding to each matrix block, where the size information includes the length and width of the matrix block; calculate the average value of the brightness values of all feature vectors in each matrix block to obtain the average brightness corresponding to each matrix block; determine the contrast of the matrix block according to the average brightness, size information, dimension, and the brightness value of each feature vector in the matrix block.

2. The method according to claim 1, wherein Dividing the first feature matrix into multiple matrix blocks includes: Divide the first feature matrix into multiple consecutive matrix blocks through a batch processing mechanism, where the area of each matrix block is a preset area, and the difference value between the contrasts of all feature vectors in each matrix block is less than a preset threshold.

3. The method according to claim 2, wherein Determining the contrast of the matrix block according to the average brightness, size information, dimension, and the brightness value of each feature vector in the matrix block includes: Input the average brightness, size information, dimension, and the brightness value of each feature vector in the matrix block corresponding to each matrix block into a preset function to obtain the contrast of the matrix block, where the preset function is as follows: In the preset function, F represents the unique identifier of each matrix block, N represents the length of the matrix block, M represents the width of the matrix block, D represents the dimension of the matrix block, and F ijk represents the luminance value of the eigenvector k at the position (i, j) in the matrix block, and avg(F) k represents the average luminance corresponding to the matrix block, and CMS represents the contrast of the matrix block.

4. The method according to claim 1, wherein Determining the target value corresponding to each matrix block according to the contrast corresponding to each matrix block includes: Perform normalization processing on the contrast corresponding to each matrix block according to the hyperbolic tangent function to obtain the normalized value corresponding to each matrix block, where there is a positive correlation between the contrast and the normalized value; Obtain the value range of the target value, where the value range includes the minimum value and the maximum value of the target value; Determine the target value corresponding to each matrix block according to the maximum value, the minimum value, and the normalized value corresponding to each matrix block, where there is a negative correlation between the normalized value and the target value.

5. The method according to claim 1, characterized in that, After determining the target value corresponding to each matrix block according to the contrast corresponding to each matrix block, the method further includes: Step 1, arbitrarily select a feature vector from the first feature matrix as the target feature vector, and detect the vector similarity between each feature vector in the second feature matrix and the target feature vector; Step 2, sort all the feature vectors in the second feature matrix according to the vector similarity to obtain a sorting result; Step 3, select N feature vectors from the second feature matrix as the matching feature vectors corresponding to the target feature vector according to the sorting result, where N is the target value; Step 4, repeat the process of Step 1 to Step 3 until the matching feature vectors corresponding to each feature vector in the first feature matrix are obtained, and obtain the position information of each matching feature vector in the second feature matrix.

6. The method according to claim 5, wherein After obtaining the matching feature vectors corresponding to each feature vector in the first feature matrix and obtaining the position information of each matching feature vector in the second feature matrix, the method further includes: Store each matching feature vector and the position information of each matching feature vector in the second feature matrix into the target video memory; Construct a similarity matrix according to the matching feature vectors and position information stored in the target video memory, where the similarity matrix represents the matching information between the first feature matrix and the second feature matrix; Input the similarity matrix and the image features of the first image into the optical flow estimation network to obtain the optical flow field corresponding to the first image.

7. A device for determining the number of matching feature vectors, characterized in that Including: An acquisition module, configured to acquire a first image and a second image in an image sequence, where the first image is the previous frame image of the second image, and the image sequence is a video stream to be subjected to optical flow estimation; A feature extraction module, configured to extract the image features of the first image and the image features in the second image through a multi-layer convolutional network, and respectively generate a first feature matrix corresponding to the first image and a second feature matrix corresponding to the second image according to the extracted image features, where the image features are used to represent the image structure information of the first image or the image structure information of the second image; A partitioning module, configured to partition the first feature matrix into multiple matrix blocks, where each matrix block corresponds to a contrast, and the contrast is used to represent the standard deviation of the image pixel intensity in the matrix block; A determination module, configured to determine the target value corresponding to each matrix block according to the contrast corresponding to each matrix block, where the target value is used to represent the number of matching feature vectors, and the matching feature vectors are the feature vectors matched by each feature vector in the first feature matrix in the second feature matrix according to the vector similarity, and there is a negative correlation between the contrast and the target value; The apparatus for determining the number of matching feature vectors further includes: a first obtaining module, configured to obtain the size information, dimension, and the brightness value of each feature vector in each of the matrix blocks corresponding to each of the matrix blocks, where the size information includes the length and width of the matrix block; a first calculating module, configured to calculate the average value of the brightness values of all the feature vectors in each of the matrix blocks to obtain the average brightness corresponding to each of the matrix blocks; and a first determining module, configured to determine the contrast of each of the matrix blocks according to the average brightness, size information, dimension, and the brightness value of each feature vector in the matrix block corresponding to each of the matrix blocks.

8. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, where the computer program is configured to execute the method for determining the number of matching feature vectors described in any one of claims 1 to 6 when running.

9. An electronic device, characterized in that, It includes one or more processors and a memory, and the memory is used to store one or more programs, where when the one or more programs are executed by the one or more processors, the one or more processors implement the method for determining the number of matching feature vectors described in any one of claims 1 to 6.

Images