Video area relocation method and device, electronic device and storage medium

Abstract

The present disclosure relates to a video area relocation method and device, electronic device and storage medium, and relates to the technical field of video area relocation. The video area relocation method comprises: obtaining a sensitive area defined by a first image in a real-time video stream and a plurality of second images in the real-time video stream; determining matching fixed point pairs of the first image and the plurality of second images according to the sensitive area; determining the sensitive area in the plurality of second images according to the coordinate transformation of the matching fixed point pairs, so as to relocate the sensitive area in the video stream in real time. The embodiments of the present disclosure can realize the relocation of sensitive areas in a video stream.

Description

Technical Field

[0001] The present disclosure relates to the technical field of video area relocation, and in particular to a video area relocation method and device, electronic equipment and storage medium. Background Art

[0002] In order to identify unsafe factors at offshore oil field operation sites, the recognition algorithms that have been developed have been deployed and applied. However, if all scenes within the camera are recognized during the application process, sensitive areas in the video stream may not be relocated due to camera movement, scene changes, and too many detected scenes. Summary of the invention

[0003] The present disclosure proposes a video area relocation method and device, an electronic device, and a storage medium technical solution.

[0004] According to one aspect of the present disclosure, a method for relocating a video area is provided, comprising:

[0005] Acquire a sensitive area defined by a first image in a real-time video stream and a plurality of second images in the real-time video stream;

[0006] Determining matching fixed point pairs of the first image and the plurality of second images according to the sensitive area;

[0007] According to the coordinate transformation of the matching fixed point pair, the sensitive areas in the plurality of second images are determined to relocate the sensitive areas in the video stream in real time.

[0008] Preferably, the method for determining matching fixed point pairs of the first image and the plurality of second images according to the sensitive area comprises:

[0009] determining the first image as a fixed image, and determining the plurality of second images as floating images;

[0010] performing a registration operation on the floating image and the fixed image according to the sensitive area to obtain matching fixed point pairs of the first image and the plurality of second images;

[0011] and / or,

[0012] Before determining the sensitive areas in the plurality of second images according to the coordinate transformation of the matching fixed point pairs, screening the matching fixed point pairs;

[0013] determining sensitive areas in the plurality of second images according to the coordinate transformation of the screened matching fixed point pairs;

[0014] and / or,

[0015] Before determining the sensitive areas in the plurality of second images according to the filtered matching fixed point pairs or the coordinate transformation of the matching fixed point pairs, coordinate transformation is performed on the filtered matching fixed point pairs or the matching fixed point pairs, wherein the coordinate transformation method comprises:

[0016] According to the linear transformation equation of the horizontal coordinate and the linear transformation equation of the vertical coordinate of the selected matching fixed point pair or the matching fixed point pair respectively;

[0017] The coordinate transformation includes: a linear transformation equation of the horizontal coordinate and a linear transformation equation of the vertical coordinate.

[0018] Preferably, the method of respectively transforming the linear transformation equation of the horizontal coordinate and the linear transformation equation of the vertical coordinate based on the screened matching fixed point pairs or the matching fixed point pairs comprises:

[0019] Determining a first slope and a first intercept according to the screened matching fixed point pair or the abscissa of the matching fixed point pair;

[0020] constructing a linear transformation equation of the abscissa according to the first slope and the first intercept;

[0021] Determining a second slope and a second intercept according to the screened matching fixed point pair or the ordinate of the matching fixed point pair;

[0022] constructing a linear transformation equation of the ordinate according to the second slope and the second intercept;

[0023] and / or,

[0024] Before determining the sensitive areas in the plurality of second images according to the coordinate transformation of the matching fixed point pairs, the matching fixed point pairs are screened, and the method for screening the matching fixed point pairs includes:

[0025] Get the set percentage value;

[0026] Respectively calculating the average value of the horizontal coordinate change amount and the average value of the vertical coordinate change amount of the matching fixed point pair;

[0027] Calculate the distances between the matching fixed point pair and the average value point composed of the average value of the horizontal coordinate change amount and the average value of the vertical coordinate change amount respectively to obtain multiple distances;

[0028] Sorting the multiple distances in ascending order to obtain multiple distance sequences;

[0029] Based on the set percentage value, the plurality of distance sequences are intercepted to obtain matching fixed point pairs corresponding to the intercepted plurality of distance sequences;

[0030] and / or,

[0031] The registration algorithm for performing the registration operation on the floating image and the fixed image is a SIFT registration algorithm;

[0032] and / or,

[0033] The method of performing a registration operation on the floating image and the fixed image according to the sensitive area to obtain matching fixed point pairs of the first image and the plurality of second images includes:

[0034] According to the sensitive area, performing a registration operation on the floating image and the fixed image to obtain a first number of first fixed points and a second number of second fixed points corresponding to the first image and the plurality of second images;

[0035] constructing a plurality of two-dimensional similarity matrices using the first fixed point and the corresponding second fixed points respectively;

[0036] Respectively calculating the effective maximum values ​​of rows and their column numbers and the effective maximum values ​​of columns and their row numbers in the multiple two-dimensional similarity matrices;

[0037] Based on the row effective maximum value and its column number and the column effective maximum value and its row number, matching fixed point pairs of the first image and the plurality of second images are determined.

[0038] Preferably, the method of respectively calculating the effective maximum values ​​of rows and column numbers in the plurality of two-dimensional similarity matrices comprises:

[0039] Get row setting threshold;

[0040] Determine the maximum and second largest values ​​of each row in each two-dimensional similarity matrix;

[0041] Calculating a first ratio of the maximum value to the second largest value;

[0042] If the first ratio is greater than the row setting threshold, the maximum value is determined as the row effective maximum value, and the column number of the row effective maximum value is recorded;

[0043] Otherwise, the row in the two-dimensional similarity matrix does not have a row effective maximum value, and the row is marked;

[0044] and / or,

[0045] The method of respectively calculating the effective maximum values ​​of columns and row numbers in the plurality of two-dimensional similarity matrices comprises:

[0046] Get column setting threshold;

[0047] Determine the maximum value and the second largest value of each column in each two-dimensional similarity matrix respectively;

[0048] Calculating a second ratio of the maximum value to the second largest value;

[0049] If the second ratio is greater than the column set threshold, the maximum value is determined as the column effective maximum value, and the row number of the column effective maximum value is recorded;

[0050] Otherwise, the row in the two-dimensional similarity matrix does not have a valid column maximum value, and the column is marked;

[0051] and / or,

[0052] The method for determining matching fixed point pairs of the first image and the plurality of second images based on the row effective maximum value and its column number and the column effective maximum value and its row number comprises:

[0053] Get the row mark and the column mark;

[0054] Based on the row mark, the column mark, the row effective maximum value and its column number, and the column effective maximum value and its row number, matching fixed point pairs of the first image and the multiple second images are determined.

[0055] Preferably, the method for determining the matching fixed point pairs of the first image and the plurality of second images based on the row mark, the column mark, the row effective maximum value and its column sequence number, and the column effective maximum value and its row sequence number comprises:

[0056] Determine the total number of rows of the two-dimensional similarity matrix;

[0057] If the number of rows of the row effective maximum value is within the total number of rows and does not have the row mark, and the row effective maximum value is the same as the column effective value, then the fixed point corresponding to the row effective maximum value of the first image is determined to match the fixed point corresponding to the column number of the row effective maximum value in the multiple second images to obtain matching fixed point pairs of the first image and the multiple second images.

[0058] Preferably, the method for determining the sensitive areas in the plurality of second images according to the coordinate transformation of the matching fixed point pairs comprises:

[0059] Determining the matching fixed points corresponding to the plurality of second images respectively according to the coordinate transformation of the matching fixed point pair;

[0060] respectively determining a plurality of image boundaries corresponding to the plurality of second images;

[0061] Sensitive areas in the plurality of second images are determined based on the matching fixed points corresponding to the plurality of second images and the plurality of image boundaries, respectively.

[0062] Preferably, the method for determining the sensitive areas in the plurality of second images based on the matching fixed points corresponding to the plurality of second images and the plurality of image boundaries respectively comprises:

[0063] Determining whether all matching fixed points corresponding to the plurality of second images are within the corresponding image boundaries;

[0064] If yes, then determining the area where the matching fixed point is located as the sensitive area in the corresponding second image;

[0065] Otherwise, coordinate correction is performed on the matching fixed points outside the image boundary to obtain the matching fixed points after coordinate correction; based on the matching fixed points after coordinate correction and the matching fixed points within the image boundary, the corresponding sensitive area in the second image is determined.

[0066] Preferably, the method for determining whether all matching fixed points corresponding to the plurality of second images are within the corresponding image boundaries comprises:

[0067] Connecting the matched fixed points in the plurality of second images respectively to form a polygon;

[0068] If the polygon has an intersection with the image boundary, determining that the matching fixed points corresponding to the plurality of second images are outside the corresponding image boundary;

[0069] Otherwise, determining that all matching fixed points corresponding to the plurality of second images are within the corresponding image boundaries;

[0070] and / or,

[0071] The method for performing coordinate correction on a matching fixed point outside the image boundary comprises:

[0072] Determining a position of a matching fixed point outside a boundary of the image;

[0073] The horizontal coordinate and the vertical coordinate of the matching fixed point outside the image boundary are updated based on the position and the image boundary, and the coordinate correction of the matching fixed point outside the image boundary is completed to obtain the matching fixed point after the coordinate correction.

[0074] Preferably, the method of updating the horizontal coordinates and coordinates of the matching fixed points outside the image boundary based on the position and the image boundary, completing the coordinate correction of the matching fixed points outside the image boundary, and obtaining the matching fixed points after the coordinate correction includes:

[0075] Get the set direction;

[0076] According to the set direction, the matching fixed point moves along the edge of the polygon formed by all the matching fixed points;

[0077] When the edge of the polygon crosses the left boundary or the right boundary of the polygon, the coordinates of the matching fixed point outside the left boundary or the right boundary are updated to the coordinates of the intersection of the edge and the left boundary or the right boundary, so as to obtain an updated polygon;

[0078] Continuing to move the matching fixed point in the updated polygon along the edge of the updated polygon in the set direction;

[0079] When the edge of the updated polygon passes through the upper boundary or lower boundary of the polygon, the coordinates of the matching fixed point outside the upper boundary or lower boundary are updated to the coordinates of the intersection of the edge and the upper boundary or lower boundary to obtain the matching fixed point with corrected coordinates.

[0080] According to one aspect of the present disclosure, a video area relocation device is provided, comprising:

[0081] An acquisition unit, configured to acquire a sensitive area defined by a first image in a real-time video stream and a plurality of second images in the real-time video stream;

[0082] A matching unit, configured to determine matching fixed point pairs of the first image and the plurality of second images according to the sensitive area;

[0083] A repositioning unit is used to determine the sensitive areas in the multiple second images according to the coordinate transformation of the matching fixed point pairs, so as to reposition the sensitive areas in the video stream in real time.

[0084] According to one aspect of the present disclosure, there is provided an electronic device, including:

[0085] processor;

[0086] a memory for storing processor-executable instructions;

[0087] Wherein, the processor is configured to: execute the above-mentioned video area relocation method.

[0088] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above-mentioned video area relocation method is implemented.

[0089] In the embodiments of the present disclosure, a method and device for relocating a video area, an electronic device and a storage medium are proposed to solve the problem of identifying unsafe factors at the current offshore oil field operation site. Specifically, the recognition algorithm that has been developed currently, during the process of deployment and application, if all scenes within the camera are recognized, there will be a problem that the detection algorithm cannot relocate sensitive areas in the video stream due to camera movement, scene changes, and too many scenes to be detected.

[0090] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

[0091] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0093] Figure 1 A flowchart showing relocation of a video area according to an embodiment of the present disclosure;

[0094] Figure 2 A schematic diagram of a two-dimensional similarity matrix according to an embodiment of the present disclosure is shown;

[0095] Figure 3 A schematic diagram of matching fixed points in a two-dimensional similarity matrix according to an embodiment of the present disclosure is shown;

[0096] Figure 4 A schematic diagram of coordinate correction according to an embodiment of the present disclosure is shown;

[0097] Figure 5 is a block diagram of an electronic device 800 according to an exemplary embodiment;

[0098] Figure 6 is a block diagram of an electronic device 1900 according to an exemplary embodiment. DETAILED DESCRIPTION

[0099] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0100] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0101] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0102] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0103] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not go into details.

[0104] In addition, the present disclosure also provides a video area repositioning device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any video area repositioning method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to in the corresponding records of the method part and will not be repeated here.

[0105] Figure 1 A flowchart showing a method for relocating a video area according to an embodiment of the present disclosure is shown. Figure 1 As shown, the video area relocation method includes: step S101: obtaining the sensitive area defined by the first image in the real-time video stream and multiple second images in the real-time video stream; step S102: determining the matching fixed point pairs of the first image and the multiple second images according to the sensitive area; step S103: determining the sensitive area in the multiple second images according to the coordinate transformation of the matching fixed point pairs, so as to relocate the sensitive area in the video stream in real time. To solve the problem of identifying unsafe factors at the current offshore oil field operation site, specifically, the recognition algorithm that has been developed, during the process of deployment and application, if all scenes in the camera are identified, there will be a problem that the detection algorithm cannot relocate the sensitive area in the video stream due to camera movement, scene changes, and too many detected scenes. Based on the above problems, the present disclosure proposes an algorithm for the coordinate transformation of the matching fixed point pairs, thereby determining the sensitive area in the multiple second images, so as to relocate the sensitive area in the video stream in real time.

[0106] Step S101: Acquire a sensitive area defined by a first image in a real-time video stream and a plurality of second images in the real-time video stream. The number of the plurality of second images is at least one, that is, acquire a sensitive area defined by a first image in a real-time video stream and at least one second image in the real-time video stream. In the specific embodiment of the present disclosure and other possible embodiments, for ease of understanding, a plurality should be understood as at least one.

[0107] In the specific embodiment of the present disclosure and other possible embodiments, the offshore oil field operation site needs to use a camera to shoot all scenes of the offshore oil field operation site in real time to form a real-time video stream.

[0108] In the specific embodiment of the present disclosure and other possible embodiments, the sensitive area defined by the first image can be defined according to the needs of the user. For example, the sensitive area can be an offshore oil well, a driving device of an oil field, or other equipment that requires real-time monitoring.

[0109] The sensitive area of ​​the first image, the method for defining the sensitive area of ​​the first image can be implemented by outlining the shape of the device to be monitored in real time in the first image, or the sensitive area of ​​the first image includes: determining a to-be-defined area including the first image in the first image, and determining the sensitive area of ​​the first image by using an edge detection algorithm in the to-be-defined area. The edge detection algorithm can be one or more of the following: Roberts gradient operator, prewitt operator, sobel operator, laplacian operator, canny operator.

[0110] Step S102: determining matching fixed point pairs of the first image and the plurality of second images according to the sensitive area.

[0111] In the present disclosure, the method for determining the matching fixed point pairs of the first image and the plurality of second images according to the sensitive area includes: determining the first image as a fixed image, and determining the plurality of second images as floating images; performing a registration operation on the floating image and the fixed image according to the sensitive area to obtain the matching fixed point pairs of the first image and the plurality of second images. The registration algorithm for performing the registration operation on the floating image and the fixed image is a SIFT registration algorithm.

[0112] The essence of the image fixed point detection algorithm is to find key points in different scale spaces, calculate the size, direction, and scale information of the key points, and use this information to form key points to describe the fixed points. The key points found by the fixed points are some very prominent "stable" feature points that will not change due to factors such as lighting, affine transformation, and noise.

[0113] The purpose of constructing a scale space is to simulate the multi-scale characteristics of image data. In scale transformation, the Gaussian convolution kernel is the only linear kernel. Assuming I(x,y) is the original image and G(x,y,δ) is a scale-variable Gaussian function, the scale space of an image can be defined as the function L(x,y,δ), as shown in the following formula:

[0114] L(x,y,δ)=G(x,y,δ)*I(x,y).

[0115] In the above formula, δ represents the scale factor, * represents the convolution operation, and the specific expression of the Gaussian kernel function G(x, y, δ) is as follows:

[0116]

[0117] When performing a convolution operation, because all pixels of the entire image must be traversed before convolution, this will lead to high time and space complexity. Using the Gaussian difference scale space (DOG scale-space), this can not only effectively locate the direction of the key coordinate points, but also reduce the complexity of time and space. The difference formula of two adjacent scales separated by a constant multiplied by a coefficient k can be calculated as follows:

[0118] D(x,y,δ)=(G(x,y,kδ)-G(x,y,δ))*I(x,y)=L(x,y,kδ)-L(x,y,δ).

[0119] (1) The original image is convolved with a Gaussian convolution kernel. Due to the different scale factors, a set of different scale spaces can be obtained to form the layers of the Gaussian pyramid. Then the first image in the first layer of the Gaussian pyramid is downsampled by 2 times the pixel distance to obtain the first image in the second layer. Then the image is convolved with Gaussian kernels of different scale factors to obtain a set of images in the second layer of the Gaussian pyramid. Similarly, according to this method, we can obtain a layer of images in each group of the Gaussian pyramid.

[0120] (2) Subtract two adjacent layers of the Gaussian pyramid image to obtain a Gaussian difference pyramid image. The extreme points extracted from the Gaussian difference pyramid hierarchical structure are used as candidate feature points. At the same time, in the DOG scale space, each point is compared with the points of adjacent scales and adjacent positions one by one. In this way, the position of the local extreme value can be obtained, which is the corresponding scale and position of the feature point.

[0121] In order to detect local extreme points in the scale space, it is necessary to compare each sampling point with the adjacent scale space (upper and lower layers) and the adjacent points in the same space (a total of 26). If a point is larger or smaller than the 26 adjacent points around it, then the point is an extreme point. In the process of extreme value comparison, the first and last two layers of each group of images cannot be effectively compared. Therefore, in order to meet the continuity of scale changes, three images are generated by Gaussian blurring on the top layer of each layer of images. Therefore, each group of Gaussian pyramids has S+3 layers of images (S is the number of layers per pyramid), and each group of Gaussian difference pyramids has S+2 layers of images. In order to make the located extreme points accurate, enhance the matching efficiency, and effectively suppress noise, it is necessary to eliminate those points with low contrast and points at the edge.

[0122] (1) Remove low-contrast points. In order to filter out low-contrast points, it is necessary to perform curve fitting on the D(x, y, δ) function, which is expanded after being processed using the Taylor formula. The specific formula is as follows:

[0123]

[0124] Among them, x represents the coordinate point, X(x,y,δ) T , perform the derivative operation on the above formula and let its expression value equal to 0, then the offset expression of the extreme point is as follows:

[0125]

[0126] To compensate for this, substitute equation (2) into equation (1) and we get:

[0127]

[0128] When the |D(X)| value of a point is too small, it is easily disturbed by noise and becomes unstable. The extreme points with |D(X)|<0.03 are deleted. At the same time, the precise position (original position plus the fitted offset) and scale of the feature points are obtained in this process.

[0129] (2) Eliminate edge response. The DOG operator will produce a strong edge response and is easily affected by noise. Generally speaking, the extreme value of the DOG function will have a relatively large principal curvature at the intersection, such as at the edge, but at the vertical edge, the principal curvature of its extreme value will be relatively small. Based on this property, unstable edge response points can be removed. The Hessian matrix H at the feature point is expressed as follows, where D ij Respectively represent the corresponding second-order partial derivatives:

[0130]

[0131] Assume that the eigenvalues ​​of H are a and b, which represent the gradients in the x and y directions respectively, Tr(H) is the sum of the diagonal elements of H, and Det(H) is the determinant of , then:

[0132] Tr(H)=D xx +D yy =a+b.

[0133] Det(H)=D xx D yy -(D xy ) 2 =ab.

[0134] Assume that the eigenvalue a is large and b is small, let γ be the ratio of the maximum magnitude eigenvalue to the minimum eigenvalue, and a = γb, then:

[0135]

[0136] The principal curvature is proportional to the eigenvalues ​​a and b. When a=b, (γ+1) in formula (10) 2 / γ has the smallest value; when the ratio of a to b is larger, (γ+1) 2 The larger the value of / γ, the larger the gradient value in one direction, and the smaller the gradient value in another direction, which just meets the edge situation. Therefore, in order to remove these edge response points, you only need to set a specific threshold γ, when:

[0137]

[0138] When the formula is true, the key point is retained, otherwise it is removed.

[0139] The basic idea of ​​determining the direction of feature points is to assign a direction to each key point, which can be achieved through the local features of the image. At the same time, in order to achieve the rotation independence of fixed points, it is also necessary to use the gradient and direction distribution characteristics of the key point neighborhood pixels. The expressions of the gradient m(x, y) and θ(x, y) directions are as follows:

[0140]

[0141] θ(x,y)=arctan(L(x,y+1)-L(x,y-1)) / L(x+1,y)-L(x-1,y)).

[0142] In the above formula, L(x,y) is the sampling point, θ(x,y) is the direction indicator, and the gradient direction of the pixels in the area around the key point is calculated. The specific method is to take the key point as the center, sample in its neighborhood window, and use the histogram to count the gradient direction of the pixels in the field, then accumulate and sum them, and finally draw a gradient direction histogram. The range of the gradient histogram is 0 to 360 degrees, with a column every 45 degrees, a total of 8 columns, that is, 8 directions, and the peak of the histogram is selected as the main direction of the key point.

[0143] Through the steps in the above sections, each key point is given three pieces of information: position, scale and direction. Then, an appropriate descriptor is established for each key point, and a set of vectors is used to describe the key point so that it does not change with changes in lighting, viewing angle, etc. At the same time, in order to improve the accuracy of feature point matching, the descriptor should also have a high degree of uniqueness. The gradient information of 8 directions is calculated in a 16*16 window in the key point scale space. This information represents the descriptor, which is a 128-dimensional feature vector.

[0144] In the actual matching process, we use 4*4=16 seed points to describe each key feature point, that is, we sample every pixel in the 16*16 window around the key point. This will enhance the robustness of the matching, so each key feature point can eventually generate 4*4*8=128 data, that is, a 128-dimensional feature vector.

[0145] After the 128-dimensional feature vectors are formed, they are normalized to remove the influence of illumination changes. Assume that the resulting 128-dimensional descriptor vector is H(h 1 ,h 2 ,...,h 128 ), the normalized eigenvector is L = (L 1 ,L 2 ,...,L 128 ),but:

[0146]

[0147] Finally, in order to reduce the impact of some large gradient values, a threshold is set to cut off larger gradient values. After vector normalization, this value is generally 0.2. Then, a normalization process is performed to improve the algorithm's ability to overcome brightness changes.

[0148] In the present disclosure, the method for performing a registration operation on the floating image and the fixed image according to the sensitive region to obtain the matching fixed-point pairs of the first image and the multiple second images includes: performing a registration operation on the floating image and the fixed image according to the sensitive region to obtain a first number of first fixed points corresponding to the first image and a second number of second multiple fixed points corresponding to the multiple second images; respectively constructing multiple two-dimensional similarity matrices with the first fixed points and the corresponding second multiple fixed points; respectively calculating the effective maximum value in each row and its column serial number and the effective maximum value in each column and its row serial number of the multiple two-dimensional similarity matrices; and determining the matching fixed-point pairs of the first image and the multiple second images based on the effective maximum value in each row and its column serial number and the effective maximum value in each column and its row serial number.

[0149] For convenience of description, the present disclosure only illustrates with one second image. To achieve the fixed-point matching of two images (the first image and the second image), it is assumed that the fixed-point detection has been performed on the two images, and there are m first fixed points and n second fixed points respectively. For ease of description, let u i represent the first feature vector of each fixed point of the first image (0 ≤ i < m - 1), and let v j represent the second feature vector of each fixed point of the second image (0 ≤ j < n - 1). The feature vector of each fixed point includes 128-dimensional features. Let u ik represent the k-th dimension feature of the first feature vector u i , and let v ik represent the k-th dimension feature of the second feature vector v i .

[0150] To facilitate the rewriting into a GPU program to reduce the running time of the algorithm, a triple-loop program structure is adopted. The first loop iterates through each first fixed point of the first image, the second loop iterates through each second fixed point of the second image, and the third loop calculates the similarity of 128-dimensional feature vectors. This implementation method has no if...else... statements and is very suitable for running on a GPU. The algorithm is as follows:

[0151]

[0152] In the above algorithm, the loop for k is used to calculate the dot product of u i and v j , take the dot product of the two as the similarity of u i and v j , and use Similarity(i, j) to represent the similarity of u i and v j . Each u i and v jThe Similarity(i,j) values ​​of compose the feature vector similarity matrix.

[0153] An example of a feature vector similarity matrix is ​​given below. In this example, the first image has 7 fixed points and the second image has 5 fixed points. The value of each element in the matrix is ​​the similarity of the corresponding row and column feature vectors.

[0154] Figure 2 FIG. 2 shows a schematic diagram of a two-dimensional similarity matrix according to an embodiment of the present disclosure. Figure 2 As shown, the similarity between the first fixed point and the second fixed point of the two images (the first image and the second image) can be analyzed by the similarity matrix corresponding to the first eigenvector and the second eigenvector, and a fixed point matching pair with high similarity can be found. In practical applications, there may be a situation where a fixed point in the first image is similar to two or more fixed points in the second image. In this case, only one point in the second image can correctly correspond to the fixed point of the first image. If one is randomly selected, the probability of incorrect matching will be relatively high. Based on this consideration, if a fixed point in the first image is similar to two or more fixed points in the second image, this feature point of the first image is not matched with any feature point of the second image to reduce the matching error rate.

[0155] Taking the above similarity matrix as an example, first find the maximum and second largest values ​​of each similarity in each row. If the maximum value is significantly greater than the second largest value, the maximum value is the valid maximum value, and the maximum value of each row is represented by a box. If the maximum value and the second largest value are close, it means that the row has no valid maximum value. In this example, rows u0, u3, u4, and u6 have no valid maximum value because the maximum value and the second largest value are close. Rows u1, u2, and u5 have valid maximum values. Each column is processed in a similar way. In this example, column v0 has no valid maximum value because the maximum value and the second largest value are close. Columns v1, v2, v3, and v4 have valid maximum values. If an element of the similarity matrix element is both the valid maximum value of the row and the valid maximum value of the column, it means that the two fixed points represented by the row and column have a corresponding relationship and are a valid fixed point matching pair.

[0156] In the present disclosure, the method for respectively calculating the effective maximum values ​​of rows and their column numbers in the multiple two-dimensional similarity matrices includes: obtaining a row setting threshold; respectively determining the maximum value and the second largest value of each row in each two-dimensional similarity matrix; calculating the first ratio of the maximum value and the second largest value; if the first ratio is greater than the row setting threshold, the maximum value is determined as the effective maximum value of the row, and the column number of the effective maximum value of the row is recorded; otherwise, the row in the two-dimensional similarity matrix does not have a row effective maximum value, and the row is marked. Among them, those skilled in the art can set the row setting threshold as needed; the second largest value is the second largest second largest value. For example, the elements of a row in the two-dimensional similarity matrix are 0.01, 0.00, 0.15, 0.28, 0.02, respectively, the maximum value is 0.28, and the second largest value is 0.15.

[0157] In the present disclosure, the method for respectively calculating the effective maximum values ​​of columns and their row numbers in the multiple two-dimensional similarity matrices includes: obtaining a column setting threshold; respectively determining the maximum value and the second maximum value of each column in each two-dimensional similarity matrix; calculating the second ratio of the maximum value and the second maximum value; if the second ratio is greater than the column setting threshold, the maximum value is determined as the effective maximum value of the column, and the row number of the effective maximum value of the column is recorded; otherwise, the row in the two-dimensional similarity matrix does not have a column effective maximum value, and the column is marked. Among them, those skilled in the art can set the column setting threshold as needed.

[0158] Figure 3 FIG. 2 shows a schematic diagram of matching fixed points in a two-dimensional similarity matrix according to an embodiment of the present disclosure. Figure 3 As shown, □ represents the valid maximum value of the row, and ○ represents the valid maximum value of the column.

[0159] Assume that S RowMax (i) represents the maximum value of the i-th row of the matrix, expressed as S RowSecondMax (i) represents the second largest value of the i-th row of the matrix, then the first ratio S RowMax (i) / S RowSecondMax (i)>T r When S RowMax (i) is the effective maximum value of the i-th row. In the above formula, T r is a threshold value (row setting threshold value). When the effective maximum value is obtained, the column number of the effective maximum value is also recorded. RowMax (i) indicates the column number of the valid maximum value in the i-th row; if there is no valid maximum value in the i-th row, then I RowMax The value of (i) is -1 (this row is marked with -1).

[0160] Assume that S ColMax (j) represents the maximum value of the jth column of the matrix, expressed as S ColSecondMax(j) represents the second largest value of the jth column of the matrix, then the second ratio S ColMax (j) / S ColSecondMax (j)>T r When S ColMax (j) is the effective maximum value of the jth column. In the above formula, T r is a threshold value (column setting threshold value). When the effective maximum value is obtained, the row number of the effective maximum value is also recorded. ColMax (j) represents the serial number of the row where the valid maximum value of the jth column is located; if there is no valid maximum value in the jth column, then I ColMax The value of (j) is -1 (this column is marked as -1).

[0161] In the present disclosure, the method for determining the matching fixed point pairs of the first image and the multiple second images based on the row effective maximum value and its column number and the column effective maximum value and its row number includes: obtaining the row mark and the column mark; determining the matching fixed point pairs of the first image and the multiple second images based on the row mark, the column mark, the row effective maximum value and its column number and the column effective maximum value and its row number.

[0162] In the specific embodiment of the present disclosure and other possible embodiments, the method for determining the matching fixed point pairs of the first image and the plurality of second images based on the row mark, the column mark, the row effective maximum value and its column number and the column effective maximum value and its row number includes: determining the total number of rows m of the two-dimensional similarity matrix; if the number of rows i of the row effective maximum value is within the total number of rows m and does not have the row mark (I RowMax (i)! = -1), and the row effective maximum value is the same as the column effective value (I ColMax (I RowMax (i))==i), then determine to match the fixed point corresponding to the row effective maximum value of the first image with the fixed point corresponding to the column number of the row effective maximum value in the plurality of second images (the i-th fixed point of the first image and the i-th fixed point of the second image) RowMax (i) fixed points are corresponding points), and matching fixed point pairs of the first image and the plurality of second images are obtained.

[0163] After obtaining the column number I of the valid maximum value of each i-th row RowMax (i) and the serial number I of the row where the valid maximum value of each j-th column is located ColMax (j), the code for calculating whether there is a corresponding relationship between the fixed points is:

[0164] For i=0,i <m,i=i+1

[0165] {If I RowMax(i) ! = -1

[0166] {If I ColMax (I RowMax (i))==I the i-th fixed point of the first image and the i-th fixed point of the second image RowMax (i) fixed points are corresponding points}}.

[0167] In the present disclosure, before determining the sensitive areas in the multiple second images based on the coordinate transformation of the matching fixed point pairs, the matching fixed point pairs are screened; and based on the coordinate transformation of the screened matching fixed point pairs, the sensitive areas in the multiple second images are determined.

[0168] In the present disclosure, before determining the sensitive areas in the multiple second images according to the coordinate transformation of the matching fixed point pairs, the matching fixed point pairs are screened, and the method for screening the matching fixed point pairs includes: obtaining a set percentage value; respectively calculating the average value of the horizontal coordinate change amount and the average value of the vertical coordinate change amount of the matching fixed point pairs; respectively calculating the distance from the matching fixed point pairs to the average value point composed of the average value of the horizontal coordinate change amount and the average value of the vertical coordinate change amount to obtain multiple distances; sorting the multiple distances in ascending order to obtain multiple distance sequences; intercepting the multiple distance sequences based on the set percentage value to obtain the matching fixed point pairs corresponding to the intercepted multiple distance sequences. Among them, the set percentage value can be 70%, and those skilled in the art can set it according to actual needs.

[0169] After calculating the fixed point positions and matching the fixed points of the first image P and the second image Q, we obtain the corresponding relationships between several pairs of fixed points. Let us assume that there are s pairs of fixed points. These s pairs of fixed points can be described as <(x P1 ,y P1 ),(x Q1 ,y Q1 )>,<(x P2 ,y P2 ),(x Q2 ,y Q2 )>,<(x P3 ,y P3 ),(x Q3 ,y Q3 )>,......,<(x Ps ,y Ps ),(x Qs ,y Qs )>. Through these s pairs of fixed points, the functional relationship between the coordinates of the corresponding points in image P and image Q can be established. Considering translation and scaling, the coordinate function relationship between the two images is a linear relationship, that is, there is a parameter k x , bx , k y , b y , so that the following formula holds:

[0170] x Qi =k x *x Pi +b x ;

[0171] y Qi =k y *y Pi +b y .

[0172] Since fixed point matching uses local features rather than global features, mismatching due to local similarity is inevitable. If mismatched points are also included in the calculation, the error will increase. x , b x , k y , b y When , we need to remove the wrong matching points first. Given an example of fixed point matching, in this example, there are 4 pairs of points that are correctly matched and 1 pair of points that are wrongly matched. Assume

[0173] Δx i =x Pi -x Qi ;

[0174] Δy i =y Pi -y Qi .

[0175] In this example, it can be found that for the correctly matched fixed point, the horizontal coordinate change Δx i The value will be closer. For the wrong matching point, the horizontal coordinate change Δx i The value of will be different from other values. i The same is true for the case. Therefore, we can analyze the change in the horizontal coordinate Δx i and the vertical coordinate change Δy i The statistical laws of the system determine which are correct matches and which are incorrect matches.

[0176] To find the correct and incorrect matching points, first calculate the horizontal coordinate change Δx i and the vertical coordinate change Δy i The average value of the horizontal axis is A Δx and the vertical coordinate change A Δy express:

[0177]

[0178] Next, calculate the (Δx i ,Δy i ) and (A Δx ,A Δy ) and is represented by the distance D(i). The smaller D(i), the closer (Δx i ,Δy i ) and (A Δx ,A Δy ) is closer.

[0179] Calculate the distances between the matching fixed point pair and the average value point composed of the average value of the horizontal coordinate and the average value of the vertical coordinate respectively to obtain multiple distances

[0180]

[0181] The D(i) values ​​of each matching fixed point pair are sorted in ascending order. The D(i) values ​​of the incorrectly matched points are generally larger and are ranked at the back. The incorrectly matched point pairs can be removed by selecting the matching fixed point pairs ranked in front according to a certain percentage (set percentage value, such as 70%).

[0182] In the present disclosure, before determining the sensitive areas in the multiple second images based on the filtered matching fixed point pairs or the coordinate transformation of the matching fixed point pairs, the filtered matching fixed point pairs or the matching fixed point pairs are subjected to coordinate transformation, and the coordinate transformation method includes: respectively using linear transformation equations of the horizontal coordinates and linear transformation equations of the vertical coordinates according to the filtered matching fixed point pairs or the matching fixed point pairs; wherein the coordinate transformation includes: the linear transformation equation of the horizontal coordinates and the linear transformation equation of the vertical coordinates.

[0183] In the present disclosure, the method of respectively determining the linear transformation equations of the horizontal coordinates and the linear transformation equations of the vertical coordinates based on the screened matching fixed point pairs or the matching fixed point pairs includes: determining a first slope and a first intercept based on the horizontal coordinates of the screened matching fixed point pairs or the matching fixed point pairs; constructing the linear transformation equation of the horizontal coordinate based on the first slope and the first intercept; determining a second slope and a second intercept based on the vertical coordinates of the screened matching fixed point pairs or the matching fixed point pairs; and constructing the linear transformation equation of the vertical coordinate based on the second slope and the second intercept.

[0184] Assume that after removing the wrong matching point pairs, there are s' pairs of fixed points. For these s' pairs of fixed points, the least squares method can be used to calculate the coefficients of the linear equation. The specific calculation process is as follows.

[0185] First, calculate the mean abscissa APx of the s' fixed points in the first image P and the mean abscissa AQx of the s' fixed points in the second image Q:

[0186]

[0187] According to the least squares method, the first slope k of the abscissa linear transformation equation x and the first intercept b x They are:

[0188]

[0189] b x =A Qx -k x *A Px .

[0190] Similarly, the linear transformation coefficient of the ordinate is calculated. First, the ordinate mean A of the s' fixed points in the first image P is calculated. Py and the mean value of the ordinates of the s' fixed points in the second image Q Qy :

[0191]

[0192]

[0193] According to the least squares method, the second slope k of the ordinate linear transformation equation y and the second intercept b y They are:

[0194]

[0195] b y =A Qy -k y *A Py .

[0196] In order to obtain the coefficient k x , b x , k y , b y After that, for any point (x, y) in the first image P, the coordinates (x', y') of the corresponding point in the second image Q can be calculated by the following formula. The coordinate transformation formula is:

[0197] x'=k x *x+b x .

[0198] y'=k y *y+b y .

[0199] Step S103: Determine the sensitive areas in the plurality of second images according to the coordinate transformation of the matching fixed point pairs, so as to relocate the sensitive areas in the video stream in real time.

[0200] After calculating the linear transformation coefficients of the coordinates of the first image P and the second image Q, the first image P is transformed by n P The polygon composed of points can be calculated to obtain the n of the second image Q P The coordinates of the points. But in the second image Q, these n Q The coordinates of a point may exceed the range of the image. In this case, the polygon cannot be used and displayed directly. For this problem, the more common method is to create a polygon mask image, and then use the mask image and the image rectangular area to perform a logical AND operation. This method has a large amount of calculation and is not suitable for real-time video processing. In the public, this problem can be solved well by correcting the coordinates of the polygon. Not only can the polygon area that meets the requirements be obtained, but the time complexity of the algorithm is also very low, which meets the requirements of real-time processing.

[0201] In the present disclosure, the method for determining the sensitive areas in the multiple second images based on the coordinate transformation of the matching fixed point pairs includes: determining the matching fixed points corresponding to the multiple second images respectively according to the coordinate transformation of the matching fixed point pairs; determining the multiple image boundaries corresponding to the multiple second images respectively; and determining the sensitive areas in the multiple second images based on the matching fixed points corresponding to the multiple second images and the multiple image boundaries respectively.

[0202] In the present disclosure, the method for determining the sensitive areas in the multiple second images based on the matching fixed points corresponding to the multiple second images and the multiple image boundaries, respectively, includes: determining whether all the matching fixed points corresponding to the multiple second images are within the corresponding image boundary; if so, determining the area where the matching fixed points are located as the sensitive area in the corresponding second image; otherwise, performing coordinate correction on the matching fixed points outside the image boundary to obtain the matching fixed points after coordinate correction; and determining the sensitive area in the corresponding second image based on the matching fixed points after coordinate correction and the matching fixed points within the image boundary.

[0203] In the present disclosure, the method for determining whether all the matching fixed points corresponding to the multiple second images are within the corresponding image boundary includes: connecting the matching fixed points in the multiple second images respectively to form a polygon; if the polygon has an intersection with the image boundary, determining that the matching fixed points corresponding to the multiple second images are outside the corresponding image boundary; otherwise, determining that all the matching fixed points corresponding to the multiple second images are within the corresponding image boundary.

[0204] In the present disclosure, the method for performing coordinate correction on the matching fixed points outside the image boundary includes: determining the position of the matching fixed points outside the image boundary; updating the horizontal coordinate and the vertical coordinate of the matching fixed points outside the image boundary based on the position and the image boundary, completing the coordinate correction of the matching fixed points outside the image boundary, and obtaining the matching fixed points after coordinate correction.

[0205] In the specific embodiments of the present disclosure and other possible embodiments,

[0206] The method of updating the horizontal coordinates and coordinates of the matching fixed points outside the image boundary based on the position and the image boundary, completing the coordinate correction of the matching fixed points outside the image boundary, and obtaining the matching fixed points after the coordinate correction includes: obtaining a set direction; according to the set direction, the matching fixed points move with the edge of the polygon formed by all the matching fixed points; when the edge of the polygon crosses the left boundary or the right boundary of the polygon, the coordinates of the matching fixed points outside the left boundary or the right boundary are updated to the coordinates of the intersection of the edge and the left boundary or the right boundary, and the updated polygon is obtained; continuing to move the matching fixed points in the updated polygon with the edge of the updated polygon according to the set direction; when the edge of the updated polygon crosses the upper boundary or the lower boundary of the polygon, the coordinates of the matching fixed points outside the upper boundary or the lower boundary are updated to the coordinates of the intersection of the edge and the upper boundary or the lower boundary, and the matching fixed points after the coordinate correction are obtained. Wherein, the set direction can be clockwise or counterclockwise.

[0207] Figure 4 FIG. 2 shows a schematic diagram of coordinate correction according to an embodiment of the present disclosure. Figure 4 As shown, the point at the lower left corner of the image boundary is defined as the origin (0, 0), and based on the origin, the coordinates of the other three points on the image boundary are determined respectively (W Q -1,0), (0,H Q -1) and (W Q -1, W Q -1); wherein said W Q -1 is the length value of the second image, and H Q is the height of the second image.

[0208] exist Figure 4 In the example (a), part of the pentagon is beyond the image range. After coordinate correction, we get Figure 4 (b) The quadrilateral shown by the black solid line. Figure 4 In the example (c), there are two discontinuous regions outside the image area. After coordinate correction, the original hexagon is obtained Figure 4(d) The octagon shown by the black solid line. Figure 4 In the example (e), the polygon exceeds the image range in both horizontal and vertical directions. After coordinate correction, we get Figure 4 (f) The polygon does not exceed the image range in both horizontal and vertical directions.

[0209] The algorithm for polygon coordinate correction is as follows:

[0210] (1) Read the coordinates of each vertex of the polygon in a clockwise direction, store the coordinates of these vertices in a circular queue, and record whether the horizontal and vertical coordinates of each point exceed the range of the image.

[0211] (2) If all points are located inside the image, no transformation is required and the algorithm ends.

[0212] (3) Assume that the coordinates are (x u ,y u ) and (x v ,y v ) are the two vertices of the polygon. If (x u ,y u ) and (x v ,y v ) and x u Does not exceed the horizontal coordinate range of the image, and x v If the point (x u ,y u ) and point (x v ,y v ) intersects the left or right border of the image, the point pair <(x u ,y u ),(x v ,y v )>Add to set S x .

[0213] (4) For the set S x For each pair of points in the image, calculate the coordinates of the intersection of the line connecting the point pairs and the left and right boundaries of the image:

[0214] If x v <0

[0215] x v '=0

[0216] y v '=y u +x u *(y v -y u ) / (x u -xv )

[0217] Then the intersection of this point pair and the left edge of the image is (x v ',y v '), and (x v ',y v ')Add to set S xN .

[0218] Assume that the width of the second image Q is W Q .

[0219] If x v >W Q -1

[0220] x v '=W Q -1

[0221] y v '=y u +(W Q -1-x u )*(y v -y u ) / (x v -x u )

[0222] Then the intersection of this point pair and the right edge of the image is (x v ',y v '), and (x v ',y v ')Add to set S xN .

[0223] (5) Delete the points whose horizontal coordinates are out of range in the polygon and convert the set S xN The intersection points in the polygon are added to the polygon. At this time, the horizontal coordinate of the polygon must not exceed the range, but the vertical coordinate may exceed the range.

[0224] (6) Clear the circular queue.

[0225] (7) Read the coordinates of each vertex of the polygon obtained in step (5) in a clockwise direction, store the coordinates of these vertices in a circular queue, and record whether the vertical coordinate of each point exceeds the range of the image.

[0226] (8) If all points are located inside the image, no transformation is required and the algorithm ends.

[0227] (9) If a point (x u ,y u ) and point (x v ,y v ) are connected, and yu Does not exceed the image ordinate range, and y v If the vertical coordinate of the image is beyond the range, the point (x u ,y u ) and point (x v ,y v ) intersects with the upper or lower boundary of the image, the point pair <(x u ,y u ),(x v ,y v )>Add to set S y .

[0228] (10) For the set S x For each pair of points in the image, calculate the coordinates of the intersection of the line connecting the point pairs and the upper and lower boundaries of the image:

[0229] If y v <0

[0230] y v '=0

[0231] x v '=x u +y u *(x v -x u ) / (y u -y v )

[0232] Then the intersection of this point pair and the lower boundary of the image is (x v ',y v '), (x v ',y v ')Add to set S yN .

[0233] Assume that the height of image Q is H Q .

[0234] If y v >H Q -1

[0235] y v '=H Q -1

[0236] x v '=x u +(H Q -1-y u )*(x v -x u ) / (y v -y u )

[0237] Then the intersection of this point pair and the upper boundary of the image is (x v ',y v '), and (x v ',y v ')Add to set S yN .

[0238] (11) Delete the points whose ordinates are out of range in the polygon and set S yN The intersection points in are added to the polygon.

[0239] The execution subject of the video area relocation method may be a video area relocation device, for example, the video area relocation method may be executed by a terminal device or a server or other processing device, wherein the terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the video area relocation method may be implemented by a processor calling a computer-readable instruction stored in a memory.

[0240] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0241] The present disclosure also proposes a video area repositioning device, which includes: an acquisition unit, used to acquire a sensitive area defined by a first image in a real-time video stream and a plurality of second images in the real-time video stream; a matching unit, used to determine matching fixed point pairs of the first image and the plurality of second images based on the sensitive area; and a repositioning unit, used to determine the sensitive area in the plurality of second images based on the coordinate transformation of the matching fixed point pairs, so as to reposition the sensitive area in the video stream in real time.

[0242] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0243] The embodiment of the present disclosure also provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above method is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0244] The embodiment of the present disclosure also provides an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to perform the above method. The electronic device can be provided as a terminal, a server or other forms of equipment.

[0245] The present disclosure solves the problem that the detection algorithm is too slow due to camera movement, scene changes, and too many scenes to be detected. Therefore, the present disclosure first marks the dangerous area in advance and then relocates the dangerous area based on the fixed point matching algorithm to achieve the purpose and application of only detecting the dangerous area. Specifically, the present disclosure first detects the fixed point: it is completed by constructing a scale space and detecting local extreme points; then, a fixed point feature vector descriptor is generated: the direction of the feature point is first determined and then the feature descriptor is generated; then fixed point matching is performed; finally, coordinate relocation and area relocation are performed. In order to meet the real-time requirements, the above algorithm is implemented using a GPU. The present disclosure not only has excellent real-time performance, but also can cope with problems such as regional scaling, geometric deformation, and night video area relocation. Compared with deep learning that consumes computing resources and has poor real-time performance, the present disclosure can not only directly relocate the video sensitive area, but also re-identify when the video sensitive area disappears for a long time.

[0246] Figure 6 8 is a block diagram of an electronic device 800 according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0247] Reference Figure 6 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0248] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0249] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0250] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

[0251] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0252] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0253] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0254] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0255] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0256] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0257] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above method.

[0258] Figure 6 1 is a block diagram of an electronic device 1900 according to an exemplary embodiment. For example, the electronic device 1900 may be provided as a server. Figure 6, the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0259] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0260] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0261] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0262] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0263] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0264] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0265] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0266] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0267] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0268] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0269] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A video area relocation method, It is characterized in that include: Acquire a sensitive area defined by a first image in a real-time video stream and a plurality of second images in the real-time video stream; Determine the matching fixed point pairs of the first image and the multiple second images according to the sensitive area; wherein, according to the sensitive area, perform a registration operation on the floating image corresponding to the multiple second images and the fixed image corresponding to the first image to obtain the matching fixed point pairs of the first image and the multiple second images, including: according to the sensitive area, perform a registration operation on the floating image and the fixed image to obtain a first number of first fixed points and a second number of second fixed points corresponding to the first image and the multiple second images; construct multiple two-dimensional similarity matrices with the first fixed points and the corresponding second fixed points respectively; obtain a row setting threshold; determine each two-dimensional similarity respectively. The maximum value and the second maximum value of each row in the matrix; calculate a first ratio of the maximum value and the second maximum value; if the first ratio is greater than the row setting threshold, determine the maximum value as the row effective maximum value, and record the column number of the row effective maximum value; obtain the column setting threshold; respectively determine the maximum value and the second maximum value of each column in each two-dimensional similarity matrix; calculate a second ratio of the maximum value and the second maximum value; if the second ratio is greater than the column setting threshold, determine the maximum value as the column effective maximum value, and record the row number of the column effective maximum value; based on the row effective maximum value and its column number and the column effective maximum value and its row number, determine the matching fixed point pairs of the first image and the multiple second images; According to the coordinate transformation of the matching fixed point pair, the sensitive areas in the plurality of second images are determined to relocate the sensitive areas in the video stream in real time.

2. The relocation method according to claim 1, It is characterized in that The method for determining matching fixed point pairs of the first image and the plurality of second images according to the sensitive area comprises: determining the first image as a fixed image, and determining the plurality of second images as floating images; According to the sensitive area, a registration operation is performed on the floating image and the fixed image to obtain matching fixed point pairs of the first image and the plurality of second images.

3. The relocation method according to any one of claims 1 to 2, It is characterized in that Before determining the sensitive areas in the plurality of second images according to the coordinate transformation of the matching fixed point pairs, screening the matching fixed point pairs; The sensitive areas in the plurality of second images are determined according to the coordinate transformation of the screened matching fixed point pairs.

4. The relocation method according to claim 3, It is characterized in that Before determining the sensitive areas in the plurality of second images according to the filtered matching fixed point pairs or the coordinate transformation of the matching fixed point pairs, coordinate transformation is performed on the filtered matching fixed point pairs or the matching fixed point pairs, wherein the coordinate transformation method comprises: constructing a linear transformation equation of the horizontal coordinate and a linear transformation equation of the vertical coordinate according to the screened matching fixed point pairs or the matching fixed point pairs respectively; The coordinate transformation includes: a linear transformation equation of the horizontal coordinate and a linear transformation equation of the vertical coordinate.

5. The relocation method according to claim 4, It is characterized in that The method of constructing the linear transformation equation of the horizontal coordinate and the linear transformation equation of the vertical coordinate according to the screened matching fixed point pairs or the matching fixed point pairs respectively includes: Determining a first slope and a first intercept according to the screened matching fixed point pair or the abscissa of the matching fixed point pair; constructing a linear transformation equation of the abscissa according to the first slope and the first intercept; Determining a second slope and a second intercept according to the screened matching fixed point pair or the ordinate of the matching fixed point pair; A linear transformation equation of the ordinate is constructed according to the second slope and the second intercept.

6. The relocation method according to claim 3, It is characterized in that Before determining the sensitive areas in the plurality of second images according to the coordinate transformation of the matching fixed point pairs, the matching fixed point pairs are screened, and the method for screening the matching fixed point pairs includes: Get the set percentage value; Respectively calculating the average value of the horizontal coordinate change amount and the average value of the vertical coordinate change amount of the matching fixed point pair; Calculate the distances between the matching fixed point pair and the average value point composed of the average value of the horizontal coordinate change amount and the average value of the vertical coordinate change amount respectively to obtain multiple distances; Sorting the multiple distances in ascending order to obtain multiple distance sequences; The plurality of distance sequences are intercepted based on the set percentage value to obtain matching fixed point pairs corresponding to the intercepted plurality of distance sequences.

7. The relocation method according to any one of claims 4 or 5, It is characterized in that Before determining the sensitive areas in the plurality of second images according to the coordinate transformation of the matching fixed point pairs, the matching fixed point pairs are screened, and the method for screening the matching fixed point pairs includes: Get the set percentage value; Respectively calculating the average value of the horizontal coordinate change amount and the average value of the vertical coordinate change amount of the matching fixed point pair; Calculate the distances between the matching fixed point pair and the average value point composed of the average value of the horizontal coordinate change amount and the average value of the vertical coordinate change amount respectively to obtain multiple distances; Sorting the multiple distances in ascending order to obtain multiple distance sequences; The plurality of distance sequences are intercepted based on the set percentage value to obtain matching fixed point pairs corresponding to the intercepted plurality of distance sequences.

8. The relocation method according to any one of claims 1, 2, 4-6, It is characterized in that If the first ratio is less than or equal to the row setting threshold, the row in the two-dimensional similarity matrix does not have a row effective maximum value, and the row is marked.

9. The relocation method according to claim 3, It is characterized in that If the first ratio is less than or equal to the row setting threshold, the row in the two-dimensional similarity matrix does not have a row effective maximum value, and the row is marked.

10. The relocation method according to claim 7, It is characterized in that If the first ratio is less than or equal to the row setting threshold, the row in the two-dimensional similarity matrix does not have a row effective maximum value, and the row is marked.

11. The relocation method according to any one of claims 1, 2, 4-6, 9, and 10, It is characterized in that If the second ratio is less than or equal to the column setting threshold, the column in the two-dimensional similarity matrix does not have a column effective maximum value, and the column is marked.

12. The relocation method according to claim 3, It is characterized in that If the second ratio is less than or equal to the column setting threshold, the column in the two-dimensional similarity matrix does not have a column effective maximum value, and the column is marked.

13. The relocation method according to claim 7, It is characterized in that If the second ratio is less than or equal to the column setting threshold, the column in the two-dimensional similarity matrix does not have a column effective maximum value, and the column is marked.

14. The relocation method according to claim 8, It is characterized in that If the second ratio is less than or equal to the column setting threshold, the column in the two-dimensional similarity matrix does not have a column effective maximum value, and the column is marked.

15. The relocation method according to any one of claims 1, 2, 4-6, 9, 10, 12-14, It is characterized in that The method for determining matching fixed point pairs of the first image and the plurality of second images based on the row effective maximum value and its column number and the column effective maximum value and its row number comprises: Get row and column labels; Based on the row mark, the column mark, the row effective maximum value and its column number, and the column effective maximum value and its row number, matching fixed point pairs of the first image and the multiple second images are determined.

16. The relocation method according to claim 3, It is characterized in that The method for determining matching fixed point pairs of the first image and the plurality of second images based on the row effective maximum value and its column number and the column effective maximum value and its row number comprises: Get row and column labels; Based on the row mark, the column mark, the row effective maximum value and its column number, and the column effective maximum value and its row number, matching fixed point pairs of the first image and the multiple second images are determined.

17. The relocation method according to claim 7, It is characterized in that The method for determining matching fixed point pairs of the first image and the plurality of second images based on the row effective maximum value and its column number and the column effective maximum value and its row number comprises: Get row and column labels; Based on the row mark, the column mark, the row effective maximum value and its column number, and the column effective maximum value and its row number, matching fixed point pairs of the first image and the multiple second images are determined.

18. The relocation method according to claim 8, It is characterized in that The method for determining matching fixed point pairs of the first image and the plurality of second images based on the row effective maximum value and its column number and the column effective maximum value and its row number comprises: Get row and column labels; Based on the row mark, the column mark, the row effective maximum value and its column number, and the column effective maximum value and its row number, matching fixed point pairs of the first image and the multiple second images are determined.

19. The relocation method according to claim 11, It is characterized in that The method for determining matching fixed point pairs of the first image and the plurality of second images based on the row effective maximum value and its column number and the column effective maximum value and its row number comprises: Get row and column labels; Based on the row mark, the column mark, the row effective maximum value and its column number, and the column effective maximum value and its row number, matching fixed point pairs of the first image and the multiple second images are determined.

20. The relocation method according to any one of claims 1, 2, 4-6, 9, 10, 12-14, 16-19, It is characterized in that The method for determining the sensitive areas in the plurality of second images according to the coordinate transformation of the matching fixed point pair comprises: Determining the matching fixed points corresponding to the plurality of second images respectively according to the coordinate transformation of the matching fixed point pair; respectively determining a plurality of image boundaries corresponding to the plurality of second images; Sensitive areas in the plurality of second images are determined based on the matching fixed points corresponding to the plurality of second images and the plurality of image boundaries, respectively.

21. The relocation method according to claim 3, It is characterized in that The method for determining the sensitive areas in the plurality of second images according to the coordinate transformation of the matching fixed point pair comprises: Determining the matching fixed points corresponding to the plurality of second images respectively according to the coordinate transformation of the matching fixed point pair; respectively determining a plurality of image boundaries corresponding to the plurality of second images; Sensitive areas in the plurality of second images are determined based on the matching fixed points corresponding to the plurality of second images and the plurality of image boundaries, respectively.

22. The relocation method according to claim 7, It is characterized in that The method for determining the sensitive areas in the plurality of second images according to the coordinate transformation of the matching fixed point pair comprises: Determining the matching fixed points corresponding to the plurality of second images respectively according to the coordinate transformation of the matching fixed point pair; respectively determining a plurality of image boundaries corresponding to the plurality of second images; Sensitive areas in the plurality of second images are determined based on the matching fixed points corresponding to the plurality of second images and the plurality of image boundaries, respectively.

23. The relocation method according to claim 8, It is characterized in that The method for determining the sensitive areas in the plurality of second images according to the coordinate transformation of the matching fixed point pair comprises: Determining the matching fixed points corresponding to the plurality of second images respectively according to the coordinate transformation of the matching fixed point pair; respectively determining a plurality of image boundaries corresponding to the plurality of second images; Sensitive areas in the plurality of second images are determined based on the matching fixed points corresponding to the plurality of second images and the plurality of image boundaries, respectively.

24. The relocation method according to claim 11, It is characterized in that The method for determining the sensitive areas in the plurality of second images according to the coordinate transformation of the matching fixed point pair comprises: Determining the matching fixed points corresponding to the plurality of second images respectively according to the coordinate transformation of the matching fixed point pair; respectively determining a plurality of image boundaries corresponding to the plurality of second images; Sensitive areas in the plurality of second images are determined based on the matching fixed points corresponding to the plurality of second images and the plurality of image boundaries, respectively.

25. The relocation method according to claim 15, It is characterized in that The method for determining the sensitive areas in the plurality of second images according to the coordinate transformation of the matching fixed point pair comprises: Determining the matching fixed points corresponding to the plurality of second images respectively according to the coordinate transformation of the matching fixed point pair; respectively determining a plurality of image boundaries corresponding to the plurality of second images; Sensitive areas in the plurality of second images are determined based on the matching fixed points corresponding to the plurality of second images and the plurality of image boundaries, respectively.

26. The relocation method according to claim 20, It is characterized in that The method for determining sensitive areas in the plurality of second images based on the matching fixed points corresponding to the plurality of second images and the plurality of image boundaries, respectively, comprises: Determining whether all matching fixed points corresponding to the plurality of second images are within the corresponding image boundaries; If yes, then determining the area where the matching fixed point is located as the sensitive area in the corresponding second image; Otherwise, coordinate correction is performed on the matching fixed points outside the image boundary to obtain the matching fixed points after coordinate correction; based on the matching fixed points after coordinate correction and the matching fixed points within the image boundary, the corresponding sensitive area in the second image is determined.

27. The relocation method according to any one of claims 21 to 25, It is characterized in that The method for determining sensitive areas in the plurality of second images based on the matching fixed points corresponding to the plurality of second images and the plurality of image boundaries, respectively, comprises: Determining whether all matching fixed points corresponding to the plurality of second images are within the corresponding image boundaries; If yes, then determining the area where the matching fixed point is located as the sensitive area in the corresponding second image; Otherwise, coordinate correction is performed on the matching fixed points outside the image boundary to obtain the matching fixed points after coordinate correction; based on the matching fixed points after coordinate correction and the matching fixed points within the image boundary, the corresponding sensitive area in the second image is determined.

28. The relocation method according to claim 26, It is characterized in that The method for determining whether all matching fixed points corresponding to the plurality of second images are within the corresponding image boundaries comprises: Connecting the matched fixed points in the plurality of second images respectively to form a polygon; If the polygon has an intersection with the image boundary, determining that the matching fixed points corresponding to the plurality of second images are outside the corresponding image boundary; Otherwise, it is determined that all matching fixed points corresponding to the plurality of second images are within the corresponding image boundaries.

29. The relocation method according to claim 27, It is characterized in that The method for determining whether all matching fixed points corresponding to the plurality of second images are within the corresponding image boundaries comprises: Connecting the matched fixed points in the plurality of second images respectively to form a polygon; If the polygon has an intersection with the image boundary, determining that the matching fixed points corresponding to the plurality of second images are outside the corresponding image boundary; Otherwise, it is determined that all matching fixed points corresponding to the plurality of second images are within the corresponding image boundaries.

30. The relocation method according to any one of claims 26, 28 or 29, It is characterized in that The method for performing coordinate correction on a matching fixed point outside the image boundary comprises: Determining a position of a matching fixed point outside a boundary of the image; The horizontal coordinate and the vertical coordinate of the matching fixed point outside the image boundary are updated based on the position and the image boundary, and the coordinate correction of the matching fixed point outside the image boundary is completed to obtain the matching fixed point after the coordinate correction.

31. The relocation method according to claim 27, It is characterized in that The method for performing coordinate correction on a matching fixed point outside the image boundary comprises: Determining a position of a matching fixed point outside a boundary of the image; The horizontal coordinate and the vertical coordinate of the matching fixed point outside the image boundary are updated based on the position and the image boundary, and the coordinate correction of the matching fixed point outside the image boundary is completed to obtain the matching fixed point after the coordinate correction.

32. A device for relocating a video area, It is characterized in that include: An acquisition unit, configured to acquire a sensitive area defined by a first image in a real-time video stream and a plurality of second images in the real-time video stream; A matching unit is used to determine matching fixed point pairs of the first image and the multiple second images according to the sensitive area; wherein, according to the sensitive area, a registration operation is performed on the floating image corresponding to the multiple second images and the fixed image corresponding to the first image to obtain matching fixed point pairs of the first image and the multiple second images, including: according to the sensitive area, a registration operation is performed on the floating image and the fixed image to obtain a first number of first fixed points and a second number of second fixed points corresponding to the first image and the multiple second images; a plurality of two-dimensional similarity matrices are constructed respectively with the first fixed points and the corresponding second fixed points; a row setting threshold is obtained; each two-dimensional similarity matrix is ​​determined respectively. The maximum value and the second maximum value of each row in the similarity matrix; calculating a first ratio of the maximum value and the second maximum value; if the first ratio is greater than the row setting threshold, determining the maximum value as the row effective maximum value, and recording the column number of the row effective maximum value; obtaining the column setting threshold; respectively determining the maximum value and the second maximum value of each column in each two-dimensional similarity matrix; calculating a second ratio of the maximum value and the second maximum value; if the second ratio is greater than the column setting threshold, determining the maximum value as the column effective maximum value, and recording the row number of the column effective maximum value; based on the row effective maximum value and its column number and the column effective maximum value and its row number, determining the matching fixed point pairs of the first image and the multiple second images; A repositioning unit is used to determine the sensitive areas in the multiple second images according to the coordinate transformation of the matching fixed point pair, so as to reposition the sensitive areas in the video stream in real time.

33. An electronic device, It is characterized in that include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method for relocating the video area as described in any one of claims 1 to 31.

34. A computer readable storage medium having computer program instructions stored thereon, It is characterized in that When the computer program instructions are executed by a processor, the method for repositioning the video area as claimed in any one of claims 1 to 31 is implemented.

Images