Video key frame extraction method, electronic device and storage medium

Abstract

The present application discloses a method for extracting key frames of a video, an electronic device and a storage medium. The method for extracting key frames of a video comprises: using visual simultaneous localization and map construction (VSLAM) technology to determine a first pose corresponding to each frame of an image in a first video; the first pose at least includes a first vector; the first vector represents the position of a camera when the corresponding image is taken; based on the distance between each two first vectors, a plurality of second poses are determined; the plurality of second poses are determined in the determined first pose corresponding to each frame of the image; and the image corresponding to each second pose is extracted from the first video to obtain a key frame corresponding to the first video.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a video key frame extraction method, electronic equipment and storage medium. Background Art

[0002] In computer vision, 3D reconstruction refers to the process of reconstructing 3D information based on single-view or multi-view images. The multi-view 3D reconstruction method first calculates the relationship between the camera's image coordinate system and the world coordinate system, and then reconstructs the 3D information using the information in multiple 2D images.

[0003] In the related art, a set number of key frames are extracted from a video at fixed time intervals, and 3D reconstruction is performed based on the extracted key frames. However, the key frames extracted at fixed time intervals may lose some image information that is important for 3D reconstruction, or there may be image redundancy, and the quality of the extracted key frames is low, resulting in low image quality or low reconstruction efficiency of the 3D reconstruction. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a video key frame extraction method, an electronic device and a storage medium to solve the technical problem of low quality of video key frames extracted in the related art.

[0005] To achieve the above purpose, the technical solution of this application is implemented as follows:

[0006] The present application provides a method for extracting video key frames, including:

[0007] Using VSLAM technology, determining a first pose corresponding to each frame of an image in a first video; the first pose at least includes a first vector; the first vector represents a camera position when shooting a corresponding image;

[0008] Based on the distance between each two first vectors, a plurality of second postures are determined; the plurality of second postures are determined in the first posture corresponding to each frame of the image;

[0009] An image corresponding to each second posture is extracted from the first video to obtain a key frame corresponding to the first video.

[0010] In the above solution, the determining of multiple second postures based on the distance between each two first vectors includes:

[0011] Determine a second vector; the second vector represents the average vector of all the first vectors;

[0012] Based on the distance between the third vector and each first vector, a plurality of second postures are determined; wherein,

[0013] The third vector represents the vector with the largest distance from the second vector among the first vectors corresponding to each frame of image determined.

[0014] In the above solution, the determining of multiple second postures includes:

[0015] Add the third vector in the first set to the second set; the initial second set is an empty set; the first set is composed of the determined first vector corresponding to each frame of the image;

[0016] Adding a fourth vector to the second set; the fourth vector represents the first vector in the first set with the largest distance from the third vector;

[0017] Based on the first distance and the second distance corresponding to each first vector in the first set, a fifth vector is added to the second set; wherein the first distance represents the distance between the first vector and the vector added to the second set for the i-th time; the second distance represents the distance between the first vector and the vector added to the second set for the i+1th time; the fifth vector represents the first vector corresponding to the maximum value of the second distances in the first set that are smaller than the corresponding first distance; i is an integer greater than or equal to 1;

[0018] If the set condition is not met currently, i is assigned a value of i+1, and the method of adding the fifth vector to the second set based on the first distance and the second distance corresponding to each first vector in the first set is performed;

[0019] When the set condition is currently satisfied, the first postures corresponding to all vectors in the second set are determined as the second posture.

[0020] In the above scheme, the setting condition includes one of the following:

[0021] A first set condition, indicating that the total number of vectors in the second set reaches a first set threshold; the first set threshold is greater than or equal to 3;

[0022] The second setting condition indicates that the maximum value of the second distances that are smaller than the corresponding first distances is smaller than a second setting threshold.

[0023] In the above solution, the first posture also includes the posture of the camera when shooting the corresponding image; when determining the distance between the two first vectors, the method includes:

[0024] determining a vector difference between two first vectors;

[0025] Calculate a first weight based on the two camera postures corresponding to the two first vectors;

[0026] A distance between two first vectors is determined based on a product of the vector difference and the first weight.

[0027] In the above solution, the first weight is calculated based on the two camera postures corresponding to the two vectors, including:

[0028] Converting a first of the two camera poses into a first rotation vector, and converting a second of the two camera poses into a second rotation vector;

[0029] A first weight is calculated based on a first setting coefficient, the first rotation vector, and the second rotation vector, and based on a norm of the first rotation vector and a norm of the second rotation vector.

[0030] In the above solution, determining the vector difference between the two first vectors includes:

[0031] The second norm between the two first vectors is calculated to obtain the vector difference.

[0032] In the above solution, when the camera is a monocular camera, before determining a plurality of second postures based on the distance between each two first vectors, the method further includes:

[0033] Using VSLAM technology, determine the point cloud map corresponding to each frame of the first video;

[0034] Determine a first scale corresponding to each frame of image based on a distance between every two positions in a point cloud map corresponding to each frame of image and a corresponding calibration distance;

[0035] Based on the first ratio corresponding to each frame of image, a first vector corresponding to the corresponding image is enlarged.

[0036] The present application also provides an electronic device, including:

[0037] An estimation unit, configured to determine a first pose corresponding to each frame of an image in a first video by using a visual simultaneous localization and mapping (VSLAM) technology; the first pose at least includes a first vector; the first vector represents a position of a camera when the corresponding image is captured;

[0038] A determination unit, configured to determine a plurality of second postures based on the distance between each two first vectors; the plurality of second postures are determined from the first posture corresponding to each frame of the image;

[0039] An extraction unit is used to extract an image corresponding to each second posture from the first video to obtain a key frame corresponding to the first video.

[0040] The present application also provides an electronic device, including: a processor and a memory for storing a computer program that can be run on the processor.

[0041] Wherein, the processor is used to execute the steps of the above-mentioned video key frame extraction method when running the computer program.

[0042] An embodiment of the present application further provides a storage medium on which a computer program is stored, wherein the computer program implements the steps of the above-mentioned video key frame extraction method when executed by a processor.

[0043] In an embodiment of the present application, based on the distance between the first vectors included in each two first poses, multiple second poses are determined in the first pose corresponding to each frame of the image; the image corresponding to each second pose is extracted from the first video to obtain the key frame corresponding to the first video. The first pose can be uniformly sampled through the distance between the first vector corresponding to each frame of the image and each other first vector to obtain multiple second poses; the multiple second poses can extract as many key frames as possible where the action of the photographed object changes, and the key frames where the scene in the image changes, which can improve the quality of the extracted key frames, and further improve the quality of the image obtained by three-dimensional reconstruction of the extracted key frames; uniform sampling of the first pose can also reduce the probability of extracting key frames with repeated images, thereby reducing the time consumed by processing key frames with repeated images and improving the efficiency of three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of the implementation flow of the video key frame extraction method provided in the embodiment of the present application;

[0045] Figure 2 A schematic diagram of the camera position and point cloud provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of a calibrated distance between two positions provided in an embodiment of the present application;

[0047] Figure 4 A schematic diagram of a timing flow of determining multiple second postures in a video key frame extraction method provided in an embodiment of the present application;

[0048] Figure 5 A schematic diagram of a timing flow of determining a plurality of second postures in a video key frame extraction method provided in another embodiment of the present application;

[0049] Figure 6 A schematic diagram of a second posture determined according to an embodiment of the present application;

[0050] Figure 7A schematic diagram of a second posture determined according to another embodiment of the present application;

[0051] Figure 8 A schematic diagram of the implementation flow of the video key frame extraction method provided in the application embodiment of the present application;

[0052] Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention;

[0053] Fig.10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] Figure 1 The following is a schematic diagram of the implementation process of the video key frame extraction method provided in the embodiment of the present application, wherein the execution subject of the process is an electronic device such as a computer or a server. The computer includes a desktop computer, a laptop computer, and a tablet computer. Figure 1 As shown, the video key frame extraction method includes:

[0056] Step 101: Determine the first position corresponding to each frame image in the first video by using the visual simultaneous localization and mapping (VSLAM) technology; the first position includes at least a first vector; the first vector represents the position of the camera when the corresponding image is captured.

[0057] Wherein, when the electronic device is equipped with a camera, the first video can be shot by the electronic device. When the electronic device is not equipped with a camera, the first video can be shot by a camera connected to the electronic device; or it can be shot by other devices equipped with a camera. When other devices equipped with a camera shoot the first video, the first video can be sent to the electronic device, or the first video can be uploaded to a cloud database so that the electronic device can download the first video from the cloud database.

[0058] It should be noted that other devices refer to devices other than the electronic device that executes the video key frame extraction method. The camera can be a monocular camera or a binocular camera.

[0059] In actual applications, the photographer moves in the shooting scene and uses a terminal with a camera such as a camera or a mobile phone to shoot the object in the shooting scene to obtain the first video. The object includes a person and / or an object.

[0060] The electronic device obtains a first video, which is a video of a key frame to be extracted; and uses Visual Simultaneous Localization And Mapping (VSLAM) technology to determine a first pose corresponding to each frame of the first video. The first pose includes at least a first vector, which represents a camera position when the corresponding image is captured.

[0061] In some embodiments, the first pose also includes a first matrix, and the first matrix represents the pose of the camera when capturing the corresponding image.

[0062] In practical applications, the first vector is a 3×1 translation vector, and the first matrix is ​​a 3×3 rotation matrix.

[0063] It should be noted that, since the image captured by the binocular camera has parallax, the camera position represented by the first vector in the first pose determined by the VSLAM technology is consistent with the corresponding actual position; while the image captured by the monocular camera has no parallax, the first vector in the first pose determined by the VSLAM technology and the point cloud map are both reduced, and it is necessary to determine the magnification ratio by the distance between the two positions in the determined point cloud map and the corresponding calibration distance, so as to correct the first vector in the first pose with the help of the determined magnification ratio; therefore, when the first video is captured by the binocular camera, the electronic device directly executes step 102 after executing step 101; when the first video is captured by the monocular camera, the electronic device also executes the following steps during the execution of step 101 and before executing step 102:

[0064] Using VSLAM technology, determine the point cloud map corresponding to each frame of the first video;

[0065] Determine a first scale corresponding to each frame of image based on a distance between every two positions in a point cloud map corresponding to each frame of image and a corresponding calibration distance;

[0066] Based on the first ratio corresponding to each frame of image, a first vector corresponding to the corresponding image is enlarged.

[0067] It should be noted that step 101 and using the VSLAM technology to determine the point cloud map corresponding to each frame image in the first video can be performed simultaneously.

[0068] Here, since the initialization of the monocular camera is completed by the epipolar geometry constraint, the scenario in which the epipolar geometry constraint is applied is that a number of matching points between two frames of images are known. Restoring the movement of the camera between the two frames of images is a 2D-2D problem. The core is to solve an essential matrix E (or a basic matrix with an intrinsic parameter matrix), that is, by performing singular value decomposition on the essential matrix E, the pose of the camera when each frame of image is taken is obtained, where the pose includes the camera posture r and the camera position t. However, the essential matrix E is scale-equivalent, and the equivalence is mainly manifested in the camera position t, that is, the camera position t multiplied by any non-zero positive number can satisfy the epipolar geometry constraint. Therefore, the camera position t obtained by performing singular value decomposition on the essential matrix E differs from the actual camera position by a scale factor. Since the camera pose and camera posture when each frame of image is taken are related to the point cloud in the point cloud map corresponding to the shooting scene, for example, Figure 2 As shown, when the camera position is x1, the relevant data of point cloud y1 and point cloud y2 are detected. Therefore, the point clouds in the point cloud map are also equivalent.

[0069] Based on this, the electronic device uses VSLAM technology to determine the point cloud map corresponding to each frame of the first video; identifies at least two marked positions from the point cloud map corresponding to each frame of the image, determines the distance between any two marked positions in the point cloud map, and obtains the corresponding calibration distance, wherein the calibration distance is as follows: Figure 3 As shown; the quotient of the distance between the two marked positions in the point cloud map corresponding to each frame image and the corresponding calibration distance is calculated to obtain the first ratio corresponding to each frame image, that is, the scale factor; based on the first ratio corresponding to each frame image, the first vector corresponding to the corresponding image is enlarged to obtain the real camera position in the shooting scene, so that multiple second postures can be determined based on the distance between each two enlarged first vectors.

[0070] It should be noted that the calibrated distance refers to the distance measured in the actual shooting scene.

[0071] In practical application, the formula Calculate the first scale. Wherein, scale represents the first scale; D real_word Characterizes the nominal distance between two locations; D map Represents the distance between two locations in a point cloud map.

[0072] In actual application, the It represents the first posture determined by VSLAM technology, r represents the first matrix, corresponding to the camera posture, and t represents the first vector.

[0073] Because r does not have scale equivalence, the first pose determined by VSLAM technology is aligned to the actual shooting scene through the first scale to obtain the real first pose

[0074] Step 102: Based on the distance between each two first vectors, a plurality of second postures are determined; the plurality of second postures are determined in the first posture corresponding to each frame of the image.

[0075] Here, the electronic device calculates the distance between every two first vectors based on the first vector included in the determined first posture; and determines multiple second postures in the determined first posture corresponding to each frame of image based on the first vector corresponding to each frame of image and the distance between each other first vector.

[0076] In practical applications, the distance between the first vectors included in the second postures determined twice adjacently is equal or approximately equal, or the distance between the first vectors included in the second postures determined twice adjacently is the largest; or the distance between the first vectors included in the second postures determined twice adjacently is the largest, and the distance is less than or equal to a set distance threshold.

[0077] It should be noted that the electronic device can randomly select a first posture from the determined first postures as the first second posture.

[0078] In order to improve the image quality of the extracted key frame, the first second pose can be determined based on the average vector of all first vectors, such as Figure 4 As shown, in some embodiments, the determining of multiple second postures based on the distance between each two first vectors includes:

[0079] Step 401: Determine a second vector; the second vector represents the average vector of all first vectors;

[0080] Step 402: Determine a plurality of second postures based on the distance between the third vector and each first vector; wherein the third vector represents the vector having the largest distance from the second vector among the first vectors corresponding to each frame of the image determined.

[0081] Here, the electronic device calculates the average vector of all first vectors based on the first vector corresponding to each frame image, and obtains the second vector; calculates the distance between each first vector and the second vector, and determines the first vector with the largest distance to the second vector as the third vector; and determines multiple second postures based on the distance between the third vector and each first vector.

[0082] Wherein, when a plurality of second postures are determined based on the distance between the third vector and each first vector, the method includes:

[0083] Based on the distance between the third vector and each of the remaining first vectors, determining a fourth vector having the largest distance from the third vector among the remaining first vectors;

[0084] determining a fifth vector based on the distance between the fourth vector and each of the remaining first vectors;

[0085] determining a sixth vector based on the distance between the fifth vector and each of the remaining first vectors;

[0086] According to the method, based on the distance between the currently determined vector and each of the remaining first vectors, the next vector is determined, thereby determining multiple vectors; the first posture to which the third vector and each of the determined vectors belong is determined as the second posture, thereby obtaining multiple second postures. That is, the electronic device determines the first posture to which the vector farthest from the currently determined vector belongs as the next second posture. The distance between each two adjacent determined vectors is the largest. Among them, the remaining first vectors refer to the first vectors other than the determined vectors.

[0087] In some embodiments, when a plurality of second postures are determined based on the distance between the third vector and each of the first vectors, the method includes:

[0088] Based on the first distance between the vector determined for the i-th time and each remaining first vector, and based on the second distance between the vector determined for the i+1-th time and each remaining first vector, the first posture to which the first vector corresponding to the maximum value of the second distances smaller than the first distance belongs is determined as the second posture.

[0089] In order to improve the image quality of the extracted key frames, such as Figure 5 As shown, in some embodiments, determining a plurality of second postures includes:

[0090] Step 501: adding the third vector in the first set to the second set; the initial second set is an empty set; the first set is composed of the determined first vector corresponding to each frame of the image;

[0091] Step 502: Add a fourth vector to the second set; the fourth vector represents the first vector in the first set that has the largest distance to the third vector;

[0092] Step 503: Add a fifth vector to the second set based on the first distance and the second distance corresponding to each first vector in the first set; wherein the first distance represents the distance between the first vector and the vector added to the second set for the i-th time; the second distance represents the distance between the first vector and the vector added to the second set for the (i+1)th time; the fifth vector represents the first vector corresponding to the maximum value of the second distances in the first set that are smaller than the corresponding first distance; i is an integer greater than or equal to 1;

[0093] Step 504: Determine whether the current setting condition is met.

[0094] Step 505: if the set condition is not met currently, assign i to i+1, execute the method based on the first distance and the second distance corresponding to each first vector in the first set, and add the fifth vector to the second set;

[0095] Step 506: When the set condition is currently satisfied, the first posture corresponding to all vectors in the second set is determined as the second posture.

[0096] Here, the electronic device generates a first set and a second set based on the determined first vectors corresponding to each frame of the image; wherein the first set includes all the first vectors, and the generated second set is an empty set.

[0097] When the electronic device determines a third vector in the first set, the third vector in the first set is added to the second set; the distance between the third vector and each first vector in the first set is calculated, and based on the calculated distance, a fourth vector is determined from the first set, and the fourth vector is added to the second set, wherein the fourth vector represents the first vector in the first set having the largest distance to the third vector.

[0098] Based on the first distance between each first vector in the first set and the third vector in the second set, and based on the second distance between each first vector in the first set and the fourth vector in the second set, a second distance smaller than the first distance is determined, and a maximum second distance is determined from the second distances smaller than the first distance, the first vector corresponding to the maximum second distance in the first set is determined as a fifth vector, and the fifth vector is added to the second set.

[0099] It is determined whether the set condition is currently met to obtain a first determination result.

[0100] When the first judgment result indicates that the set condition is not currently met, a second distance smaller than the first distance is determined based on the first distance between each first vector in the first set and the fourth vector in the second set, and based on the second distance between each first vector in the first set and the fifth vector in the second set, and a maximum second distance is determined from the second distances smaller than the first distance, the first vector corresponding to the maximum second distance in the first set is determined as the sixth vector, and the sixth vector is added to the second set.

[0101] When the first judgment result indicates that the set condition is currently satisfied, the first postures corresponding to all vectors in the second set are determined as the second posture. At this time, the first posture to which the third vector belongs, the first posture to which the fourth vector belongs, and the first posture to which the fifth vector belongs are all determined as the second posture.

[0102] When the sixth vector is added to the second set, it is determined whether the set condition is currently satisfied to obtain a second determination result.

[0103] When the second judgment result indicates that the set condition is not currently met, a second distance smaller than the first distance is determined based on the first distance between each first vector in the first set and the fifth vector in the second set, and based on the second distance between each first vector in the first set and the sixth vector in the second set, and a maximum second distance is determined from the second distances smaller than the first distance, the first vector corresponding to the maximum second distance in the first set is determined as the seventh vector, and the seventh vector is added to the second set.

[0104] When the second judgment result indicates that the set condition is currently satisfied, the first postures corresponding to all vectors in the second set are determined as the second posture. At this time, the first posture to which the third vector belongs, the first posture to which the fourth vector belongs, the first posture to which the fifth vector belongs, and the first posture to which the sixth vector belongs are all determined as the second posture.

[0105] When the seventh vector is added to the second set, it is determined whether the set condition is currently satisfied to obtain a third determination result.

[0106] When the third judgment result indicates that the set condition is not currently met, a second distance smaller than the first distance is determined based on the first distance between each first vector in the first set and the sixth vector in the second set, and based on the second distance between each first vector in the first set and the seventh vector in the second set, and a maximum second distance is determined from the second distances smaller than the first distance, the first vector corresponding to the maximum second distance in the first set is determined as the eighth vector, and the eighth vector is added to the second set.

[0107] When the third judgment result indicates that the set conditions are currently met, at this time, the first posture belonging to the third vector, the first posture belonging to the fourth vector, the first posture belonging to the fifth vector, the first posture belonging to the sixth vector and the first posture belonging to the seventh vector are all determined as the second posture.

[0108] Thus, according to the above-mentioned iterative method, a plurality of second postures are determined.

[0109] It should be noted that the vector added to the second set for the first time is the third vector, the vector added to the second set for the second time is the fourth vector, the vector added to the second set for the third time is the fifth vector, the vector added to the second set for the fourth time is the sixth vector, and the vector added to the second set for the fifth time is the seventh vector.

[0110] In practical applications, the electronic device determines multiple second postures according to the following steps:

[0111] 1) Generate a first set and a second set based on the determined first vectors corresponding to each frame of the image; wherein the first set includes all the first vectors, and the generated second set is an empty set.

[0112] 2) Calculate the average vector of all first vectors in the first set to obtain the second vector.

[0113] 3) The first vector in the first set with the largest distance from the second vector is determined as the third vector

[0114] 4) The third vector Add to second collection

[0115] 5) Calculate the first distance between each first vector in the first set and the vector added to the second set for the i-th time to obtain an N-dimensional array L; determine the first vector corresponding to the largest distance in the first set; add the determined vector to the second set S; the value of N is equal to the total number of first vectors in the first set.

[0116] When 5) is executed for the first time, i is equal to 1, and the vector added to the second set for the i-th time is When the first vector corresponding to the largest distance in the first set is determined to be The second set

[0117] 6) Calculate the second distance between each first vector in the first set and the vector added to the second set for the i+1th time. For each first vector, when the second distance corresponding to the first vector is less than the corresponding first distance, update the distance corresponding to the first vector in the N-dimensional array to the corresponding second distance.

[0118] Therefore, the distance stored in the N-dimensional array L is always the minimum distance between the first vector in the first set and the vector in the second set S.

[0119] 7) Determine the maximum second distance from the updated N-dimensional array L, and add the first vector corresponding to the determined maximum second distance to the second set S.

[0120] When the first vector corresponding to the determined maximum second distance is The second set

[0121] 8) If the set condition is not met, i is incremented by 1 and the process returns to 5) to 7).

[0122] 9) When the set conditions are currently met, the first pose to which each vector in the second set S belongs is determined as the second pose.

[0123] In some embodiments, the setting condition includes one of the following:

[0124] A first set condition, indicating that the total number of vectors in the second set reaches a first set threshold; the first set threshold is greater than or equal to 3;

[0125] The second setting condition indicates that the maximum value of the second distances that are smaller than the corresponding first distances is smaller than a second setting threshold.

[0126] Here, the electronic device can exit the above-mentioned iterative loop process when the total number of vectors in the second set is equal to the first set threshold. Or the electronic device can exit the above-mentioned iterative loop process when the maximum value of the second distance that is less than the corresponding first distance is less than the second set threshold. Among them, the second set threshold represents the minimum sampling spacing, thereby, the first posture can be uniformly sampled according to the minimum sampling spacing, thereby determining multiple second postures, and being able to extract a set of key frames with the best quality from the first video.

[0127] In practical applications, the second threshold is 1 or 2. Different values ​​of the second threshold have different distances between each two second postures, and different densities of all the second postures. The larger the second threshold, the larger the distance between each two second postures, and the sparser the second postures. Figure 6 This is a schematic diagram of a second posture determined when the second threshold is set to 1. Figure 7 This is a schematic diagram of a second posture determined when the second threshold is set to 2. Figure 6 The density of the second pose in is less than Figure 5 The density of the second pose in .

[0128] In order to improve the accuracy of the calculated distance between each two first vectors and further improve the image quality of the extracted key frame, in some embodiments, the first posture also includes the posture of the camera when the corresponding image is taken; when determining the distance between the two first vectors, the method includes:

[0129] determining a vector difference between two first vectors;

[0130] Calculate a first weight based on the two camera postures corresponding to the two first vectors;

[0131] A distance between two first vectors is determined based on a product of the vector difference and the first weight.

[0132] Here, the electronic device calculates the vector difference between the two first vectors, and calculates the first weight based on the difference between the camera postures between the two first vectors; based on the product between the calculated vector difference and the calculated first weight, the distance between the corresponding two first vectors is determined.

[0133] Therefore, the electronic device calculates the distance between every two first vectors according to the above steps.

[0134] In practical applications, the product of the calculated vector difference and the calculated first weight can be determined as the distance between the two first vectors; or the sum of the product and a set constant can be determined as the distance between the two first vectors. The set constant is used to fine-tune the distance between the two vectors.

[0135] In order to improve the accuracy of the calculated distance between each two first vectors and further improve the image quality of the extracted key frame, in some embodiments, the first weight is calculated based on the two camera postures corresponding to the two vectors, including:

[0136] Converting a first of the two camera poses into a first rotation vector, and converting a second of the two camera poses into a second rotation vector;

[0137] A first weight is calculated based on a first setting coefficient, the first rotation vector, and the second rotation vector, and based on a norm of the first rotation vector and a norm of the second rotation vector.

[0138] Among them, the camera posture is a rotation matrix.

[0139] The electronic device converts a first posture of the two camera postures into a first rotation vector, and converts a second posture of the two camera postures into a second rotation vector; calculates a first product between the first rotation vector and the second rotation vector; calculates a norm of the first rotation vector and a norm of the second rotation vector, and calculates a second product between the norm of the first rotation vector and the norm of the second rotation vector; divides the first product by the second product to obtain a quotient between the first product and the second product, and calculates a first weight based on a first set coefficient and the calculated quotient.

[0140] In actual application, the first weight is calculated using the following formula:

[0141]

[0142] Wherein, weight(r1,r2) represents the first weight calculated by the first posture r1 and the first posture r2; k represents the first setting coefficient; ξ r1 Characterizes the first rotation vector; ξ r2 Characterizes the second rotation vector; ||ξ r1 || represents the norm of the first rotation vector; ||ξ r2 || represents the norm of the second rotation vector. In practical applications, the first setting coefficient is 0.5, and of course it can also be set to other decimals.

[0143] In order to improve the accuracy of the calculated distance between each two vectors, thereby accurately extracting the key frame where the picture changes according to the distance between the two vectors, and further improving the image quality of the extracted key frame, in some embodiments, determining the vector difference between the two first vectors includes:

[0144] The second norm between the two first vectors is calculated to obtain the vector difference.

[0145] Here, the electronic device calculates the second norm between the two first vectors according to the formula l(t1, t2) = ||t1-t2||2 to obtain the vector difference. Among them, t1 represents one of the two first vectors, t2 represents the other of the two first vectors; l(t1, t2) represents the vector difference between t1 and t2. In actual application, the distance between each two first vectors is: D(t1, t2) = weight(r1, r2) × l(t1, t2). D(t1, t2) represents the vector difference between t1 and t2.

[0146] Step 103: extracting an image corresponding to each second posture from the first video to obtain a key frame corresponding to the first video.

[0147] Here, when the electronic device determines the second posture, an image corresponding to each second posture is extracted from the first video to obtain a group of key frames corresponding to the first video.

[0148] In the embodiment of the present application, VSLAM technology is used to determine the first pose corresponding to each frame of the first video; based on the distance between the first vectors included in each two first poses, multiple second poses are determined in the first pose corresponding to each frame of the determined image; the image corresponding to each second pose is extracted from the first video to obtain the key frame corresponding to the first video. Through the distance between the first vector corresponding to each frame of the image and each other first vector, the first pose can be uniformly sampled to obtain multiple second poses; from the multiple second poses, as many key frames as possible where the action of the photographed object changes, and key frames where the scene in the image changes, can be extracted, which can improve the quality of the extracted key frames, and further improve the quality of the image obtained by three-dimensional reconstruction of the extracted key frames; uniform sampling of the first pose can also reduce the probability of extracting key frames with repeated images, thereby reducing the time consumed by processing key frames with repeated images and improving the efficiency of three-dimensional reconstruction.

[0149] In addition, a first weight is determined based on the two camera postures corresponding to the two first vectors, and a distance between the two vectors is determined based on the product of the vector difference between the two first vectors and the first weight. Since the difference between the camera positions and the difference between the camera postures are comprehensively considered when determining the distance between the two first vectors, key frames in which the action of the photographed object changes and key frames in which the scene in the image changes can be accurately extracted from the first video, further improving the quality of the extracted key frames.

[0150] Figure 8 A schematic diagram of the implementation process of the video key frame extraction method provided in the application embodiment of the present application, wherein the execution subject of the process is an electronic device such as a computer or a server

[0151] Step 801: Using VSLAM technology, determine the first pose and point cloud map corresponding to each frame image in the first video; the first pose includes at least a first vector; the first vector represents the camera position when the corresponding image is taken; the camera is a monocular camera.

[0152] Step 802: Determine a first scale corresponding to each frame of image based on the distance between each two positions in the point cloud map corresponding to each frame of image and the corresponding calibration distance.

[0153] Step 803: Based on the first ratio corresponding to each frame of image, amplify the first vector corresponding to the corresponding image.

[0154] Step 804: Determine a second vector; the second vector represents the average vector of all the amplified first vectors.

[0155] Step 805: Add the third vector in the first set to the second set; the initial second set is an empty set; the first set is composed of all the enlarged first vectors; the third vector represents the vector with the largest distance from the second vector among the first vectors corresponding to each frame of the image.

[0156] Step 806: Add a fourth vector to the second set; the fourth vector represents the first vector in the first set that has the largest distance to the third vector.

[0157] Step 807: Add a fifth vector to the second set based on the first distance and the second distance corresponding to each first vector in the first set; wherein the first distance represents the distance between the first vector and the vector added to the second set for the i-th time; the second distance represents the distance between the first vector and the vector added to the second set for the (i+1)th time; the fifth vector represents the first vector corresponding to the maximum value of the second distances in the first set that are less than the corresponding first distances; and i is an integer greater than or equal to 1.

[0158] Step 808: Determine whether the set conditions are currently met.

[0159] Step 809: when the set condition is not currently met, assign i a value of i+1, execute the first distance and the second distance corresponding to each first vector in the first set, and add the fifth vector to the second set.

[0160] Step 810: When the set condition is currently satisfied, the first posture corresponding to all vectors in the second set is determined as the second posture.

[0161] Step 811: extracting an image corresponding to each second posture from the first video to obtain a key frame corresponding to the first video.

[0162] In order to implement the video key frame extraction method of the embodiment of the present application, the embodiment of the present application also provides an electronic device, such as Fig. 9 As shown, the electronic device includes:

[0163] The estimation unit 91 is used to determine the first pose corresponding to each frame of the first video by using the visual simultaneous localization and mapping (VSLAM) technology; the first pose includes at least a first vector; the first vector represents the position of the camera when the corresponding image is captured;

[0164] A determination unit 92, configured to determine a plurality of second postures based on the distance between each two first vectors; the plurality of second postures are determined from the first posture corresponding to each frame of the image;

[0165] The extraction unit 93 is used to extract the image corresponding to each second posture from the first video to obtain the key frame corresponding to the first video.

[0166] In some embodiments, the determining unit 92 is specifically configured to:

[0167] Determine a second vector; the second vector represents the average vector of all the first vectors;

[0168] Based on the distance between the third vector and each first vector, a plurality of second postures are determined; wherein,

[0169] The third vector represents the vector with the largest distance from the second vector among the first vectors corresponding to each frame of image determined.

[0170] In some embodiments, the determining unit 92 is specifically configured to:

[0171] Add the third vector in the first set to the second set; the initial second set is an empty set; the first set is composed of the determined first vector corresponding to each frame of the image;

[0172] Adding a fourth vector to the second set; the fourth vector represents the first vector in the first set with the largest distance from the third vector;

[0173] Based on the first distance and the second distance corresponding to each first vector in the first set, a fifth vector is added to the second set; wherein the first distance represents the distance between the first vector and the vector added to the second set for the i-th time; the second distance represents the distance between the first vector and the vector added to the second set for the i+1th time; the fifth vector represents the first vector corresponding to the maximum value of the second distances in the first set that are smaller than the corresponding first distance; i is an integer greater than or equal to 1;

[0174] If the set condition is not met currently, i is assigned a value of i+1, and the method of adding the fifth vector to the second set based on the first distance and the second distance corresponding to each first vector in the first set is performed;

[0175] When the set condition is currently satisfied, the first postures corresponding to all vectors in the second set are determined as the second posture.

[0176] In some embodiments, the setting condition includes one of the following:

[0177] A first set condition, indicating that the total number of vectors in the second set reaches a first set threshold; the first set threshold is greater than or equal to 3;

[0178] The second setting condition indicates that the maximum value of the second distances that are smaller than the corresponding first distances is smaller than a second setting threshold.

[0179] In some embodiments, the determining unit 92 is specifically configured to:

[0180] determining a vector difference between two first vectors;

[0181] Calculate a first weight based on the two camera postures corresponding to the two first vectors;

[0182] A distance between two first vectors is determined based on a product of the vector difference and the first weight.

[0183] In some embodiments, the determining unit 92 is specifically configured to:

[0184] Converting a first of the two camera poses into a first rotation vector, and converting a second of the two camera poses into a second rotation vector;

[0185] A first weight is calculated based on a first setting coefficient, the first rotation vector, and the second rotation vector, and based on a norm of the first rotation vector and a norm of the second rotation vector.

[0186] In some embodiments, the determination unit 92 is specifically used to calculate the second norm between the two first vectors to obtain a vector difference.

[0187] In some embodiments, the electronic device further comprises:

[0188] A second estimation unit is used to determine a point cloud map corresponding to each frame image in the first video by using VSLAM technology;

[0189] A calculation unit, configured to determine a first scale corresponding to each frame of image based on a distance between every two positions in a point cloud map corresponding to each frame of image and a corresponding calibration distance;

[0190] The enlarging unit is used to enlarge the first vector corresponding to the corresponding image based on the first ratio corresponding to each frame of the image.

[0191] In actual application, the estimation unit 91, the determination unit 92, the extraction unit 93, the second estimation unit, the calculation unit and the amplification unit can be implemented by a processor in the terminal, such as a central processing unit (CPU), a digital signal processor (DSP), a microcontroller unit (MCU) or a programmable gate array (FPGA).

[0192] It should be noted that: when the electronic device provided in the above embodiment performs video key frame extraction, only the division of the above program modules is used as an example. In actual application, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the above-described processing. In addition, the electronic device provided in the above embodiment and the video key frame extraction method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0193] Based on the hardware implementation of the above program modules and in order to implement the method of the embodiment of the present application, the embodiment of the present invention also provides an electronic device. Fig.10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown in FIG. Fig.10 As shown, the electronic device 10 includes:

[0194] The communication interface 110 is capable of exchanging information with other devices such as a terminal for shooting the first video;

[0195] The processor 120 is connected to the communication interface 110 to implement information exchange with other devices and is used to execute the video key frame extraction method provided by one or more technical solutions when running a computer program. The computer program is stored in the memory 130.

[0196] Of course, in actual application, the various components in the electronic device 10 are coupled together through the bus system 140. It can be understood that the bus system 140 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 140 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Fig.10 Various buses are labeled as bus system 140 .

[0197] The memory 130 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device 10. Examples of such data include: any computer program used to operate on the electronic device 10.

[0198] It can be understood that the memory 130 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAM bus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 130 described in the embodiments of the present application is intended to include but is not limited to these and any other suitable types of memories.

[0199] The method disclosed in the above embodiment of the present application can be applied to the processor 120, or implemented by the processor 120. The processor 120 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor 120 or an instruction in the form of software. The above processor 120 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 120 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in the memory 130, and the processor 120 reads the program in the memory 130 and completes the steps of the above method in combination with its hardware.

[0200] Optionally, when the processor 120 executes the program, it implements the corresponding processes implemented by the terminal in each method of the embodiments of the present application, which will not be described here for the sake of brevity.

[0201] In an exemplary embodiment, the present application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, for example, including a first memory 130 storing a computer program, and the computer program can be executed by the processor 120 of the terminal to complete the steps of the aforementioned method. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface storage, optical disk, or CD-ROM.

[0202] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0203] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0204] In addition, all functional units in the embodiments of the present application may be integrated into one processing module, or each unit may be separately configured as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0205] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0206] It should be noted that the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0207] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A video key frame extraction method, characterized in that: include: Determine the first position corresponding to each frame of the first video by using the visual simultaneous localization and mapping (VSLAM) technology; the first position at least includes a first vector and a camera posture when shooting the corresponding image; The first vector represents the position of the camera when the corresponding image is captured; Based on the distance between each two first vectors, a plurality of second postures are determined; the plurality of second postures are determined in the first posture corresponding to each frame of the image; Extracting an image corresponding to each second posture from the first video to obtain a key frame corresponding to the first video; Wherein, when determining the distance between every two first vectors, the method further includes: Determining a first weight based on the two camera postures corresponding to each two first vectors; The distance between each two first vectors is determined based on the vector difference between each two first vectors and the first weight.

2. The method according to claim 1, characterized in that The determining of a plurality of second postures based on the distance between each two first vectors comprises: Determine a second vector; the second vector represents the average vector of all the first vectors; Based on the distance between the third vector and each first vector, a plurality of second postures are determined; wherein, The third vector represents the vector with the largest distance from the second vector among the first vectors corresponding to each frame of image determined.

3. The method according to claim 2, characterized in that The determining of a plurality of second postures comprises: Add the third vector in the first set to the second set; the initial second set is an empty set; the first set is composed of the determined first vector corresponding to each frame of the image; Adding a fourth vector to the second set; the fourth vector represents the first vector in the first set with the largest distance from the third vector; Based on the first distance and the second distance corresponding to each first vector in the first set, a fifth vector is added to the second set; wherein the first distance represents the distance between the first vector and the vector added to the second set for the i-th time; the second distance represents the distance between the first vector and the vector added to the second set for the i+1th time; the fifth vector represents the first vector corresponding to the maximum value of the second distances in the first set that are smaller than the corresponding first distance; i is an integer greater than or equal to 1; If the set condition is not met currently, i is assigned a value of i+1, and the method of adding the fifth vector to the second set based on the first distance and the second distance corresponding to each first vector in the first set is performed; When the set condition is currently satisfied, the first postures corresponding to all vectors in the second set are determined as the second posture.

4. The method according to claim 3, characterized in that The setting condition includes one of the following: A first set condition, indicating that the total number of vectors in the second set reaches a first set threshold; the first set threshold is greater than or equal to 3; The second setting condition indicates that the maximum value of the second distances that are smaller than the corresponding first distances is smaller than a second setting threshold.

5. The method according to any one of claims 1 to 4, characterized in that: When determining the distance between two first vectors, the method comprises: determining a vector difference between two first vectors; Calculate a first weight based on the two camera postures corresponding to the two first vectors; A distance between two first vectors is determined based on a product of the vector difference and the first weight.

6. The method according to claim 5, characterized in that The calculating a first weight based on the two camera postures corresponding to the two vectors includes: Converting a first of the two camera poses into a first rotation vector, and converting a second of the two camera poses into a second rotation vector; A first weight is calculated based on a first setting coefficient, the first rotation vector, and the second rotation vector, and based on a norm of the first rotation vector and a norm of the second rotation vector.

7. The method according to claim 5, characterized in that The determining of the vector difference between the two first vectors comprises: The second norm between the two first vectors is calculated to obtain the vector difference.

8. The method according to claim 1, characterized in that: In the case where the camera is a monocular camera, before determining a plurality of second postures based on the distance between each two first vectors, the method further includes: Using VSLAM technology, determine the point cloud map corresponding to each frame of the first video; Determine a first scale corresponding to each frame of image based on a distance between every two positions in a point cloud map corresponding to each frame of image and a corresponding calibration distance; Based on the first ratio corresponding to each frame of image, a first vector corresponding to the corresponding image is enlarged.

9. An electronic device, characterized in that: include: An estimation unit, configured to determine a first pose corresponding to each frame of an image in a first video by using a visual simultaneous localization and mapping (VSLAM) technology; the first pose at least includes a first vector and a camera pose when the corresponding image is captured; The first vector represents the position of the camera when the corresponding image is captured; A determination unit, configured to determine a plurality of second postures based on the distance between each two first vectors; The plurality of second postures are determined from the first posture corresponding to each frame of the determined image; An extraction unit, configured to extract an image corresponding to each second posture from the first video to obtain a key frame corresponding to the first video; The determining unit is further configured to determine a first weight based on two camera postures corresponding to each two first vectors when determining the distance between each two first vectors; The distance between each two first vectors is determined based on the vector difference between each two first vectors and the first weight.

10. An electronic device, characterized in that: include: a processor and a memory for storing a computer program capable of being executed on the processor, Wherein, when the processor is used to run the computer program, it executes the steps of the video key frame extraction method described in any one of claims 1 to 8.

11. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the video key frame extraction method according to any one of claims 1 to 8 are implemented.

Images