Method for extracting video reference frame, related device and computer program product
By extracting the first and last frames of the target video from the video platform and determining the reference frame position through multiple iterations to form gradient acquisition, the problem of low efficiency and resource waste in the existing technology is solved, and the effect of uniform acquisition and efficient provision of reference frames is achieved.
Patent Information
- Application Number
- CN202410942213.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-12
- Publication Date
- 2024-11-15
AI Technical Summary
In existing technologies, video platforms are inefficient and waste resources when extracting reference frames. Furthermore, the cost of obtaining reference frames is high for business users, and the extraction quality is uneven, making it difficult to meet the needs of different businesses.
The first and last frames of the target video are extracted as the first round of reference frames. Multiple rounds of extraction are performed in a loop. The position of the reference frames in each round is determined based on the historical extraction results, forming gradient acquisition, which is stored in the reference frame database and provides reference frames in response to business requests.
It achieves uniform acquisition of reference frames, reduces the acquisition cost for the business side, improves extraction efficiency and quality, decouples the extraction instructions from the business side, and reduces resource waste.
Smart Images

Figure CN118972639A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a method and device for extracting a video reference frame, an electronic device, a computer-readable medium, and a computer program product. Background Art
[0002] With the development of computer technology, people are gradually entering the Internet and digital society. In this process, video has become an indispensable part of people's lives. Through mobile Internet technology, users can watch videos provided by video platforms and other users online or by downloading through video platforms.
[0003] In this context, in order to facilitate users to quickly understand the video content they are watching, and to facilitate business parties to promote, publicize and use video content, video platform service providers and business parties often choose to extract at least part of the video frames from the video as reference frames to analyze the video content and produce reference content. Therefore, how to extract reference frames with higher quality and provide reference frames more efficiently is worthy of attention and urgent need. Summary of the invention
[0004] Multiple aspects of the present application provide a method, device, electronic device, computer-readable storage medium, and computer program product for extracting video reference frames, which can extract and store reference frames in advance through a "gradient" acquisition method after acquiring the video. As a result, reference frames can be evenly acquired for all types of videos, ensuring the acquisition quality while reducing the acquisition consumption of reference frame demanders such as business parties, and providing them with the required reference frames conveniently and efficiently.
[0005] In one aspect of the present application, a method for extracting video reference frames is provided, comprising: extracting the first frame and the last frame of a target video to obtain a first reference frame set as a result of a first round of extraction; looping through the following extraction process until a target round is reached and then exiting to obtain a second reference frame set corresponding to each extraction round: based on the extraction results of a historical extraction round, obtaining a second reference frame set corresponding to the current extraction round, wherein the positions of the reference frames in the second reference frame set of the current round in the target video are located within a frame interval determined based on adjacent historical reference frames that have been extracted; in response to exiting after reaching the target round, storing the first reference frame set and each second reference frame set corresponding to the target video in a reference frame database, wherein the reference frame database is used to respond to a reference frame retrieval request sent by a business party and provide the business party with a set of reference frame sets for the target video.
[0006] In another aspect of the present application, there is provided an apparatus for extracting video reference frames, comprising: a first reference frame extraction unit, configured to extract the first frame and the last frame of a target video, and obtain a first reference frame set as a result of a first round of extraction; a second reference frame extraction unit, configured to cyclically execute the following extraction process until a target round is reached and then jump out, and obtain a second reference frame set corresponding to each extraction round: based on the extraction results of historical extraction rounds, a second reference frame set corresponding to the current extraction round is obtained, wherein the positions of the reference frames in the second reference frame set of the current round in the target video are located within a frame interval determined based on adjacent historical reference frames that have been extracted; a reference frame result storage unit, configured to, in response to jumping out after reaching the target round, store the first reference frame set and each second reference frame set corresponding to the target video in a reference frame database, wherein the reference frame database is used to respond to a reference frame retrieval request sent by a business party and provide the business party with a set of reference frame sets for the target video.
[0007] Another aspect of the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method for extracting video reference frames provided above.
[0008] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method for extracting a video reference frame as provided above.
[0009] In another aspect of the present application, a computer program product includes a computer program having computer program instructions stored thereon. When the computer program is executed by a processor, the method for extracting a video reference frame as provided above can be implemented.
[0010] In the scheme provided by the embodiment of the present application, the first frame and the last frame of the target video are extracted to obtain the first reference frame set as the result of the first round of extraction; the following extraction process is executed cyclically until the target round is reached and then the second reference frame set corresponding to each extraction round is obtained: based on the extraction result of the historical extraction round, the second reference frame set corresponding to the current extraction round is obtained, wherein the position of the reference frame in the second reference frame set of this round in the target video is located within the frame interval determined based on the adjacent historical reference frame that has been extracted; in response to the target round being reached and then the first reference frame set and each second reference frame set are stored in the reference frame database corresponding to the target video, wherein the reference frame database is used to respond to the reference frame retrieval request sent by the business party, and provide the business party with a set of reference frame sets for the target video. In this way, after acquiring the video, the reference frame can be extracted and stored in advance by the "gradient" acquisition method. As a result, reference frames can be evenly acquired for all types of videos, while ensuring the acquisition quality, reducing the acquisition consumption of reference frame demanders such as business parties, and providing them with the required reference frames conveniently and efficiently. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0012] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:
[0013] Figure 1 A schematic diagram of a process of extracting a video reference frame provided by an embodiment of the present application;
[0014] Figure 2 A schematic diagram of a process of extracting a video reference frame in an application scenario provided by another embodiment of the present application;
[0015] Figure 3 A schematic diagram of the structure of a device for extracting a video reference frame provided in an embodiment of the present application;
[0016] Figure 4 The present invention is a schematic diagram of the structure of an electronic device suitable for implementing the solution in the embodiment of the present application.
[0017] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION
[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0019] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.
[0020] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0021] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.
[0022] In general scenarios, the reference frame provider (e.g., the service provider of the video platform) will extract the reference frames required by the business party from the target video according to the instructions of the reference frame demander (e.g., the business party), and provide these reference frames for the business party to use. Therefore, as discussed above, how to extract reference frames with higher quality and provide reference frames more efficiently is worthy of attention and urgent need.
[0023] In some solutions, the business party can choose to provide the required number of reference frames to the service provider of the video platform (hereinafter referred to as the platform party) based on actual needs. Then, the platform party will evenly split the target video based on this number to determine the positions of each frame extraction, or in other words, determine the video frames that need to be extracted as reference frames. For example, in the case where the business party needs 10 reference frames, the platform party can choose to equidistantly split the target video into 9 video segments of equal length, and then determine the frames at the endpoints of the 9 video segments as the "reference frames" that need to be extracted, and extract the reference frames.
[0024] In this way, the start of the frame extraction process is very dependent on the needs of the business party and the synchronization of the frame extraction needs of the business party, which not only leads to low frame extraction efficiency on the platform side (often frame extraction can only be actually performed after the business party provides the needs), but also causes a lot of resource waste for the platform side (for example, it is necessary to extract frames independently according to the needs provided by each business party, and there may be problems of repeated frame extraction between business parties). Moreover, this method is not only too consuming for the business party (for example, the business party may need to additionally calculate the video duration, the length of the split interval, etc.), but also requires the business party to wait for frame extraction after providing the needs. For the business party, the efficiency of providing and obtaining reference frames is low. For example, in the equidistant splitting method discussed above, the logic of the reference frame is actually "fixed and equidistant". In such a context, if the business party or the platform party adjusts the extraction position of one of the extracted frames, for example, moving the position of a reference frame determined by equidistant forward or backward, the extraction positions of the reference frames after the reference frame will all change, forcing the platform party to re-extract subsequent reference frames. Such "re-extraction" will not only consume additional memory and time, but may also cause unexpected "offsets" in the reference frames extracted later, reducing the cache hit rate and having a negative impact on the extraction quality of the reference frames.
[0025] In this regard, an embodiment of the present application provides a method for extracting video reference frames, which extracts the first frame and the last frame of the target video to obtain a first reference frame set as the first round of extraction results; loops through the following extraction process until the target round is reached and then jumps out, obtaining a second reference frame set corresponding to each extraction round: based on the extraction results of the historical extraction round, a second reference frame set corresponding to the current extraction round is obtained, wherein the position of the reference frame in the second reference frame set of this round in the target video is located within a frame interval determined based on the adjacent historical reference frames that have been extracted; in response to jumping out after reaching the target round, the first reference frame set and each second reference frame set are stored in a reference frame database corresponding to the target video, wherein the reference frame database is used to respond to the reference frame retrieval request sent by the business party, and provide the business party with a set of reference frame sets for the target video. In this way, after acquiring the video, reference frames can be extracted and stored in advance by "gradient" acquisition. As a result, reference frames can be collected evenly for all types of videos, ensuring the collection quality while reducing the acquisition consumption of reference frame demanders such as business parties, and providing them with the required reference frames conveniently and efficiently.
[0026] In actual scenarios, the execution subject of the method can be a user device, or a device formed by integrating a user device and a network device through a network, or an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing (Cloud Computing), where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.
[0027] When the execution subject is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here.
[0028] Figure 1 A process 100 for extracting a video reference frame provided by an embodiment of the present application is shown. The process 100 includes at least the following processing steps:
[0029] (Step) S101, extract the first frame and the last frame of the target video to obtain a first reference frame set as a result of the first round of extraction.
[0030] In an embodiment of the present application, after the execution subject obtains the target video (for example, a video uploaded by a user or provided by a platform, which can be requested as a reference frame by a business party or a user), the first frame and the last frame of the target video frame are first extracted to obtain two reference frames, forming a reference frame set (described as a first reference frame set for ease of understanding). Accordingly, the first reference frame set can be understood as the extraction result of the execution subject for the first round of reference frame extraction for the target video.
[0031] S102, whether the detection extraction round reaches the target round?
[0032] In the embodiments of the present application, as discussed above, the execution subject can continuously extract reference frames from the target video by means of a "gradient" (which can be understood as, as the number of extraction rounds increases, more reference frames can be provided in a more fine-grained manner to achieve a "gradient") and multiple rounds. For example, after the "first reference frame set" is extracted based on the above S103, it can be regarded as the first round, or the reference frame (result) extracted by the first round. Subsequently, the execution subject can obtain the reference frame results extracted by the second round, the third round, and so on, or the reference frame set, respectively, by means of multiple rounds of extraction. Accordingly, for the convenience of description, the reference frame sets extracted by each round after the first round can be described as second reference frame sets. These second reference frame sets correspond to the extraction rounds respectively and individually, or in other words, each round after the first round can extract a second reference frame set corresponding to the "current round".
[0033] Accordingly, the execution subject may detect in step S102 whether the total number of extraction rounds currently executed, or the total number of extraction rounds for extracting the reference frame, has reached the target round.
[0034] It should be understood that the target round can actually be "once", or "one round". That is, when the target round is set to "one round", after the execution subject determines that the above S101 is completed and the first reference frame set is extracted, all required extraction rounds have been completed, and in the current state, "the total number of extraction rounds has reached the target round".
[0035] Typically, the target round can be preset based on the number of reference frames that the platform expects to provide. For example, the execution entity can determine the "target round" based on the total number of reference frames that need to be stored and the matching relationship between the number of reference frames extracted in each round. For example, if X reference frames can be extracted in each round, if the execution entity determines that a total number of "5 X" reference frames needs to be cached, it can set the target round to "5".
[0036] It should be understood that the number of reference frames that can be extracted in each round may be different (i.e., based on the difference in the "extraction logic" involved in the extraction, the number of reference frames obtained between each round may be different). For example, the "second reference frame set" extracted in the second round may be the same as the "first reference frame set", and the reference frame set includes 2 reference frames. The "second reference frame set" extracted in the third round may include 4 reference frames, which is different from the number of reference frames included in the reference frame sets obtained in the first and second rounds.
[0037] In some embodiments, the target round may be determined based on the video length of the target video, and the numerical value of the target round is positively correlated with the numerical value of the video length.
[0038] Specifically, when determining the target round, the execution subject may select the video length of the reference target video to determine its corresponding round. Compared with the target video with a short video length, the target video with a longer video length may have more target rounds correspondingly, so that the target video with a longer video length may have richer and sufficient reference frames.
[0039] For example, the execution entity can pre-divide multiple video length levels (each level corresponds to a numerical range of allowed video length), and then divide and configure corresponding target rounds according to the video length levels. Then, after obtaining the target video, the execution entity determines the corresponding video length level based on the numerical range that the length of the target video falls into, and determines the number of "target rounds" accordingly.
[0040] In this step, the execution subject may choose to execute S103 to continue to perform the next round of reference frame extraction when detecting that the target round has not been reached.
[0041] Alternatively, the execution subject may execute S104 when detecting that the target round has been reached.
[0042] S103, based on the extraction results of the historical extraction rounds, obtain a second reference frame set corresponding to the current extraction round.
[0043] In the embodiment of the present application, based on the above S102, the execution subject continues to perform the reference frame extraction operation when the extraction round does not meet the requirement (i.e., the target extraction round is not reached). That is, when the target round is two or more rounds, the execution subject continues to perform the next or next several rounds of reference frame extraction after obtaining the first reference frame set.
[0044] When the execution entity extracts reference frames in each round, for the reference frames in the second reference frame set of this round, the execution entity can determine the position of the reference frames by obtaining adjacent historical reference frames in the second reference frame set corresponding to the current extraction round based on the extraction results of historical extraction rounds (for example, the first reference frame set or a combination of the first reference frame set and the second reference frame set).
[0045] For example, taking this round as the second round of extraction operation as an example, the reference frames in the second reference frame set of this round, their positions in the target video are located within the frame interval determined based on the adjacent historical reference frames in the first reference frame set extracted in the first extraction round (in fact, in this example, the frame interval consists of all video frames of the target video except the first frame and the last frame).
[0046] That is, for a scene in which the first reference frame set is used as the reference of the "previous extraction round", in the extraction round of extracting the second reference frame set, the reference frames of the current extraction round can be extracted in the frame interval starting from the first frame of the target video and ending at the last frame. For example, when the number of extractions is 3, the extracted reference frames can be Z1, Z2 and Z3.
[0047] Correspondingly, in the next round of extraction process, based on the combination result of the first reference frame set and the second reference frame set obtained in the second round, the four frame intervals of "first frame-Z1", "Z1-Z2", "Z2-Z3" and "Z3-last frame" of the target video can be determined to extract reference frames respectively to perform the "third round of extraction operation". For example, Z4 is extracted from "first frame-Z1", Z5 is extracted from "Z1-Z2", and so on.
[0048] In some embodiments, in the process of performing the current round of extraction based on the extraction results of the previous rounds, in order to evenly extract reference frames in each frame interval, the execution entity may choose to extract a fixed number of reference frames in each frame interval each time.
[0049] As a result, the reference frames in each round of extraction can be more evenly distributed, and a "gradient" can be better formed through multiple rounds (for example, in some scenarios, if the number of reference frames extracted in the frame interval of the first frame-Z1 is greater than or much greater than the frame interval of Z3-the last frame, the reference frames extracted in this round may be distributed "more inclined to and concentrated on the first frame", resulting in an unbalanced distribution).
[0050] In some embodiments, in order to make the reference frames more evenly distributed, the positions of the reference frames extracted in this round may be determined based on the binary idea.
[0051] Specifically, during the extraction process of this round, the execution subject selects the middle position in the frame interval determined by the adjacent historical reference frames in the reference frame set extracted in the historical round as the extraction position of the reference frame of this round. That is, each reference frame obtained in this extraction round is actually a frame at the middle position of each frame interval determined based on the extraction results in the historical round. Therefore, through the binary division method, not only can the reference frame extraction results of each round of extraction be more "evenly" distributed relative to the complete target video, but also as the number of extraction rounds increases, the number of reference frames extracted in each round can be gradually increased, thereby improving the granularity of reference frame extraction.
[0052] Accordingly, after the current round of extraction is completed, the process may jump back to S102 to determine whether it is necessary to continue the subsequent round of extraction, thereby cyclically extracting one or more second reference frame sets.
[0053] S104: Store the first reference frame set and each second reference frame set corresponding to the target video into a reference frame database.
[0054] In an embodiment of the present application, when the execution subject determines that the extraction round has reached the target round, the execution subject jumps out and stores the first reference frame set and each second reference frame set corresponding to the target video in the reference frame database. The reference frame database is used to respond to the reference frame retrieval request sent by the business party and provide the business party with a set of reference frame sets for the target video. That is, the execution subject can use the reference frame database to respond to the reference frame retrieval request sent by the business party and provide it with reference frames.
[0055] It should be understood that the execution subject usually does not choose to "break up" the original set form in the process of storing the first reference frame set and each second reference frame set in the reference frame database. That is to say, in the process of storing, the execution subject still performs storage in units or dimensions such as "first reference frame set" and "second reference frame set". Accordingly, the reference frames actually included in the "first reference frame set" and "second reference frame set" are subsequently acquired and extracted as a whole through the hit results of the "reference frame set". Accordingly, after responding to the reference frame retrieval request sent by the business party, the execution subject can form a group of reference frame sets with the "first reference frame set" and "second reference frame set", and provide the reference frame to the business party by returning and providing the group of reference frame sets to the business party.
[0056] Therefore, the present application can decouple the "extraction logic" from the extraction indication part of the "business party" by pre-extraction, so that the business party can directly obtain the reference frame by providing the required number of reference frames to improve acquisition efficiency and reduce storage pressure (for example, the extraction results of the stored reference frame can be used by multiple business parties and provided to multiple business parties, and there is no need to maintain a "reference frame storage" independently for each business party).
[0057] Moreover, this approach will not, as discussed above, cause the original extraction logic to be completely destroyed and re-extracted when the business side adjusts the extraction position of the "reference frame", thereby wasting extraction resources and reducing the cache hit rate.
[0058] For easier understanding, you can also refer to Figure 2 . Figure 2 A schematic diagram of a process 200 for extracting a video reference frame in an application scenario provided by another embodiment of the present application. In the process 200, four extraction rounds are exemplarily used as "target rounds" for example.
[0059] In the process 200, for the target video 210, its video frame sequence may be shown as a sequence 215. In the sequence 215, the target video 210 is exemplarily composed of 18 video frames.
[0060] When executing the first extraction round, the execution subject may extract the 1st video frame and the 18th video frame from the sequence 215 to obtain a reference frame set 221 .
[0061] Then, the execution subject determines that the target extraction round has not been reached, and then executes the second extraction round. Specifically, after the execution subject determines the frame interval based on the first video frame and the eighteenth video frame, it extracts the ninth video frame based on the binary result of the frame interval to obtain the second round of extraction results, that is, the extraction result is the reference frame set 222.
[0062] Next, the execution subject determines that the target extraction round has not been reached, and then executes the third extraction round. Specifically, the execution subject determines the frame interval based on the first two extraction results, the 1st video frame, the 9th video frame, and the 18th video frame, and extracts the 5th video frame and the 14th video frame based on the binary results of each frame interval, and obtains the extraction result of the third round, that is, the extraction result is the reference frame set 223.
[0063] Next, the execution subject determines that the target extraction round has not been reached, and then executes the fourth extraction round. Specifically, the execution subject determines the frame interval based on the first three extraction results, using the first video frame, the fifth video frame, the ninth video frame, the fourteenth video frame, and the eighteenth video frame, and extracts the third video frame, the seventh video frame, the eleventh video frame, and the sixteenth video frame based on the binary results of each frame interval, and obtains the fourth round of extraction results, that is, the extraction result is the reference frame set 224.
[0064] Next, the execution subject determines that the target extraction round has been reached and chooses to exit. Then, the reference frame sets 221-224 are independently and completely stored in the reference frame database 230 corresponding to the target video 210.
[0065] Then, the method for extracting video reference frames provided by the present application extracts the first frame and the last frame of the target video to obtain a first reference frame set as the result of the first round of extraction; the following extraction process is executed cyclically until the target round is reached and then the second reference frame set corresponding to each extraction round is obtained: based on the extraction results of the historical extraction round, the second reference frame set corresponding to the current extraction round is obtained, wherein the position of the reference frame in the second reference frame set of this round in the target video is located within the frame interval determined based on the adjacent historical reference frames that have been extracted; in response to the target round being reached and then the first reference frame set and each second reference frame set are stored in the reference frame database corresponding to the target video, wherein the reference frame database is used to respond to the reference frame retrieval request sent by the business party, and provide the business party with a set of reference frame sets for the target video. In this way, after acquiring the video, the reference frame can be extracted and stored in advance by the "gradient" acquisition method. As a result, reference frames can be collected evenly for all types of videos, ensuring the collection quality while reducing the acquisition consumption of reference frame demanders such as business parties, and providing them with the required reference frames conveniently and efficiently.
[0066] In some embodiments, when the execution subject receives a reference frame retrieval request, the execution subject may respond to it and determine the number of target reference frames requested by the reference frame retrieval request. That is, the business party may determine the required number of target reference frames according to its own needs, and then communicate with the execution subject to request the execution subject to provide at least this number of (target) reference frames.
[0067] Accordingly, the execution subject determines a set of reference frame sets from the reference frame database based on the target reference frame quantity, starting with the first reference frame set and according to the extraction round order.
[0068] Specifically, the execution subject may extract reference frame sets (i.e., the first reference frame set and the second reference frame set corresponding to each round) in sequence and continuously based on the order of the extraction rounds to obtain a set of reference frame sets. For example, the execution subject continuously extracts the first reference frame set obtained in the first extraction round, the second reference frame set obtained in the second extraction round, and the second reference frame set obtained in the third extraction round, until the total number of reference frames corresponding to a set of reference frame sets (i.e., the sum of the reference frames included in each reference frame set) is greater than or equal to the above-mentioned target number of reference frames.
[0069] Finally, the execution entity can provide the extracted set of reference frames to the business party for their use. Thus, the execution entity can preferentially and continuously choose to provide reference frame extraction results with a larger "span" based on the "gradient" reference frame extraction results until the needs of the business party are met. Thus, not only is the "uneven distribution" problem that may exist due to methods such as random extraction avoided, but the business party can also obtain reference frames based on the entire video content range and span by only providing the quantity, which can reduce the user consumption of the business party. Moreover, compared with the "equidistant" method in the above-mentioned "some solutions", it can provide reference frames with a larger "span" range under the same quantity, thereby improving the quality and scope of content acquisition for the business party.
[0070] It should be understood that in some scenarios, the number of reference frames required by the business party (i.e., the target number of reference frames) may have exceeded all the reference frames that the reference frame database can provide. In this case, the execution entity can provide all the first reference frame sets and second reference frame sets stored in the reference frame database as a group of reference frame sets to the business party, and at the same time send a prompt message to the business party to indicate that the "reference frames" stored in its current reference frame database have been exhausted.
[0071] Accordingly, in such a case, if the business party still needs reference frames, it may choose to communicate with the execution subject and instruct the execution subject to "continue to extract frames". For example, the execution subject is instructed to extract at least one supplementary second reference frame set based on the target second reference frame set corresponding to the last round of extraction results of the target video stored in the reference frame database through the extraction process described above (for example, after determining the frame interval based on the reference frames in the target second reference frame set, the reference frames in the supplementary second reference frame set are determined based on the frame interval).
[0072] In other words, the business party can continue to instruct the execution subject to use the currently stored "second reference frame set extracted in the last round" as a reference to continue extracting "reference frames" to enrich the storage of the target video in the reference frame database. Thus, more reference frames can be obtained by continuing and supplementing the extraction to meet the business party's usage needs.
[0073] Accordingly, in order to reduce the user consumption of the service side, the execution subject may also actively perform such a "supplementation process". Specifically, the execution subject may detect the number of times all second reference frame sets are used in the process of determining a set of reference frame sets within a preset time after storing the first reference frame set and each second reference frame set in the reference frame database. If the number of times all second reference frame sets are used is equal to a preset quantity threshold (for example, it may be determined in advance based on the situation that the platform side believes that more reference frames need to be provided), the execution subject actively extracts at least one supplementary second reference frame set through an extraction process based on the target second reference frame set corresponding to the extraction result of the last round of the target video stored in the reference frame database. For example, the purpose of supplementing more than two supplementary second reference frame sets may also be achieved by looping two or more rounds. Considering that the process of extracting (supplementing) reference frames is similar to that discussed above, it will not be repeated here.
[0074] Then, the execution entity may also store the supplementary second reference frame set corresponding to the target video into the above-mentioned reference frame database to supplement the reference frames corresponding to the target video in the reference frame database.
[0075] In some embodiments, as discussed above, reference frames are provided in the form of a "reference frame set", that is, reference frames are provided to the business party in the form of a set of reference frame sets. In such a case, redundancy may exist. For example, the total number of reference frames corresponding to a set of reference frame sets is actually greater than the target number of reference frames requested and required by the business party. In such a case, the execution entity can respond to this by selecting and determining the target reference frame set of the last extraction round in a set of reference frame sets, and then updating a set of reference frame sets by deleting some reference frames in the target reference frame set, so that the total number of reference frames corresponding to the updated set of reference frame sets is equal to the target number of reference frames.
[0076] For example, for the target reference frame set extracted at the end of the round (depending on the setting of the target round, it may be the first reference frame set or the second reference frame set), the execution subject may choose to delete some reference frames in the target reference frame set in a set of reference frames after determining a set of reference frames, so that the total number of reference frames corresponding to the set of reference frames provided to the business party corresponds to the needs of the business party. In this way, it is avoided to provide "redundant" content to the business party and occupy the storage and communication resources of the business party, and avoid negative impact on the business party.
[0077] Typically, the executor may choose to randomly delete a redundant number of reference frames from a target reference frame set (for example, when the total number of reference frames corresponding to a current set of reference frames exceeds Y, the executor may choose to delete Y reference frames from the target reference frame set to update a set of reference frames).
[0078] In some embodiments, the execution entity may also communicate with the business party to provide description information, thumbnail information, etc. of each reference frame in the target reference frame set, and require the business party to refer to the description information and thumbnail information to select the reference frames to be deleted.
[0079] Furthermore, when the executing entity is updating a set of reference frame sets by deleting some reference frames in the target reference frame set, if it can refer to the historical records of reference frames in the target reference frame set being deleted, it can obtain and determine the frequency of providing the reference frames in the target reference frame set to the business party based on such historical records.
[0080] Then, the executing entity may choose to sort the reference frames in the target reference set in ascending order based on the frequency with which the reference frames in the target reference frame set are provided to the business party, and continuously select the reference frames to be deleted from the target reference set according to the frequency of provision from low to high (i.e., reference frames that are less frequently used and not provided to the business party are preferentially deleted) until the number of remaining reference frames in the target reference set meets the requirement.
[0081] As a result, the execution entity can prioritize and recommend reference frames with greater "usage popularity" based on the frequency with which the reference frames are provided in historical data, so as to improve the quality of providing reference frames.
[0082] On the basis of any of the above embodiments, the execution subject may also choose to add a round identifier and a quantity identifier for the first reference frame set and each second reference frame set in the reference frame database in the process of storing the first reference frame set and each second reference frame set corresponding to the target video. The round identifier is used to indicate the number of extraction rounds required to extract the corresponding reference frame set, and the quantity identifier is used to indicate the number of reference frames included in the corresponding reference frame set. Thus, the execution subject can subsequently determine the selection order for each reference frame set through the round representation, and judge whether a set of reference frame sets composed of the currently selected reference frame sets can meet the requirements based on the quantity identifier.
[0083] In some embodiments, the execution subject may also provide the round identifier and quantity identifier recorded in the reference frame database to the corresponding target device (e.g., the device of the business party, the device of the platform party maintenance personnel) according to the instruction. Thus, the subjects using these target devices can intuitively and quantitatively understand the situation of the reference frames maintained by the execution subject, so as to assist these subjects in making decisions (e.g., adjusting the number of reference frames required, adjusting and updating the reference frames in the reference frame database, etc.).
[0084] The present application also provides a device for extracting a video reference frame. The structure of the device is as follows: Figure 3 The device 300 shown. The device 300 includes: a first reference frame extraction unit 310, configured to extract the first frame and the last frame of the target video to obtain a first reference frame set as the first round of extraction results; a second reference frame extraction unit 320, configured to loop through the following extraction process until the target round is reached and then jump out, to obtain a second reference frame set corresponding to each extraction round: based on the extraction results of the historical extraction round, obtain a second reference frame set corresponding to the current extraction round, wherein the position of the reference frames in the second reference frame set of the current round in the target video is located within a frame interval determined based on the adjacent historical reference frames that have been extracted; a reference frame result storage unit 330, configured to jump out after reaching the target round, and store the first reference frame set and each second reference frame set corresponding to the target video in a reference frame database, wherein the reference frame database is used to respond to the reference frame retrieval request sent by the business party and provide the business party with a set of reference frame sets for the target video.
[0085] In some embodiments, the positions of the reference frames in the second reference frame set of this round in the target video are located at the middle positions within the frame interval determined based on the adjacent historical reference frames that have been extracted.
[0086] In some embodiments, the target round is determined based on the video length of the target video, and the numerical value of the target round is positively correlated with the numerical value of the video length.
[0087] In some embodiments, the device 300 also includes: a reference frame quantity determination unit, configured to determine the target number of reference frames requested by the reference frame retrieval request in response to receiving the reference frame retrieval request; a reference frame set determination unit, configured to determine a group of reference frame sets from the reference frame database based on the target number of reference frames, starting with the first reference frame set and according to the order of extraction rounds, wherein the total number of reference frames corresponding to a group of reference frame sets is greater than or equal to the target number of reference frames; and a reference frame set providing unit, configured to provide a group of reference frame sets to the business party.
[0088] In some embodiments, in response to a total number of reference frames corresponding to a set of reference frame sets being greater than a target number of reference frames, the device 300 further includes: a last-round set determination unit, configured to determine a target reference frame set of the last extraction round in a set of reference frame sets; a reference frame set updating unit, configured to update a set of reference frame sets by deleting some reference frames in the target reference frame set so that the total number of reference frames corresponding to the updated set of reference frame sets is equal to the target number of reference frames; and the reference frame set providing unit is further configured to provide the updated set of reference frame sets to the business party.
[0089] In some embodiments, a group of reference frame sets are updated by deleting some reference frames in the target reference frame set, including: sorting the reference frames in the target reference frame set in ascending order based on the frequency with which the reference frames in the target reference frame set are provided to the service party, and continuously selecting the deleted reference frames from the target reference set according to the frequency of provision from low to high.
[0090] In some embodiments, the device 300 also includes: a supplementary reference frame extraction unit, configured to extract at least one supplementary second reference frame set through an extraction process based on the target second reference frame set corresponding to the last round of extraction results of the target video stored in the reference frame database in response to the number of times all second reference frame sets are used in the process of determining a group of reference frame sets being equal to a preset quantity threshold within a preset time after the first reference frame set and each second reference frame set are stored in the reference frame database; and a supplementary reference frame storage unit, configured to store the supplementary second reference frame set corresponding to the target video in the reference frame database.
[0091] In some embodiments, the device 300 also includes: an identifier adding unit, configured to add a round identifier and a quantity identifier for the first reference frame set and each second reference frame set in the reference frame database, wherein the round identifier is used to indicate the extraction round required to obtain the corresponding reference frame set, and the quantity identifier is used to indicate the number of reference frames included in the corresponding reference frame set.
[0092] Based on the same inventive concept, an electronic device, a readable storage medium and a computer program product are also provided in an embodiment of the present application. The method corresponding to the electronic device may be the method for extracting a video reference frame in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.
[0093] The electronic device may be a user device, or a device formed by integrating a user device and a network device through a network, or an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.
[0094] Figure 4 The structure of an electronic device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown, and the electronic device 400 includes a central processing unit (CPU, Central Processing Unit) 401, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 402 or the program loaded from the storage part 408 to the random access memory (RAM, Random Access Memory) 403. In RAM 403, various programs and data required for system operation are also stored. CPU 401, ROM 402 and RAM 403 are connected to each other through bus 404. Input / output (I / O, Input / Output) interface 405 is also connected to bus 404.
[0095] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, etc., and a speaker, etc.; a storage section 408 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet.
[0096] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 401, the above functions defined in the method of the present application are executed.
[0097] Another embodiment of the present application further provides a computer-readable storage medium and a computer program product, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application.
[0098] Specifically, the present embodiment can adopt any combination of one or more computer-readable media. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, systems, devices or devices including but not limited to electricity, magnetism, light, electromagnetic, infrared, or semiconductors, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.
[0099] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0100] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0101] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0102] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0103] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0104] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules and units is only a logical function division. There may be other division methods in actual implementation. For example, with units as an example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0105] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0106] In addition, each functional module or unit in each embodiment of the present application may be integrated into one processing module or unit, or each module or unit may exist physically separately, or two or more units may be integrated into one module or unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules or units.
[0107] The above-mentioned integrated modules and units implemented in the form of software function modules and units can be stored in a computer-readable storage medium. The above-mentioned software function modules and units are stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform some steps of the methods of each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.
[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
[0109] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.
Claims
1. A method for extracting a video reference frame, comprising: Extract the first frame and the last frame of the target video to obtain a first reference frame set as the first round of extraction results; The following extraction process is executed repeatedly until the target round is reached and then the second reference frame set corresponding to each extraction round is obtained: Based on the extraction results of the historical extraction rounds, a second reference frame set corresponding to the current extraction round is obtained, wherein the positions of the reference frames in the second reference frame set of the current round in the target video are located within the frame interval determined based on the adjacent historical reference frames that have been extracted; In response to jumping out after reaching the target round, the first reference frame set and each of the second reference frame sets are stored in a reference frame database corresponding to the target video, wherein the reference frame database is used to respond to a reference frame retrieval request sent by a business party and provide the business party with a set of reference frame sets for the target video.
2. The method according to claim 1, wherein: The positions of the reference frames in the second reference frame set of the current round in the target video are located at the middle positions within the frame interval determined based on the adjacent historical reference frames that have been extracted.
3. The method according to claim 1, wherein: The target round is determined based on the video length of the target video, and the numerical value of the target round is positively correlated with the numerical value of the video length.
4. The method according to any one of claims 1 to 3, further comprising: In response to receiving the reference frame retrieval request, determining the number of target reference frames requested by the reference frame retrieval request; Based on the target number of reference frames, starting with the first reference frame set, a set of reference frame sets is determined from the reference frame database according to an extraction round order, wherein the total number of reference frames corresponding to the set of reference frame sets is greater than or equal to the target number of reference frames; The set of reference frames is provided to the service party.
5. The method according to claim 4, in response to the total number of reference frames corresponding to the set of reference frames being greater than the target number of reference frames, further comprising: Determine a target reference frame set of the last extraction round in the set of reference frame sets; Updating the set of reference frames by deleting some of the reference frames in the target reference frame set, so that the total number of reference frames corresponding to the updated set of reference frames is equal to the target number of reference frames; as well as The providing the set of reference frames to the service party comprises: The updated set of reference frames is provided to the service party.
6. The method according to claim 5, wherein: The updating of the set of reference frames by deleting some of the reference frames in the target reference frame set comprises: The reference frames in the target reference frame set are sorted in ascending order based on the frequency with which the reference frames in the target reference frame set are provided to the service party, and the reference frames to be deleted are selected successively from the target reference set according to the frequency of provision from low to high.
7. The method according to claim 4, further comprising: In response to the number of times all the second reference frame sets are used in the process of determining the set of reference frame sets being equal to a preset number threshold within a preset time after the first reference frame set and each of the second reference frame sets are stored in the reference frame database, extracting at least one supplementary second reference frame set through the extraction process based on a target second reference frame set corresponding to an extraction result of a last round of the target video stored in the reference frame database; The supplementary second reference frame set is stored in the reference frame database corresponding to the target video.
8. The method according to claim 1, further comprising: A round identifier and a quantity identifier are added to the reference frame database for the first reference frame set and each of the second reference frame sets, wherein the round identifier is used to indicate the extraction round required to obtain the corresponding reference frame set, and the quantity identifier is used to indicate the number of reference frames included in the corresponding reference frame set.
9. A device for extracting a video reference frame, comprising: A first reference frame extraction unit is configured to extract the first frame and the last frame of the target video to obtain a first reference frame set as a first round of extraction results; The second reference frame extraction unit is configured to cyclically execute the following extraction process until a target round is reached and then exit, to obtain a set of second reference frames corresponding to each extraction round: Based on the extraction results of the historical extraction rounds, a second reference frame set corresponding to the current extraction round is obtained, wherein the positions of the reference frames in the second reference frame set of the current round in the target video are located within the frame interval determined based on the adjacent historical reference frames that have been extracted; The reference frame result storage unit is configured to, in response to jumping out after reaching the target round, store the first reference frame set and each of the second reference frame sets corresponding to the target video in a reference frame database, wherein the reference frame database is used to respond to a reference frame retrieval request sent by a business party and provide the business party with a set of reference frame sets for the target video.
10. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.
11. A computer readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 8.
12. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8.
Citation Information
Patent Citations
Adaptive KF (keyframe) extraction method based on sub-lens segmentation
CN102314681A
Streaming media speed multiplication playing method and apparatus
CN106454417A
Video frame extraction methods and device, equipment and medium
CN109618225A
Video storage method and device and computer readable storage medium
CN114245167A
Multimedia retrieval method and device and computer equipment
CN114357203A
