Video acquisition method and device, electronic equipment and computer readable medium
Through video frame extraction and clustering processing of multiple target videos, the problem of inefficient human video slicing is solved, and the method of efficient and accurate acquisition of target video clips is realized, ensuring the integrity and accuracy of video clips.
Patent Information
- Application Number
- CN202411206550.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-30
AI Technical Summary
In the prior art, artificial video slicing is inefficient, resulting in incomplete video clips and difficult to accurately obtain video information of the target object.
By acquiring multiple target videos, video frame extraction process is performed to generate video frame sequences, and frame vector sequences are generated using clustering algorithms, and video clips of the target object are accurately obtained through two clustering processing and feature label generation.
It realizes efficient and accurate acquisition of video clip sets related to target objects from multiple target videos, improving the accuracy and completeness of video clips.
Smart Images

Figure CN118945422A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a video acquisition method, device, electronic device, and computer-readable medium. Background Art
[0002] At present, camera videos play a vital monitoring role in people's daily life. For obtaining the relevant video of the target object, the method usually adopted is: manually slicing the shot video to obtain the video clip including the target object.
[0003] However, when the above method is used to obtain the video clip corresponding to the target object, the following technical problems often occur:
[0004] First, manual video slicing is inefficient and the resulting video segments may not be complete.
[0005] Second, it is crucial to accurately determine the corresponding object recognition information for at least one object vector under the same vector cluster.
[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention
[0007] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.
[0008] Some embodiments of the present disclosure propose a video acquisition method, an apparatus, an electronic device, and a computer-readable medium to solve one or more of the technical problems mentioned in the above background technology section.
[0009] In a first aspect, some embodiments of the present disclosure provide a video acquisition method, comprising: acquiring multiple target videos for a target space, wherein multiple cameras corresponding to the multiple target videos are located at different positions in the target space; performing video frame extraction processing on each of the multiple target videos to generate a video frame sequence to obtain multiple video frame sequences; generating multiple frame vector sequences for the multiple video frame sequences; using a first clustering algorithm, performing clustering processing on each frame vector in the multiple frame vector sequences to generate a first clustering result; based on the first clustering result, performing feature labeling on each frame vector in the multiple frame vector sequences to generate multiple first frame vector sequences with labels added; using a second clustering algorithm, performing clustering processing on each first frame vector in the multiple first frame vector sequences to generate a second clustering result; in response to receiving a video acquisition request for a target object and the second clustering result passing verification, acquiring a set of video clips corresponding to the target object from the multiple target videos based on the second clustering result.
[0010] In a second aspect, some embodiments of the present disclosure provide a video acquisition device, comprising: a first acquisition unit, configured to acquire multiple target videos for a target space, wherein multiple cameras corresponding to the multiple target videos are located at different positions in the target space; a frame extraction unit, configured to perform video frame extraction processing on each of the multiple target videos to generate a video frame sequence to obtain multiple video frame sequences; a first generation unit, configured to generate multiple frame vector sequences for the multiple video frame sequences; a first execution unit, configured to use a first clustering algorithm to perform clustering processing on each frame vector in the multiple frame vector sequences. processing to generate a first clustering result; a labeling unit is configured to feature label each frame vector in the multiple frame vector sequences according to the first clustering result to generate multiple labeled first frame vector sequences; a second execution unit is configured to perform clustering processing on each first frame vector in the multiple first frame vector sequences using a second clustering algorithm to generate a second clustering result; a second acquisition unit is configured to respond to receiving a video acquisition request for a target object and the second clustering result passes verification, and acquire a set of video clips corresponding to the target object from the multiple target videos according to the second clustering result.
[0011] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.
[0012] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner in the first aspect is implemented.
[0013] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the video acquisition method of some embodiments of the present disclosure, a set of video clips related to the target object can be accurately and efficiently acquired from multiple target videos. Specifically, the reason why the acquisition of the relevant video clips is not accurate and efficient is that the efficiency of artificial video slicing is low, and the obtained video clips may not be complete. Based on this, the video acquisition method of some embodiments of the present disclosure first acquires multiple target videos for the target space as the video source of the target object to subsequently acquire the video clips for the target object. Among them, the multiple camera devices corresponding to the above-mentioned multiple target videos are in different positions of the above-mentioned target space. Then, each target video in the above-mentioned multiple target videos is subjected to video frame extraction processing to generate a video frame sequence, and multiple video frame sequences are obtained, so as to facilitate the determination of the video clips from the frame perspective, thereby ensuring the accuracy of the subsequent video clips. Then, multiple frame vector sequences for the above-mentioned multiple video frame sequences can be accurately generated for subsequent clustering processing. Then, using the first clustering algorithm, clustering processing for each frame vector in the above-mentioned multiple frame vector sequences can be accurately performed to generate a first clustering result. Here, through the first clustering algorithm, vectors with a certain feature can be clustered together, so as to facilitate the subsequent determination of the corresponding video clips for the target object. Furthermore, according to the above-mentioned first clustering result, each frame vector in the above-mentioned multiple frame vector sequences is marked with a feature label to generate multiple first frame vector sequences after adding labels, so as to facilitate the subsequent second clustering processing and improve the accuracy of the subsequent video clip acquisition. Further, using the second clustering algorithm, the clustering processing of each first frame vector in the above-mentioned multiple first frame vector sequences can be accurately performed to generate a second clustering result to obtain a label for each video frame, which is convenient for the subsequent video clip acquisition. Finally, in response to receiving a video acquisition request for the target object and the above-mentioned second clustering result passing the verification, according to the above-mentioned second clustering result, the video clip set corresponding to the above-mentioned target object can be accurately obtained from the above-mentioned multiple target videos. In summary, through the two-stage clustering processing, the label set corresponding to each frame image in the multiple target videos can be obtained. Based on this, after receiving a video acquisition request for a target object and the second clustering result passes verification, a set of video clips corresponding to the target object can be efficiently and accurately acquired from the multiple target videos. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.
[0015] Figure 1 is a flow chart of some embodiments of the video acquisition method according to the present disclosure;
[0016] Figure 2 is a schematic structural diagram of some embodiments of the video acquisition device according to the present disclosure;
[0017] Figure 3 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION
[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0019] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.
[0020] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0021] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0022] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0023] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.
[0024] refer to Figure 1, shows a process 100 of some embodiments of the video acquisition method according to the present disclosure. The video acquisition method comprises the following steps:
[0025] Step 101, obtaining multiple target videos for a target space.
[0026] In some embodiments, the execution subject of the above-mentioned video acquisition method can acquire multiple target videos for the target space through a wired connection method or a wireless connection method. Among them, the target space can be a space where video is shot. For example, the target space can be a factory area. There is a one-to-one shooting relationship between the target videos in the multiple target videos and the camera devices in the multiple camera devices. And the multiple camera devices corresponding to the above-mentioned multiple target videos are in different positions of the above-mentioned target space.
[0027] Step 102: Perform video frame extraction processing on each of the multiple target videos to generate a video frame sequence, thereby obtaining multiple video frame sequences.
[0028] In some embodiments, the execution subject may perform video frame extraction processing on each of the multiple target videos to generate a video frame sequence, thereby obtaining multiple video frame sequences.
[0029] As an example, the execution subject may set a frame extraction duration to perform video frame extraction processing on each of the multiple target videos according to the frame extraction duration to generate a video frame sequence, thereby obtaining multiple video frame sequences.
[0030] Step 103: Generate multiple frame vector sequences for the multiple video frame sequences.
[0031] In some embodiments, the execution subject may generate a plurality of frame vector sequences for the plurality of video frame sequences. The frame vector may be information in a vector representation corresponding to at least one video frame.
[0032] In some optional implementations of some embodiments, the generating of the plurality of frame vector sequences for the plurality of video frame sequences may include the following steps:
[0033] The first step is to obtain the frame extraction frequency information corresponding to the above-mentioned multiple target videos. The frame extraction frequency information may be the frequency of extracting frames for the target videos. The frame extraction frequency information may be one frame extracted every 0.5 seconds.
[0034] The second step is to generate the number of frame feature merging according to the above frame extraction frequency information.
[0035] As an example, the execution subject may multiply the number of frame extraction frequencies by a preset value to generate the number of frame feature combinations. The preset value may be a preset value. For example, the preset value may be a value of 2. The preset value may be obtained through relevant experimental data.
[0036] In the third step, at least two adjacent video frames in the above-mentioned multiple video frame sequences are grouped according to the above-mentioned frame feature merging number to generate a video frame group and obtain multiple video frame group sequences. The number of video frames included in the video frame group is the same as the above-mentioned frame feature merging number. For example, the video frame sequence includes: a first video frame, a second video frame, a third video frame, a fourth video frame, a fifth video frame, a sixth video frame, a seventh video frame and an eighth video frame. The frame feature merging number can be 4. Then the corresponding first video frame group can include: a first video frame, a second video frame, a third video frame, and a fourth video frame. The corresponding first video frame group can include: a fifth video frame, a sixth video frame, a seventh video frame and an eighth video frame.
[0037] Step 4: for each video frame group in the plurality of video frame group sequences, perform the following first generation step:
[0038] In the first sub-step, each video frame in the above video frame group is input into a pre-trained image feature extraction model to generate video frame feature information and obtain a video frame feature information group. The image feature extraction model may be a neural network model for extracting image feature information. In practice, the image feature extraction model may be a convolutional neural network model with multiple layers connected in series. The image feature extraction model may be trained based on a conventional model training method.
[0039] The second sub-step is to generate combined feature information for the above-mentioned video frame feature information group as a frame vector corresponding to the above-mentioned video frame group.
[0040] As an example, based on the chronological order, the execution subject may sequentially splice the respective video frame feature information in the video frame feature information group to generate a splicing vector as a frame vector corresponding to the video frame group.
[0041] Step 104: using a first clustering algorithm, perform clustering processing on each frame vector in the plurality of frame vector sequences to generate a first clustering result.
[0042] In some embodiments, the execution subject may use a first clustering algorithm to perform clustering processing on each frame vector in the plurality of frame vector sequences to generate a first clustering result. The first clustering algorithm may be, but is not limited to, at least one of the following: an FCM clustering algorithm, a Kmeans clustering algorithm. The first clustering result may include: a frame vector cluster set and a cluster label corresponding to each frame vector cluster.
[0043] In some optional implementations of some embodiments, the step of using the first clustering algorithm to perform clustering processing on each frame vector in the plurality of frame vector sequences to generate a first clustering result may include the following steps:
[0044] The first step is to extract the object vector corresponding to each frame vector in the above multiple frame vector sequences.
[0045] As an example, the above-mentioned execution subject can input the frame vector into the object vector generation model included in the object information recognition model to generate the object vector. The object information recognition model can be a neural network that generates object recognition information. The object recognition information can include: object type and object position. The object vector generation model can be a neural network model that generates object vectors. The object vector can be information in the form of a vector that characterizes the semantics related to the object in the frame image. In practice, the object vector generation model can be a residual model with multiple layers connected in series.
[0046] In the second step, the first clustering algorithm is used to perform clustering processing on each object vector in the obtained plurality of object vector sequences to generate an object vector cluster set, wherein each object vector cluster in the object vector cluster set has corresponding at least one object information.
[0047] Step 3: For each object vector cluster in the above object vector cluster set, perform the following second generation step:
[0048] The first sub-step is to determine at least one object vector centered at the corresponding cluster center of the object vector cluster and having a corresponding vector distance less than a target value. The vector distance may be a cosine vector distance. The target value may be a preset value. For example, the target value may be 3.2.
[0049] The second sub-step is to determine the distance between the at least one object vector and the cluster center to obtain at least one distance. There is a one-to-one correspondence between the object vector in the at least one object vector and the distance in the at least one distance. The distance in the at least one distance is a cosine distance.
[0050] The third sub-step is to input the at least one object vector and the at least one distance into an object recognition information generation model to generate object recognition information. The object recognition information generation model may be a neural network model for generating object recognition information. In practice, the object recognition information generation model may be a YOLO model. The object recognition information may include: an object identifier.
[0051] The fourth sub-step is to determine the object identification information as a cluster label corresponding to the object vector cluster.
[0052] The fourth step is to generate a first clustering result based on the obtained object vector cluster set and the corresponding cluster label set.
[0053] As an example, the execution subject determines the correspondence between the object vector cluster set and the cluster label set as the first clustering result.
[0054] Optionally, inputting the at least one object vector and the at least one distance into an object recognition information generation model to generate object recognition information comprises the following steps:
[0055] In the first step, each object vector in the at least one object vector is input into the object recognition information generation model to generate initial object recognition information, thereby obtaining at least one initial object recognition information. The initial object recognition information includes: an object category probability set corresponding to an object category set, wherein there is a one-to-one correspondence between the object category in the object category set and the object category probability in the object category probability set. The object category probability can represent the probability that the object category corresponding to the object vector is the corresponding object category. The larger the object category probability, the greater the probability of representing the corresponding object category.
[0056] Step 2. For each object vector in the at least one object vector, determine the distance weight of the distance corresponding to the object vector to obtain at least one distance weight. The distance weight can represent the importance of the corresponding feature of the corresponding vector.
[0057] As an example, the execution subject may determine the distance weight corresponding to the distance of the object vector by querying an association table to obtain at least one distance weight, wherein the association table may represent the corresponding relationship between the distance and the distance weight.
[0058] In the third step, for each object category in the at least one object category, the following first information generation step is performed:
[0059] In the first step, at least one object category probability corresponding to the object category is determined, wherein an object category probability in the at least one object category probability and an object vector in the at least one object vector have a one-to-one correspondence.
[0060] In the second step, weighted summation is performed on the at least one object category probability and the at least one distance weight to generate a weighted sum value.
[0061] In the fourth step, object categories whose corresponding weighted sum values are within the target number are selected from at least one object category to obtain an object category set.
[0062] The fifth step is to sort the object categories in the object category set according to the weighted sum value to obtain the object category sequence.
[0063] The sixth step is to determine an object vector cluster corresponding to the at least one object vector as a target object vector cluster.
[0064] The seventh step is to determine the category vector corresponding to each object category in the above object category set to obtain a category vector set.
[0065] Step 8: For each category vector in the category vector set, perform the following second information generation step:
[0066] The first sub-step is to determine the vector distance between the target object vector in the target object vector cluster and the above category vector to obtain a vector distance set.
[0067] In the second sub-step, according to the object category position in the object category sequence, the object category position corresponding to the category vector is determined as the target object category position.
[0068] The third sub-step is to determine a preset distance weight corresponding to the target object category position as the target distance weight.
[0069] In a fourth sub-step, a vector distance whose corresponding vector distance is less than a preset vector distance is selected from the vector distance set as a target vector distance, thereby obtaining at least one target vector distance.
[0070] The fifth sub-step is to add each target vector distance in the at least one target vector distance to obtain a sum value.
[0071] The sixth sub-step is to multiply the added value by the target distance weight to obtain a multiplied value.
[0072] In the ninth step, the object category with the largest corresponding multiplication distance is selected from the object category set as the object recognition information.
[0073] The above-mentioned "optionally", as an inventive point of the present disclosure, solves the technical problem mentioned in the background technology: "It is crucial to determine the corresponding object recognition information accurately for at least one object vector under the same vector cluster." Based on this, the present disclosure first determines the most likely object category sequence through the object category probability, and then, on this basis, accurately determines the object recognition information through the vector distance between the object category and the object vector.
[0074] Optionally, the object recognition information generation model includes: an attention mechanism model based on a first number of convolutional layers, an attention mechanism model based on a second number of convolutional layers, an attention mechanism model based on a third number of convolutional layers, and an output layer, wherein the first number is greater than the second number, the second number is greater than the third number, the output weight corresponding to the attention mechanism model based on the first number of convolutional layers is greater than the output weight corresponding to the attention mechanism model based on the second number of convolutional layers, the output weight corresponding to the attention mechanism model based on the second number of convolutional layers is greater than the output weight corresponding to the attention mechanism model based on the third number of convolutional layers, the vector level corresponding to the attention mechanism model based on the first number of convolutional layers is the first level, the vector level corresponding to the attention mechanism model based on the second number of convolutional layers is the second level, and the vector level corresponding to the attention mechanism model based on the third number of convolutional layers is the third level. For example, the first number may be 20. The second number may be 15. The third number may be 10. The attention mechanism model based on the first number of convolutional layers may include: 20 layers of convolutional layers connected in series + a multi-head attention mechanism model. The attention mechanism model based on the second number of convolutional layers may include: 15 layers of convolutional layers connected in series + a multi-head attention mechanism model. The attention mechanism model based on the third number of convolutional layers may include: 10 layers of convolutional layers connected in series + a multi-head attention mechanism model. For example, the output weight corresponding to the attention mechanism model based on the first number of convolutional layers may be 0.5. The output weight corresponding to the attention mechanism model based on the second number of convolutional layers may be 0.3. The output weight corresponding to the attention mechanism model based on the third number of convolutional layers may be 0.2. The output layer may be a fully connected layer.
[0075] Optionally, the step of inputting the at least one object vector and the at least one distance into an object recognition information generation model to generate object recognition information may include the following steps:
[0076] In the first step, according to the at least one distance, the at least one object vector is divided into vector levels to generate an object vector group set. Each object vector group has a corresponding vector level, and each vector level has a corresponding distance range. For vector levels of the first level, the second level, and the third level, the corresponding object vector group set includes: an object vector group corresponding to the first level, an object vector group corresponding to the second level, and an object vector group corresponding to the third level.
[0077] In the second step, for each object vector group in the above object vector group set, the following fifth generation step is performed:
[0078] The first sub-step is to execute the following sixth generation step for each object vector in the above object vector group:
[0079] Sub-step 1: determine at least one object vector centered on the object vector and having a vector distance with the object vector less than a predetermined distance as at least one target object vector. For example, the predetermined distance may be 1.5.
[0080] Sub-step 2: generating, based on the at least one target object vector and the object vector, a label object vector included in the object vector and corresponding to a cluster label corresponding to the object vector cluster.
[0081] As an example, first, the execution subject may perform vector fusion on at least one target object vector and an object vector to generate a vector set, and then input the vector set into an object feature extraction model to generate an object feature vector as a label object vector.
[0082] In the second sub-step, the vector level corresponding to the object vector group is determined as the target vector level.
[0083] The third sub-step, in response to determining that the target vector level is the first level, inputs the label object vector group corresponding to the object vector group into the attention mechanism model based on the first number of convolutional layers to generate a first attention vector.
[0084] The fourth sub-step is to multiply the first attention vector by the output weight corresponding to the attention mechanism model based on the first number of convolutional layers to obtain a multiplication result.
[0085] The fifth sub-step, in response to determining that the target vector level is the second level, inputs the label object vector group corresponding to the object vector group into the attention mechanism model based on the second number of convolutional layers to generate a second attention vector.
[0086] The sixth sub-step is to multiply the second attention vector by the output weight corresponding to the attention mechanism model based on the second number of convolutional layers to obtain a multiplication result.
[0087] The seventh sub-step, in response to determining that the target vector level is the third level, inputs the label object vector group corresponding to the object vector group into the attention mechanism model based on the third number of convolutional layers to generate a third attention vector.
[0088] In an eighth sub-step, the third attention vector is multiplied by the output weight corresponding to the attention mechanism model based on the third number of convolutional layers to obtain a multiplication result.
[0089] The third step is to add up the multiplication results in the obtained multiplication result set to obtain the addition result.
[0090] Fourthly, the addition result is input to the output layer to generate the object recognition information.
[0091] Step 105 , based on the first clustering result, feature labels are added to each frame vector in the plurality of frame vector sequences to generate a plurality of labeled first frame vector sequences.
[0092] In some embodiments, the execution subject may perform feature labeling on each frame vector in the plurality of frame vector sequences according to the first clustering result to generate a plurality of first frame vector sequences after labeling, wherein the label corresponding to the first frame vector after labeling is the cluster label corresponding to the corresponding object vector cluster.
[0093] Step 106: Use a second clustering algorithm to perform clustering processing on each first frame vector in the plurality of first frame vector sequences to generate a second clustering result.
[0094] In some embodiments, the execution entity may utilize a second clustering algorithm to perform clustering processing on each first frame vector in the plurality of first frame vector sequences to generate a second clustering result. The second clustering result may include: a cluster set of re-clustered frame vectors and a cluster label corresponding to each re-clustered frame vector cluster. In practice, the second clustering algorithm may be the same clustering algorithm as the first clustering algorithm, or may be a different clustering algorithm. For example, the second clustering algorithm may be, but is not limited to, at least one of the following: an FCM clustering algorithm, a Kmeans clustering algorithm.
[0095] In some optional implementations of some embodiments, after step 106, the steps further include:
[0096] In the first step, the second clustering algorithm is used to perform clustering processing on each first frame vector in the plurality of first frame vector sequences to generate a third clustering result.
[0097] In the second step, according to the third clustering result, each first frame vector in the plurality of first frame vector sequences is feature-labeled to generate a plurality of second frame vector sequences after labeling.
[0098] As an example, first, the execution subject may determine the feature label corresponding to each cluster center in the cluster center set according to the third clustering result, and obtain a feature label set. According to the feature label set, each first frame vector in the plurality of first frame vector sequences is feature labeled to generate a plurality of second frame vector sequences after adding labels.
[0099] The third step is to use the first clustering algorithm to perform clustering processing on each second frame vector in the plurality of second frame vector sequences to generate a fourth clustering result.
[0100] The feature labels corresponding to each cluster in the cluster set corresponding to the fourth clustering result are the same. The same target feature label corresponding to each cluster is obtained by marking the feature labels.
[0101] In the fourth step, a third clustering algorithm is used to perform clustering processing on each frame vector in the plurality of frame vector sequences to generate a fifth clustering result. The fusion clustering algorithm is a clustering algorithm different from the first clustering algorithm and the second clustering algorithm. For example, the third clustering algorithm may be a DBSCAN algorithm.
[0102] The fifth step is to verify the second clustering result based on the fourth clustering result and the fifth clustering result to generate a verification result.
[0103] As an example, the execution subject may determine whether the clustering situation corresponding to the second clustering result is significantly different from the clustering situation corresponding to the fourth clustering result and the clustering situation corresponding to the fifth clustering result. If the difference is significant, a verification result indicating that the verification has not been passed is generated. If the difference is minor, a verification result indicating that the verification has been passed is generated.
[0104] Step 107 , in response to receiving a video acquisition request for the target object and the second clustering result passing verification, acquiring a set of video segments corresponding to the target object from the multiple target videos according to the second clustering result.
[0105] In some embodiments, in response to receiving a video acquisition request for a target object and the second clustering result passing verification, the execution subject may acquire a video segment set corresponding to the target object from the plurality of target videos according to the second clustering result. The video segments in the video segment set correspond to display objects including the target object. The video acquisition request may be a request to acquire a video segment corresponding to the target object.
[0106] In some optional implementations of some embodiments, after step 107, the steps further include:
[0107] In the first step, according to the time information corresponding to each video segment in the video segment set, each video segment in the video segment set is time-arranged to generate an arranged video. Each video frame in the arranged video is arranged in a time sequence.
[0108] The second step is to set a video tag for the above-placed video. The video tag may be an object tag corresponding to the target object. For example, the video tag is "video clip of the target object".
[0109] The third step is to store the above-arranged videos and the above-mentioned video tags in a target server correspondingly, wherein the target server may be a pre-deployed server.
[0110] In some optional implementations of some embodiments, the step of acquiring a set of video segments corresponding to the target object from the plurality of target videos according to the second clustering result may include the following steps:
[0111] In the first step, according to the second clustering result, each first frame vector in the plurality of labeled first frame vector sequences is labeled to generate a plurality of labeled third frame vector sequences, wherein the characteristic labels of the third frame vectors are cluster labels corresponding to the second clustering result.
[0112] In the second step, for each third frame vector in the plurality of third frame vector sequences, the following fourth generation step is performed:
[0113] The first sub-step is to determine the video frame group corresponding to the third frame vector as the target video frame group.
[0114] The second sub-step is to determine at least one feature label corresponding to the third frame vector as at least one feature label corresponding to the target video frame group.
[0115] In the third step, video frames whose corresponding feature labels are the corresponding object labels of the target objects are selected from the multiple target videos to obtain multiple video frame sets.
[0116] In the fourth step, each video frame in the above-mentioned multiple video frame sets is integrated to generate a video segment set.
[0117] As an example, the execution subject may integrate the video frames in the plurality of video frame sets according to a temporal sequence to generate a video segment set.
[0118] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the video acquisition method of some embodiments of the present disclosure, a set of video clips related to the target object can be accurately and efficiently acquired from multiple target videos. Specifically, the reason why the acquisition of the relevant video clips is not accurate and efficient is that the efficiency of artificial video slicing is low, and the obtained video clips may not be complete. Based on this, the video acquisition method of some embodiments of the present disclosure first acquires multiple target videos for the target space as the video source of the target object to subsequently acquire the video clips for the target object. Among them, the multiple camera devices corresponding to the above-mentioned multiple target videos are in different positions of the above-mentioned target space. Then, each target video in the above-mentioned multiple target videos is subjected to video frame extraction processing to generate a video frame sequence, and multiple video frame sequences are obtained, so as to facilitate the determination of the video clips from the frame perspective, thereby ensuring the accuracy of the subsequent video clips. Then, multiple frame vector sequences for the above-mentioned multiple video frame sequences can be accurately generated for subsequent clustering processing. Then, using the first clustering algorithm, clustering processing for each frame vector in the above-mentioned multiple frame vector sequences can be accurately performed to generate a first clustering result. Here, through the first clustering algorithm, vectors with a certain feature can be clustered together, so as to facilitate the subsequent determination of the corresponding video clips for the target object. Furthermore, according to the above-mentioned first clustering result, each frame vector in the above-mentioned multiple frame vector sequences is marked with a feature label to generate multiple first frame vector sequences after adding labels, so as to facilitate the subsequent second clustering processing and improve the accuracy of the subsequent video clip acquisition. Further, using the second clustering algorithm, the clustering processing of each first frame vector in the above-mentioned multiple first frame vector sequences can be accurately performed to generate a second clustering result to obtain a label for each video frame, which is convenient for the subsequent video clip acquisition. Finally, in response to receiving a video acquisition request for the target object and the above-mentioned second clustering result passing the verification, according to the above-mentioned second clustering result, the video clip set corresponding to the above-mentioned target object can be accurately obtained from the above-mentioned multiple target videos. In summary, through the two-stage clustering processing, the label set corresponding to each frame image in the multiple target videos can be obtained. Based on this, after receiving a video acquisition request for a target object and the second clustering result passes verification, a set of video clips corresponding to the target object can be efficiently and accurately acquired from the multiple target videos.
[0119] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a video acquisition device, and these device embodiments are Figure 1 Corresponding to the method embodiments shown, the video acquisition device can be specifically applied to various electronic devices.
[0120] like Figure 2As shown, a video acquisition device 200 includes: a first acquisition unit 201, a frame extraction unit 202, a first generation unit 203, a first execution unit 204, a labeling unit 205, a second execution unit 206 and a second acquisition unit 207. The first acquisition unit 201 is configured to acquire multiple target videos for a target space, wherein multiple cameras corresponding to the multiple target videos are located at different positions in the target space; the frame extraction unit 202 is configured to perform video frame extraction processing on each of the multiple target videos to generate a video frame sequence to obtain multiple video frame sequences; the first generation unit 203 is configured to generate multiple frame vector sequences for the multiple video frame sequences; the first execution unit 204 is configured to perform clustering processing on each frame vector in the multiple frame vector sequences using a first clustering algorithm to generate a first clustering result; The labeling unit 205 is configured to label each frame vector in the above-mentioned multiple frame vector sequences according to the above-mentioned first clustering result, so as to generate multiple labeled first frame vector sequences; the second execution unit 206 is configured to use a second clustering algorithm to perform clustering processing on each first frame vector in the above-mentioned multiple first frame vector sequences, so as to generate a second clustering result; the second acquisition unit 207 is configured to respond to receiving a video acquisition request for a target object and the above-mentioned second clustering result passes verification, and according to the above-mentioned second clustering result, acquire a set of video clips corresponding to the above-mentioned target object from the above-mentioned multiple target videos.
[0121] It is understandable that the units described in the video acquisition device 200 are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the video acquisition device 200 and the units included therein, and will not be described in detail here.
[0122] Reference below Figure 3 , which shows a structural schematic diagram of an electronic device (eg, an electronic device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0123] like Figure 3As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.
[0124] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.
[0125] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.
[0126] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0127] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.
[0128] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains multiple target videos for the target space, wherein the multiple cameras corresponding to the multiple target videos are located at different positions in the target space; performs video frame extraction processing on each of the multiple target videos to generate a video frame sequence to obtain multiple video frame sequences; generates multiple frame vector sequences for the multiple video frame sequences; uses a first clustering algorithm to perform clustering processing on each frame vector in the multiple frame vector sequences to generate a first clustering result; based on the first clustering result, features are labeled on each frame vector in the multiple frame vector sequences to generate multiple first frame vector sequences with labels added; uses a second clustering algorithm to perform clustering processing on each first frame vector in the multiple first frame vector sequences to generate a second clustering result; in response to receiving a video acquisition request for a target object and the second clustering result passes verification, obtains a set of video clips corresponding to the target object from the multiple target videos based on the second clustering result.
[0129] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0130] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0131] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor including a first acquisition unit, a frame extraction unit, a first generation unit, a first execution unit, a labeling unit, a second execution unit, and a second acquisition unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the first acquisition unit may also be described as a "unit for acquiring multiple target videos for a target space".
[0132] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0133] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.
Claims
1. A video acquisition method, comprising: Acquire a plurality of target videos for a target space, wherein a plurality of camera devices corresponding to the plurality of target videos are located at different positions of the target space; Performing video frame extraction processing on each target video of the multiple target videos to generate a video frame sequence, thereby obtaining multiple video frame sequences; generating a plurality of frame vector sequences for the plurality of video frame sequences; Using a first clustering algorithm, performing clustering processing on each frame vector in the plurality of frame vector sequences to generate a first clustering result; According to the first clustering result, each frame vector in the plurality of frame vector sequences is subjected to a feature label to generate a plurality of first frame vector sequences after adding labels; Using a second clustering algorithm, performing clustering processing on each first frame vector in the plurality of first frame vector sequences to generate a second clustering result; In response to receiving a video acquisition request for a target object and the second clustering result passing verification, a set of video segments corresponding to the target object is acquired from the multiple target videos according to the second clustering result.
2. The method according to claim 1, wherein: The method further comprises: According to the time information corresponding to each video segment in the video segment set, performing time placement processing on each video segment in the video segment set to generate a placed video; Setting a video tag for the placed video; The arranged videos and the video tags are stored correspondingly on the target server.
3. The method according to claim 2, wherein: The generating a plurality of frame vector sequences for the plurality of video frame sequences comprises: Obtaining frame extraction frequency information corresponding to the multiple target videos; Generate a frame feature merging number according to the frame extraction frequency information; According to the combined number of frame features, at least two adjacent video frames in the plurality of video frame sequences are grouped to generate a video frame group, thereby obtaining a plurality of video frame group sequences, wherein the number of video frames included in the video frame group is the same as the combined number of frame features; For each video frame group in the plurality of video frame group sequences, the following first generating step is performed: Inputting each video frame in the video frame group into a pre-trained image feature extraction model to generate video frame feature information to obtain a video frame feature information group; Generate combined feature information for the video frame feature information group as a frame vector corresponding to the video frame group.
4. The method according to claim 3, wherein: The using a first clustering algorithm to perform clustering processing on each frame vector in the plurality of frame vector sequences to generate a first clustering result includes: For each frame vector in the plurality of frame vector sequences, extracting an object vector corresponding to the frame vector; Using a first clustering algorithm, clustering each object vector in the obtained plurality of object vector sequences is performed to generate an object vector cluster set; For each object vector cluster in the object vector cluster set, the following second generation step is performed: Determine at least one object vector centered at the corresponding cluster center of the object vector cluster and having a corresponding vector distance less than a target value; determining a distance between the at least one object vector and the cluster center to obtain at least one distance; inputting the at least one object vector and the at least one distance into an object recognition information generation model to generate object recognition information; The object identification information is determined as a cluster label corresponding to the object vector cluster; and a first clustering result is generated according to the obtained object vector cluster set and the corresponding cluster label set.
5. The method according to claim 4, wherein: After performing clustering processing on each first frame vector in the plurality of first frame vector sequences by using the second clustering algorithm to generate a second clustering result, the method further includes: Using the second clustering algorithm, performing clustering processing on each first frame vector in the plurality of first frame vector sequences to generate a third clustering result; According to the third clustering result, each first frame vector in the plurality of first frame vector sequences is subjected to a feature label to generate a plurality of second frame vector sequences after adding the label; Using the first clustering algorithm, perform clustering processing on each second frame vector in the plurality of second frame vector sequences to generate a fourth clustering result; Using a fusion clustering algorithm, performing clustering processing on each frame vector in the plurality of frame vector sequences to generate a fifth clustering result, wherein the fusion clustering algorithm is a clustering algorithm different from the first clustering algorithm and the second clustering algorithm; According to the fourth clustering result and the fifth clustering result, a result verification is performed on the second clustering result to generate a verification result.
6. The method according to claim 5, wherein: The acquiring, according to the second clustering result, a set of video segments corresponding to the target object from the multiple target videos comprises: According to the second clustering result, each first frame vector in the plurality of labeled first frame vector sequences is subjected to a feature label to generate a plurality of labeled third frame vector sequences; For each third frame vector in the plurality of third frame vector sequences, the following fourth generating step is performed: Determine a video frame group corresponding to the third frame vector as a target video frame group; Determining at least one feature label corresponding to the third frame vector as at least one feature label corresponding to the target video frame group; Filter out video frames whose corresponding feature labels are object labels corresponding to the target objects from the multiple target videos to obtain multiple video frame sets; The video frames in the plurality of video frame sets are integrated to generate a video segment set.
7. The method according to claim 6, wherein: The object recognition information generation model includes: an attention mechanism model based on a first number of convolutional layers, an attention mechanism model based on a second number of convolutional layers, an attention mechanism model based on a third number of convolutional layers, and an output layer, wherein the first number is greater than the second number, the second number is greater than the third number, the output weight corresponding to the attention mechanism model based on the first number of convolutional layers is greater than the output weight corresponding to the attention mechanism model based on the second number of convolutional layers, the output weight corresponding to the attention mechanism model based on the second number of convolutional layers is greater than the output weight corresponding to the attention mechanism model based on the third number of convolutional layers, the vector level corresponding to the attention mechanism model based on the first number of convolutional layers is a first level, the vector level corresponding to the attention mechanism model based on the second number of convolutional layers is a second level, and the vector level corresponding to the attention mechanism model based on the third number of convolutional layers is a third level; and The step of inputting the at least one object vector and the at least one distance into an object recognition information generation model to generate object recognition information comprises: According to the at least one distance, the at least one object vector is divided into vector levels to generate a set of object vector groups, wherein each object vector group has a corresponding vector level, and each vector level has a corresponding distance range; For each object vector group in the object vector group set, the following fifth generation step is performed: For each object vector in the object vector group, the following sixth generation step is performed: determining at least one object vector centered on the object vector and having a vector distance with the object vector less than a predetermined distance as at least one target object vector; Generate, according to the at least one target object vector and the object vector, a label object vector included in the object vector and corresponding to a cluster label corresponding to the object vector cluster; Determine a vector level corresponding to the object vector group as a target vector level; In response to determining that the target vector level is the first level, inputting the label object vector group corresponding to the object vector group into the attention mechanism model based on the first number of convolutional layers to generate a first attention vector; Multiplying the first attention vector by the output weight corresponding to the attention mechanism model based on the first number of convolutional layers to obtain a multiplication result; In response to determining that the target vector level is the second level, inputting the label object vector group corresponding to the object vector group into the attention mechanism model based on the second number of convolutional layers to generate a second attention vector; Multiplying the second attention vector by the output weight corresponding to the attention mechanism model based on the second number of convolutional layers to obtain a multiplication result; In response to determining that the target vector level is the third level, inputting the label object vector group corresponding to the object vector group into the attention mechanism model based on the third number of convolutional layers to generate a third attention vector; Multiplying the third attention vector by the output weight corresponding to the attention mechanism model based on the third number of convolutional layers to obtain a multiplication result; Adding each multiplication result in the obtained multiplication result set to obtain an addition result; The addition result is input to the output layer to generate the object recognition information.
8. A video acquisition device, comprising: A first acquisition unit is configured to acquire a plurality of target videos for a target space, wherein a plurality of camera devices corresponding to the plurality of target videos are located at different positions of the target space; A frame extraction unit is configured to perform a video frame extraction process on each target video of the multiple target videos to generate a video frame sequence, thereby obtaining multiple video frame sequences; A first generating unit, configured to generate a plurality of frame vector sequences for the plurality of video frame sequences; A first execution unit is configured to perform clustering processing on each frame vector in the plurality of frame vector sequences using a first clustering algorithm to generate a first clustering result; a labeling unit, configured to label each frame vector in the plurality of frame vector sequences with a feature label according to the first clustering result, so as to generate a plurality of first frame vector sequences after labeling; A second execution unit is configured to perform clustering processing on each first frame vector in the plurality of first frame vector sequences using a second clustering algorithm to generate a second clustering result; The second acquisition unit is configured to, in response to receiving a video acquisition request for a target object and the second clustering result passing verification, acquire a set of video segments corresponding to the target object from the multiple target videos according to the second clustering result.
9. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.
10. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Object clustering method and device, computer readable medium and electronic equipment
CN111667018A
Subtitle tracking method and device and electronic equipment
CN112954455A
Video recommendation method, system and device
CN112989116A
Multi-video abstract generation method, system and device and storage medium
CN114880519A
Neural networks implementing attention over object embeddings for object-centric visual reasoning
US20240020972A1
