Video abstract generation method for open-air non-coal mine and related equipment

Abstract

The invention relates to the technical field of video abstract generation, and provides a video abstract generation method for an open-air non-coal mine and related equipment. The video abstract generation method comprises the steps of performing feature extraction on a target video to obtain a time feature, a spatial feature and an optical flow feature, and performing feature enhancement by using a feature enhancement model to obtain a final time feature, a final spatial feature and a final optical flow feature; generating a video abstract by using an abstract generation model according to the final time feature, the final space feature and the final optical flow feature; constructing a loss function based on all video abstracts, and optimizing the feature enhancement model and the abstract generation model by using the loss function; and obtaining a current video corresponding to the to-be-processed non-coal mine, and generating a final video abstract corresponding to the to-be-processed non-coal mine based on the current video by using the optimized feature enhancement model and the optimized abstract generation model. According to the video abstract generation method, the accuracy of the video abstract can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of video summary generation, and in particular to a video summary generation method and related equipment for an open-pit non-coal mine. Background Art

[0002] In the field of open-pit non-coal mines where intelligence and automation are constantly improving, video surveillance has become one of the key safety management tools. The amount of video data in the open-pit non-coal mine environment is huge, which not only contains real-time monitoring images, but also records every detail of the operation process. However, with the surge in the amount of video data, how to quickly extract valuable information from massive video content has become a major challenge to improve the safety management efficiency of open-pit non-coal mines. Users need a tool that can quickly identify key events and generate summaries to assist in rapid decision-making and response.

[0003] At present, video summarization technology is increasingly used in the field of open-pit non-coal mines. Its goal is to provide mining engineers and safety supervisors with a time-saving and information-rich video overview by screening and summarizing the most critical and meaningful parts of the video. Existing video summarization technologies are mainly divided into traditional image processing technology and deep learning-based methods. Video summarization technologies based on traditional methods include keyframe-based methods, shot switching-based methods, content analysis-based methods, motion analysis-based methods, and clustering-based methods. Keyframe-based methods select visual features such as inter-frame changes, color, and texture. However, they cannot fully express the dynamic changes and time series information in the video. The shot switching-based method is also an effective technology, but this method is prone to missing important content or selecting irrelevant content, and may not effectively capture complex changes in the lens. When using content analysis-based methods for video summarization, because of their limited understanding of content semantics and high reliance on the accuracy of feature extraction, they cannot work effectively for complex scenes. The motion analysis-based method is limited by the moving targets and trajectories in the video, and the method has certain limitations. When these methods are applied to video summarization of open-pit non-coal mines, they can capture the static and dynamic features of the video to a certain extent, but they are often difficult to adapt to the complex and changeable environment of open-pit non-coal mines, and have limited deep semantic understanding of the video content. At the same time, most of them rely on manual features and cannot capture the complexity and diversity of video content.

[0004] For the above reasons, there are many methods for video summarization. Most traditional methods rely on manual features, which are difficult to fully capture the complexity of video content; most deep learning methods rely on single features or dual features, and the selected features cannot fully capture the diversity of video content. Whether it is traditional methods or deep learning methods, they often rely on a single or limited feature set when processing video summarization, which limits the model's ability to capture the complexity and diversity of video content. It can be seen that the current video summary generation method has the problem of low accuracy of video summarization. Summary of the invention

[0005] The present application provides a method for generating a video summary of an open-pit non-coal mine and related equipment, which can solve the problem of low accuracy of the video summary.

[0006] In a first aspect, an embodiment of the present application provides a method for generating a video summary of an open-pit non-coal mine, the method comprising:

[0007] Acquire multiple target videos corresponding to each target non-coal mine;

[0008] For each target video, feature extraction is performed on the target video to obtain the temporal feature, spatial feature and optical flow feature of the target video, and the feature enhancement model is used to enhance the temporal feature, spatial feature and optical flow feature to obtain the final temporal feature, final spatial feature and final optical flow feature of the target video; the feature enhancement model includes a first feature enhancement sub-model and a second feature enhancement sub-model connected in sequence, the first feature enhancement sub-model and the second feature enhancement sub-model each include a feature enhancement sub-model, the feature enhancement sub-model includes a first enhancement unit, a second enhancement unit, a first convolution unit, a third enhancement unit, a second convolution unit, and a fourth enhancement unit connected in sequence, the input end of the first enhancement unit is the input end of the feature enhancement sub-model, the output end of the fourth enhancement unit is the output end of the feature enhancement sub-model, the first enhancement unit, the second enhancement unit, the third enhancement unit, and the fourth enhancement unit each include an enhancement unit, the enhancement unit includes a convolution sub-unit, a dense sub-unit, a pooling sub-unit and an attention sub-unit connected in sequence, the input end of the convolution sub-unit is the input end of the enhancement unit, and the output end of the attention sub-unit is the output end of the enhancement unit;

[0009] Generate a video summary of each target video using a summary generation model according to the final temporal features, final spatial features, and final optical flow features of each target video;

[0010] A loss function is constructed based on all video summaries, and the feature enhancement model and the summary generation model are optimized using the loss function to obtain an optimized feature enhancement model and an optimized summary generation model; the loss function is used to describe the accuracy of all video summaries;

[0011] The current video corresponding to the non-coal mine to be processed is obtained, and the final video summary corresponding to the non-coal mine to be processed is generated based on the current video by using the optimized feature enhancement model and the optimized summary generation model.

[0012] Optionally, the feature enhancement model is used to enhance the temporal features, spatial features, and optical flow features respectively to obtain the final temporal features, final spatial features, and final optical flow features of the target video, including:

[0013] By formula:

[0014]

[0015] Get the final features of the target video

[0016] Among them, when l = S, Represents the final spatial features of the target video. When l = T, represents the final spatial features of the target video. When l = F, Represents the final optical flow features of the target video, represents the operation of the feature enhancement model, represents the operation of the first feature enhancer model, represents the operation of the second feature enhancer model, P l represents the initial features of the target video. When l = S, P S represents the spatial features of the target video. When l = T, P T represents the temporal characteristics of the target video. When l = F, P F Represents the optical flow features of the target video.

[0017] Optionally, the attention subunit includes a global average pooling layer, a spatial attention layer, a temporal attention layer, a first addition layer, and a second addition layer;

[0018] The input end of the global average pooling layer and the input end of the first addition layer are both the input ends of the attention sub-unit, and the output end of the second addition layer is the output end of the attention sub-unit;

[0019] The output end of the global average pooling layer is connected to the input end of the spatial attention layer and the input end of the temporal attention layer respectively. The output end of the temporal attention layer is connected to the input end of the first addition layer. The output end of the spatial attention layer is connected to the input end of the second addition layer. The output end of the first addition layer is connected to the input end of the second addition layer.

[0020] Optionally, the spatial attention layer includes a spatial pooling block, a first dimensionality reduction block, a first fully connected block, a first activation function block, and a first conversion block connected in sequence;

[0021] The input end of the spatial pooling block is the input end of the spatial attention layer, and the output end of the first conversion block is the output end of the spatial attention layer;

[0022] The temporal attention layer includes a temporal pooling block, a second dimensionality reduction block, a second fully connected block, a second activation function block, and a second conversion block connected in sequence;

[0023] The input end of the temporal pooling block is the input end of the temporal attention layer, and the output end of the second conversion block is the output end of the temporal attention layer.

[0024] Optionally, the summary generation model includes an addition sub-model, a first linear sub-model, a first activation sub-model, a regularization sub-model, a standardization sub-model, a second linear sub-model, and a second activation sub-model connected in sequence;

[0025] The input end of the addition sub-model is the input end of the summary generation model, and the output end of the second activation sub-model is the output end of the summary generation model.

[0026] Optionally, the loss function is:

[0027] Loss CE =-C label logC out -(1-C label )log(1-C out )

[0028] Among them, Loss CE represents the value of the loss function, C label represents the set of true summaries of all target videos, C out Represents the collection of video summaries for all target videos.

[0029] Optionally, a loss function is constructed based on all video summaries, and the feature enhancement model and the summary generation model are optimized using the loss function to obtain an optimized feature enhancement model and an optimized summary generation model, including:

[0030] Determine whether the value of the loss function is less than the preset value of the loss function;

[0031] If yes, the feature enhancement model is used as the optimized feature enhancement model, and the summary generation model is used as the optimized summary generation model;

[0032] Otherwise, adjust the parameters in the feature enhancement model and the summary generation model, and return to the step of extracting features of the target video for each target video to obtain the temporal features, spatial features, and optical flow features of the target video, and use the feature enhancement model to enhance the temporal features, spatial features, and optical flow features to obtain the final temporal features, final spatial features, and final optical flow features of the target video.

[0033] In a second aspect, an embodiment of the present application provides a video summary generation device for an open-pit non-coal mine, comprising:

[0034] An acquisition module, used for acquiring multiple target videos corresponding to each target non-coal mine;

[0035] A feature extraction module is used to extract features of each target video, obtain the temporal features, spatial features and optical flow features of the target video, and use a feature enhancement model to enhance the temporal features, spatial features and optical flow features, respectively, to obtain the final temporal features, final spatial features and final optical flow features of the target video; the feature enhancement model includes a first feature enhancement sub-model and a second feature enhancement sub-model connected in sequence, the first feature enhancement sub-model and the second feature enhancement sub-model each include a feature enhancement sub-model, the feature enhancement sub-model includes a first enhancement unit, a second enhancement unit, a first convolution unit, a third enhancement unit, a second convolution unit, and a fourth enhancement unit connected in sequence, the input end of the first enhancement unit is the input end of the feature enhancement sub-model, the output end of the fourth enhancement unit is the output end of the feature enhancement sub-model, the first enhancement unit, the second enhancement unit, the third enhancement unit, and the fourth enhancement unit each include an enhancement unit, the enhancement unit includes a convolution sub-unit, a dense sub-unit, a pooling sub-unit and an attention sub-unit connected in sequence, the input end of the convolution sub-unit is the input end of the enhancement unit, and the output end of the attention sub-unit is the output end of the enhancement unit;

[0036] A first generation module is used to generate a video summary of each target video using a summary generation model according to a final temporal feature, a final spatial feature, and a final optical flow feature of each target video;

[0037] An optimization module is used to construct a loss function based on all video summaries, and use the loss function to optimize the feature enhancement model and the summary generation model to obtain an optimized feature enhancement model and an optimized summary generation model; the loss function is used to describe the accuracy of all video summaries;

[0038] The second generation module is used to obtain the current video corresponding to the non-coal mine to be processed, and use the optimized feature enhancement model and the optimized summary generation model to generate a final video summary corresponding to the non-coal mine to be processed based on the current video.

[0039] In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for generating a video summary of an open-pit non-coal mine when executing the above-mentioned computer program.

[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned method for generating a video summary of an open-pit non-coal mine is implemented.

[0041] The above solution of the present application has the following beneficial effects:

[0042] In some embodiments of the present application, multiple target videos corresponding to each target non-coal mine are obtained, and then for each target video, feature extraction is performed on the target video to obtain the time feature, spatial feature and optical flow feature of the target video, and the feature enhancement model is used to enhance the time feature, spatial feature and optical flow feature to obtain the final time feature, final spatial feature and final optical flow feature of the target video, and then a summary generation model is used to generate a video summary of each target video based on the final time feature, final spatial feature and final optical flow feature of each target video, and then a loss function is constructed based on all video summaries, and the loss function is used to optimize the feature enhancement model and the summary generation model to obtain the optimized feature enhancement model and the optimized summary generation model, and finally the current video corresponding to the non-coal mine to be processed is obtained, and the optimized feature enhancement model and the optimized summary generation model are used to generate the final video summary corresponding to the non-coal mine to be processed based on the current video. Among them, the attention mechanism and dense convolution mechanism are introduced into the feature enhancement model, which can improve the feature processing ability of the feature enhancement model and capture the information of the features in the depth direction, thereby improving the information richness and representation ability of the final temporal features, final spatial features and final optical flow features. The video summary is obtained based on the information-rich final temporal features, final spatial features and final optical flow features, which can improve the accuracy of the video summary and take into account the information of time, space and optical flow. Therefore, the video summary generation method of the present application can further improve the accuracy of the video summary.

[0043] Other beneficial effects of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 A flowchart of a method for generating a video summary of an open-pit non-coal mine provided in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of the structure of a feature enhancer model provided in one embodiment of the present application;

[0047] Figure 3 A schematic diagram of the structure of an attention subunit provided in one embodiment of the present application;

[0048] Figure 4 A schematic diagram of the structure of a summary generation model provided in an embodiment of the present application;

[0049] Figure 5 A schematic diagram of a video summary provided in an embodiment of the present application;

[0050] Figure 6 A schematic diagram of the structure of a video summary generation device for an open-pit non-coal mine provided in an embodiment of the present application;

[0051] Figure 7 A schematic diagram of the structure of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0052] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0053] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0054] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0055] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0056] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0057] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0058] In view of the problem of low accuracy of existing video summaries, an embodiment of the present application provides a method for generating video summaries for open-pit non-coal mines. The method obtains multiple target videos corresponding to each target non-coal mine, and then extracts features of the target video for each target video to obtain the time feature, spatial feature and optical flow feature of the target video, and uses a feature enhancement model to enhance the time feature, spatial feature and optical flow feature to obtain the final time feature, final spatial feature and final optical flow feature of the target video, and then uses a summary generation model to generate a video summary of each target video based on the final time feature, final spatial feature and final optical flow feature of each target video, and then constructs a loss function based on all video summaries, and uses the loss function to optimize the feature enhancement model and the summary generation model to obtain an optimized feature enhancement model and an optimized summary generation model, and finally obtains the current video corresponding to the non-coal mine to be processed, and uses the optimized feature enhancement model and the optimized summary generation model to generate a final video summary corresponding to the non-coal mine to be processed based on the current video. Among them, the attention mechanism and dense convolution mechanism are introduced into the feature enhancement model, which can improve the feature processing ability of the feature enhancement model and capture the information of the features in the depth direction, thereby improving the information richness and representation ability of the final temporal features, final spatial features and final optical flow features. The video summary is obtained based on the information-rich final temporal features, final spatial features and final optical flow features, which can improve the accuracy of the video summary and take into account the information of time, space and optical flow. Therefore, the video summary generation method of the present application can further improve the accuracy of the video summary.

[0059] Next, an exemplary description is given of the method for generating a video summary of an open-pit non-coal mine provided in this application.

[0060] like Figure 1 As shown, the video summary generation method of the open-pit non-coal mine provided by this application includes the following steps:

[0061] Step 11, obtaining multiple target videos corresponding to each target non-coal mine.

[0062] It should be noted that the above-mentioned target non-coal mine is an open-pit non-coal mine with a corresponding target video, and the target video is a relevant video of the target non-coal mine, such as a monitoring video during mining operations.

[0063] In some embodiments of the present application, a plurality of target videos of a target non-coal mine may be acquired using a camera or other equipment.

[0064] Step 12, for each target video, feature extraction is performed on the target video to obtain the temporal features, spatial features and optical flow features of the target video, and the temporal features, spatial features and optical flow features are enhanced by using a feature enhancement model to obtain the final temporal features, final spatial features and final optical flow features of the target video.

[0065] The above-mentioned temporal features are used to describe information on the temporal dimension, such as the sequence of images in the target video and the gap between two consecutive frames of images. The spatial features are used to describe information on the spatial dimension, such as the color and pixel value of each frame of the target video. The optical flow features are used to describe information on the pixel displacement between two consecutive frames of the target video.

[0066] In some embodiments of the present application, the target video can be preprocessed (such as characterizing the interference caused by the complex environment in the target video by vectors, and converting the pixels of each frame of the target video into a feature representation with perceptual significance and interference-related), and then the temporal features, spatial features, and optical flow features can be obtained using models such as Transformer. The above preprocessing process can be expressed as:

[0067]

[0068] Among them, E i represents the embedded representation of the i-th frame image in the target video, f att () represents the multi-head self-attention mechanism, f FFN () represents the feedforward network, f L () indicates layer normalization, Represents the probability code corresponding to the i-th frame image, A′ i represents the masked multi-head self-attention result of the i-th frame image, represents the encoder-decoder attention mechanism, f i represents the preprocessing result of the i-th frame image, i=1,2,...,t, and t represents the number of frames of the target video.

[0069] The above-mentioned feature enhancement model includes a first feature enhancer model and a second feature enhancer model connected in sequence, the first feature enhancer model and the second feature enhancer model each include a feature enhancer model, the feature enhancer model includes a first enhancement unit, a second enhancement unit, a first convolution unit, a third enhancement unit, a second convolution unit, and a fourth enhancement unit connected in sequence, the input end of the first enhancement unit is the input end of the feature enhancer model, the output end of the fourth enhancement unit is the output end of the feature enhancer model, the first enhancement unit, the second enhancement unit, the third enhancement unit, and the fourth enhancement unit each include an enhancement unit, the enhancement unit includes a convolution subunit, a dense subunit, a pooling subunit and an attention subunit connected in sequence, the input end of the convolution subunit is the input end of the enhancement unit, and the output end of the attention subunit is the output end of the enhancement unit.

[0070] The above-mentioned first convolution unit, second convolution unit, and convolution subunit are all convolution operations, the dense subunit is a dense convolution network, and the pooling subunit is a pooling operation.

[0071] It should be noted that the above feature enhancement model is used to enhance the temporal features, spatial features and optical flow features respectively, and the final temporal features, final spatial features and final optical flow features of the target video can be expressed as:

[0072] By formula:

[0073]

[0074] Get the final features of the target video

[0075] Among them, when l = S, Represents the final spatial features of the target video. When l = T, represents the final spatial features of the target video. When l = F, Represents the final optical flow features of the target video, represents the operation of the feature enhancement model, represents the operation of the first feature enhancer model, represents the operation of the second feature enhancer model, P l represents the initial features of the target video. When l = S, P S represents the spatial features of the target video. When l = T, P T represents the temporal characteristics of the target video. When l = F, P F Represents the optical flow features of the target video.

[0076] It should be noted that the expression of the convolution operation in the feature enhancement model can be:

[0077]

[0078] in, represents the output of the convolution operation, P l′ Represents the input of the convolution operation, W (1) represents the weight, b (1) Indicates paranoia, P l′ The dimension is H×W×3×t, where H represents the total height of each frame of the target video, W represents the total width of each frame of the target video, and t represents P l′ The sequence number of the corresponding image in the target video.

[0079] The expression of the pooling subunit can be:

[0080]

[0081] in, represents the output of the pooling subunit, f pool (a+m,b+n) represents the value of the pixel with coordinates (a+m,b+n) in the input of the pooling subunit, and M×N represents the size of the pooling window.

[0082] The above-mentioned attention subunit includes a global average pooling layer, a spatial attention layer, a temporal attention layer, a first addition layer, and a second addition layer.

[0083] The input of the global average pooling layer and the input of the first addition layer are both the input of the attention sub-unit, and the output of the second addition layer is the output of the attention sub-unit.

[0084] The output end of the global average pooling layer is connected to the input end of the spatial attention layer and the input end of the temporal attention layer respectively. The output end of the temporal attention layer is connected to the input end of the first addition layer. The output end of the spatial attention layer is connected to the input end of the second addition layer. The output end of the first addition layer is connected to the input end of the second addition layer.

[0085] The above-mentioned spatial attention layer includes a spatial pooling block, a first dimensionality reduction block, a first fully connected block, a first activation function block, and a first conversion block connected in sequence. The input end of the spatial pooling block is the input end of the spatial attention layer, and the output end of the first conversion block is the output end of the spatial attention layer. The spatial pooling block is used to pool the input data in the spatial dimension, the first dimensionality reduction block is used to reduce the dimension of the input data, and the first conversion block is used to rearrange the dimension of the input data.

[0086] The temporal attention layer includes a temporal pooling block, a second dimensionality reduction block, a second fully connected block, a second activation function block, and a second conversion block connected in sequence. The input end of the temporal pooling block is the input end of the temporal attention layer, and the output end of the second conversion block is the output end of the temporal attention layer. The temporal pooling block is used to pool the input data in the time dimension, the second dimensionality reduction block is used to reduce the dimension of the input data, and the second conversion block is used to rearrange the dimension of the input data.

[0087] In some embodiments of the present application, the expression of the above attention subunit is:

[0088]

[0089] in, represents the output of the attention subunit, represents the input of the attention subunit, f cGAP () represents the operation of the global average pooling layer, f s () represents the operation of the spatial attention layer, f T () represents the operation of the temporal attention layer:

[0090]

[0091] f s =f softmax,1 (W S *f sGAP +b S )

[0092] f T =f softmax,2 (W T *f tGAP +b T )

[0093] Among them, f cGAP represents the output of the global average pooling layer, f sGAP represents the output of the spatial pooling block, f tGAP represents the output of the temporal pooling block, f s represents the output of the first fully connected layer, W S represents the weight of the first fully connected layer, b S represents the bias of the first fully connected layer, f T represents the output of the second fully connected layer, W T represents the weight of the second fully connected layer, b T represents the bias of the second fully connected layer, f softmax,1 () represents the operation of the first activation function block, f softmax,2 () represents the operation of the second activation function block.

[0094] The operations of the first dimensionality reduction block, the first conversion block, the second dimensionality reduction block, and the second conversion block are not reflected in the above formula. After obtaining the output of the spatial pooling block, it is necessary to first use the first dimensionality reduction block to reduce the dimension, and then input it to the first activation function block and the first fully connected layer. After obtaining f s , it is necessary to use the first conversion block for processing. After obtaining the output of the time pooling block, it is necessary to use the second dimensionality reduction block for dimensionality reduction, and then input it into the second activation function block and the second fully connected layer. After obtaining f T , it needs to be processed by the second conversion block.

[0095] Exemplarily, the first dimensionality reduction block and the second dimensionality reduction block are both Flatten functions, the first fully connected layer and the second fully connected layer are both FC functions, the first activation function block and the second activation function block are both Softmax functions, and the first conversion block and the second conversion block are both Reshape functions.

[0096] It is worth mentioning that the attention mechanism and dense convolution mechanism are introduced in the feature enhancement model, which can improve the feature processing ability of the feature enhancement model and capture the information of the features in the depth direction, thereby improving the information richness and representation ability of the final temporal features, final spatial features and final optical flow features.

[0097] The above-mentioned feature enhancer model is exemplified below with reference to a specific example.

[0098] like Figure 2 As shown, the feature enhancement sub-model includes a first enhancement unit, a second enhancement unit, a first convolution unit, a third enhancement unit, a second convolution unit, and a fourth enhancement unit connected in sequence. Figure 2 In the figure, Conv1a represents the convolution subunit in the first enhancement unit, Dense Block represents the dense subunit, Pool1 represents the pooling subunit in the first enhancement unit, Mul-Atteneion represents the attention subunit, Conv2a represents the convolution subunit in the second enhancement unit, Pool2 represents the pooling subunit in the second enhancement unit, Conv3a represents the first convolution unit, Conv3b represents the convolution subunit in the third enhancement unit, Pool3 represents the pooling subunit in the third enhancement unit, Conv4a represents the second convolution unit, Conv4b represents the convolution subunit in the fourth enhancement unit, and Pool4 represents the pooling subunit in the fourth enhancement unit.

[0099] Figure 2 The structure of the attention subunit in the feature enhancer model is shown in Figure 3 As shown, it includes a global average pooling layer, a spatial attention layer, a temporal attention layer, a first addition layer, and a second addition layer. Figure 3In which cGAP represents the global average pooling layer, tGAP represents the temporal pooling block, sGAP represents the spatial pooling block, Flatten1 represents the first dimensionality reduction block, Flatten2 represents the second dimensionality reduction block, FC1 represents the first fully connected block, FC2 represents the second fully connected block, Softmax1 represents the first activation function block, Softmax2 represents the second activation function block, Reshape1 represents the first transformation block, Reshape2 represents the second transformation block, and + represents the addition in the first addition layer and the second addition layer.

[0100] Step 13, generating a video summary of each target video using a summary generation model according to the final temporal features, final spatial features and final optical flow features of each target video.

[0101] In some embodiments of the present application, the summary generation model includes an additive sub-model, a first linear sub-model, a first activation sub-model, a regularization sub-model, a normalization sub-model, a second linear sub-model, and a second activation sub-model connected in sequence; the input end of the additive sub-model is the input end of the summary generation model, and the output end of the second activation sub-model is the output end of the summary generation model.

[0102] Exemplarily, the first activation submodel is a ReLu activation function, the regularization submodel is a drop out function, the standardization submodel is a Normalization function, the second activation submodel is a sigmoid function, and the first linear submodel and the second linear submodel are both nn.linear functions.

[0103] It should be noted that the addition sub-model is used to add and fuse the final temporal features, final spatial features and final optical flow features of the input, the first linear sub-model is used to perform linear transformation on the input data, the first activation sub-model is used to enhance nonlinear relationships, the regularization sub-model is used to regularize the input data, the standardization sub-model is used to standardize the input data, and the second activation sub-model is used to capture the nonlinear factors of the input data.

[0104] The above summary generation model is exemplarily described below with reference to a specific example.

[0105] like Figure 4 As shown, the summary generation model includes an additive sub-model, a first linear sub-model, a first activation sub-model, a regularization sub-model, a standardization sub-model, a second linear sub-model, and a second activation sub-model connected in sequence. Figure 4In it, LinearLayer1 represents the first linear sub-model, ReLu represents the first activation sub-model, dropout represents the regularization sub-model, Normalization represents the standardization sub-model, Linear Layer2 represents the second linear sub-model, sigmoid represents the second activation sub-model, and + represents the addition sub-model.

[0106] The above video summary is exemplarily described below with reference to a specific example.

[0107] like Figure 5 As shown, the original video (i.e., the target video mentioned above) includes multiple video frames. The original video is preprocessed (i.e., the steps of obtaining time features, spatial features, and optical flow features mentioned above), and then feature enhancement is performed to obtain the final time features, final spatial features, and final optical flow features. Feature fusion is then performed, and the fused features are processed to obtain the video summary of the original video.

[0108] Step 14: construct a loss function based on all video summaries, and use the loss function to optimize the feature enhancement model and the summary generation model to obtain an optimized feature enhancement model and an optimized summary generation model.

[0109] The above loss function is used to describe the accuracy of all video summaries.

[0110] Specifically, the loss function is:

[0111] Loss CE =-C label logC out -(1-C label )log(1-C out )

[0112] Among them, Loss CE represents the value of the loss function, C label represents the set of true summaries of all target videos, C out Represents the set of video summaries of all target videos (i.e., the video summaries output by the summary generation model).

[0113] In some embodiments of the present application, the step of optimizing the feature enhancement model and the summary generation model using the loss function to obtain the optimized feature enhancement model and the optimized summary generation model includes:

[0114] Determine whether the value of the loss function is less than the preset value of the loss function.

[0115] If so, the feature enhancement model is used as the optimized feature enhancement model, and the summary generation model is used as the optimized summary generation model.

[0116] Otherwise, adjust the parameters in the feature enhancement model and the summary generation model, and return to the step of extracting features of the target video for each target video to obtain the temporal features, spatial features, and optical flow features of the target video, and use the feature enhancement model to enhance the temporal features, spatial features, and optical flow features to obtain the final temporal features, final spatial features, and final optical flow features of the target video.

[0117] Exemplarily, the value of the loss function is 3.8, which is greater than the preset value of the loss function of 1.3. The parameters in the feature enhancement model and the summary generation model are adjusted, and the steps of extracting features of the target video for each target video, obtaining the temporal features, spatial features, and optical flow features of the target video, and using the feature enhancement model to enhance the temporal features, spatial features, and optical flow features, respectively, to obtain the final temporal features, final spatial features, and final optical flow features of the target video are returned. At this time, the value of the loss function is 0.7, which is less than the preset value of the loss function. The feature enhancement model at this time is used as the optimized feature enhancement model, and the summary generation model is used as the optimized summary generation model.

[0118] It is worth mentioning that the loss function is used to optimize the feature enhancement model and the summary generation model, so that the performance of the feature enhancement model and the summary generation model meets expectations.

[0119] Step 15, obtaining the current video corresponding to the non-coal mine to be processed, and using the optimized feature enhancement model and the optimized summary generation model to generate a final video summary corresponding to the non-coal mine to be processed based on the current video.

[0120] The non-coal mine to be processed is a non-coal mine for which a corresponding video summary needs to be obtained, and the current video is a video for which a video summary needs to be generated, such as the latest acquired monitoring video of the mining operation of the non-coal mine to be processed.

[0121] In some embodiments of the present application, the above-mentioned steps of using the optimized feature enhancement model and the optimized summary generation model to generate the final video summary corresponding to the non-coal mine to be processed based on the current video are specifically as follows:

[0122] The temporal features, spatial features and optical flow features of the current video are obtained, and then the temporal features, spatial features and optical flow features are enhanced respectively by using the optimized feature enhancement model to obtain the final temporal features, final spatial features and final optical flow features of the current video, and then the final temporal features, final spatial features and final optical flow features are input into the optimized summary generation model to obtain the final video summary of the current video.

[0123] Exemplarily, the optimized feature enhancement model and the optimized summary generation model can be run using computer software such as matlab and python to obtain the final video summary of the current video.

[0124] It is worth mentioning that the attention mechanism and dense convolution mechanism are introduced in the feature enhancement model, which can improve the feature processing ability of the feature enhancement model and capture the information of the features in the depth direction, thereby improving the information richness and representation ability of the final temporal features, final spatial features and final optical flow features. The accuracy of the video summary obtained based on the information-rich final temporal features, final spatial features and final optical flow features is improved, and the information of time, space and optical flow is taken into account, which further improves the accuracy of the video summary.

[0125] The video summary generation method of an open-pit non-coal mine of the present application is exemplarily described below with reference to a specific example.

[0126] F1-score is used as the evaluation indicator. F1-score is the harmonic mean of precision and recall. It provides a single evaluation value for these two indicators. The expression is:

[0127]

[0128] Among them, F1 represents the value of F1-score, Precision represents the accuracy, Recall represents the recall rate, TP represents the number of correct segments in the generated video summary, FP represents the number of irrelevant segments incorrectly included in the generated video summary, and FN represents the number of segments that are not extracted in the actual summary.

[0129] Using the method of the present application, a long short-term memory model for video summarization (vsLSTM, Video Summarization with Long Short-Term Memory), a deterministic point process long short-term memory model (dppLSTM, Determinantal Point Process Long Short-Term Memory), a generative adversarial network and a deterministic point process video summarization (SUM-GANdpp, SUM-GAN with Determinantal Point Process), a generative adversarial network video summarization based on a supervised signal (SUM-GANsup, SUM-GAN with Supervision), a video summarization based on additive attention (A-AVA, Additive Attention-based Video Summarization), a video summarization based on multiplicative attention (M-AVS, Multiplicative Attention-based Video Summarization), a semantic attention video summarization network (SASUM, Semantic Attended Video Summarization Network), a semantic attended video summarization network based on a supervised signal (SASUMsup, semantic attended video summarization networkwith supervision), a temporal-spatial-transition network (TS-STN, Temporal-Spatial-Transition Network), Fully Convolutional Siamese Networks (FCSN), Video Attention-based Summarization Network (VASNet), Detect-to-summarize network for video summarization (DSNet), Multi-Source Visual Attention (MSVA), use the datasets SumMe and TVSum for video summary generation, and the obtained F1-score values ​​are shown in Table 1.

[0130] Model SumMe TVSum vs LSTM 37.6 54.2 dppLSTM 38.6 54.7 SUM-GANdpp 39.1 51.7 SUM-GANsup 41.7 56.3 A-AVA 43.9 59.4 M-AVS 44.4 61.0 SASUM 40.6 53.9 SASUMsup 45.3 58.2 DR-DSN 41.4 57.6 DR-DSNsup 42.1 58.1 TS-STN 46.1 60.0 FCSN 48.8 58.4 VASNet 49.7 61.4 DSNet 50.2 62.1 MSVA 54.5 62.8 This application 57.07 64.03

[0131] Table 1

[0132] It can be seen from Table 1 that the video summary generation results of the method of this application are better than those of other models.

[0133] The learning rate when optimizing the model and the value of apeature in the self-attention mechanism are adjusted, and the video summary is generated using the method of this application. The obtained F1-score values ​​are shown in Table 2.

[0134]

[0135] Table 2

[0136] It can be seen from Table 2 that for the SumMe dataset, as the learning rate increases, the effect of generating video summaries is gradually increasing, but too large a learning rate will lead to overfitting problems in the process of learning features. The opposite is true for the TVSum dataset.

[0137] It can be seen that when the method of the present application is used to generate a video summary, the accuracy of the video summary can be improved, and setting parameters according to the actual situation of the task can further improve the accuracy of the video summary.

[0138] The following is an exemplary description of the video summary generation device for an open-pit non-coal mine provided in the present application.

[0139] like Figure 6 As shown, the embodiment of the present application provides a video summary generation device for an open-pit non-coal mine, and the video summary generation device 600 for an open-pit non-coal mine includes:

[0140] An acquisition module 601 is used to acquire multiple target videos corresponding to each target non-coal mine;

[0141] The feature extraction module 602 is used to extract features of each target video, obtain the temporal features, spatial features and optical flow features of the target video, and use the feature enhancement model to enhance the temporal features, spatial features and optical flow features, respectively, to obtain the final temporal features, final spatial features and final optical flow features of the target video; the feature enhancement model includes a first feature enhancement sub-model and a second feature enhancement sub-model connected in sequence, the first feature enhancement sub-model and the second feature enhancement sub-model each include a feature enhancement sub-model, the feature enhancement sub-model includes a first enhancement unit, a second enhancement unit, a first convolution unit, a third enhancement unit, a second convolution unit, and a fourth enhancement unit connected in sequence, the input end of the first enhancement unit is the input end of the feature enhancement sub-model, the output end of the fourth enhancement unit is the output end of the feature enhancement sub-model, the first enhancement unit, the second enhancement unit, the third enhancement unit, and the fourth enhancement unit each include an enhancement unit, the enhancement unit includes a convolution sub-unit, a dense sub-unit, a pooling sub-unit and an attention sub-unit connected in sequence, the input end of the convolution sub-unit is the input end of the enhancement unit, and the output end of the attention sub-unit is the output end of the enhancement unit;

[0142] A first generation module 603 is used to generate a video summary of each target video using a summary generation model according to the final temporal features, the final spatial features and the final optical flow features of each target video;

[0143] The optimization module 604 is used to construct a loss function based on all video summaries, and use the loss function to optimize the feature enhancement model and the summary generation model to obtain an optimized feature enhancement model and an optimized summary generation model; the loss function is used to describe the accuracy of all video summaries;

[0144] The second generation module 605 is used to obtain the current video corresponding to the non-coal mine to be processed, and use the optimized feature enhancement model and the optimized summary generation model to generate a final video summary corresponding to the non-coal mine to be processed based on the current video.

[0145] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0146] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0147] like Figure 7 As shown, an embodiment of the present application provides a terminal device. The terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 7 Only one processor is shown in the figure), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 implements the steps of any of the above-mentioned method embodiments when executing the computer program D102.

[0148] Specifically, when the processor D100 executes the computer program D102, it obtains multiple target videos corresponding to each target non-coal mine, and then extracts features of each target video to obtain the time feature, spatial feature and optical flow feature of the target video, and uses the feature enhancement model to enhance the time feature, spatial feature and optical flow feature to obtain the final time feature, final spatial feature and final optical flow feature of the target video, and then uses the summary generation model to generate a video summary of each target video according to the final time feature, final spatial feature and final optical flow feature of each target video, and then constructs a loss function based on all video summaries, and uses the loss function to optimize the feature enhancement model and the summary generation model to obtain the optimized feature enhancement model and the optimized summary generation model, and finally obtains the current video corresponding to the non-coal mine to be processed, and uses the optimized feature enhancement model and the optimized summary generation model to generate the final video summary corresponding to the non-coal mine to be processed based on the current video. Among them, the attention mechanism and dense convolution mechanism are introduced into the feature enhancement model, which can improve the feature processing ability of the feature enhancement model and capture the information of the features in the depth direction, thereby improving the information richness and representation ability of the final temporal features, final spatial features and final optical flow features. The video summary is obtained based on the information-rich final temporal features, final spatial features and final optical flow features, which can improve the accuracy of the video summary and take into account the information of time, space and optical flow. Therefore, the video summary generation method of the present application can further improve the accuracy of the video summary.

[0149] The processor D100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0150] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may also be an external storage device of the terminal device D10, such as a plug-in hard disk, a smart memory card (SMC, SmartMedia Card), a secure digital (SD, Secure Digital) card, a flash card (Flash Card), etc. equipped on the terminal device D10. Further, the memory D101 may also include both an internal storage unit of the terminal device D10 and an external storage device. The memory D101 is used to store an operating system, an application program, a boot loader (BootLoader), data and other programs, such as the program code of the computer program, etc. The memory D101 may also be used to temporarily store data that has been output or is to be output.

[0151] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0152] An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0153] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium that can carry the computer program code to the video summary generation method device / terminal device of the open-pit non-coal mine. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0154] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0155] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for generating video summaries of open-pit non-coal mines, characterized in that: include: Acquire multiple target videos corresponding to each target non-coal mine; For each of the target videos, feature extraction is performed on the target video to obtain the temporal feature, spatial feature and optical flow feature of the target video, and feature enhancement is performed on the temporal feature, spatial feature and optical flow feature by using a feature enhancement model to obtain the final temporal feature, final spatial feature and final optical flow feature of the target video; The feature enhancement model includes a first feature enhancement sub-model and a second feature enhancement sub-model connected in sequence, the first feature enhancement sub-model and the second feature enhancement sub-model each include a feature enhancement sub-model, the feature enhancement sub-model includes a first enhancement unit, a second enhancement unit, a first convolution unit, a third enhancement unit, a second convolution unit, and a fourth enhancement unit connected in sequence, the input end of the first enhancement unit is the input end of the feature enhancement sub-model, the output end of the fourth enhancement unit is the output end of the feature enhancement sub-model, the first enhancement unit, the second enhancement unit, the third enhancement unit, and the fourth enhancement unit each include an enhancement unit, the enhancement unit includes a convolution sub-unit, a dense sub-unit, a pooling sub-unit, and an attention sub-unit connected in sequence, the input end of the convolution sub-unit is the input end of the enhancement unit, and the output end of the attention sub-unit is the output end of the enhancement unit; Generate a video summary of each target video using a summary generation model according to the final temporal features, the final spatial features, and the final optical flow features of each target video; A loss function is constructed based on all video summaries, and the feature enhancement model and the summary generation model are optimized using the loss function to obtain an optimized feature enhancement model and an optimized summary generation model; the loss function is used to describe the accuracy of all video summaries; A current video corresponding to the non-coal mine to be processed is obtained, and a final video summary corresponding to the non-coal mine to be processed is generated based on the current video by using the optimized feature enhancement model and the optimized summary generation model.

2. The video summary generation method according to claim 1, characterized in that: The feature enhancement model is used to enhance the temporal features, spatial features and optical flow features respectively to obtain the final temporal features, final spatial features and final optical flow features of the target video, including: By formula: Get the final features of the target video Among them, when l = S, represents the final spatial feature of the target video. When l = T, represents the final spatial feature of the target video. When l=F, represents the final optical flow feature of the target video, represents the operation of the feature enhancement model, represents the operation of the first feature enhancer model, represents the operation of the second feature enhancer model, P l represents the initial features of the target video. When l=S, P S represents the spatial features of the target video. When l = T, P T represents the temporal characteristics of the target video. When l = F, P F Represents the optical flow features of the target video.

3. The video summary generation method according to claim 1, characterized in that: The attention subunit includes a global average pooling layer, a spatial attention layer, a temporal attention layer, a first addition layer, and a second addition layer; The input end of the global average pooling layer and the input end of the first addition layer are both input ends of the attention sub-unit, and the output end of the second addition layer is the output end of the attention sub-unit; The output end of the global average pooling layer is respectively connected to the input end of the spatial attention layer and the input end of the temporal attention layer, the output end of the temporal attention layer is connected to the input end of the first addition layer, the output end of the spatial attention layer is connected to the input end of the second addition layer, and the output end of the first addition layer is connected to the input end of the second addition layer.

4. The video summary generation method according to claim 3, characterized in that: The spatial attention layer includes a spatial pooling block, a first dimensionality reduction block, a first fully connected block, a first activation function block, and a first conversion block connected in sequence; The input end of the spatial pooling block is the input end of the spatial attention layer, and the output end of the first conversion block is the output end of the spatial attention layer; The temporal attention layer includes a temporal pooling block, a second dimensionality reduction block, a second fully connected block, a second activation function block, and a second conversion block connected in sequence; The input end of the temporal pooling block is the input end of the temporal attention layer, and the output end of the second conversion block is the output end of the temporal attention layer.

5. The video summary generation method according to claim 1, characterized in that: The summary generation model includes an addition sub-model, a first linear sub-model, a first activation sub-model, a regularization sub-model, a standardization sub-model, a second linear sub-model, and a second activation sub-model connected in sequence; The input end of the addition sub-model is the input end of the summary generation model, and the output end of the second activation sub-model is the output end of the summary generation model.

6. The video summary generation method according to claim 1, characterized in that: The loss function is: Loss CE =-C label logC out -(1-C label )log(1-C out ) Among them, Loss CE represents the value of the loss function, C label represents the set of true summaries of all target videos, C out Represents the collection of video summaries for all target videos.

7. The video summary generation method according to claim 1, characterized in that: The loss function is constructed based on all video summaries, and the feature enhancement model and the summary generation model are optimized by using the loss function to obtain an optimized feature enhancement model and an optimized summary generation model, including: Determine whether the value of the loss function is less than a preset value of the loss function; If yes, the feature enhancement model is used as the optimized feature enhancement model, and the summary generation model is used as the optimized summary generation model; Otherwise, the parameters in the feature enhancement model and the summary generation model are adjusted, and the step of extracting features from the target video for each target video to obtain the temporal features, spatial features and optical flow features of the target video is returned to, and the step of enhancing the temporal features, spatial features and optical flow features by using the feature enhancement model to obtain the final temporal features, final spatial features and final optical flow features of the target video is obtained.

8. A video summary generation device for an open-pit non-coal mine, characterized in that: include: An acquisition module, used for acquiring multiple target videos corresponding to each target non-coal mine; A feature extraction module is used to extract features of each target video to obtain the temporal features, spatial features and optical flow features of the target video, and to enhance the temporal features, spatial features and optical flow features by using a feature enhancement model to obtain the final temporal features, final spatial features and final optical flow features of the target video; The feature enhancement model includes a first feature enhancement sub-model and a second feature enhancement sub-model connected in sequence, the first feature enhancement sub-model and the second feature enhancement sub-model each include a feature enhancement sub-model, the feature enhancement sub-model includes a first enhancement unit, a second enhancement unit, a first convolution unit, a third enhancement unit, a second convolution unit, and a fourth enhancement unit connected in sequence, the input end of the first enhancement unit is the input end of the feature enhancement sub-model, the output end of the fourth enhancement unit is the output end of the feature enhancement sub-model, the first enhancement unit, the second enhancement unit, the third enhancement unit, and the fourth enhancement unit each include an enhancement unit, the enhancement unit includes a convolution sub-unit, a dense sub-unit, a pooling sub-unit, and an attention sub-unit connected in sequence, the input end of the convolution sub-unit is the input end of the enhancement unit, and the output end of the attention sub-unit is the output end of the enhancement unit; A first generation module is used to generate a video summary of each target video using a summary generation model according to the final temporal features, the final spatial features and the final optical flow features of each target video; An optimization module, used to construct a loss function based on all video summaries, and use the loss function to optimize the feature enhancement model and the summary generation model to obtain an optimized feature enhancement model and an optimized summary generation model; the loss function is used to describe the accuracy of all video summaries; The second generation module is used to obtain the current video corresponding to the non-coal mine to be processed, and use the optimized feature enhancement model and the optimized summary generation model to generate a final video summary corresponding to the non-coal mine to be processed based on the current video.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for generating a video summary of an open-pit non-coal mine as claimed in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for generating a video summary of an open-pit non-coal mine as claimed in any one of claims 1 to 7 is implemented.

Images