Video frame recognition method, device, readable medium and electronic device

Abstract

The present disclosure relates to a video border recognition method, device, readable medium and electronic device. The video border recognition method divides a target video into multiple sub-videos; extracts a preset number of frames of to-be-recognized images from each of the sub-videos to obtain multiple frames of to-be-recognized images; then obtains a multi-scale feature map set corresponding to the multiple frames of to-be-recognized images through a preset border recognition model, and determines whether the target video has a border according to a feature map with the smallest spatial resolution in the feature map set; and when it is determined that the target video has a border, fuses feature maps of different scales in the feature map set to obtain a pending feature map, and predicts the target position according to the pending feature map; the method can simultaneously consider the temporal information and spatial information of the video, and can also effectively fuse the deep semantic features and shallow spatial features of the video, so as to effectively improve the accuracy of the border position recognition result.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular, to a video border recognition method, device, readable medium and electronic device. Background Art

[0002] In real-life scenarios, it is often necessary to perform border recognition operations on videos with borders. For example, in video publishing platforms, it is often encountered that videos uploaded by users have borders. If the video is to be published on devices suitable for vertical videos as well as devices suitable for horizontal videos, it is usually necessary to first effectively identify the video border in the video with the border, determine the main area of ​​the video with the border (i.e., the video image area after removing the border), and then adjust the size according to the main area to meet the playback requirements of devices suitable for vertical videos or meet the playback requirements of devices suitable for horizontal videos. However, current video border recognition methods usually have the problem that the accuracy of the border position recognition result is poor, which is not conducive to determining the position of the video body. Summary of the invention

[0003] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] The present disclosure provides a video frame recognition method, device, readable medium and electronic device.

[0005] In a first aspect, the present disclosure provides a video border recognition method, the method comprising:

[0006] Obtain a target video to be identified, and divide the target video into multiple sub-videos;

[0007] Extracting a preset number of frames of images to be recognized from each of the sub-videos to obtain multiple frames of images to be recognized;

[0008] Inputting the multiple frames of images to be recognized into a preset border recognition model so that the preset border recognition model outputs a border recognition result, wherein the border recognition result is used to characterize whether the target video has a border and, if the target video has a border, the target position of the border;

[0009] Among them, the preset border recognition model is used to obtain a multi-scale feature map set corresponding to the multiple frames of images to be recognized, and determine whether the target video has a border based on the feature map with the smallest spatial resolution in the feature map set; and when it is determined that the target video has a border, the feature maps of different scales in the feature map set are merged to obtain a pending feature map, and the target position is predicted based on the pending feature map.

[0010] In a second aspect, the present disclosure provides a video border recognition device, the device comprising:

[0011] A first acquisition module is configured to acquire a target video to be identified and divide the target video into a plurality of sub-videos;

[0012] A second acquisition module is configured to extract a preset number of frames of images to be recognized from each of the sub-videos to obtain multiple frames of images to be recognized;

[0013] A determination module is configured to input the multiple frames of images to be identified into a preset border recognition model so that the preset border recognition model outputs a border recognition result, wherein the border recognition result is used to characterize whether there is a border in the target video and, if there is a border in the target video, a target position of the border;

[0014] Among them, the preset border recognition model is used to obtain a multi-scale feature map set corresponding to the multiple frames of images to be recognized, and determine whether the target video has a border based on the feature map with the smallest spatial resolution in the feature map set; and when it is determined that the target video has a border, the feature maps of different scales in the feature map set are merged to obtain a pending feature map, and the target position is predicted based on the pending feature map.

[0015] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect above.

[0016] In a fourth aspect, the present disclosure provides an electronic device, including:

[0017] a storage device having a computer program stored thereon;

[0018] A processing device is used to execute the computer program in the storage device to implement the steps of the method described in the first aspect above.

[0019] The above technical scheme divides the target video into multiple sub-videos; extracts a preset number of frames of to-be-recognized images from each of the sub-videos to obtain multiple frames of to-be-recognized images; then obtains a multi-scale feature map set corresponding to the multiple frames of to-be-recognized images through a preset border recognition model, and determines whether the target video has a border according to the feature map with the smallest spatial resolution in the feature map set; and when it is determined that the target video has a border, fuses the feature maps of different scales in the feature map set to obtain a pending feature map, and predicts the target position according to the pending feature map; in this way, by performing multi-scale feature extraction on the multiple frames of to-be-recognized images corresponding to the multiple sub-videos, the temporal information and spatial information of the video can be considered at the same time, and by fusing the feature maps of different scales, the effective fusion of the deep semantic features and shallow spatial features of the video can be achieved, and then the target position is predicted according to the pending feature map, which can effectively improve the accuracy of the border position recognition result.

[0020] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings:

[0022] Figure 1 is a flow chart of a video border recognition method shown in an exemplary embodiment of the present disclosure;

[0023] Figure 2 According to this disclosure Figure 1 A block diagram of a preset border recognition model shown in the illustrated embodiment;

[0024] Figure 3 According to this disclosure Figure 2 The illustrated embodiment shows a flow chart of a method for identifying a video border;

[0025] Figure 4 is a schematic diagram of a recognition process of a frame recognition method shown in an exemplary embodiment of the present disclosure;

[0026] Figure 5 is a flow chart of a method for training a preset bounding box recognition model shown in an exemplary embodiment of the present disclosure;

[0027] Figure 6 is a block diagram of a video border recognition device shown in an exemplary embodiment of the present disclosure;

[0028] Figure 7 It is a block diagram of an electronic device shown in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0030] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0031] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0032] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0033] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0034] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0035] Before introducing the specific implementation methods of the present invention in detail, the application scenarios of the present invention are first described as follows. The present invention can be applied to the recognition process of video borders. A common video border recognition scenario is when switching between horizontal and vertical versions of a video. It is usually necessary to first identify the main area of ​​the video in the original video, and then add a border suitable for the target version on the basis of the main area, or adjust the pixel ratio of the main area of ​​the video to obtain a target version that meets the playback requirements. However, the video border recognition method in the related art usually extracts several frames of pictures from the video, and then detects whether there is a border frame by frame, and finally uses a post-processing algorithm to obtain the target version that meets the playback requirements. Through artificially formulated recognition conditions (for example, if each extracted frame has a border, it is determined that there is a border, or if the pictures with borders in multiple extracted frames account for 90% of the total extracted amount, it is determined that the entire video has a border, etc.), it is summarized and inferred from the picture detection results whether the video contains a border and the position of the border of the entire video. For example, if some clips in a video have borders and some clips do not have borders, the border recognition method in the related art can easily regard the border of the video with a border as the border of the entire video. Subsequent video version conversion based on the recognition result is likely to cause the wrong cropping of the borderless part, which is not conducive to improving the viewing effect of the entire video. The inventor found that the reason why the border recognition method in the related art has the problem of poor accuracy of the border recognition result is mainly because each detection is only for a static image of a single frame, and multiple detections are independent of each other. The model cannot perceive the dynamic changes of the video and cannot combine the timing information and spatiotemporal change characteristics of the video.

[0036] In order to solve the above technical problems, the present disclosure provides a video border recognition method, device, readable medium and electronic device. The video border recognition method divides the target video into multiple sub-videos; extracts a preset number of frames of to-be-recognized images from each of the sub-videos to obtain multiple frames of to-be-recognized images; then obtains a multi-scale feature map set corresponding to the multiple frames of to-be-recognized images through a preset border recognition model, and determines whether the target video has a border according to the feature map with the smallest spatial resolution in the feature map set; and when it is determined that the target video has a border, the feature maps of different scales in the feature map set are fused to obtain a pending feature map, and the target position is predicted according to the pending feature map; in this way, by performing multi-scale feature extraction on the multiple frames of to-be-recognized images corresponding to the multiple sub-videos, the temporal information and spatial information of the video can be considered at the same time, and by fusing the feature maps of different scales, the deep semantic features and shallow spatial features of the video can be effectively fused, and then the target position is predicted according to the pending feature map, which can effectively improve the accuracy of the border position recognition result.

[0037] The technical solution of the present disclosure is described in detail below in conjunction with specific embodiments.

[0038] Figure 1 is a flow chart of a video border recognition method shown in an exemplary embodiment of the present disclosure; Figure 1 As shown, the method may include the following steps:

[0039] Step 101: obtain a target video to be identified, and divide the target video into multiple sub-videos.

[0040] In this step, the target video can be divided into a front video, a middle video and a back video according to the timing of the target video; for example, for a 15-second video, the first 3 seconds are used as a sub-video, the last 4 seconds are used as a sub-video, and the middle 8 seconds are used as a sub-video, among which the first 3 seconds of the sub-video are used as the front video, the middle 8 seconds of the sub-video are used as the middle video, and the last 4 seconds of the sub-video are used as the back video.

[0041] Step 102 : extracting a preset number of frames of images to be recognized from each of the sub-videos to obtain a plurality of frames of images to be recognized.

[0042] The preset numbers corresponding to different sub-videos may be the same or different.

[0043] For example, still taking the example shown in the above step 101 as an example, 3 frames of images to be recognized are extracted from the front video and the back video respectively, and 5 frames of images to be recognized are extracted from the middle video, so as to obtain 11 frames of images to be recognized.

[0044] Step 103, input the multiple frames of images to be recognized into a preset border recognition model, so that the preset border recognition model outputs a border recognition result, and the border recognition result is used to characterize whether there is a border in the target video, and when there is a border in the target video, the target position of the border.

[0045] Among them, the preset border recognition model is used to obtain a multi-scale feature map set corresponding to the multiple frames of images to be recognized, and determine whether the target video has a border based on the feature map with the smallest spatial resolution in the feature map set; and when it is determined that the target video has a border, the feature maps of different scales in the feature map set are fused to obtain a pending feature map, and the target position is predicted based on the pending feature map.

[0046] It should be noted that the preset border recognition model can be used to perform multiple convolution operations on a preset number of images to be recognized corresponding to each sub-video to achieve multi-scale feature extraction, and obtain feature maps of multiple different scales corresponding to each sub-video, thereby obtaining the feature map set, and these feature maps of different scales can form a feature pyramid after being sorted according to the scale size. Among them, in the feature pyramid, the feature maps at the low level (shallow layer) have rich detail information, and the feature maps at the high level (deep layer) have rich semantic information. It can be understood that the higher the level of the feature map, the smaller the spatial resolution, the more details it loses, and the richer the semantic information it contains.

[0047] In this step, the feature map with the smallest spatial resolution can be used to determine whether the target video has a border, and only when it is determined that the target video has a border, the feature maps of different scales in the feature map set are fused, and the target position of the border is determined according to the fused pending feature map. When it is determined that the target video does not have a border, there is no need to perform feature fusion, and only the recognition result for characterizing that the target video does not have a border is output. In this way, determining whether there is a border through semantically rich deep features can effectively improve the accuracy of the determination result, and only when it is determined that there is a border, feature maps of different scales are fused for position recognition, which can effectively improve the efficiency of the model and effectively ensure the accuracy of the recognized border position.

[0048] In addition, in this step, the implementation method of fusing feature maps of different scales in the feature map set can be to upsample and convolve the feature map with smaller spatial resolution to obtain a feature map of the same scale as the feature map with larger spatial resolution, and then splice the feature map with larger spatial resolution with the feature map obtained after upsampling and convolution operations, upsample and perform multiple convolution operations to obtain the pending feature map; after obtaining the pending feature map, the border position can be directly predicted based on the pending feature map to obtain the target position of the border, and the pending feature map can also be row pooled to obtain a first target feature map, and the pending feature map is column pooled to obtain a second target feature map, and then the target position of the border is predicted based on the first target feature map and the second target feature map.

[0049] The above technical scheme can perform multi-scale feature extraction on multiple frames of images to be identified corresponding to multiple sub-videos, can simultaneously consider the temporal information and spatial information of the video, and by fusing feature maps of different scales, achieve the effect of effectively fusing the deep semantic features and shallow spatial features of the video, and then predict the target position based on the undetermined feature map, which can effectively improve the accuracy of the border position recognition result.

[0050] Figure 2 According to this disclosure Figure 1 The embodiment shown is a block diagram of a preset border recognition model; Figure 2 As shown, the preset border recognition model includes a backbone network module 201, a feature fusion module 202, a first classification module 203 and a second classification module 204, the backbone network module 201 includes a multi-level feature extraction sub-network, and the feature extraction sub-networks at different levels are used to extract feature maps of different scales; the backbone network module 201 is coupled with the feature fusion module 202 and the first classification module 203, and the feature fusion module 202 is also coupled with the second classification module 204.

[0051] It should be noted that the backbone network module can be ResNet (residual network), for example, it can be ResNet50, ResNet101, etc. Taking ResNet50 as an example, the multi-level feature extraction subnetwork can be stage1 (first stage), stage2 (second stage), stage3 (third stage), stage4 (fourth stage), and stage5 (fifth stage) in ResNet50. Stage1 to stage 5 are used to extract feature maps of different scales, and the feature map output by stage 5 is the feature map with the smallest spatial resolution.

[0052] Furthermore, the processing flow of the preset border recognition model for the image to be recognized can be seen as follows Figure 3 As shown, Figure 3 According to this disclosure Figure 2 The embodiment shown is a flow chart of a video border recognition method, such as Figure 3 As shown, the preset border recognition model is used for:

[0053] Step 301, obtaining a multi-channel feature map corresponding to each frame of the image to be recognized.

[0054] The pair of multi-channel feature maps may be an R channel feature map, a G channel feature map, and a B channel feature map, or may be at least two of the R channel feature map, the G channel feature map, the B channel feature map, and the A channel feature map, and the A channel feature map is used to measure the transparency of the image.

[0055] For example, Figure 4 For example, Figure 4It is a schematic diagram of the recognition process of a border recognition method shown in an exemplary embodiment of the present disclosure. Three frames of images to be recognized are respectively extracted from the front video and the back video, and five frames of images to be recognized are extracted from the middle video, so as to obtain 11 frames of images to be recognized. The R channel feature map, the G channel feature map and the B channel feature map of each frame of the image to be recognized can be obtained, and nine feature maps corresponding to the front video (three per frame, and nine for three frames) are obtained, that is, a multi-channel feature map corresponding to the front video is obtained. Similarly, the multi-channel feature maps corresponding to the back video and the middle video can be obtained.

[0056] Step 302, concatenating the multi-channel feature maps corresponding to the multiple sub-videos and inputting them into the backbone network module, so that the multi-level feature extraction sub-network outputs multiple feature maps of specified scales to obtain the feature map set.

[0057] The multiple feature maps of specified scales include a first feature map of a first scale, a second feature map of a second scale, and a third feature map of a third scale, the first scale is smaller than the second scale, and the second scale is smaller than the third scale.

[0058] For example, when the backbone network module is ResNet50, the first feature map may be a feature map output by stage5 (fifth stage), the multi-channel feature maps are all 128×128 feature maps, the first scale may be 4×4, the second feature map may be a feature map output by stage4 (fourth and fifth stages), the second scale may be 8×8, the third feature map may be a feature map output by stage3 (third stage), and the third scale may be 16×16. Figure 4 For example, the 9 feature maps corresponding to the front video can be concat (splicing) and input into the stage1 (first stage), the 9 feature maps corresponding to the back video can be spliced ​​and input into the stage1 (first stage), and the 15 feature maps corresponding to the middle video (3 per frame, 15 feature maps for 5 frames) can be spliced ​​and input into the stage1 (first stage). If the channel feature maps are all 128×128 feature maps, the feature maps output by the stage1 (first stage) can be 64×64 feature maps, and then stage3 (third stage), stage4 (fourth stage), and stage5 (fifth stage) respectively output feature maps of multiple sub-videos of different scales to obtain feature map sets corresponding to stage3 (third stage), stage4 (fourth stage), and stage5 (fifth stage).

[0059] Step 303: Determine whether the target video has a border according to the feature map with the smallest spatial resolution in the feature map set through the first classification module.

[0060] In this step, Figure 4 For example, the feature maps output by stage5 (fifth stage) in the feature map set can be spliced, and then multiple convolution operations can be performed, and then the recognition result of whether there is a border can be obtained through the GAP layer or the fully connected layer.

[0061] Optionally, the first classification module can also determine the border category of the target video based on the feature map with the smallest spatial resolution in the feature map set. The border category can include pure white border, image border, pure black border, other borders, etc.

[0062] Still Figure 4 For example, the feature maps output by stage5 (fifth stage) in the feature map set can be concat, and then multiple convolution operations can be performed, and then another GAP (global average pooling) layer or another fully connected layer can be used to obtain the recognition result of the border category.

[0063] Step 304: when it is determined that the target video has a border, the feature fusion module fuses the feature maps of the multiple specified scales to obtain the undetermined feature map;

[0064] In this step, the first feature maps corresponding to multiple sub-videos can be spliced ​​and then subjected to convolution operation and upsampling processing to obtain a first designated feature map with the same scale as the second feature map; the second feature maps corresponding to multiple sub-videos can be spliced ​​and then subjected to convolution operation to obtain a second designated feature map; a third designated feature map with the same scale as the third feature map is determined based on the first designated feature map and the second designated feature map; the third designated feature map and the third feature map are convoluted and upsampled to obtain the undetermined feature map of the target scale, and the target scale can be the same scale as the multi-channel feature map, or half the length and width of the multi-channel feature map, or other scales.

[0065] For example, Figure 4For example, after the first feature maps corresponding to the three sub-videos (front video, back video and middle video) output by the stage5 (fifth stage) are spliced, a 1×1 convolution operation and a 2-fold upsampling process are performed, and then a 1×1 convolution operation is performed to obtain a first designated feature map of 8×8. After the second feature maps corresponding to the three sub-videos output by the stage4 (fourth stage) are spliced, a convolution operation is performed to obtain a second designated feature map, and a convolution operation is performed on the first designated feature map after a convolution operation is performed, and then the second designated feature map is spliced, and then multiple convolutions are performed and then a 2-fold upsampling process is performed to obtain a third designated feature map of 16×16. After the third feature maps corresponding to the three sub-videos of 16×16 output by the stage3 (third stage) are spliced, a convolution operation is performed, and then the convolution operation is continued after splicing with the third designated feature map, and a 4-fold upsampling process is performed after multiple convolution operations, and a 1×1 convolution operation is performed after 4-fold upsampling to obtain the undetermined feature map.

[0066] Step 305: Perform row pooling processing on the undetermined feature map through the second classification module to obtain a first target feature map, and perform column pooling processing on the undetermined feature map to obtain a second target feature map.

[0067] In this step, the mean value of each row element in the pending feature map can be obtained to obtain the first target feature map; the mean value of each column element in the pending feature map can be obtained to obtain the second target feature map.

[0068] For example, if the size of the undetermined feature map F is H×W×C, the i-th element in the first target feature map can be calculated by the following formula: The jth element in the second target feature map can be calculated by the following formula Among them, 0 <i<H,0<j<W, i,j It is the element in the i-th row and j-th column of a certain layer in the undetermined feature map of the C-layer.

[0069] Step 306, predicting the vertical coordinates of the corner points in the frame according to the first target feature map, and predicting the horizontal coordinates of the corner points in the frame according to the second target feature map.

[0070] Wherein, the second classification module includes a first classifier, a second classifier, a third classifier and a fourth classifier. In this step, the first target feature map can be input into the first classifier and the second classifier respectively, so that the first classifier outputs the vertical coordinate of the upper left corner of the video, and the second classifier outputs the vertical coordinate of the lower right corner of the video. The second target feature map is input into the third classifier and the fourth classifier respectively, so that the third classifier outputs the horizontal coordinate of the upper left corner of the video, and the fourth classifier outputs the horizontal coordinate of the lower right corner of the video.

[0071] The above technical scheme can perform multi-scale feature extraction on multiple frames of images to be identified corresponding to multiple sub-videos, can simultaneously consider the temporal information and spatial information of the video, and by fusing feature maps of different scales, achieve the effect of effectively fusing the deep semantic features and shallow spatial features of the video, and then predict the target position based on the undetermined feature map, which can effectively improve the accuracy of the border position recognition result.

[0072] Figure 5 is a flow chart of a method for training a preset bounding box recognition model shown in an exemplary embodiment of the present disclosure; Figure 5 As shown above Figure 1-4 The preset bounding box recognition model shown in can be trained by the following steps:

[0073] S1, obtaining a plurality of video sample data, wherein the video sample data includes a sample video, first annotation data on whether the sample video has a border, and second annotation data on the position of the border.

[0074] S2, dividing each of the sample videos into a plurality of sub-video samples, and extracting a preset number of frames of to-be-recognized image samples from each of the sub-video samples to obtain a plurality of frames of to-be-recognized image samples.

[0075] In this step, each sample video may be divided into a plurality of sub-video samples according to the time sequence; when extracting frames, the preset numbers corresponding to different sub-video samples may be the same or different.

[0076] S3, input the multiple frames of image samples to be identified into a preset initial model, so that the preset initial model obtains a multi-scale feature map set corresponding to the multiple frames of image samples to be identified, and determines the prediction probability of whether the target video has a border according to the feature map with the smallest spatial resolution in the feature map set; when it is determined that the sample video has a border, the feature maps of different scales in the feature map set are merged to obtain a pending feature map sample, the pending feature map sample is row pooled to obtain a first target feature map sample, and the pending feature map is column pooled to obtain a second target feature map sample; the predicted position of the border is determined according to the first target feature map sample and the second target feature map sample.

[0077] Among them, the preset initial model may include an initial backbone network, and the initial backbone network extracts feature maps of different scales corresponding to multiple frames of the image samples to be identified to obtain the feature map set.

[0078] S4, obtaining a first loss value through a first preset loss function according to the predicted probability and the first labeled data, and obtaining a second loss value through a second preset loss function according to the predicted position and the second labeled data.

[0079] Among them, the first preset loss function and the second preset loss function can both be cross entropy loss functions.

[0080] S5, iteratively training the preset initial model according to the first loss value and the second loss value to obtain the preset bounding box recognition model.

[0081] In this step, the model parameters of the preset initial model are adjusted according to the first loss value and the second loss value, and then the iterative calculation is continued until the obtained loss function meets the preset threshold requirement, the iteration is stopped, and the updated preset initial model is used as the preset border recognition model.

[0082] The above technical solution can train a border recognition model with high recognition efficiency and strong reliability of recognition results. Since the preset border recognition model can fuse feature maps of different scales, it can effectively fuse the deep semantic features and shallow spatial features of the video, thereby effectively improving the accuracy of the border position recognition results.

[0083] Figure 6 is a block diagram of a video border recognition device shown in an exemplary embodiment of the present disclosure; Figure 6 As shown, the device may include:

[0084] The first acquisition module 601 is configured to acquire a target video to be identified and divide the target video into a plurality of sub-videos;

[0085] The second acquisition module 602 is configured to extract a preset number of frames of to-be-recognized images from each of the sub-videos to obtain multiple frames of to-be-recognized images;

[0086] The determination module 603 is configured to input the multiple frames of images to be recognized into a preset border recognition model so that the preset border recognition model outputs a border recognition result, and the border recognition result is used to characterize whether the target video has a border, and if the target video has a border, the target position of the border;

[0087] Among them, the preset border recognition model is used to obtain a multi-scale feature map set corresponding to the multiple frames of images to be recognized, and determine whether the target video has a border based on the feature map with the smallest spatial resolution in the feature map set; and when it is determined that the target video has a border, the feature maps of different scales in the feature map set are fused to obtain a pending feature map, and the target position is predicted based on the pending feature map.

[0088] The above technical scheme can simultaneously consider the temporal information and spatial information of the video by performing multi-scale feature extraction on multiple frames of images to be identified corresponding to multiple sub-videos, and can achieve the effect of effectively fusing the deep semantic features and shallow spatial features of the video by fusing feature maps of different scales. The target position is then predicted based on the feature map to be determined, which can effectively improve the accuracy of the border position recognition result.

[0089] Optionally, the preset border recognition model includes a backbone network module, a feature fusion module, a first classification module and a second classification module, the backbone network module includes a multi-level feature extraction subnetwork, and the feature extraction subnetworks at different levels are used to extract feature maps of different scales; the backbone network module is coupled with the feature fusion module and the first classification module, and the feature fusion module is also coupled with the second classification module; the preset border recognition model is used to:

[0090] Obtain a multi-channel feature map corresponding to each frame of the image to be recognized;

[0091] The multi-channel feature maps corresponding to the multiple sub-videos are spliced ​​and input into the backbone network module, so that the multi-level feature extraction sub-network outputs multiple feature maps of specified scales to obtain the feature map set;

[0092] Determine whether the target video has a border according to the feature map with the smallest spatial resolution in the feature map set by the first classification module;

[0093] When it is determined that the target video has a border, the feature fusion module fuses the feature maps of the multiple specified scales to obtain the undetermined feature map;

[0094] Performing row pooling processing on the undetermined feature map through the second classification module to obtain a first target feature map, and performing column pooling processing on the undetermined feature map to obtain a second target feature map;

[0095] The horizontal coordinates of the corner points in the frame are predicted according to the first target feature map, and the vertical coordinates of the corner points in the frame are predicted according to the second target feature map.

[0096] Optionally, the multiple feature maps of specified scales include a first feature map of a first scale, a second feature map of a second scale, and a third feature map of a third scale, the first scale is smaller than the second scale, the second scale is smaller than the third scale, and the preset bounding box recognition model is used to:

[0097] The first feature maps corresponding to the plurality of sub-videos are concatenated and then subjected to convolution and upsampling to obtain a first designated feature map having the same scale as the second feature map;

[0098] Concatenate the second feature maps corresponding to the multiple sub-videos and perform a convolution operation to obtain a second specified feature map;

[0099] Determine, according to the first designated feature map and the second designated feature map, a third designated feature map having the same scale as the third feature map;

[0100] The third designated feature map and the third feature map are convolved and up-sampled to obtain the undetermined feature map having the same scale as the multi-channel feature map.

[0101] Optionally, the preset border recognition model is used to:

[0102] Obtaining the mean of each row of elements in the pending feature map to obtain the first target feature map;

[0103] The mean of each column element in the undetermined feature map is obtained to obtain the second target feature map.

[0104] Optionally, the second classification module includes a first classifier, a second classifier, a third classifier and a fourth classifier, and the preset border recognition model is used to:

[0105] Inputting the first target feature map into the first classifier and the second classifier respectively, so that the first classifier outputs the vertical coordinate of the upper left corner of the video, and the second classifier outputs the vertical coordinate of the lower right corner of the video;

[0106] The second target feature map is input into the third classifier and the fourth classifier respectively, so that the third classifier outputs the horizontal coordinate of the upper left corner of the video, and the fourth classifier outputs the horizontal coordinate of the lower right corner of the video.

[0107] Optionally, the first acquisition module 601 is configured to:

[0108] According to the time sequence of the target video, the target video is divided into a front video, a middle video and a back video.

[0109] Optionally, the preset border recognition model is further used for:

[0110] The border category of the target video is determined according to the feature map with the smallest spatial resolution in the feature map set.

[0111] Optionally, the preset border recognition model is trained in the following manner:

[0112] Acquire multiple video sample data, the video sample data including a sample video, first annotation data of whether the sample video has a border, and second annotation data of the position of the border;

[0113] Dividing each sample video into a plurality of sub-video samples, and extracting a preset number of frames of to-be-recognized image samples from each sub-video sample to obtain a plurality of frames of to-be-recognized image samples;

[0114] The multiple frames of image samples to be identified are input into a preset initial model so that the preset initial model obtains a multi-scale feature map set corresponding to the multiple frames of image samples to be identified, and determines the prediction probability of whether the target video has a border according to the feature map with the smallest spatial resolution in the feature map set; and when it is determined that the sample video has a border, the feature maps of different scales in the feature map set are merged to obtain a pending feature map sample, the pending feature map sample is subjected to row pooling processing to obtain a first target feature map sample, and the pending feature map is subjected to column pooling processing to obtain a second target feature map sample; and the predicted position of the border is determined according to the first target feature map sample and the second target feature map sample;

[0115] Obtain a first loss value through a first preset loss function according to the predicted probability and the first labeled data, and obtain a second loss value through a second preset loss function according to the predicted position and the second labeled data;

[0116] The preset initial model is iteratively trained according to the first loss value and the second loss value to obtain the preset bounding box recognition model.

[0117] The above technical solution can train a border recognition model with high recognition efficiency and strong reliability of recognition results. Since the preset border recognition model can fuse feature maps of different scales, it can effectively fuse the deep semantic features and shallow spatial features of the video, thereby effectively improving the accuracy of the border position recognition results.

[0118] Reference below Figure 7 , which shows a schematic diagram of the structure of an electronic device 700 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0119] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0120] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0121] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0122] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0123] In some embodiments, any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol) can be used for communication, and can be interconnected with any form or medium of digital data communication (e.g., communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), internets (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

[0124] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0125] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a target video to be identified and divides the target video into multiple sub-videos; extracts a preset number of frames of images to be identified from each of the sub-videos to obtain multiple frames of images to be identified; inputs the multiple frames of images to be identified into a preset border recognition model so that the preset border recognition model outputs a border recognition result, which is used to characterize whether the target video has a border and, if the target video has a border, the target position of the border; wherein the preset border recognition model is used to obtain a multi-scale feature map set corresponding to the multiple frames of images to be identified, determine whether the target video has a border according to the feature map with the smallest spatial resolution in the feature map set; and, if it is determined that the target video has a border, fuse the feature maps of different scales in the feature map set to obtain a feature map to be determined, and predict the target position according to the feature map to be determined.

[0126] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0127] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0128] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of the module does not limit the module itself in some cases. For example, the first acquisition module may also be described as "a module for acquiring a target video to be identified and dividing the target video into multiple sub-videos".

[0129] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0130] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] According to one or more embodiments of the present disclosure, Example 1 provides a video border recognition method, the method comprising:

[0132] Obtain a target video to be identified, and divide the target video into multiple sub-videos;

[0133] Extracting a preset number of frames of images to be recognized from each of the sub-videos to obtain multiple frames of images to be recognized;

[0134] Inputting the multiple frames of images to be recognized into a preset border recognition model so that the preset border recognition model outputs a border recognition result, wherein the border recognition result is used to characterize whether the target video has a border and, if the target video has a border, the target position of the border;

[0135] Among them, the preset border recognition model is used to obtain a multi-scale feature map set corresponding to the multiple frames of images to be recognized, and determine whether the target video has a border based on the feature map with the smallest spatial resolution in the feature map set; and when it is determined that the target video has a border, the feature maps of different scales in the feature map set are merged to obtain a pending feature map, and the target position is predicted based on the pending feature map.

[0136] According to one or more embodiments of the present disclosure, Example 2 provides the method described in Example 1, wherein the preset border recognition model includes a backbone network module, a feature fusion module, a first classification module, and a second classification module, wherein the backbone network module includes a multi-level feature extraction subnetwork, and the feature extraction subnetworks at different levels are used to extract feature maps of different scales; the backbone network module is coupled with the feature fusion module and the first classification module, and the feature fusion module is also coupled with the second classification module; the preset border recognition model is used to:

[0137] Obtain a multi-channel feature map corresponding to each frame of the image to be recognized;

[0138] The multi-channel feature maps corresponding to the plurality of sub-videos are spliced ​​and input into the backbone network module, so that the multi-level feature extraction sub-network outputs a plurality of feature maps of specified scales to obtain the feature map set;

[0139] Determining whether a border exists in the target video according to the feature map with the smallest spatial resolution in the feature map set by the first classification module;

[0140] When it is determined that the target video has a border, the feature fusion module fuses the feature maps of the multiple specified scales to obtain the undetermined feature map;

[0141] Performing row pooling processing on the undetermined feature map through the second classification module to obtain a first target feature map, and performing column pooling processing on the undetermined feature map to obtain a second target feature map;

[0142] The vertical coordinates of the corner points in the frame are predicted according to the first target feature map, and the horizontal coordinates of the corner points in the frame are predicted according to the second target feature map.

[0143] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 2, wherein the multiple feature maps of specified scales include a first feature map of a first scale, a second feature map of a second scale, and a third feature map of a third scale, the first scale is smaller than the second scale, and the second scale is smaller than the third scale, and the feature fusion module is used to fuse the feature maps of the multiple specified scales to obtain the undetermined feature map, including:

[0144] The first feature maps corresponding to the plurality of sub-videos are concatenated and then subjected to convolution and upsampling to obtain a first designated feature map having the same scale as the second feature map;

[0145] Concatenate the second feature maps corresponding to the plurality of sub-videos and perform a convolution operation thereon to obtain a second designated feature map;

[0146] Determine, according to the first designated feature map and the second designated feature map, a third designated feature map having the same scale as the third feature map;

[0147] A convolution operation and an upsampling process are performed on the third designated feature map and the third feature map to obtain the undetermined feature map of the target scale.

[0148] According to one or more embodiments of the present disclosure, Example 4 provides the method of Example 2, wherein the second classification module performs row pooling processing on the undetermined feature map to obtain a first target feature map, and performs column pooling processing on the undetermined feature map to obtain a second target feature map, including:

[0149] Obtaining the mean of the elements in each row of the pending feature map to obtain the first target feature map;

[0150] The mean of the elements in each column of the undetermined feature map is obtained to obtain the second target feature map.

[0151] According to one or more embodiments of the present disclosure, Example 5 provides the method described in Example 2, wherein the second classification module includes a first classifier, a second classifier, a third classifier, and a fourth classifier, and the predicting the ordinate of the corner point in the frame according to the first target feature map, and predicting the abscissa of the corner point in the frame according to the second target feature map includes:

[0152] Inputting the first target feature map into the first classifier and the second classifier respectively, so that the first classifier outputs the vertical coordinate of the upper left corner of the video, and the second classifier outputs the vertical coordinate of the lower right corner of the video;

[0153] The second target feature map is input into the third classifier and the fourth classifier respectively, so that the third classifier outputs the horizontal coordinate of the upper left corner of the video, and the fourth classifier outputs the horizontal coordinate of the lower right corner of the video.

[0154] According to one or more embodiments of the present disclosure, Example 6 provides the method described in Example 1, wherein obtaining a target video to be identified and dividing the target video into a plurality of sub-videos includes:

[0155] According to the time sequence of the target video, the target video is divided into a front video, a middle video and a back video.

[0156] According to one or more embodiments of the present disclosure, Example 7 provides the method described in Example 1, wherein the preset border recognition model is further used to:

[0157] Determine the border category of the target video according to the feature map with the smallest spatial resolution in the feature map set.

[0158] According to one or more embodiments of the present disclosure, Example 8 provides a method according to any one of Examples 1-7, characterized in that the preset border recognition model is trained in the following manner:

[0159] Acquire multiple video sample data, wherein the video sample data includes a sample video, first annotation data of whether the sample video has a border, and second annotation data of the position of the border;

[0160] Dividing each of the sample videos into a plurality of sub-video samples, and extracting a preset number of frames of to-be-recognized image samples from each of the sub-video samples to obtain a plurality of frames of to-be-recognized image samples;

[0161] Input the multiple frames of image samples to be identified into a preset initial model, so that the preset initial model obtains a multi-scale feature map set corresponding to the multiple frames of image samples to be identified, and determines the prediction probability of whether the target video has a border according to the feature map with the smallest spatial resolution in the feature map set; and when it is determined that the sample video has a border, fuses the feature maps of different scales in the feature map set to obtain a pending feature map sample, performs row pooling processing on the pending feature map sample to obtain a first target feature map sample, and performs column pooling processing on the pending feature map to obtain a second target feature map sample; and determines the predicted position of the border according to the first target feature map sample and the second target feature map sample;

[0162] Obtain a first loss value through a first preset loss function according to the predicted probability and the first labeled data, and obtain a second loss value through a second preset loss function according to the predicted position and the second labeled data;

[0163] The preset initial model is iteratively trained according to the first loss value and the second loss value to obtain the preset bounding box recognition model.

[0164] According to one or more embodiments of the present disclosure, Example 9 provides a video border recognition device, the device comprising:

[0165] A first acquisition module is configured to acquire a target video to be identified and divide the target video into a plurality of sub-videos;

[0166] A second acquisition module is configured to extract a preset number of frames of images to be recognized from each of the sub-videos to obtain multiple frames of images to be recognized;

[0167] A determination module is configured to input the multiple frames of images to be identified into a preset border recognition model so that the preset border recognition model outputs a border recognition result, wherein the border recognition result is used to characterize whether there is a border in the target video and, if there is a border in the target video, a target position of the border;

[0168] Among them, the preset border recognition model is used to obtain a multi-scale feature map set corresponding to the multiple frames of images to be recognized, and determine whether the target video has a border based on the feature map with the smallest spatial resolution in the feature map set; and when it is determined that the target video has a border, the feature maps of different scales in the feature map set are merged to obtain a pending feature map, and the target position is predicted based on the pending feature map.

[0169] According to one or more embodiments of the present disclosure, Example 10 provides a computer-readable medium having a computer program stored thereon, which implements the steps of the method described in any one of Examples 1-8 above when executed by a processing device.

[0170] According to one or more embodiments of the present disclosure, Example 11 provides an electronic device, including:

[0171] a storage device having a computer program stored thereon;

[0172] A processing device is used to execute the computer program in the storage device to implement the steps of the method described in any one of Examples 1-8 above.

[0173] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0174] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0175] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A video border recognition method, characterized in that: The method comprises: Obtain a target video to be identified, and divide the target video into multiple sub-videos; Extracting a preset number of frames of images to be recognized from each of the sub-videos to obtain multiple frames of images to be recognized; Inputting the multiple frames of images to be recognized into a preset border recognition model so that the preset border recognition model outputs a border recognition result, wherein the border recognition result is used to characterize whether the target video has a border and, if the target video has a border, the target position of the border; The preset border recognition model is used to obtain a multi-scale feature map set corresponding to the multiple frames of images to be recognized, and determine whether the target video has a border according to the feature map with the smallest spatial resolution in the feature map set; and when it is determined that the target video has a border, the feature maps of different scales in the feature map set are merged to obtain a pending feature map, and the target position is predicted according to the pending feature map; The preset border recognition model includes a backbone network module, a feature fusion module, a first classification module and a second classification module. The backbone network module includes a multi-level feature extraction subnetwork, and the feature extraction subnetworks at different levels are used to extract feature maps of different scales; the backbone network module is coupled with the feature fusion module and the first classification module, and the feature fusion module is also coupled with the second classification module; the preset border recognition model is used to: Obtain a multi-channel feature map corresponding to each frame of the image to be recognized; The multi-channel feature maps corresponding to the plurality of sub-videos are spliced ​​and input into the backbone network module, so that the multi-level feature extraction sub-network outputs a plurality of feature maps of specified scales to obtain the feature map set; Determining whether a border exists in the target video according to the feature map with the smallest spatial resolution in the feature map set by the first classification module; When it is determined that the target video has a border, the feature fusion module fuses the feature maps of the multiple specified scales to obtain the undetermined feature map; Performing row pooling processing on the undetermined feature map through the second classification module to obtain a first target feature map, and performing column pooling processing on the undetermined feature map to obtain a second target feature map; Predicting the ordinates of the corner points in the frame according to the first target feature map, and predicting the abscissas of the corner points in the frame according to the second target feature map; The second classification module includes a first classifier, a second classifier, a third classifier and a fourth classifier, and the predicting the ordinate of the corner point in the frame according to the first target feature map and the predicting the abscissa of the corner point in the frame according to the second target feature map includes: Inputting the first target feature map into the first classifier and the second classifier respectively, so that the first classifier outputs the vertical coordinate of the upper left corner of the video, and the second classifier outputs the vertical coordinate of the lower right corner of the video; The second target feature map is input into the third classifier and the fourth classifier respectively, so that the third classifier outputs the horizontal coordinate of the upper left corner of the video, and the fourth classifier outputs the horizontal coordinate of the lower right corner of the video.

2. The method according to claim 1, characterized in that The multiple feature maps of the specified scales include a first feature map of a first scale, a second feature map of a second scale, and a third feature map of a third scale, the first scale is smaller than the second scale, and the second scale is smaller than the third scale, and fusing the feature maps of the multiple specified scales by the feature fusion module to obtain the undetermined feature map, including: The first feature maps corresponding to the plurality of sub-videos are concatenated and then subjected to convolution and upsampling to obtain a first designated feature map having the same scale as the second feature map; Concatenate the second feature maps corresponding to the plurality of sub-videos and perform a convolution operation thereon to obtain a second designated feature map; Determine, according to the first designated feature map and the second designated feature map, a third designated feature map having the same scale as the third feature map; A convolution operation and an upsampling process are performed on the third designated feature map and the third feature map to obtain the undetermined feature map of the target scale.

3. The method according to claim 1, characterized in that The step of performing row pooling processing on the undetermined feature map by the second classification module to obtain a first target feature map, and performing column pooling processing on the undetermined feature map to obtain a second target feature map includes: Obtaining the mean of the elements in each row of the pending feature map to obtain the first target feature map; The mean of the elements in each column of the undetermined feature map is obtained to obtain the second target feature map.

4. The method according to claim 1, characterized in that: The step of obtaining a target video to be identified and dividing the target video into a plurality of sub-videos includes: According to the time sequence of the target video, the target video is divided into a front video, a middle video and a back video.

5. The method according to claim 1, characterized in that The preset border recognition model is also used for: Determine the border category of the target video according to the feature map with the smallest spatial resolution in the feature map set.

6. The method according to any one of claims 1 to 5, characterized in that: The preset border recognition model is trained in the following way: Acquire multiple video sample data, wherein the video sample data includes a sample video, first annotation data of whether the sample video has a border, and second annotation data of the position of the border; Dividing each of the sample videos into a plurality of sub-video samples, and extracting a preset number of frames of to-be-recognized image samples from each of the sub-video samples to obtain a plurality of frames of to-be-recognized image samples; Inputting the multiple frames of image samples to be identified into a preset initial model so that the preset initial model obtains a multi-scale feature map set corresponding to the multiple frames of image samples to be identified, and determining a prediction probability of whether the target video has a border according to a feature map with the smallest spatial resolution in the feature map set; When it is determined that the sample video has a border, feature maps of different scales in the feature map set are merged to obtain a pending feature map sample, row pooling is performed on the pending feature map sample to obtain a first target feature map sample, and column pooling is performed on the pending feature map to obtain a second target feature map sample; Determining a predicted position of the border according to the first target feature map sample and the second target feature map sample; Obtain a first loss value through a first preset loss function according to the predicted probability and the first labeled data, and obtain a second loss value through a second preset loss function according to the predicted position and the second labeled data; The preset initial model is iteratively trained according to the first loss value and the second loss value to obtain the preset bounding box recognition model.

7. A video border recognition device, characterized in that: The device comprises: A first acquisition module is configured to acquire a target video to be identified and divide the target video into a plurality of sub-videos; A second acquisition module is configured to extract a preset number of frames of images to be recognized from each of the sub-videos to obtain multiple frames of images to be recognized; A determination module is configured to input the multiple frames of images to be identified into a preset border recognition model so that the preset border recognition model outputs a border recognition result, wherein the border recognition result is used to characterize whether there is a border in the target video and, if there is a border in the target video, a target position of the border; Wherein, the preset border recognition model is used to obtain a multi-scale feature map set corresponding to the multiple frames of images to be recognized, and determine whether the target video has a border according to the feature map with the smallest spatial resolution in the feature map set; and when it is determined that the target video has a border, the feature maps of different scales in the feature map set are fused to obtain a pending feature map, and the target position is predicted according to the pending feature map; the preset border recognition model includes a backbone network module, a feature fusion module, a first classification module and a second classification module, the backbone network module includes a multi-level feature extraction subnetwork, and the feature extraction subnetworks at different levels are used to extract feature maps of different scales; the backbone network module is coupled with the feature fusion module and the first classification module, and the feature fusion module is also coupled with the second classification module; the preset border recognition model is used to: Obtain a multi-channel feature map corresponding to each frame of the image to be recognized; The multi-channel feature maps corresponding to the plurality of sub-videos are spliced ​​and input into the backbone network module, so that the multi-level feature extraction sub-network outputs a plurality of feature maps of specified scales to obtain the feature map set; Determining whether a border exists in the target video according to the feature map with the smallest spatial resolution in the feature map set by the first classification module; When it is determined that the target video has a border, the feature fusion module fuses the feature maps of the multiple specified scales to obtain the undetermined feature map; Performing row pooling processing on the undetermined feature map through the second classification module to obtain a first target feature map, and performing column pooling processing on the undetermined feature map to obtain a second target feature map; Predicting the ordinates of the corner points in the frame according to the first target feature map, and predicting the abscissas of the corner points in the frame according to the second target feature map; The second classification module includes a first classifier, a second classifier, a third classifier and a fourth classifier, and the predicting the ordinate of the corner point in the frame according to the first target feature map and the predicting the abscissa of the corner point in the frame according to the second target feature map includes: Inputting the first target feature map into the first classifier and the second classifier respectively, so that the first classifier outputs the vertical coordinate of the upper left corner of the video, and the second classifier outputs the vertical coordinate of the lower right corner of the video; The second target feature map is input into the third classifier and the fourth classifier respectively, so that the third classifier outputs the horizontal coordinate of the upper left corner of the video, and the fourth classifier outputs the horizontal coordinate of the lower right corner of the video.

8. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processing device, the steps of the method described in any one of claims 1 to 6 are implemented.

9. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 6.

Images