A shot boundary detection method, model training method and related device

Abstract

The present application discloses a method for shot boundary detection, including: obtaining a video to be detected, the video to be detected includes at least one video clip; obtaining the shot type corresponding to the video clip in the video to be detected, the shot type is a first shot type or a second shot type; if the shot type belongs to the first shot type, determining the image change value corresponding to the video to be detected, wherein the image change value is obtained by calculating two adjacent frames of images in the video to be detected; determining the shot boundary information of the video to be detected according to the image change value, wherein the shot boundary information includes the shot change type and the shot change position. The present application also discloses a model training method and device. The present application combines a deep learning algorithm and a method for counting adjacent image frames, which can provide richer information for the detection of shot boundaries, thereby reducing the missed detection rate of shot boundaries and improving the detection effect of shot boundaries.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a shot boundary detection method, a model training method, and related devices. Background Art

[0002] The basis for shot boundary detection is that the visual content within a shot is similar, while the video content between different shots varies significantly, with distinct features. Shot changes in videos are primarily categorized as abrupt cuts and gradual cuts. A sudden cut occurs when the last frame of the previous shot is immediately followed by the first frame of the next shot without a transition, while a gradual cut occurs when the shot transition is completed gradually over a period of time, typically lasting from a few to a dozen frames.

[0003] Currently, traditional shot detection algorithms typically analyze changes in video content, such as detecting shot boundaries based on changes in image intensity. Another approach is to analyze shot boundaries based on histograms.

[0004] However, existing shot detection algorithms have a high missed detection rate for gradual shots. For sudden shots, if the color content of the shots on both sides is similar, they are easily mistaken for the same shot, resulting in poor detection of shot boundaries. Summary of the Invention

[0005] The embodiments of the present application provide a method, model training method, and related devices for shot boundary detection. Combined with a deep learning algorithm and a method for counting adjacent image frames, these methods can provide richer information for shot boundary detection, thereby reducing the missed detection rate of shot boundaries and improving the detection effect of shot boundaries.

[0006] In view of this, a first aspect of the present application provides a method for shot boundary detection, comprising:

[0007] Acquire a video to be detected, wherein the video to be detected includes at least one video segment, and the video segment includes multiple frames of continuous images;

[0008] Obtaining a shot type corresponding to a video clip in the video to be detected, wherein the shot type is a first shot type or a second shot type, the first shot type indicating a shot with changes, and the second shot type indicating a shot without changes;

[0009] If the shot type belongs to the first shot type, determining an image change value corresponding to the video segment in the video to be detected, wherein the image change value is obtained by calculating two adjacent frames of images in the video segment in the video to be detected;

[0010] The shot boundary information of the video segment in the to-be-detected video is determined according to the image change value, wherein the shot boundary information includes a shot change type and a shot change position.

[0011] A second aspect of the present application provides a model training method, comprising:

[0012] Acquire a set of video segments to be trained, wherein the set of video segments to be trained includes at least one video segment to be trained;

[0013] Obtaining a predicted shot type corresponding to each to-be-trained video segment in the to-be-trained video segment set, wherein the predicted shot type is a first shot type or a second shot type, the first shot type indicating a shot with a change, and the second shot type indicating a shot without a change;

[0014] Training the shot boundary prediction model to be trained using a loss function according to the predicted shot type and the real shot type corresponding to each video clip to be trained, to obtain model parameters, wherein the real shot type is obtained by pre-labeling the video clip to be trained;

[0015] A shot boundary prediction model is generated according to the model parameters.

[0016] A third aspect of the present application provides a shot boundary detection device, comprising:

[0017] An acquisition module is configured to acquire a video to be detected, wherein the video to be detected includes at least one video segment, and the video segment includes multiple frames of continuous images;

[0018] The acquisition module is further configured to acquire a shot type corresponding to a video clip in the video to be detected, wherein the shot type is a first shot type or a second shot type, the first shot type indicating a shot with changes, and the second shot type indicating a shot without changes;

[0019] a determination module configured to determine, if the shot type acquired by the acquisition module belongs to the first shot type, an image change value corresponding to the video segment in the video to be detected, wherein the image change value is obtained by calculating two adjacent frames of images in the video segment in the video to be detected;

[0020] The determination module is further configured to determine shot boundary information of the video segment in the to-be-detected video according to the image change value, wherein the shot boundary information includes a shot change type and a shot change position.

[0021] In a possible design, in a first implementation of the third aspect of the embodiments of the present application,

[0022] The acquisition module is specifically configured to acquire segment extraction information, wherein the segment extraction information includes a batch value of the video segment, a number of channels, a height of an image in the video segment, a width of an image in the video segment, and a number of frames of the image in the video segment;

[0023] The video segment is acquired according to the segment extraction information, wherein the video segment belongs to the video to be detected.

[0024] In a possible design, in the second implementation of the third aspect of the embodiments of the present application,

[0025] The acquisition module is specifically configured to acquire first segment extraction information, wherein the segment extraction information includes a batch value of the first video segment, a number of channels, a height of an image in the first video segment, a width of the image in the first video segment, and a number of frames of the image in the first video segment;

[0026] Acquire second segment extraction information, wherein the segment extraction information includes a batch value of the second video segment, a number of channels, a height of an image in the second video segment, a width of the image in the second video segment, and a number of frames of the image in the second video segment;

[0027] Acquire the first video segment according to the first segment extraction information, wherein the first video segment includes M consecutive images, and the first video segment belongs to the video to be detected;

[0028] The second video segment is obtained according to the second segment extraction information, wherein the second video segment includes M continuous images, the second video segment belongs to the video to be detected, and the first video segment and the second video segment have the same N continuous images, where N is an integer greater than or equal to 1 and less than M.

[0029] In a possible design, in a third implementation of the third aspect of the embodiments of the present application,

[0030] The acquisition module is specifically configured to acquire the shot type corresponding to the first video clip through a shot boundary prediction model;

[0031] Obtaining a shot type corresponding to the second video clip by using a shot boundary prediction model;

[0032] The determining module is specifically configured to generate a target video segment based on the first video segment and the second video segment if the shot type corresponding to the first video segment and the shot type corresponding to the second video segment are both the first shot type, wherein the target video segment includes (2M-N) consecutive images;

[0033] The image change value is determined according to the target video segment.

[0034] In a possible design, in a fourth implementation of the third aspect of the embodiments of the present application,

[0035] The determining module is specifically configured to obtain a first image frame, a second image frame, and a third image frame from the video segment of the video to be detected, wherein the first image frame is an image frame immediately preceding the second image frame, and the second image frame is an image frame immediately preceding the third image frame;

[0036] Obtaining a first color model corresponding to the first image frame, a second color model corresponding to the second image frame, and a third color model corresponding to the third image frame, wherein the first color model includes a first hue parameter, a first saturation parameter, and a first lightness parameter, the second color model includes a second hue parameter, a second saturation parameter, and a second lightness parameter, and the third color model includes a third hue parameter, a third saturation parameter, and a third lightness parameter;

[0037] Calculating a first hue difference value based on the first hue parameter and the second hue parameter, calculating a first saturation difference value based on the first saturation parameter and the second saturation parameter, and calculating a first lightness difference value based on the first lightness parameter and the second lightness parameter;

[0038] Calculating a first average difference value according to the first hue difference value, the first saturation difference value, and the first lightness difference value;

[0039] Calculating a second hue difference value based on the second hue parameter and the third hue parameter, calculating a second saturation difference value based on the second saturation parameter and the third saturation parameter, and calculating a second lightness difference value based on the second lightness parameter and the third lightness parameter;

[0040] Calculating a first average difference value according to the second hue difference value, the second saturation difference value, and the second lightness difference value;

[0041] An image change value corresponding to the video segment in the video to be detected is determined according to the first average difference and the second average difference.

[0042] In a possible design, in a fifth implementation of the third aspect of the embodiments of the present application,

[0043] The determining module is specifically configured to determine that the shot change type of the video segment in the video to be detected is a shot mutation if the image change value corresponding to the video segment in the video to be detected is greater than or equal to a first threshold;

[0044] If the image change value corresponding to the video segment in the to-be-detected video is less than the first threshold, determining that the shot change type of the video segment in the to-be-detected video is a shot gradient;

[0045] The lens change position is determined according to the lens change type.

[0046] In a possible design, in a sixth implementation of the third aspect of the embodiments of the present application,

[0047] The determining module is specifically configured to determine that the shot change position is between the first image frame and the second image frame if the shot change type is the sudden shot change and the first average difference is greater than the second average difference;

[0048] If the shot change type is the sudden shot change, and the second average difference is greater than the first average difference, the shot change position is determined to be between the second image frame and the third image frame.

[0049] In a possible design, in a seventh implementation of the third aspect of the embodiments of the present application,

[0050] The determining module is specifically configured to determine that the shot change position includes the first image frame and the second image frame if the shot change type is the shot gradual change and the first average difference is greater than a second threshold;

[0051] If the lens change type is the lens gradual change, and the second average difference is greater than the second threshold, determining that the lens change position includes the second image frame and the third image frame;

[0052] If the lens change type is the lens gradual change, and the first average difference and the second average difference are both greater than the second threshold, it is determined that the lens change position includes the first image frame, the second image frame, and the third image frame.

[0053] A fourth aspect of the present application provides a model training device, comprising:

[0054] An acquisition module, configured to acquire a set of video segments to be trained, wherein the set of video segments to be trained includes at least one video segment to be trained;

[0055] The acquisition module is further configured to acquire a predicted shot type corresponding to each to-be-trained video segment in the to-be-trained video segment set, wherein the predicted shot type is a first shot type or a second shot type, the first shot type indicating a shot with a change, and the second shot type indicating a shot without a change;

[0056] a training module configured to train the shot boundary prediction model to be trained using a loss function based on the predicted shot type and the real shot type corresponding to each video clip to be trained obtained by the acquisition module, to obtain model parameters, wherein the real shot type is obtained by pre-labeling the video clip to be trained;

[0057] A generation module is used to generate a shot boundary prediction model according to the model parameters.

[0058] In one possible design, in a first implementation of the fourth aspect of the embodiments of the present application, the model training device includes a determination module;

[0059] The determining module is configured to determine, after the acquiring module acquires the set of video segments to be trained, a real shot type corresponding to each of the video segments to be trained based on the set of video segments to be trained, wherein the real shot type is a first sub-shot type, a second sub-shot type, or the first shot type, and both the first sub-shot type and the second sub-shot type belong to the first shot type;

[0060] The determining module is further configured to determine the number of the second shot type, the number of the first sub-shot type, and the number of the second sub-shot type according to the real shot type corresponding to each of the video segments to be trained;

[0061] The determining module is further configured to determine a first pre-weighting of the second lens type, a second pre-weighting of the first sub-lens type, and a third pre-weighting of the second sub-lens type based on the number of the second lens type, the number of the first sub-lens type, and the number of the second sub-lens type;

[0062] The training module is specifically configured to train the shot boundary prediction model to be trained using the loss function according to the first pre-weight, the second pre-weight, and the third pre-weight to obtain model parameters.

[0063] A fifth aspect of the present application provides a terminal device, comprising: a memory, a transceiver, a processor, and a bus system;

[0064] Wherein, the memory is used to store programs;

[0065] The processor is configured to execute the program in the memory, comprising the following steps:

[0066] Acquire a video to be detected, wherein the video to be detected includes at least one video segment, and the video segment includes multiple frames of continuous images;

[0067] Obtaining a shot type corresponding to a video clip in the video to be detected, wherein the shot type is a first shot type or a second shot type, the first shot type indicating a shot with changes, and the second shot type indicating a shot without changes;

[0068] If the shot type belongs to the first shot type, determining an image change value corresponding to the video segment in the video to be detected, wherein the image change value is obtained by calculating two adjacent frames of images in the video segment in the video to be detected;

[0069] Determining shot boundary information of a video segment in the to-be-detected video according to the image change value, wherein the shot boundary information includes a shot change type and a shot change position;

[0070] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0071] A sixth aspect of the present application provides a server, comprising: a memory, a transceiver, a processor, and a bus system;

[0072] Wherein, the memory is used to store programs;

[0073] The processor is configured to execute the program in the memory, comprising the following steps:

[0074] Acquire a set of video segments to be trained, wherein the set of video segments to be trained includes at least one video segment to be trained;

[0075] Obtaining a predicted shot type corresponding to each to-be-trained video segment in the to-be-trained video segment set, wherein the predicted shot type is a first shot type or a second shot type, the first shot type indicating a shot with a change, and the second shot type indicating a shot without a change;

[0076] Training the shot boundary prediction model to be trained using a loss function according to the predicted shot type and the real shot type corresponding to each video clip to be trained, to obtain model parameters, wherein the real shot type is obtained by pre-labeling the video clip to be trained;

[0077] generating a shot boundary prediction model according to the model parameters;

[0078] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0079] A seventh aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the methods described in the above aspects.

[0080] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0081] In an embodiment of the present application, a method for shot boundary detection is provided. First, a video to be detected is obtained, and then the shot type corresponding to a video clip in the video to be detected is obtained, wherein the shot type is either a first shot type or a second shot type, wherein the first shot type indicates a shot with changes, and the second shot type indicates a shot without changes. If the shot type belongs to the first shot type, an image change value corresponding to the video to be detected is determined, and finally, shot boundary information of the video to be detected is determined based on the image change value, wherein the shot boundary information includes the shot change type and the shot change position. Through the above method, a two-layer detection mechanism is designed for different shot types. First, a deep learning algorithm is used to make a preliminary shot type determination on the video clip. For video clips with shot changes, further analysis is performed to determine whether they are sudden shot changes or gradual shot changes. The specific change position is detected based on different shots. Therefore, combining the deep learning algorithm with the adjacent image frame statistics method can provide richer information for shot boundary detection, thereby reducing the missed detection rate of shot boundaries and improving the detection effect of shot boundaries. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 This is a schematic diagram of the architecture of a shot boundary detection system in an embodiment of the present application;

[0083] Figure 2 A flowchart of a method for shot boundary detection in an embodiment of the present application;

[0084] Figure 3 This is a schematic diagram of an embodiment of a method for shot boundary detection in an embodiment of the present application;

[0085] Figure 4 A schematic diagram of an embodiment of selecting a video clip in an embodiment of the present application;

[0086] Figure 5 This is a schematic diagram of an embodiment of a lens mashing error in an embodiment of the present application;

[0087] Figure 6 This is a schematic diagram of a process for generating shot boundary information based on a shot change in an embodiment of the present application;

[0088] Figure 7 This is a schematic diagram of an embodiment of a lens mutation in the embodiment of the present application;

[0089] Figure 8 This is a schematic diagram of an embodiment of lens gradient in the embodiment of the present application;

[0090] Figure 9 This is a schematic diagram of an embodiment of the model training method in the embodiment of the present application;

[0091] Figure 10 Schematic diagram of a shot boundary prediction model in an embodiment of the present application;

[0092] Figure 11 A structural diagram of a basic block in an embodiment of the present application;

[0093] Figure 12 This is a schematic diagram of an embodiment of a shot boundary detection device in an embodiment of the present application;

[0094] Figure 13 This is a schematic diagram of an embodiment of a model training device in an embodiment of the present application;

[0095] Figure 14 This is a schematic diagram of another embodiment of the model training device in the embodiment of the present application;

[0096] Figure 15 This is a schematic diagram of the structure of a terminal device in an embodiment of the present application;

[0097] Figure 16 This is a structural diagram of the server in an embodiment of the present application. DETAILED DESCRIPTION

[0098] The embodiments of the present application provide a method, model training method, and related devices for shot boundary detection. Combined with a deep learning algorithm and a method for counting adjacent image frames, these methods can provide richer information for shot boundary detection, thereby reducing the missed detection rate of shot boundaries and improving the detection effect of shot boundaries.

[0099] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the numbers used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "corresponding to" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0100] With the significant increase in network transmission speed, multimedia technology has been widely used, greatly enriching people's daily lives. Among multimedia information, video data accounts for a large proportion. As video data increases day by day, the effective organization, management, query and retrieval of massive video information has become an urgent need.

[0101] It should be understood that the shot boundary detection method provided in this application can be used for key frame extraction in videos. According to the order of video content granularity from coarse to fine, it can be divided into four levels: video, scene, shot, and key frame. Among them, scene refers to a group of shots that are semantically related and temporally adjacent. It is the basic unit of video at the semantic layer and can express high-level concepts or stories contained in the video. Shot refers to a sequence of images generated by a single operation of a camera. A video is usually composed of one or more shots. The structuring of video includes shot segmentation and key frame extraction. Shot segmentation is also called shot change detection and is the basis of video structure hierarchy. Key frames can be extracted based on the shot segmentation results. Key frames are representatives of the visual content of the shot scene. The selection of key frames greatly reduces the amount of data in the video index, greatly improves the transmission, storage, and management of video information in the network, and effectively reduces network transmission pressure. Visual feature extraction is performed on the key frames and the motion features of the shots to which they belong are extracted, which are stored in a large-scale video database. The features of the user-input video are extracted and matched with the video features in the database to achieve video retrieval. Shot segmentation can also provide shot data and shot-based keyframe data for action and animation recognition tasks. In security interception, shot segmentation provides sampled shot boundaries for logo detection and horror detection tasks. Keyframes derived from shot segmentation can also be used to generate video cover images, displaying representative cover images.

[0102] For ease of understanding, this application proposes a shot boundary detection method, which is applied to Figure 1 The shot boundary detection system shown in Figure 1 , Figure 1 This is a schematic diagram of the architecture of the shot boundary detection system in an embodiment of the present application. As shown in the figure, the present application can implement shot boundary detection in both offline and online modes. In the offline state, the client obtains the video to be detected shot by the user, or directly extracts the video to be detected from a locally stored video, and then inputs the video to be detected into a trained shot boundary prediction model. A video to be detected can be divided into at least one video segment, and the shot boundary prediction model outputs the shot type corresponding to each video segment. For variable shot types, further shot boundary information detection is required. In the online state, the client obtains the video to be detected shot by the user, or directly extracts the video to be detected from a locally stored video, or downloads the video to be detected from a server, and then inputs the video to be detected into a trained shot boundary prediction model. The shot boundary prediction model is stored on the server side. Therefore, the video to be detected needs to be sent to the server, and the server outputs the shot type corresponding to each video segment in the video to be detected through the shot boundary prediction model.

[0103] It should be noted that the client is deployed on a terminal device, wherein the terminal device includes but is not limited to a tablet computer, a laptop computer, a PDA, a mobile phone, a voice interaction device and a personal computer (PC), which is not limited here.

[0104] It should be understood that this application provides a shot boundary detection process, please refer to Figure 2 , Figure 2 The following is a flow chart of a method for detecting shot boundaries according to an embodiment of the present application. Specifically:

[0105] In step S1, a video is first obtained, which may be a video to be detected;

[0106] In step S2, images in time sequence are obtained from the video;

[0107] In step S3, video segments are collected from the time sequence. At least one video segment may be collected. Each video segment has a time overlap, and the overlap may be set to half the number of frames in the video segment.

[0108] In step S4, the video clip is input into a shot boundary prediction model;

[0109] In step S5, the shot boundary prediction model outputs the shot change type of the video segment, and then performs key frame positioning on the video segment with shot changes;

[0110] In step S6, the specific key frame position is determined.

[0111] In combination with the above introduction, the following will introduce the method of shot boundary detection in this application. Figure 3 In one embodiment of the present application, a method for detecting shot boundaries includes:

[0112] 101. Acquire a video to be detected, wherein the video to be detected includes at least one video segment, and each video segment includes multiple frames of continuous images;

[0113] In this embodiment, a shot boundary detection device obtains a video to be detected. The shot boundary detection device is typically deployed on a terminal device, but in practical applications, it can also be deployed on a server, without limitation. The video to be detected can be captured by a user, stored in a database, or downloaded from the cloud, without limitation. The types of videos to be detected include, but are not limited to, sports videos, news, and movies.

[0114] The video to be detected includes at least one video segment, which is divided from the video to be detected in time sequence. Assuming that the video to be detected has a total of 160 frames of images and a video segment has 16 frames of images, then the video to be detected can be divided into 10 video segments. It can be understood that it can also be divided into more than 10 video segments, and each video segment has a certain degree of overlap.

[0115] 102. Obtain a shot type corresponding to a video clip in the video to be detected, where the shot type is a first shot type or a second shot type, where the first shot type indicates a shot with changes, and the second shot type indicates a shot without changes;

[0116] In this embodiment, after acquiring each video segment in the video to be detected, the shot boundary detection device can input each video segment into a shot boundary prediction model, or can input a portion of the video segments into the shot boundary prediction model. The shot boundary prediction model then outputs the shot type corresponding to each video segment. It is understood that the shot type can be a first shot type or a second shot type, where the first shot type represents a shot with variations, and the second shot type represents a shot without variations. Using the shot boundary prediction model for detection not only eliminates the need for manual reference, thereby improving detection efficiency, but also provides a stronger detection capability, thereby increasing the accuracy of the detection results.

[0117] A shot is the complete segment between the previous optical transition and the next. Shot segmentation involves dividing the video to be detected into several shots. Shots typically have clear boundaries between them, known as boundary frames. Shots are segmented and contain key frames that represent each shot.

[0118] It is understandable that the shot boundary detection device may also detect shot types in other ways, such as detecting whether a video segment sends a shot change every 0.5 seconds.

[0119] The shot boundary prediction model can use the resnet-18 network, which specifically includes a convolutional layer, a batch normalization layer, an activation function layer, a maximum pooling layer, a basic block, an average pooling layer, and a linear regression layer.

[0120] It is understandable that the last layer can also be a softmax classifier. This application uses a linear regression layer, but this should not be understood as a limitation to this application.

[0121] 103. If the shot type belongs to the first shot type, determine an image change value corresponding to the video segment in the video to be detected, wherein the image change value is obtained by calculating two adjacent frames of images in the video segment in the video to be detected;

[0122] In this embodiment, if a shot type of a video segment in the video to be detected is detected as belonging to the first shot type, the shot boundary detection device determines the image change value of the video segment. The image change value is calculated based on two adjacent frames in the video segment. Assuming that the video to be detected includes 10 video segments, 3 of which belong to the first shot type, the corresponding image change values ​​are calculated based on two adjacent frames in the video segment. The other 7 video segments belong to the second shot type, meaning that image change values ​​do not need to be calculated for these 7 video segments. Therefore, the entire video to be detected includes the image change values ​​corresponding to these 3 video segments.

[0123] 104. Determine shot boundary information of the video segment in the to-be-detected video according to the image change value, wherein the shot boundary information includes a shot change type and a shot change position.

[0124] In this embodiment, the shot boundary detection device further detects the shot boundary information of video segments with shot changes in the video to be detected, thereby obtaining the shot boundary information of the video to be detected. Assuming that the video to be detected includes 10 video segments, 3 of which belong to the first shot type, the corresponding image change values ​​are calculated based on two adjacent frames of images in the video segments. The shot boundary information of each video segment is then determined based on the image change values ​​of these three video segments, thereby obtaining the shot boundary information of the entire video to be detected. The shot boundary information includes the shot change type and the shot change position. The shot change type can be a sudden shot change or a gradual shot change, and the shot change position can refer to the change intersection point in the case of a sudden shot change or the change area in the case of a gradual shot change.

[0125] In an embodiment of the present application, a method for shot boundary detection is provided. First, a video to be detected is obtained, and then the shot type corresponding to a video clip in the video to be detected is obtained, wherein the shot type is either a first shot type or a second shot type, wherein the first shot type indicates a shot with changes, and the second shot type indicates a shot without changes. If the shot type belongs to the first shot type, an image change value corresponding to the video to be detected is determined, and finally, shot boundary information of the video to be detected is determined based on the image change value, wherein the shot boundary information includes the shot change type and the shot change position. Through the above method, a two-layer detection mechanism is designed for different shot types. First, a deep learning algorithm is used to make a preliminary shot type determination on the video clip. For video clips with shot changes, further analysis is performed to determine whether they are sudden shot changes or gradual shot changes. The specific change position is detected based on different shots. Therefore, combining the deep learning algorithm with the adjacent image frame statistics method can provide richer information for shot boundary detection, thereby reducing the missed detection rate of shot boundaries and improving the detection effect of shot boundaries.

[0126] Optionally, in the above Figure 3 Based on the corresponding embodiments, in an optional embodiment of the shot boundary detection method provided in the embodiments of the present application, obtaining a video to be detected may include:

[0127] Obtaining segment extraction information, wherein the segment extraction information includes a batch value of the video segment, a number of channels, a height of an image in the video segment, a width of an image in the video segment, and a number of frames of the image in the video segment;

[0128] A video segment is obtained according to the segment extraction information, wherein the video segment belongs to the video to be detected.

[0129] This embodiment introduces a method for determining a video segment from a video to be detected. First, segment extraction information is obtained. The segment extraction information can be expressed as BatchCHWD. For example, BatchCHWD = batch1*3*128*128*16, which indicates that the batch value is batch1, the number of channels is 3, the height of the image in the video segment is 128 pixels, the width of the image in the video segment is 128 pixels, and the number of frames in the image in the video segment is 16. The video segment is determined from the video to be detected based on the segment extraction information.

[0130] It is understandable that the batch value, number of channels, image height, image width and number of image frames of the video clip are only for reference and can be set to other values ​​according to actual conditions and are not limited here.

[0131] Based on the above introduction, multiple video clips can be determined from the video to be detected. These video clips can all be used for shot boundary detection without removing some video clips. This is because the shot boundary prediction model uses a graphics processing unit (GPU). Experiments have shown that the residual neural network (Residual NeuralNetwork, ResNet)-18 can be used. ResNet-18 has achieved a processing frame rate of 1430 frames per second (Frames Per Second, FPS), which is more than one thousand even after considering the transmission speed. Preliminary screening of video clips will not significantly speed up the processing speed and will bring the risk of missing gradient shots. Of course, in practical applications, these video clips can also be screened.

[0132] Secondly, an embodiment of the present application provides a method for obtaining a video segment, that is, obtaining a video segment based on segment extraction information, wherein the video segment belongs to the video to be detected. In this way, the set segment extraction information can be used to accurately define at least one video segment from the video to be detected, thereby facilitating subsequent detection.

[0133] Optionally, in the above Figure 3 Based on the corresponding embodiments, in an optional embodiment of the shot boundary detection method provided in the embodiments of the present application, obtaining a video to be detected may include:

[0134] Obtaining first segment extraction information, wherein the segment extraction information includes a batch value of the first video segment, a number of channels, a height of an image in the first video segment, a width of the image in the first video segment, and a number of frames of the image in the first video segment;

[0135] Obtaining second segment extraction information, wherein the segment extraction information includes a batch value of the second video segment, a number of channels, a height of an image in the second video segment, a width of the image in the second video segment, and a number of frames of the image in the second video segment;

[0136] Acquire a first video segment according to the first segment extraction information, wherein the first video segment includes M consecutive images and belongs to the video to be detected;

[0137] A second video segment is obtained according to the second segment extraction information, wherein the second video segment includes M continuous images, the second video segment belongs to the video to be detected, and the first video segment and the second video segment have the same N continuous images, where N is an integer greater than or equal to 1 and less than M.

[0138] In this embodiment, another method for determining multiple video segments from a video to be detected is introduced. For ease of understanding, please refer to Figure 4 , Figure 4 This is a schematic diagram of an embodiment of selecting a video clip in an embodiment of the present application. As shown in the figure, first clip extraction information and second clip extraction information are obtained. The first clip extraction information can be expressed as BatchCHWD1, and the second clip extraction information can be expressed as BatchCHWD2, where BatchCHWD1 = batch1*3*128*128*16, BatchCHWD2 = batch2*3*128*128*16, that is, the first video clip and the second video clip both include 16 frames of images. In this case, M is 16, as shown in FIG. Figure 4 As shown, when M is 16, the first video segment includes images from the 1st frame to the 16th frame, and the second video segment includes images from the 9th frame to the 24th frame, wherein the images from the 9th frame to the 16th frame appear in both the first video segment and the second video segment. Therefore, these 8 frames of images are overlapping images, that is, N is 8.

[0139] It is understandable that there is no need to overlap the sampling of two video clips during model training, but in actual prediction, half of the image data needs to be overlapped, that is, the sampling of the previous video clip (for example, 16 images) and the sampling start position of the next video clip are the middle frame of the previous video clip (sampling 16 images backward from the 9th image of the previous clip).

[0140] It can be understood that the batch value, number of channels, image height, image width, number of image frames M and number of overlapping image frames N of the first video clip and the second video clip are all for reference only and can also be set to other values ​​according to actual conditions and are not limited here.

[0141] Secondly, the present embodiment provides another method for obtaining video segments. When there are multiple video segments in the video to be detected, a sliding window method is also required to extract different video segments. This method can significantly reduce the possibility that the connection points of the sampled video segments may be shot cut points, thereby improving the accuracy of shot boundary detection.

[0142] Optionally, in the above Figure 3 On the basis of the corresponding embodiments, in an optional embodiment of the shot boundary detection method provided by the embodiments of the present application, obtaining the shot type corresponding to the video clip in the video to be detected may include:

[0143] Obtaining a shot type corresponding to the first video clip through a shot boundary prediction model;

[0144] Obtaining a shot type corresponding to the second video clip through a shot boundary prediction model;

[0145] If the shot type belongs to the first shot type, determining the image change value corresponding to the video segment in the video to be detected may include:

[0146] If the shot type corresponding to the first video clip and the shot type corresponding to the second video clip are both the first shot type, generating a target video clip according to the first video clip and the second video clip, wherein the target video clip includes (2M-N) consecutive images;

[0147] An image change value is determined according to the target video segment.

[0148] This embodiment introduces a method for fusing two consecutive video clips with shot changes. Specifically, if two adjacent video clips both have shot changes, they are merged into a single video clip, which is then used as the target video clip. It should be understood that if two or more consecutive video clips all have shot changes, these video clips can be merged into a single video clip to obtain the target video clip.

[0149] For easier understanding, see Figure 5 , Figure 5 This is a schematic diagram of an embodiment of a shot mashup error in an embodiment of the present application. As shown in the figure, a first video clip and a second video clip are respectively input into a shot boundary prediction model. If the shot boundary prediction model outputs a result that the first video clip and the second video clip both belong to the first shot type, then the two video clips can be merged. Figure 5The shaded portion in represents the overlapping portion of the images of the first video clip and the second video clip. Assume that the first video clip has M frames of images (for example, 16 frames of images), the second video clip also has M frames of images (for example, 16 frames), and the overlapping portion of the images has N frames of images (for example, 8 frames of images). The target video clip obtained after merging has (2M-N) frames of images (for example, 24 frames of images).

[0150] Furthermore, in an embodiment of the present application, a method for merging video clips is provided. Specifically, if the shot type corresponding to a first video clip and the shot type corresponding to a second video clip are both the first shot type, a target video clip is generated based on the first video clip and the second video clip, and an image change value is determined based on the target video clip. In this manner, considering that the distance between adjacent video clips with shot changes may be less than the sampling interval, thereby causing a mashup error, such adjacent video clips can be merged, and the merged video clip is subsequently processed together with the other video clips to avoid mashup errors.

[0151] Optionally, in the above Figure 3 Based on the corresponding embodiments, in an optional embodiment of the shot boundary detection method provided by the embodiments of the present application, determining the image change value corresponding to the video segment in the video to be detected may include:

[0152] Acquire a first image frame, a second image frame, and a third image frame from a video clip of a video to be detected, wherein the first image frame is an image frame immediately preceding the second image frame, and the second image frame is an image frame immediately preceding the third image frame;

[0153] Obtaining a first color model corresponding to the first image frame, a second color model corresponding to the second image frame, and a third color model corresponding to the third image frame, wherein the first color model includes a first hue parameter, a first saturation parameter, and a first lightness parameter, the second color model includes a second hue parameter, a second saturation parameter, and a second lightness parameter, and the third color model includes a third hue parameter, a third saturation parameter, and a third lightness parameter;

[0154] Calculating a first hue difference value based on the first hue parameter and the second hue parameter, calculating a first saturation difference value based on the first saturation parameter and the second saturation parameter, and calculating a first lightness difference value based on the first lightness parameter and the second lightness parameter;

[0155] Calculating a first average difference value based on the first hue difference value, the first saturation difference value, and the first lightness difference value;

[0156] Calculating a second hue difference value based on the second hue parameter and the third hue parameter, calculating a second saturation difference value based on the second saturation parameter and the third saturation parameter, and calculating a second lightness difference value based on the second lightness parameter and the third lightness parameter;

[0157] Calculating a first average difference value based on the second hue difference value, the second saturation difference value, and the second lightness difference value;

[0158] An image change value corresponding to the video segment in the video to be detected is determined according to the first average difference and the second average difference.

[0159] This embodiment introduces a method for calculating image change values ​​corresponding to video segments in a video to be detected. Based on the image change values ​​of the video segments, the shot change type of the video segments is determined. A video to be detected may include multiple video segments, resulting in multiple image change values, which can be used to determine shot boundary information within the video to be detected.

[0160] For easier understanding, see Figure 6 , Figure 6 This is a flow chart of generating shot boundary information based on shot change in an embodiment of the present application. As shown in the figure, in step A1, a video clip is extracted from the video to be detected, and the video clip includes multiple frames of images of three channels of red, green, and blue (RGB). In step A2, each RGB image is converted into a color model (hue saturation value, HSV). In step A3, the sum of the three channel differences between each two adjacent images in the video clip is calculated respectively. In step A4, the image change value of the video clip is calculated based on the result of step A3. If the image change value is greater than or equal to the threshold, step A5 is entered to determine that the video clip is a shot mutation, and then the change position of the shot mutation is determined. If the image change value is less than the threshold, step A6 is entered to determine that the video clip is a shot gradient, and then the area of ​​the shot gradient is determined.

[0161] RGB images are designed based on the principle of color luminescence. Each of the three color channels, red, green, and blue, is divided into 256 levels of brightness. When the grayscale values ​​of the three colors are the same, different grayscale values ​​are produced. That is, when the grayscale values ​​of all three colors are 0, the darkest black tone is produced; when the grayscale values ​​of all three colors are 255, the brightest white tone is produced.

[0162] The HSV image model is a relatively intuitive color model. The color parameters in this model are hue, saturation, and value. Hue is measured in degrees, ranging from 0° to 360°. Starting from red and moving counterclockwise, red is 0°, green is 120°, and blue is 240°. Their complementary colors are yellow at 60°, cyan at 180°, and magenta at 300°. Saturation indicates the degree to which a color resembles a spectral color. A color can be thought of as a mixture of a spectral color and white. The greater the proportion of the spectral color, the closer the color is to the spectral color, and the higher the saturation. High saturation results in a deep and vivid color. Saturation is highest when the white light component of a spectral color is zero. Values ​​typically range from 0% to 100%, with higher values ​​indicating more saturated colors. Value indicates the brightness of a color. For light source color, the value is related to the brightness of the light source. For object color, the value is related to the object's transmittance or reflectance. The value usually ranges from 0% (black) to 100% (white).

[0163] Specifically, first select a video clip from the video to be detected. If the video to be detected only includes one video clip, then directly extract the image frames of the video to be detected. Taking the extraction of three frames of images as an example, the three frames of images are the first image frame, the second image frame, and the third image frame. The first image frame is the previous image frame adjacent to the second image frame, and the second image frame is the previous image frame adjacent to the third image frame. Then, a first color model corresponding to the first image frame, a second color model corresponding to the second image frame, and a third color model corresponding to the third image frame are generated respectively. Please refer to Table 1, which is a schematic diagram of the color models corresponding to multiple frames of images in a video clip.

[0164] Table 1

[0165] Hue (H) channel Saturation (S) channel Value (V) channel First image frame 200 250 180 Second image frame 100 230 170 The third image frame 50 140 120

[0166] As shown in Table 1, the first hue parameter of the first image frame is 200, the first saturation parameter is 250, and the first brightness parameter is 180. The second hue parameter of the second image frame is 100, the second saturation parameter is 230, and the second brightness parameter is 170. The third hue parameter of the third image frame is 50, the third saturation parameter is 140, and the third brightness parameter is 120. Calculate the average difference between the first image frame and the second image frame. First, calculate the first hue difference based on the first hue parameter and the second hue parameter, that is:

[0167] 200-100=100

[0168] The first saturation difference is calculated according to the first saturation parameter and the second saturation parameter, that is:

[0169] 250-230=20

[0170] The first brightness difference is calculated according to the first brightness parameter and the second brightness parameter, that is:

[0171] 180-170=10

[0172] According to the first hue difference, the first saturation difference, and the first lightness difference, a first average difference is calculated, that is:

[0173] (100+20+10) / 3=43.33

[0174] That is, the average difference between the first image frame and the second image frame is 43.33, that is, the first average difference is 43.33.

[0175] Calculate the average difference between the second image frame and the third image frame. First, calculate the second tone difference based on the second tone parameter and the third tone parameter, that is:

[0176] 100-50=50

[0177] The second saturation difference is calculated based on the second saturation parameter and the third saturation parameter, namely:

[0178] 230-140=90

[0179] The second brightness difference is calculated based on the second brightness parameter and the third brightness parameter, namely:

[0180] 170-120=50

[0181] The second average difference is calculated based on the second hue difference, the second saturation difference, and the second lightness difference, that is:

[0182] (50+90+50) / 3=63.33

[0183] That is, the average difference between the second image frame and the third image frame is 63.33, that is, the second average difference is 63.33.

[0184] Based on the first average difference and the second average difference, the image change value can be calculated. Assuming that the video to be detected includes only one video clip, and the video to be detected only includes the first image frame, the second image frame and the third image frame, then the average calculated according to the first average difference and the second average difference is (43.33+63.33) / 2=53.33.

[0185] S 2 =[(43.33-53.33) 2 +(63.33-53.33) 2 ] / 2=100

[0186] That is, the image change value of the three frames of images is 100. It can be understood that in actual applications, the video to be detected can also include multiple video clips, each video clip includes multiple frames of images, and the image change values ​​of these video clips are calculated as described above and will not be repeated here.

[0187] Secondly, an embodiment of the present application provides a method for calculating image change values ​​based on video segments in the video to be detected. Specifically, the method first obtains the color model corresponding to each image frame, calculates the average difference between each adjacent frame based on the color model, and then calculates the image change value based on multiple average differences. This method can calculate the image change value of each video segment in the video to be detected, and thus can determine the type of shot change based on the image change value, thereby improving the feasibility and operability of the solution.

[0188] Optionally, in the above Figure 3 Based on the corresponding embodiments, in an optional embodiment of the shot boundary detection method provided by the embodiments of the present application, determining shot boundary information of a video segment in a to-be-detected video based on an image change value may include:

[0189] If the image change value corresponding to the video segment in the video to be detected is greater than or equal to the first threshold, determining that the shot change type of the video segment in the video to be detected is a shot mutation;

[0190] If the image change value corresponding to the video segment in the video to be detected is less than the first threshold, determining that the shot change type of the video segment in the video to be detected is a shot gradient;

[0191] The shot change position is determined based on the shot change type.

[0192] In this embodiment, a method for detecting the shot change type of a video segment in a video to be detected is introduced. Specifically, taking a video segment in the video to be detected as an example, assuming that the image change value of the video segment is S1, it is necessary to determine whether S1 is greater than a first threshold value. If the image change value S1 is greater than or equal to the first threshold value, the shot change type of the video segment in the video to be detected is determined to be a sudden shot change. If the image change value S1 is less than the first threshold value, the shot change type of the video segment in the video to be detected is determined to be a gradual shot change. Based on the image change values ​​of different video segments in the video to be detected, some video segments with sudden shot changes and some video segments with gradual shot changes may appear, thereby obtaining the shot change type of the entire video to be detected. After determining the shot change type of the video to be detected, the shot change position can be determined based on the shot change types of different video segments.

[0193] It is understandable that the first threshold value can be set to 2.6. In actual applications, the first threshold value can also be set to other values. This is only an illustration and should not be understood as a limitation of this application.

[0194] Furthermore, in an embodiment of the present application, a method for determining the type of shot change is provided. If the image change value corresponding to the video to be detected is greater than or equal to a first threshold, the shot change type of the video to be detected is determined to be a sudden shot change. If the image change value corresponding to the video to be detected is less than the first threshold, the shot change type of the video to be detected is determined to be a gradual shot change. By using the image change value to determine the shot change type in this manner, the image change value can well reflect the amplitude of the shot change and the fluctuation of the image data, thereby improving the reliability of detecting the shot change type.

[0195] Optionally, in the above Figure 3 Based on the corresponding embodiments, in an optional embodiment of the shot boundary detection method provided by the embodiments of the present application, determining the shot change position according to the shot change type may include:

[0196] If the shot change type is a sudden shot change, and the first average difference is greater than the second average difference, determining the shot change position to be between the first image frame and the second image frame;

[0197] If the shot change type is a sudden shot change, and the second average difference is greater than the first average difference, the shot change position is determined to be between the second image frame and the third image frame.

[0198] In this embodiment, a method for determining the position of a shot change based on a shot mutation is introduced. Figure 7 , Figure 7 This is a schematic diagram of an embodiment of a shot mutation in an embodiment of the present application. As shown in the figure, when two frames of images undergo significant changes within a short period of time, it is represented as a shot mutation.

[0199] Specifically, for sudden shot changes, the location with the largest average difference is selected as the sudden shot cut point. For example, consider three frames from a video clip to be detected: the first, second, and third frames. The video clip is determined to be a sudden shot change. The first and second frames correspond to a first average difference, while the second and third frames correspond to a second average difference. If the first average difference is greater than the second average difference, the shot change location is determined to be between the first and second frames. If the second average difference is greater than the first average difference, the shot change location is determined to be between the second and third frames.

[0200] Furthermore, an embodiment of the present application provides a method for determining the position of a shot change. Specifically, if the shot change type is a sudden shot change and the first average difference is greater than the second average difference, the shot change position is determined to be between the first and second image frames. If the shot change type is a sudden shot change and the second average difference is greater than the first average difference, the shot change position is determined to be between the second and third image frames. This method selects the location with the largest average difference as the sudden shot change intersection point for sudden shot changes, thereby obtaining a more appropriate shot change position.

[0201] Optionally, in the above Figure 3 Based on the corresponding embodiments, in an optional embodiment of the shot boundary detection method provided by the embodiments of the present application, determining the shot change position according to the shot change type may include:

[0202] If the lens change type is a gradual lens change and the first average difference is greater than a second threshold, determining that the lens change position includes the first image frame and the second image frame;

[0203] If the lens change type is a gradual lens change, and the second average difference is greater than a second threshold, determining that the lens change position includes the second image frame and the third image frame;

[0204] If the shot change type is a gradual shot change, and both the first average difference and the second average difference are greater than the second threshold, it is determined that the shot change position includes the first image frame, the second image frame, and the third image frame.

[0205] In this embodiment, a method for determining the lens change position based on lens gradual change is introduced. Figure 8 , Figure 8 This is a schematic diagram of an embodiment of lens gradual change in an embodiment of the present application. As shown in the figure, when two frames of images undergo slight changes within a period of time, it is represented as a lens gradual change.

[0206] Specifically, for a shot transition, an area with an average difference greater than a second threshold is selected as the transition area. For example, three frames of a video clip in a video to be detected, namely, a first frame, a second frame, and a third frame, are selected. The video clip is determined to be a shot transition. The first and second frames correspond to a first average difference, and the second and third frames correspond to a second average difference. If the first average difference is greater than the second threshold, the shot change location is determined to include the first and second frames, meaning both the first and second frames belong to the transition area. If the second average difference is greater than the second threshold, the shot change location is determined to include the second and third frames, meaning both the second and third frames belong to the transition area. If both the first and second average differences are greater than the second threshold, the shot change location is determined to include the first, second, and third frames, meaning both the first, second, and third frames belong to the transition area.

[0207] It is understandable that the second threshold value can be set to 10. In actual applications, the second threshold value can also be set to other values. This is only an illustration and should not be understood as a limitation of this application.

[0208] Furthermore, in an embodiment of the present application, a method for determining a lens change position is provided. Specifically, if the lens change type is a gradual lens change and the first average difference is greater than a second threshold, the lens change position is determined to include the first and second image frames. If the lens change type is a gradual lens change and the second average difference is greater than the second threshold, the lens change position is determined to include the second and third image frames. If the lens change type is a gradual lens change and both the first and second average differences are greater than the second threshold, the lens change position is determined to include the first, second, and third image frames. In this manner, for a gradual lens change, images with an average difference greater than the threshold are selected as the gradual change region, thereby obtaining a more appropriate lens change position.

[0209] Combined with the above introduction, the following will introduce the model training method in this application. Please refer to Figure 9 , an embodiment of the model training method in the embodiment of the present application includes:

[0210] 201. Obtain a set of video segments to be trained, wherein the set of video segments to be trained includes at least one video segment to be trained;

[0211] In this embodiment, the model training device obtains a set of video clips to be trained, wherein the set of video clips to be trained includes at least one video clip to be trained, each video clip to be trained may include 16 frames of images, and each video clip to be trained does not include the same image.

[0212] 202. Obtaining a predicted shot type corresponding to each to-be-trained video segment in the to-be-trained video segment set using the to-be-trained shot boundary prediction model, wherein the predicted shot type is a first shot type or a second shot type, where the first shot type indicates a shot with a change, and the second shot type indicates a shot without a change;

[0213] In this embodiment, the model training device inputs each video clip to be trained into the shot boundary prediction model to be trained, thereby outputting a corresponding predicted shot type. The predicted shot type is a first shot type or a second shot type. The first shot type represents a shot with changes, and the second shot type represents a shot without changes.

[0214] The data format for a training video segment is represented as videopath begin_frame_indicelabel, where videopath begin_frame represents the starting frame number of the training segment, and indice label represents the actual label of the training segment. When collecting the training segment, all frames of data from begin_frame_indice (the starting frame number) to sample_duration (the number of sample frames) are labels.

[0215] When labeling training video clips, three types of labels can be used, namely no shot change (label = 0), gradual shot change (label = 1), and sudden shot change (label = 2). Among them, gradual shot change and sudden shot change both belong to the first shot type, and no shot change belongs to the second shot type.

[0216] 203. Based on the predicted shot type and the real shot type corresponding to each to-be-trained video clip, a loss function is used to train the to-be-trained shot boundary prediction model to obtain model parameters, wherein the real shot type is obtained by pre-labeling the to-be-trained video clip;

[0217] In this embodiment, the model training device uses a loss function to train the shot boundary prediction model based on the predicted shot type and the actual shot type corresponding to each video clip to be trained, thereby obtaining model parameters. The loss function represents the difference between the predicted and actual results, and the parameters of each network layer are reversely derived from the loss function.

[0218] 204. Generate a shot boundary prediction model based on the model parameters.

[0219] In this embodiment, the model training device generates a shot boundary prediction model based on the model parameters. Figure 10 as well as Figure 11 , Figure 10 Schematic diagram of a shot boundary prediction model in an embodiment of the present application. Figure 11 This is a structural diagram of the basic block in an embodiment of the present application. As shown in the figure, the shot boundary prediction model can use the ResNet-18 network, which specifically includes a convolutional layer, a batch normalization layer, an activation function (Rectified Linear Units, ReLu) layer, a maximum pooling layer, a basic block, an average pooling layer, and a linear regression layer. That is, the last linear regression layer replaces the softmax classifier, and the output category is the number of classifications.

[0220] It is understandable that the last layer can also be a softmax classifier. This application uses a linear regression layer, but this should not be understood as a limitation to this application.

[0221] In an embodiment of the present application, a model training method is provided. First, a set of video clips to be trained is obtained. Then, a shot boundary prediction model to be trained is used to obtain a predicted shot type for each video clip in the set. Based on the predicted shot type and the actual shot type corresponding to each video clip, a loss function is used to train the shot boundary prediction model to obtain model parameters. Finally, a shot boundary prediction model is generated based on the model parameters. This method enables training based on the number of samples in the set of video clips to be trained, thereby improving the accuracy of model training.

[0222] Optionally, in the above Figure 9 On the basis of the corresponding embodiments, in an optional embodiment of the model training method provided in the embodiments of the present application, after obtaining the set of video clips to be trained, the following steps may be further included:

[0223] Determining, based on the set of video clips to be trained, a real shot type corresponding to each video clip to be trained, wherein the real shot type is a first sub-shot type, a second sub-shot type, or a first shot type, and both the first sub-shot type and the second sub-shot type belong to the first shot type;

[0224] Determining the number of second shot types, the number of first sub-shot types, and the number of second sub-shot types according to the real shot type corresponding to each video clip to be trained;

[0225] determining, according to the number of the second lens type, the number of the first sub-lens type, and the number of the second sub-lens type, a first pre-weight for the second lens type, a second pre-weight for the first sub-lens type, and a third pre-weight for the second sub-lens type;

[0226] The loss function is used to train the shot boundary prediction model to obtain model parameters, which may include:

[0227] The shot boundary prediction model to be trained is trained using a loss function according to the first pre-weight, the second pre-weight, and the third pre-weight to obtain model parameters.

[0228] In this embodiment, a method for setting weights based on the sample size of different categories of data is introduced. Specifically, for a set of video clips to be trained, there is often a serious data imbalance problem. For example, Table 2 shows a statistical representation of the number of each label category.

[0229] Table 2

[0230] Label 0 1 2 Sample size 247374 25414 224312

[0231] As shown in Table 2, a label of 0 indicates no shot change (i.e., the second shot type), a label of 1 indicates a gradual shot change (the first sub-shot type), and a label of 2 indicates an abrupt shot change (the second sub-shot type). The number of shots without shot changes is relatively high, and there is a significant imbalance between the number of abrupt shot changes and the number of gradual shot changes. Therefore, we use the following method to balance the data: we take the inverse of the number of labels in each category as the pre-weight for each category.

[0232] For example, the first pre-weight is calculated as:

[0233] a=(247374+25414+224312) / 247374=2.01

[0234] The second pre-weight is calculated as follows:

[0235] b=(247374+25414+224312) / 25414=19.56

[0236] The third pre-weight is calculated as follows:

[0237] c=(247374+25414+224312) / 224312=2.22

[0238] Based on the first pre-weight, the second pre-weight, and the third pre-weight, a loss function is used to train the shot boundary prediction model to obtain model parameters, that is, the loss function can be expressed as:

[0239] L=a*L1+b*L2+c*L3

[0240] Among them, L represents the loss function, a represents the first pre-weight, L1 represents the loss function based on the second lens type, b represents the second pre-weight, L2 represents the loss function based on the first sub-lens type, c represents the third pre-weight, and L3 represents the loss function based on the second sub-lens type.

[0241] Secondly, this embodiment of the present application provides a method for considering the sample size of different data categories. Specifically, a loss function is used to train the shot boundary prediction model to be trained based on the first, second, and third pre-weights to obtain model parameters. This approach balances the number of samples, resolving the issue of severe data imbalance and improving the accuracy of model training.

[0242] Based on the shot boundary detection methods provided in the above embodiments, the following will be specifically described using online test data. In the experiment, 22 videos were randomly selected, and 421 shots were detected using the shot boundary prediction model provided by this application. Among them, there were 7 gradual change shots with a change of more than 5 frames, and there were no missed detections, and the detection results were accurate. There were also 32 gradual change shots (or slow mutation shots) within 5 frames, 5 of which were missed. All of them were news-related (i.e., the content was about dense crowds), and the detection results were all correct. There were 382 fast mutation shots, 11 of which were detected incorrectly (the errors were all animation or large-character poster broadcast clips), and 58 shots were missed. Please refer to Table 3, which shows the detection status of each indicator.

[0243] Table 3

[0244] The solution provided by this application Accuracy Recall F1 score index 0.973871734 0.869834711 0.918917912

[0245] As shown in Table 3, the test dataset is online data, and the test metrics are measured using the TRECVID (International Authoritative Evaluation of Video Retrieval) standard. Recall reflects the shot boundary prediction model's ability to identify positive samples; higher recall indicates a stronger ability for positive samples. Precision reflects the shot boundary prediction model's ability to distinguish negative samples; higher precision indicates a stronger ability for negative samples. The F1 score is a combination of precision and recall; a higher F1 score indicates a more robust shot boundary prediction model.

[0246] The shot boundary detection device in this application is described in detail below. Figure 12 , Figure 12 This is a schematic diagram of an embodiment of a shot boundary detection device in an embodiment of the present application. The shot boundary detection device 30 includes:

[0247] An acquisition module 301 is configured to acquire a video to be detected, wherein the video to be detected includes at least one video segment, and each video segment includes multiple frames of continuous images;

[0248] The acquisition module 301 is further configured to acquire a shot type corresponding to a video clip in the video to be detected, wherein the shot type is a first shot type or a second shot type, the first shot type indicating a shot with a change, and the second shot type indicating a shot without a change;

[0249] a determination module 302 configured to determine an image change value corresponding to a video segment in the video to be detected if the shot type obtained by the acquisition module 301 belongs to the first shot type, wherein the image change value is obtained by calculating two adjacent frames of images in the video segment in the video to be detected;

[0250] The determining module 302 is further configured to determine shot boundary information of the video segment in the to-be-detected video according to the image change value, wherein the shot boundary information includes a shot change type and a shot change position.

[0251] In this embodiment, the acquisition module 301 acquires a video to be detected, wherein the video to be detected includes at least one video segment, and the video segment includes multiple frames of continuous images. The acquisition module 301 acquires a shot type corresponding to the video segment in the video to be detected, wherein the shot type is a first shot type or a second shot type, the first shot type represents a shot with changes, and the second shot type represents a shot without changes. If the shot type acquired by the acquisition module 301 belongs to the first shot type, the determination module 302 determines an image change value corresponding to the video segment in the video to be detected, wherein the image change value is obtained by calculating two adjacent frames of images in the video segment in the video to be detected, and the determination module 302 determines the shot boundary information of the video segment in the video to be detected according to the image change value, wherein the shot boundary information includes a shot change type and a shot change position.

[0252] In an embodiment of the present application, a shot boundary detection device is provided. The device first obtains a video to be detected, then obtains the shot type corresponding to a video clip in the video to be detected, wherein the shot type is either a first shot type or a second shot type, wherein the first shot type indicates a shot with a change, and the second shot type indicates a shot without a change. If the shot type belongs to the first shot type, the image change value corresponding to the video to be detected is determined, and finally, shot boundary information of the video to be detected is determined based on the image change value, wherein the shot boundary information includes the shot change type and the shot change position. Through the above method, a two-layer detection mechanism is designed for different shot types. First, a deep learning algorithm is used to make a preliminary shot type determination on the video clip. For video clips with shot changes, further analysis is performed to determine whether they are sudden shot changes or gradual shot changes. The specific change position is detected based on different shots. Therefore, the combination of the deep learning algorithm and the adjacent image frame statistics method can provide richer information for shot boundary detection, thereby reducing the missed detection rate of shot boundaries and improving the detection effect of shot boundaries.

[0253] Optionally, in the above Figure 12 On the basis of the corresponding embodiment, in another embodiment of the shot boundary detection device 30 provided in the embodiment of the present application,

[0254] The acquisition module 301 is specifically configured to acquire segment extraction information, wherein the segment extraction information includes a batch value of the video segment, a number of channels, a height of an image in the video segment, a width of an image in the video segment, and a number of frames of the image in the video segment;

[0255] The video segment is acquired according to the segment extraction information, wherein the video segment belongs to the video to be detected.

[0256] Secondly, an embodiment of the present application provides a method for obtaining a video segment, that is, obtaining a video segment based on segment extraction information, wherein the video segment belongs to the video to be detected. In this way, the set segment extraction information can be used to accurately define at least one video segment from the video to be detected, thereby facilitating subsequent detection.

[0257] Optionally, in the above Figure 12 On the basis of the corresponding embodiment, in another embodiment of the shot boundary detection device 30 provided in the embodiment of the present application,

[0258] The acquisition module 301 is specifically configured to acquire first segment extraction information, wherein the segment extraction information includes a batch value of the first video segment, a number of channels, a height of an image in the first video segment, a width of an image in the first video segment, and a number of frames of the image in the first video segment;

[0259] Acquire second segment extraction information, wherein the segment extraction information includes a batch value of the second video segment, a number of channels, a height of an image in the second video segment, a width of the image in the second video segment, and a number of frames of the image in the second video segment;

[0260] Acquire the first video segment according to the first segment extraction information, wherein the first video segment includes M consecutive images, and the first video segment belongs to the video to be detected;

[0261] The second video segment is obtained according to the second segment extraction information, wherein the second video segment includes M continuous images, the second video segment belongs to the video to be detected, and the first video segment and the second video segment have the same N continuous images, where N is an integer greater than or equal to 1 and less than M.

[0262] Secondly, the present embodiment provides another method for obtaining video segments. When there are multiple video segments in the video to be detected, a sliding window method is also required to extract different video segments. This method can significantly reduce the possibility that the connection points of the sampled video segments may be shot cut points, thereby improving the accuracy of shot boundary detection.

[0263] Optionally, in the above Figure 12 On the basis of the corresponding embodiment, in another embodiment of the shot boundary detection device 30 provided in the embodiment of the present application,

[0264] The acquisition module 301 is specifically configured to acquire the shot type corresponding to the first video clip through a shot boundary prediction model;

[0265] Obtaining a shot type corresponding to the second video clip by using a shot boundary prediction model;

[0266] The determining module 302 is specifically configured to generate a target video segment based on the first video segment and the second video segment if the shot type corresponding to the first video segment and the shot type corresponding to the second video segment are both the first shot type, wherein the target video segment includes (2M-N) consecutive images;

[0267] The image change value is determined according to the target video segment.

[0268] Furthermore, in an embodiment of the present application, a method for merging video clips is provided. Specifically, if the shot type corresponding to a first video clip and the shot type corresponding to a second video clip are both the first shot type, a target video clip is generated based on the first video clip and the second video clip, and an image change value is determined based on the target video clip. In this manner, considering that the distance between adjacent video clips with shot changes may be less than the sampling interval, thereby causing a mashup error, such adjacent video clips can be merged, and the merged video clip is subsequently processed together with the other video clips to avoid mashup errors.

[0269] Optionally, in the above Figure 12 On the basis of the corresponding embodiment, in another embodiment of the shot boundary detection device 30 provided in the embodiment of the present application,

[0270] The determining module 302 is specifically configured to obtain a first image frame, a second image frame, and a third image frame from the video segment of the video to be detected, wherein the first image frame is an image frame immediately preceding the second image frame, and the second image frame is an image frame immediately preceding the third image frame;

[0271] Obtaining a first color model corresponding to the first image frame, a second color model corresponding to the second image frame, and a third color model corresponding to the third image frame, wherein the first color model includes a first hue parameter, a first saturation parameter, and a first lightness parameter, the second color model includes a second hue parameter, a second saturation parameter, and a second lightness parameter, and the third color model includes a third hue parameter, a third saturation parameter, and a third lightness parameter;

[0272] Calculating a first hue difference value based on the first hue parameter and the second hue parameter, calculating a first saturation difference value based on the first saturation parameter and the second saturation parameter, and calculating a first lightness difference value based on the first lightness parameter and the second lightness parameter;

[0273] Calculating a first average difference value according to the first hue difference value, the first saturation difference value, and the first lightness difference value;

[0274] Calculating a second hue difference value based on the second hue parameter and the third hue parameter, calculating a second saturation difference value based on the second saturation parameter and the third saturation parameter, and calculating a second lightness difference value based on the second lightness parameter and the third lightness parameter;

[0275] Calculating a first average difference value according to the second hue difference value, the second saturation difference value, and the second lightness difference value;

[0276] An image change value corresponding to the video segment in the video to be detected is determined according to the first average difference and the second average difference.

[0277] Secondly, an embodiment of the present application provides a method for calculating image change values ​​based on video segments in the video to be detected. Specifically, the method first obtains the color model corresponding to each image frame, calculates the average difference between each adjacent frame based on the color model, and then calculates the image change value based on multiple average differences. This method can calculate the image change value of each video segment in the video to be detected, and thus can determine the type of shot change based on the image change value, thereby improving the feasibility and operability of the solution.

[0278] Optionally, in the above Figure 12 On the basis of the corresponding embodiment, in another embodiment of the shot boundary detection device 30 provided in the embodiment of the present application,

[0279] The determining module 302 is specifically configured to determine that the shot change type of the video segment in the video to be detected is a shot mutation if the image change value corresponding to the video segment in the video to be detected is greater than or equal to a first threshold;

[0280] If the image change value corresponding to the video segment in the to-be-detected video is less than the first threshold, determining that the shot change type of the video segment in the to-be-detected video is a shot gradient;

[0281] The lens change position is determined according to the lens change type.

[0282] Furthermore, in an embodiment of the present application, a method for determining the type of shot change is provided. If the image change value corresponding to the video to be detected is greater than or equal to a first threshold, the shot change type of the video to be detected is determined to be a sudden shot change. If the image change value corresponding to the video to be detected is less than the first threshold, the shot change type of the video to be detected is determined to be a gradual shot change. By using the image change value to determine the shot change type in this manner, the image change value can well reflect the amplitude of the shot change and the fluctuation of the image data, thereby improving the reliability of detecting the shot change type.

[0283] Optionally, in the above Figure 12 On the basis of the corresponding embodiment, in another embodiment of the shot boundary detection device 30 provided in the embodiment of the present application,

[0284] The determining module 302 is specifically configured to determine that the shot change position is between the first image frame and the second image frame if the shot change type is the sudden shot change and the first average difference is greater than the second average difference;

[0285] If the shot change type is the sudden shot change, and the second average difference is greater than the first average difference, the shot change position is determined to be between the second image frame and the third image frame.

[0286] Furthermore, an embodiment of the present application provides a method for determining the position of a shot change. Specifically, if the shot change type is a sudden shot change and the first average difference is greater than the second average difference, the shot change position is determined to be between the first and second image frames. If the shot change type is a sudden shot change and the second average difference is greater than the first average difference, the shot change position is determined to be between the second and third image frames. This method selects the location with the largest average difference as the sudden shot change intersection point for sudden shot changes, thereby obtaining a more appropriate shot change position.

[0287] Optionally, in the above Figure 12 On the basis of the corresponding embodiment, in another embodiment of the shot boundary detection device 30 provided in the embodiment of the present application,

[0288] The determining module 302 is specifically configured to determine that the shot change position includes the first image frame and the second image frame if the shot change type is the shot gradual change and the first average difference is greater than a second threshold;

[0289] If the lens change type is the lens gradual change, and the second average difference is greater than the second threshold, determining that the lens change position includes the second image frame and the third image frame;

[0290] If the lens change type is the lens gradual change, and the first average difference and the second average difference are both greater than the second threshold, it is determined that the lens change position includes the first image frame, the second image frame, and the third image frame.

[0291] Furthermore, in an embodiment of the present application, a method for determining a lens change position is provided. Specifically, if the lens change type is a gradual lens change and the first average difference is greater than a second threshold, the lens change position is determined to include the first and second image frames. If the lens change type is a gradual lens change and the second average difference is greater than the second threshold, the lens change position is determined to include the second and third image frames. If the lens change type is a gradual lens change and both the first and second average differences are greater than the second threshold, the lens change position is determined to include the first, second, and third image frames. In this manner, for a gradual lens change, images with an average difference greater than the threshold are selected as the gradual change region, thereby obtaining a more appropriate lens change position.

[0292] The model training device in this application is described in detail below. Figure 13, Figure 12 This is a schematic diagram of an embodiment of a model training device in an embodiment of the present application. The model training device 40 includes:

[0293] An acquisition module 401 is configured to acquire a set of video segments to be trained, wherein the set of video segments to be trained includes at least one video segment to be trained;

[0294] The acquisition module 401 is further configured to acquire a predicted shot type corresponding to each to-be-trained video segment in the to-be-trained video segment set, wherein the predicted shot type is a first shot type or a second shot type, wherein the first shot type indicates a shot with a change, and the second shot type indicates a shot without a change;

[0295] A training module 402 is configured to train the shot boundary prediction model to be trained using a loss function based on the predicted shot type and the real shot type corresponding to each of the video segments to be trained obtained by the obtaining module 401, to obtain model parameters, wherein the real shot type is obtained by pre-labeling the video segments to be trained;

[0296] The generating module 403 is configured to generate a shot boundary prediction model according to the model parameters trained by the training module 402 .

[0297] In this embodiment, the acquisition module 401 acquires a set of video clips to be trained, wherein the set of video clips to be trained includes at least one video clip to be trained. The acquisition module 401 acquires a predicted shot type corresponding to each video clip to be trained in the set of video clips to be trained, wherein the predicted shot type is a first shot type or a second shot type, wherein the first shot type represents a shot with changes, and the second shot type represents a shot without changes. The training module 402 trains the shot boundary prediction model to be trained using a loss function according to the predicted shot type and the real shot type corresponding to each video clip to be trained acquired by the acquisition module 401, to obtain model parameters, wherein the real shot type is obtained after pre-labeling the video clips to be trained. The generation module 403 generates a shot boundary prediction model according to the model parameters trained by the training module 402.

[0298] In an embodiment of the present application, a model training method is provided. First, a set of video clips to be trained is obtained. Then, a shot boundary prediction model to be trained is used to obtain a predicted shot type for each video clip in the set. Based on the predicted shot type and the actual shot type corresponding to each video clip, a loss function is used to train the shot boundary prediction model to obtain model parameters. Finally, a shot boundary prediction model is generated based on the model parameters. This method enables training based on the number of samples in the set of video clips to be trained, thereby improving the accuracy of model training.

[0299] Optionally, in the above Figure 13 Based on the corresponding embodiment, please refer to Figure 14 In another embodiment of the model training device 40 provided in the embodiment of the present application, the model training device 40 includes a determination module 404;

[0300] The determining module 404 is configured to determine, after the acquiring module 401 acquires the set of video segments to be trained, a real shot type corresponding to each of the video segments to be trained based on the set of video segments to be trained, wherein the real shot type is the first sub-shot type, the second sub-shot type, or the first shot type, and both the first sub-shot type and the second sub-shot type belong to the first shot type;

[0301] The determining module 404 is further configured to determine the number of the second shot type, the number of the first sub-shot type, and the number of the second sub-shot type according to the real shot type corresponding to each of the video segments to be trained;

[0302] The determining module 404 is further configured to determine a first pre-weighting of the second lens type, a second pre-weighting of the first sub-lens type, and a third pre-weighting of the second sub-lens type based on the number of the second lens type, the number of the first sub-lens type, and the number of the second sub-lens type;

[0303] The training module 402 is specifically configured to train the shot boundary prediction model to be trained using the loss function according to the first pre-weight, the second pre-weight, and the third pre-weight to obtain model parameters.

[0304] Secondly, this embodiment of the present application provides a method for considering the sample size of different data categories. Specifically, a loss function is used to train the shot boundary prediction model to be trained based on the first, second, and third pre-weights to obtain model parameters. This approach balances the number of samples, resolving the issue of severe data imbalance and improving the accuracy of model training.

[0305] The present application also provides another shot boundary detection device, such as Figure 15 For ease of explanation, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal device can be any terminal device including a mobile phone, tablet computer, personal digital assistant (PDA), point of sales (POS), car computer, etc., taking the mobile phone as an example:

[0306] Figure 15 The block diagram shows a partial structure of a mobile phone related to the terminal device provided in the embodiment of the present application. Figure 15 The mobile phone includes components such as a radio frequency (RF) circuit 510, a memory 520, an input unit 530, a display unit 540, a sensor 550, an audio circuit 560, a wireless fidelity (WiFi) module 570, a processor 580, and a power supply 590. Those skilled in the art will understand that Figure 15 The mobile phone structure shown in the figure does not constitute a limitation to the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0307] The following combination Figure 15 A detailed introduction to the various components of a mobile phone:

[0308] The RF circuit 510 can be used to receive and send signals during information transmission or calls. In particular, after receiving the downlink information from the base station, it is sent to the processor 580 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 510 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 510 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0309] The memory 520 can be used to store software programs and modules. The processor 580 executes the various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 520. The memory 520 may mainly include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area may store data created based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory 520 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0310] The input unit 530 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 530 may include a touch panel 531 and other input devices 532. The touch panel 531, also known as a touch screen, can collect user touch operations on or near it (such as operations performed by the user using any suitable object or accessory such as a finger, stylus, etc. on or near the touch panel 531) and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 531 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch direction and detects the signal caused by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device and converts it into touch point coordinates, which are then sent to the processor 580. It can also receive commands sent by the processor 580 and execute them. In addition, the touch panel 531 can be implemented using various types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch panel 531, the input unit 530 may further include other input devices 532. Specifically, the other input devices 532 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick.

[0311] The display unit 540 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 540 may include a display panel 541. Optionally, the display panel 541 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 531 may cover the display panel 541. When the touch panel 531 detects a touch operation on or near it, it is transmitted to the processor 580 to determine the type of touch event. Subsequently, the processor 580 provides corresponding visual output on the display panel 541 according to the type of touch event. Although in Figure 15 In the embodiment, the touch panel 531 and the display panel 541 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 531 and the display panel 541 can be integrated to realize the input and output functions of the mobile phone.

[0312] The mobile phone may also include at least one sensor 550, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 541 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 541 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that the mobile phone can also be configured with, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be described here.

[0313] Audio circuit 560, speaker 561, and microphone 562 provide an audio interface between the user and the phone. Audio circuit 560 converts received audio data into electrical signals and transmits them to speaker 561, which then converts them into sound signals for output. Microphone 562, on the other hand, converts collected sound signals into electrical signals, which are then received by audio circuit 560 and converted into audio data. The audio data is then processed by processor 580 and transmitted via RF circuit 510 to, for example, another phone. Alternatively, the audio data can be output to memory 520 for further processing.

[0314] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse the web and access streaming media through the WiFi module 570. It provides users with wireless broadband Internet access. Figure 15 A WiFi module 570 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0315] The processor 580 is the control center of the mobile phone. It connects all parts of the mobile phone using various interfaces and circuits. It executes the software programs and / or modules stored in the memory 520 and accesses data stored in the memory 520 to perform various functions of the mobile phone and process data. Optionally, the processor 580 may include one or more processing units. Alternatively, the processor 580 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor does not need to be integrated into the processor 580.

[0316] The mobile phone also includes a power supply 590 (such as a battery) for supplying power to various components. Optionally, the power supply can be logically connected to the processor 580 through a power management system, thereby managing charging, discharging, and power consumption through the power management system.

[0317] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0318] In the embodiment of the present application, the processor 580 included in the terminal device also has the following functions:

[0319] Acquire a video to be detected, wherein the video to be detected includes at least one video segment, and the video segment includes multiple frames of continuous images;

[0320] Obtaining a shot type corresponding to a video clip in the video to be detected, wherein the shot type is a first shot type or a second shot type, the first shot type indicating a shot with changes, and the second shot type indicating a shot without changes;

[0321] If the shot type belongs to the first shot type, determining an image change value corresponding to the video segment in the video to be detected, wherein the image change value is obtained by calculating two adjacent frames of images in the video segment in the video to be detected;

[0322] The shot boundary information of the video segment in the to-be-detected video is determined according to the image change value, wherein the shot boundary information includes a shot change type and a shot change position.

[0323] Optionally, the processor 580 is specifically configured to perform the following steps:

[0324] The shot type corresponding to the video clip in the video to be detected is obtained through a shot boundary prediction model, wherein the shot boundary prediction model includes a linear regression layer, and the shot boundary prediction model is trained based on a first pre-weight, a second pre-weight, and a third pre-weight, the first pre-weight is associated with the number of the second shot type, the second pre-weight is associated with the first sub-shot type, the third pre-weight is associated with the second sub-shot type, and the first sub-shot type and the second sub-shot type both belong to the first shot type.

[0325] Optionally, the processor 580 is specifically configured to perform the following steps:

[0326] Acquire segment extraction information, wherein the segment extraction information includes a batch value of the video segment, a number of channels, a height of an image in the video segment, a width of the image in the video segment, and a number of frames of the image in the video segment;

[0327] The video segment is acquired according to the segment extraction information, wherein the video segment belongs to the video to be detected.

[0328] Optionally, the processor 580 is specifically configured to perform the following steps:

[0329] Obtaining first segment extraction information, wherein the segment extraction information includes a batch value of the first video segment, a number of channels, a height of an image in the first video segment, a width of the image in the first video segment, and a number of frames of the image in the first video segment;

[0330] Acquire second segment extraction information, wherein the segment extraction information includes a batch value of the second video segment, a number of channels, a height of an image in the second video segment, a width of the image in the second video segment, and a number of frames of the image in the second video segment;

[0331] Acquire the first video segment according to the first segment extraction information, wherein the first video segment includes M consecutive images, and the first video segment belongs to the video to be detected;

[0332] The second video segment is obtained according to the second segment extraction information, wherein the second video segment includes M continuous images, the second video segment belongs to the video to be detected, and the first video segment and the second video segment have the same N continuous images, where N is an integer greater than or equal to 1 and less than M.

[0333] Optionally, the processor 580 is specifically configured to perform the following steps:

[0334] Obtaining a shot type corresponding to the first video clip through a shot boundary prediction model;

[0335] Obtaining a shot type corresponding to the second video clip by using a shot boundary prediction model;

[0336] If the shot type corresponding to the first video clip and the shot type corresponding to the second video clip are both the first shot type, generating a target video clip according to the first video clip and the second video clip, wherein the target video clip includes (2M-N) consecutive images;

[0337] The image change value is determined according to the target video segment.

[0338] Optionally, the processor 580 is specifically configured to perform the following steps:

[0339] Acquire a first image frame, a second image frame, and a third image frame from the video segment of the video to be detected, wherein the first image frame is an image frame immediately preceding the second image frame, and the second image frame is an image frame immediately preceding the third image frame;

[0340] Obtaining a first color model corresponding to the first image frame, a second color model corresponding to the second image frame, and a third color model corresponding to the third image frame, wherein the first color model includes a first hue parameter, a first saturation parameter, and a first lightness parameter, the second color model includes a second hue parameter, a second saturation parameter, and a second lightness parameter, and the third color model includes a third hue parameter, a third saturation parameter, and a third lightness parameter;

[0341] Calculating a first hue difference value based on the first hue parameter and the second hue parameter, calculating a first saturation difference value based on the first saturation parameter and the second saturation parameter, and calculating a first lightness difference value based on the first lightness parameter and the second lightness parameter;

[0342] Calculating a first average difference value according to the first hue difference value, the first saturation difference value, and the first lightness difference value;

[0343] Calculating a second hue difference value based on the second hue parameter and the third hue parameter, calculating a second saturation difference value based on the second saturation parameter and the third saturation parameter, and calculating a second lightness difference value based on the second lightness parameter and the third lightness parameter;

[0344] Calculating a first average difference value according to the second hue difference value, the second saturation difference value, and the second lightness difference value;

[0345] An image change value corresponding to the video segment in the video to be detected is determined according to the first average difference and the second average difference.

[0346] Optionally, the processor 580 is specifically configured to perform the following steps:

[0347] If the image change value corresponding to the video segment in the video to be detected is greater than or equal to a first threshold, determining that the shot change type of the video segment in the video to be detected is a shot mutation;

[0348] If the image change value corresponding to the video segment in the to-be-detected video is less than the first threshold, determining that the shot change type of the video segment in the to-be-detected video is a shot gradient;

[0349] The lens change position is determined according to the lens change type.

[0350] Optionally, the processor 580 is specifically configured to perform the following steps:

[0351] If the shot change type is the sudden shot change, and the first average difference is greater than the second average difference, determining the shot change position to be between the first image frame and the second image frame;

[0352] If the shot change type is the sudden shot change, and the second average difference is greater than the first average difference, the shot change position is determined to be between the second image frame and the third image frame.

[0353] Optionally, the processor 580 is specifically configured to perform the following steps:

[0354] If the lens change type is the lens gradual change, and the first average difference is greater than a second threshold, determining that the lens change position includes the first image frame and the second image frame;

[0355] If the lens change type is the lens gradual change, and the second average difference is greater than the second threshold, determining that the lens change position includes the second image frame and the third image frame;

[0356] If the lens change type is the lens gradual change, and the first average difference and the second average difference are both greater than the second threshold, it is determined that the lens change position includes the first image frame, the second image frame, and the third image frame.

[0357] Figure 16 6 is a schematic diagram of a server structure provided by an embodiment of the present invention. The server 600 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 622 (for example, one or more processors) and a memory 632, and one or more storage media 630 (for example, one or more mass storage devices) for storing application programs 642 or data 644. Among them, the memory 632 and the storage medium 630 can be temporary storage or permanent storage. The program stored in the storage medium 630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 622 can be configured to communicate with the storage medium 630 to execute a series of instruction operations in the storage medium 630 on the server 600.

[0358] The server 600 may also include one or more power supplies 626, one or more wired or wireless network interfaces 650, one or more input and output interfaces 658, and / or one or more operating systems 641, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0359] The steps performed by the server in the above embodiment can be based on the Figure 16 The server structure shown.

[0360] In the embodiment of the present application, the CPU 622 included in the server also has the following functions:

[0361] Acquire a video to be detected, wherein the video to be detected includes at least one video segment, and the video segment includes multiple frames of continuous images;

[0362] Obtaining a shot type corresponding to a video clip in the video to be detected, wherein the shot type is a first shot type or a second shot type, the first shot type indicating a shot with changes, and the second shot type indicating a shot without changes;

[0363] If the shot type belongs to the first shot type, determining an image change value corresponding to the video segment in the video to be detected, wherein the image change value is obtained by calculating two adjacent frames of images in the video segment in the video to be detected;

[0364] The shot boundary information of the video segment in the to-be-detected video is determined according to the image change value, wherein the shot boundary information includes a shot change type and a shot change position.

[0365] In the embodiment of the present application, the CPU 622 included in the server also has the following functions:

[0366] Acquire a set of video segments to be trained, wherein the set of video segments to be trained includes at least one video segment to be trained;

[0367] Obtaining a predicted shot type corresponding to each to-be-trained video segment in the to-be-trained video segment set, wherein the predicted shot type is a first shot type or a second shot type, the first shot type indicating a shot with a change, and the second shot type indicating a shot without a change;

[0368] Training the shot boundary prediction model to be trained using a loss function according to the predicted shot type and the real shot type corresponding to each video clip to be trained, to obtain model parameters, wherein the real shot type is obtained by pre-labeling the video clip to be trained;

[0369] A shot boundary prediction model is generated according to the model parameters.

[0370] Optionally, the CPU 622 is further configured to perform the following steps:

[0371] Determining, based on the set of video clips to be trained, a real shot type corresponding to each of the video clips to be trained, wherein the real shot type is a first sub-shot type, a second sub-shot type, or the first shot type, and both the first sub-shot type and the second sub-shot type belong to the first shot type;

[0372] Determining the number of the second shot type, the number of the first sub-shot type, and the number of the second sub-shot type according to the real shot type corresponding to each of the video clips to be trained;

[0373] determining, according to the number of the second lens type, the number of the first sub-lens type, and the number of the second sub-lens type, a first pre-weighting of the second lens type, a second pre-weighting of the first sub-lens type, and a third pre-weighting of the second sub-lens type;

[0374] The loss function is used to train the shot boundary prediction model to be trained to obtain model parameters, including:

[0375] The shot boundary prediction model to be trained is trained using the loss function according to the first pre-weight, the second pre-weight, and the third pre-weight to obtain model parameters.

[0376] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0377] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0378] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0379] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0380] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0381] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for shot boundary detection, characterized in that: include: Acquire a video to be detected, wherein the video to be detected includes at least one video segment, and the video segment includes multiple frames of continuous images; Acquire a shot type corresponding to a video clip in the to-be-detected video, wherein the shot type is a first shot type or a second shot type, the first shot type represents a shot with a change, and the second shot type represents a shot without a change; If the shot type belongs to the first shot type, determining an image change value corresponding to the video segment in the to-be-detected video, wherein the image change value is obtained by calculating two adjacent frames of images in the video segment in the to-be-detected video; Determining shot boundary information of a video segment in the to-be-detected video according to the image change value, wherein the shot boundary information includes a shot change type and a shot change position; The determining of the image change value corresponding to the video segment in the video to be detected includes: Acquire a first image frame, a second image frame, and a third image frame from the video segment of the video to be detected, wherein the first image frame is an image frame adjacent to the previous image frame of the second image frame, and the second image frame is an image frame adjacent to the previous image frame of the third image frame; Acquire a first color model corresponding to the first image frame, a second color model corresponding to the second image frame, and a third color model corresponding to the third image frame, wherein the first color model includes a first hue parameter, a first saturation parameter, and a first lightness parameter, the second color model includes a second hue parameter, a second saturation parameter, and a second lightness parameter, and the third color model includes a third hue parameter, a third saturation parameter, and a third lightness parameter; Calculating a first hue difference value according to the first hue parameter and the second hue parameter, calculating a first saturation difference value according to the first saturation parameter and the second saturation parameter, and calculating a first brightness difference value according to the first brightness parameter and the second brightness parameter; Calculating a first average difference value according to the first hue difference value, the first saturation difference value, and the first lightness difference value; Calculate a second hue difference value according to the second hue parameter and the third hue parameter, calculate a second saturation difference value according to the second saturation parameter and the third saturation parameter, and calculate a second brightness difference value according to the second brightness parameter and the third brightness parameter; Calculating a second average difference value according to the second hue difference value, the second saturation difference value, and the second lightness difference value; Determine, according to the first average difference and the second average difference, an image change value corresponding to the video segment in the video to be detected, wherein the image change value is a variance between the first average difference and the second average difference; The step of determining the shot boundary information of the video segment in the to-be-detected video according to the image change value includes: If the image change value corresponding to the video segment in the video to be detected is greater than or equal to a first threshold, determining that the shot change type of the video segment in the video to be detected is a shot mutation; If the image change value corresponding to the video segment in the to-be-detected video is less than the first threshold, determining that the shot change type of the video segment in the to-be-detected video is a shot gradient; The lens change position is determined according to the lens change type.

2. The method according to claim 1, characterized in that The step of obtaining the shot type corresponding to the video clip in the video to be detected includes: The shot type corresponding to the video clip in the video to be detected is obtained through a shot boundary prediction model, wherein the shot boundary prediction model includes a linear regression layer, and the shot boundary prediction model is trained according to a first pre-weight, a second pre-weight, and a third pre-weight, the first pre-weight is associated with the number of the second shot type, the second pre-weight is associated with the number of the first sub-shot type, the third pre-weight is associated with the number of the second sub-shot type, and the first sub-shot type and the second sub-shot type both belong to the first shot type.

3. The method according to claim 1, characterized in that The step of obtaining the video to be detected includes: Acquire segment extraction information, wherein the segment extraction information includes a batch value of the video segment, a number of channels, a height of an image in the video segment, a width of the image in the video segment, and a number of frames of the image in the video segment; The video segment is acquired according to the segment extraction information, wherein the video segment belongs to the video to be detected.

4. The method according to claim 1, characterized in that: The step of obtaining the video to be detected includes: Acquire first segment extraction information, wherein the first segment extraction information includes a batch value of the first video segment, a number of channels, a height of an image in the first video segment, a width of the image in the first video segment, and a number of frames of the image in the first video segment; Acquire second segment extraction information, wherein the second segment extraction information includes a batch value of the second video segment, a number of channels, a height of an image in the second video segment, a width of an image in the second video segment, and a number of frames of the image in the second video segment; Acquire the first video segment according to the first segment extraction information, wherein the first video segment includes M consecutive images, and the first video segment belongs to the video to be detected; The second video segment is obtained according to the second segment extraction information, wherein the second video segment includes M continuous images, the second video segment belongs to the video to be detected, and the first video segment and the second video segment have the same N continuous images, where N is an integer greater than or equal to 1 and less than M.

5. The method according to claim 4, characterized in that The step of obtaining the shot type corresponding to the video clip in the video to be detected includes: Acquire the shot type corresponding to the first video clip by using a shot boundary prediction model; Acquire a shot type corresponding to the second video clip by using a shot boundary prediction model; If the shot type belongs to the first shot type, determining an image change value corresponding to a video segment in the to-be-detected video includes: If the shot type corresponding to the first video segment and the shot type corresponding to the second video segment are both the first shot type, generating a target video segment according to the first video segment and the second video segment, wherein the target video segment includes (2M-N) consecutive images; The image change value is determined according to the target video segment.

6. The method according to claim 1, characterized in that The determining the lens change position according to the lens change type includes: If the shot change type is the sudden shot change, and the first average difference is greater than the second average difference, determining the shot change position to be between the first image frame and the second image frame; If the shot change type is the sudden shot change, and the second average difference is greater than the first average difference, the shot change position is determined to be between the second image frame and the third image frame.

7. The method according to claim 1, characterized in that The determining the lens change position according to the lens change type includes: If the lens change type is the lens gradual change, and the first average difference is greater than a second threshold, determining that the lens change position includes the first image frame and the second image frame; If the lens change type is the lens gradual change, and the second average difference is greater than the second threshold, determining that the lens change position includes the second image frame and the third image frame; If the lens change type is the lens gradual change, and the first average difference and the second average difference are both greater than the second threshold, it is determined that the lens change position includes the first image frame, the second image frame, and the third image frame.

8. The method according to claim 2, characterized in that: The training process of the shot boundary prediction model includes: Acquire a set of video segments to be trained, wherein the set of video segments to be trained includes at least one video segment to be trained; Obtaining a predicted shot type corresponding to each to-be-trained video segment in the to-be-trained video segment set, wherein the predicted shot type is a first shot type or a second shot type, the first shot type represents a shot with a change, and the second shot type represents a shot without a change; According to the predicted shot type and the real shot type corresponding to each video clip to be trained, a loss function is used to train the shot boundary prediction model to be trained to obtain model parameters, wherein the real shot type is obtained by pre-labeling the video clip to be trained; A shot boundary prediction model is generated according to the model parameters.

9. The method according to claim 8, characterized in that After obtaining the set of video clips to be trained, the method further includes: Determining, according to the set of video clips to be trained, a real shot type corresponding to each of the video clips to be trained, wherein the real shot type is a first sub-shot type, a second sub-shot type, or the first shot type, and both the first sub-shot type and the second sub-shot type belong to the first shot type; Determining the number of the second shot type, the number of the first sub-shot type, and the number of the second sub-shot type according to the real shot type corresponding to each of the video clips to be trained; Determining, according to the number of the second lens type, the number of the first sub-lens type, and the number of the second sub-lens type, a first pre-weighting of the second lens type, a second pre-weighting of the first sub-lens type, and a third pre-weighting of the second sub-lens type; The adopting of the loss function to train the to-be-trained shot boundary prediction model to obtain model parameters includes: The shot boundary prediction model to be trained is trained using the loss function according to the first pre-weight, the second pre-weight, and the third pre-weight to obtain model parameters.

10. A shot boundary detection device, characterized in that: include: An acquisition module, used for acquiring a video to be detected, wherein the video to be detected includes at least one video segment, and the video segment includes multiple frames of continuous images; The acquisition module is further used to acquire a shot type corresponding to a video clip in the video to be detected, wherein the shot type is a first shot type or a second shot type, the first shot type represents a shot with a change, and the second shot type represents a shot without a change; a determination module, configured to determine, if the shot type acquired by the acquisition module belongs to the first shot type, an image change value corresponding to the video segment in the to-be-detected video, wherein the image change value is obtained by calculating two adjacent frames of images in the video segment in the to-be-detected video; The determination module is further used to determine the shot boundary information of the video segment in the to-be-detected video according to the image change value, wherein the shot boundary information includes a shot change type and a shot change position; The determination module is specifically configured to obtain a first image frame, a second image frame, and a third image frame from the video segment of the video to be detected, wherein the first image frame is an image frame adjacent to the previous image frame of the second image frame, and the second image frame is an image frame adjacent to the previous image frame of the third image frame; Acquire a first color model corresponding to the first image frame, a second color model corresponding to the second image frame, and a third color model corresponding to the third image frame, wherein the first color model includes a first hue parameter, a first saturation parameter, and a first lightness parameter, the second color model includes a second hue parameter, a second saturation parameter, and a second lightness parameter, and the third color model includes a third hue parameter, a third saturation parameter, and a third lightness parameter; Calculating a first hue difference value according to the first hue parameter and the second hue parameter, calculating a first saturation difference value according to the first saturation parameter and the second saturation parameter, and calculating a first brightness difference value according to the first brightness parameter and the second brightness parameter; Calculating a first average difference value according to the first hue difference value, the first saturation difference value, and the first lightness difference value; Calculate a second hue difference value according to the second hue parameter and the third hue parameter, calculate a second saturation difference value according to the second saturation parameter and the third saturation parameter, and calculate a second brightness difference value according to the second brightness parameter and the third brightness parameter; Calculating a second average difference value according to the second hue difference value, the second saturation difference value, and the second lightness difference value; Determine, according to the first average difference and the second average difference, an image change value corresponding to the video segment in the video to be detected, wherein the image change value is a variance between the first average difference and the second average difference; The determination module is specifically configured to determine that the shot change type of the video segment in the video to be detected is a shot mutation if the image change value corresponding to the video segment in the video to be detected is greater than or equal to a first threshold; If the image change value corresponding to the video segment in the to-be-detected video is less than the first threshold, determining that the shot change type of the video segment in the to-be-detected video is a shot gradient; The lens change position is determined according to the lens change type.

11. The device according to claim 10, characterized in that: The acquisition module is specifically used for: The shot type corresponding to the video clip in the video to be detected is obtained through a shot boundary prediction model, wherein the shot boundary prediction model includes a linear regression layer, and the shot boundary prediction model is trained according to a first pre-weight, a second pre-weight, and a third pre-weight, the first pre-weight is associated with the number of the second shot type, the second pre-weight is associated with the number of the first sub-shot type, the third pre-weight is associated with the number of the second sub-shot type, and the first sub-shot type and the second sub-shot type both belong to the first shot type.

12. The device according to claim 10, characterized in that: The acquisition module is specifically used for: The acquisition module is specifically used to acquire segment extraction information, wherein the segment extraction information includes a batch value of the video segment, a number of channels, a height of an image in the video segment, a width of an image in the video segment, and a number of frames of the image in the video segment; The video segment is acquired according to the segment extraction information, wherein the video segment belongs to the video to be detected.

13. The device according to claim 10, characterized in that: The acquisition module is specifically used for: The acquisition module is specifically used to acquire first segment extraction information, wherein the first segment extraction information includes a batch value of the first video segment, a number of channels, a height of an image in the first video segment, a width of an image in the first video segment, and a number of frames of an image in the first video segment; Acquire second segment extraction information, wherein the second segment extraction information includes a batch value of the second video segment, a number of channels, a height of an image in the second video segment, a width of an image in the second video segment, and a number of frames of the image in the second video segment; Acquire the first video segment according to the first segment extraction information, wherein the first video segment includes M consecutive images, and the first video segment belongs to the video to be detected; The second video segment is obtained according to the second segment extraction information, wherein the second video segment includes M continuous images, the second video segment belongs to the video to be detected, and the first video segment and the second video segment have the same N continuous images, where N is an integer greater than or equal to 1 and less than M.

14. The device according to claim 13, characterized in that: The acquisition module is specifically used for: The acquisition module is specifically configured to acquire the shot type corresponding to the first video clip through a shot boundary prediction model; Acquire a shot type corresponding to the second video clip by using a shot boundary prediction model; The determining module is specifically configured to generate a target video segment according to the first video segment and the second video segment if the shot type corresponding to the first video segment and the shot type corresponding to the second video segment are both the first shot type, wherein the target video segment includes (2M-N) consecutive images; The image change value is determined according to the target video segment.

15. The device according to claim 10, characterized in that: The determining module is specifically configured to determine that the shot change position is between the first image frame and the second image frame if the shot change type is the sudden shot change and the first average difference is greater than the second average difference; If the shot change type is the sudden shot change, and the second average difference is greater than the first average difference, the shot change position is determined to be between the second image frame and the third image frame.

16. The device according to claim 10, characterized in that: The determining module is specifically configured to determine that the lens change position includes the first image frame and the second image frame if the lens change type is the lens gradual change and the first average difference is greater than a second threshold; If the lens change type is the lens gradual change, and the second average difference is greater than the second threshold, determining that the lens change position includes the second image frame and the third image frame; If the lens change type is the lens gradual change, and the first average difference and the second average difference are both greater than the second threshold, it is determined that the lens change position includes the first image frame, the second image frame, and the third image frame.

17. The device according to claim 11, characterized in that: The shot boundary detection device further includes a model training device, and the model training device includes: An acquisition module, used for acquiring a set of video segments to be trained, wherein the set of video segments to be trained includes at least one video segment to be trained; The acquisition module is further used to acquire a predicted shot type corresponding to each to-be-trained video segment in the to-be-trained video segment set, wherein the predicted shot type is a first shot type or a second shot type, the first shot type represents a shot with a change, and the second shot type represents a shot without a change; A training module, configured to train a shot boundary prediction model to be trained using a loss function according to the predicted shot type and the real shot type corresponding to each video segment to be trained obtained by the acquisition module, to obtain model parameters, wherein the real shot type is obtained by pre-labeling the video segment to be trained; A generating module is used to generate a shot boundary prediction model according to the model parameters trained by the training module.

18. The device according to claim 17, characterized in that: The model training device includes a determination module; The determining module is used for determining the real shot type corresponding to each of the video segments to be trained according to the video segments to be trained after the acquiring module acquires the video segments to be trained set, wherein the real shot type is the first sub-shot type, the second sub-shot type or the first shot type, and the first sub-shot type and the second sub-shot type both belong to the first shot type; The determination module is further used to determine the number of the second lens type, the number of the first sub-lens type, and the number of the second sub-lens type according to the real lens type corresponding to each video segment to be trained; The determining module is further configured to determine a first pre-weighting of the second lens type, a second pre-weighting of the first sub-lens type, and a third pre-weighting of the second sub-lens type according to the number of the second lens type, the number of the first sub-lens type, and the number of the second sub-lens type; The training module is specifically used to train the shot boundary prediction model to be trained according to the first pre-weight, the second pre-weight and the third pre-weight using the loss function to obtain model parameters.

19. A terminal device, characterized in that: include: memories, transceivers, processors, and bus systems; Wherein, the memory is used to store programs; The processor is used to execute the program in the memory, comprising the following steps: Acquire a video to be detected, wherein the video to be detected includes at least one video segment, and the video segment includes multiple frames of continuous images; Acquire a shot type corresponding to a video clip in the to-be-detected video, wherein the shot type is a first shot type or a second shot type, the first shot type represents a shot with a change, and the second shot type represents a shot without a change; If the shot type belongs to the first shot type, determining an image change value corresponding to the video segment in the to-be-detected video, wherein the image change value is obtained by calculating two adjacent frames of images in the video segment in the to-be-detected video; Determining shot boundary information of a video segment in the to-be-detected video according to the image change value, wherein the shot boundary information includes a shot change type and a shot change position; The determining of the image change value corresponding to the video segment in the video to be detected includes: Acquire a first image frame, a second image frame, and a third image frame from the video segment of the video to be detected, wherein the first image frame is an image frame adjacent to the previous image frame of the second image frame, and the second image frame is an image frame adjacent to the previous image frame of the third image frame; Acquire a first color model corresponding to the first image frame, a second color model corresponding to the second image frame, and a third color model corresponding to the third image frame, wherein the first color model includes a first hue parameter, a first saturation parameter, and a first lightness parameter, the second color model includes a second hue parameter, a second saturation parameter, and a second lightness parameter, and the third color model includes a third hue parameter, a third saturation parameter, and a third lightness parameter; Calculating a first hue difference value according to the first hue parameter and the second hue parameter, calculating a first saturation difference value according to the first saturation parameter and the second saturation parameter, and calculating a first brightness difference value according to the first brightness parameter and the second brightness parameter; Calculating a first average difference value according to the first hue difference value, the first saturation difference value, and the first lightness difference value; Calculate a second hue difference value according to the second hue parameter and the third hue parameter, calculate a second saturation difference value according to the second saturation parameter and the third saturation parameter, and calculate a second brightness difference value according to the second brightness parameter and the third brightness parameter; Calculating a second average difference value according to the second hue difference value, the second saturation difference value, and the second lightness difference value; Determine, according to the first average difference and the second average difference, an image change value corresponding to the video segment in the video to be detected, wherein the image change value is a variance between the first average difference and the second average difference; The step of determining the shot boundary information of the video segment in the to-be-detected video according to the image change value includes: If the image change value corresponding to the video segment in the video to be detected is greater than or equal to a first threshold, determining that the shot change type of the video segment in the video to be detected is a shot mutation; If the image change value corresponding to the video segment in the to-be-detected video is less than the first threshold, determining that the shot change type of the video segment in the to-be-detected video is a shot gradient; Determining the lens change position according to the lens change type; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

20. A server, characterized in that: include: memories, transceivers, processors, and bus systems; Wherein, the memory is used to store programs; The processor is used to execute the program in the memory, comprising the following steps: Acquire a video to be detected, wherein the video to be detected includes at least one video segment, and the video segment includes multiple frames of continuous images; Acquire a shot type corresponding to a video clip in the to-be-detected video, wherein the shot type is a first shot type or a second shot type, the first shot type represents a shot with a change, and the second shot type represents a shot without a change; If the shot type belongs to the first shot type, determining an image change value corresponding to the video segment in the to-be-detected video, wherein the image change value is obtained by calculating two adjacent frames of images in the video segment in the to-be-detected video; Determining shot boundary information of a video segment in the to-be-detected video according to the image change value, wherein the shot boundary information includes a shot change type and a shot change position; The determining of the image change value corresponding to the video segment in the video to be detected includes: Acquire a first image frame, a second image frame, and a third image frame from the video segment of the video to be detected, wherein the first image frame is an image frame adjacent to the previous image frame of the second image frame, and the second image frame is an image frame adjacent to the previous image frame of the third image frame; Acquire a first color model corresponding to the first image frame, a second color model corresponding to the second image frame, and a third color model corresponding to the third image frame, wherein the first color model includes a first hue parameter, a first saturation parameter, and a first lightness parameter, the second color model includes a second hue parameter, a second saturation parameter, and a second lightness parameter, and the third color model includes a third hue parameter, a third saturation parameter, and a third lightness parameter; Calculating a first hue difference value according to the first hue parameter and the second hue parameter, calculating a first saturation difference value according to the first saturation parameter and the second saturation parameter, and calculating a first brightness difference value according to the first brightness parameter and the second brightness parameter; Calculating a first average difference value according to the first hue difference value, the first saturation difference value, and the first lightness difference value; Calculate a second hue difference value according to the second hue parameter and the third hue parameter, calculate a second saturation difference value according to the second saturation parameter and the third saturation parameter, and calculate a second brightness difference value according to the second brightness parameter and the third brightness parameter; Calculating a second average difference value according to the second hue difference value, the second saturation difference value, and the second lightness difference value; Determine, according to the first average difference and the second average difference, an image change value corresponding to the video segment in the video to be detected, wherein the image change value is a variance between the first average difference and the second average difference; The step of determining the shot boundary information of the video segment in the to-be-detected video according to the image change value includes: If the image change value corresponding to the video segment in the video to be detected is greater than or equal to a first threshold, determining that the shot change type of the video segment in the video to be detected is a shot mutation; If the image change value corresponding to the video segment in the to-be-detected video is less than the first threshold, determining that the shot change type of the video segment in the to-be-detected video is a shot gradient; Determining the lens change position according to the lens change type; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

21. A computer-readable storage medium comprising instructions, which, when executed on a computer, causes the computer to perform the method according to any one of claims 1 to 9.

Images