Scene switching recognition method, device, equipment, medium and computer product

Abstract

The present application discloses a scene switching recognition method, device, equipment, medium and computer product, which relate to the field of video processing. The method comprises: obtaining a first image frame and a second image frame, wherein the first image frame corresponds to a first timestamp and the second image frame corresponds to a second timestamp; performing pixel position transformation on the first image frame based on a target timestamp to obtain a first intermediate frame, wherein the target timestamp is a timestamp between the first timestamp and the second timestamp; performing pixel position transformation on the second image frame based on the target timestamp to obtain a second intermediate frame; and determining the scene switching recognition result between the first image frame and the second image frame based on the difference between the first intermediate frame and the second intermediate frame. The scene switching recognition result determined by the first intermediate frame and the second intermediate frame will not be misjudged due to the large displacement of the screen content of the first image frame and the second image frame, thereby improving the accuracy of scene switching recognition in the video screen.

Description

Technical Field

[0001] The present application relates to the field of video processing, and in particular to a scene switching recognition method, device, equipment, medium and computer product. Background Art

[0002] Scene switching refers to the visual interruption point in a continuous video, which usually comes from the interruption and switching of the camera lens during video shooting, or the insertion point during post-video editing. Scene switching can essentially be regarded as the segmentation of the video sequence in the time dimension. Therefore, the video can be divided into a collection of different scenes, each of which is the smallest unit for retrieving a video. Scene switching detection is used in many video analysis and processing methods and is a basic processing step, such as video compression, video super-resolution, video interpolation, etc.

[0003] In the related art, the scene switching detection method is generally implemented through a pixel difference method or a histogram matching method, that is, the frame difference between two adjacent frames of images is calculated through the statistical value of the pixel grayscale or color. If the frame difference reaches a preset threshold, it is considered that there is a scene switch between the two frames of images. If the frame difference does not reach the preset threshold, it is considered that the two frames of images are images of the same scene.

[0004] However, the scene switching detection method implemented in the above manner is very sensitive to large motion displacements. If the content displayed in the video undergoes a large displacement between the above two frames of images, the corresponding histogram will also change significantly, which will lead to a misjudgment of the scene switching. That is, even if no scene switching occurs between the above two frames of images, the detection system will also determine that a scene switching occurs, thereby reducing the accuracy of scene switching detection. Summary of the invention

[0005] The embodiments of the present application provide a scene switching recognition method, device, equipment, medium and computer product, which can improve the accuracy of scene switching recognition in video images. The technical solution is as follows:

[0006] In one aspect, a method for identifying a scene switch is provided, the method comprising:

[0007] Acquire a first image frame and a second image frame, where the first image frame and the second image frame are two adjacent frames of images in a video to be identified, the first image frame corresponds to a first timestamp, and the second image frame corresponds to a second timestamp;

[0008] Performing pixel position transformation on the first image frame based on a target timestamp to obtain a first intermediate frame, wherein the target timestamp is a timestamp between the first timestamp and the second timestamp;

[0009] Performing pixel position transformation on the second image frame based on the target timestamp to obtain a second intermediate frame;

[0010] Based on the difference between the first intermediate frame and the second intermediate frame, a scene switch recognition result between the first image frame and the second image frame is determined.

[0011] On the other hand, a scene switching recognition device is provided, the device comprising:

[0012] An acquisition module, used to acquire a first image frame and a second image frame, where the first image frame and the second image frame are two adjacent frames of images in a video to be identified, the first image frame corresponds to a first timestamp, and the second image frame corresponds to a second timestamp;

[0013] a generating module, configured to transform the pixel position of the first image frame based on a target timestamp to obtain a first intermediate frame, wherein the target timestamp is a timestamp between the first timestamp and the second timestamp;

[0014] The generating module is further used to perform pixel position transformation on the second image frame based on the target timestamp to obtain a second intermediate frame;

[0015] A determination module is used to determine a scene switching recognition result between the first image frame and the second image frame based on a difference between the first intermediate frame and the second intermediate frame.

[0016] On the other hand, a computer device is provided, wherein the terminal includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the scene switching recognition method described in any one of the embodiments of the present application.

[0017] On the other hand, a computer-readable storage medium is provided, wherein at least one program code is stored in the computer-readable storage medium, and the program code is loaded and executed by a processor to implement the scene switching recognition method described in any one of the embodiments of the present application.

[0018] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the scene switching recognition method described in any of the above embodiments.

[0019] The technical solution provided by this application includes at least the following beneficial effects:

[0020] When identifying scene switching in a video to be identified, pixel positions of a first image frame and a second image frame in the video to be identified are transformed based on a target timestamp, and a first intermediate frame corresponding to the first image frame and a second intermediate frame corresponding to the second image frame are obtained, wherein the first intermediate frame can represent the display status of the content in the first image frame at the target timestamp after motion estimation, and the second intermediate frame can represent the display status of the content in the second image frame at the target timestamp after motion estimation. The scene switching recognition result between the first image frame and the second image frame is determined according to the difference between the first intermediate frame and the second intermediate frame. The scene switching recognition result will not be misjudged due to the large displacement of the picture content of the first image frame and the second image frame, thereby improving the accuracy of scene switching recognition in the video picture. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 It is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;

[0023] Figure 2 is a flow chart of a scene switching recognition method provided by an exemplary embodiment of the present application;

[0024] Figure 3 is a flow chart of a scene switching recognition method provided by another exemplary embodiment of the present application;

[0025] Figure 4 is a schematic diagram of detection data provided by an exemplary embodiment of the present application;

[0026] Figure 5 is a flow chart of a video frame insertion method provided by an exemplary embodiment of the present application;

[0027] Figure 6 is a schematic diagram of a frame insertion network structure provided by an exemplary embodiment of the present application;

[0028] Figure 7 is a structural block diagram of a scene switching recognition device provided by an exemplary embodiment of the present application;

[0029] Figure 8 is a structural block diagram of a scene switching recognition device provided by another exemplary embodiment of the present application;

[0030] Fig. 9 It is a schematic diagram of the structure of a server provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0032] First, the nouns involved in the embodiments of the present application are briefly introduced:

[0033] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0034] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0035] Computer vision technology (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further perform image processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, autonomous driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0036] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0037] Optical flow refers to the instantaneous speed of the pixel movement of a moving object in space on the observation imaging plane (such as images and video frames). In video sequences, due to the small time interval, the displacement of the same pixel in the time domain between adjacent frames is usually approximated as the instantaneous speed of optical flow, which represents the movement of the pixel.

[0038] In an embodiment of the present application, scene switching in a video is identified through computer vision technology, optical flow information corresponding to two adjacent frames in the video to be identified is obtained based on machine learning, and position alignment is performed based on the optical flow information to obtain predicted intermediate frames corresponding to the two adjacent frames. The difference between the two predicted intermediate frames is used to determine whether there is a scene switch in the picture content of the two adjacent frames, thereby improving the accuracy of scene switch identification.

[0039] In combination with the above explanations, the application scenarios of the scene switching recognition method provided by the present application are schematically described. The above method can be applied to the following scenarios including but not limited to:

[0040] (I) The scene switching recognition method can be applied to the video frame interpolation technology.

[0041] Video interpolation technology is also known as frame rate conversion technology. It generates high frame rate video based on low frame rate video (N frames are added between two frames, N=1, 2, ...). By adding one or more frames to every two frames of the original picture display, the display time between each frame is shortened, thereby improving the smoothness of the video and achieving a better visual effect. Currently, video interpolation technology is widely used in old movie restoration, video live broadcast, video playback, etc.

[0042] When the target video needs to be interpolated, the two image frames before and after the target timestamp to be interpolated, i.e., the first image frame and the second image frame, are obtained, the first image frame and the second image frame are fused, an interpolated image is generated, and the interpolated image is inserted into the target timestamp. Since the picture content in the target video may have scene switching, if the first image frame and the second image frame correspond to the picture content of different scenes, if the two are directly fused to obtain the interpolated image, problems will arise, such as the interpolated image produced is irrelevant to the video content, or the interpolated image is a garbled image. Therefore, before interpolation, it is necessary to detect the scene switching in the target video. In the present application, it is determined whether a scene switching occurs between the image frames before and after the target timestamp to be interpolated. If a scene switching occurs, the interpolated image corresponding to the target timestamp needs special processing, for example, the previous frame is copied to replace the inaccurate interpolated image. If no scene switching occurs, the interpolated image obtained by fusion of the previous and next image frames is used. Since the false alarm rate of scene switching detection is reduced, the interpolation effect corresponding to the interpolated video finally obtained will also be improved.

[0043] (ii) The scene switching recognition method can be applied to video segmentation technology.

[0044] Video segmentation refers to dividing a video sequence into regions according to certain standards. The purpose is to separate meaningful entities from the video sequence. These meaningful entities become video objects in digital videos. Video segmentation plays a very important role in many fields, such as video coding, video object extraction in WEB technology, pattern recognition, computer vision, video retrieval, etc.

[0045] Video segmentation usually requires the use of information on the space and time axis of the video image for segmentation. The same or similar object segmentation method can be used for video images of the same scene. Therefore, the image sequence of the target video needs to be grouped according to the scene. Before grouping the image sequence of the target video according to the scene, it is necessary to determine which image sequences in the target video belong to the same scene, that is, to detect the dividing points between scenes. The scene switching recognition method provided in this application can be used to determine the nodes where scene switching occurs in the target video, so as to realize the grouping of image sequences according to scenes.

[0046] (iii) The scene switching recognition method can be applied to video compression technology.

[0047] Video compression, also known as video encoding, refers to the method of converting files in the original video format into files in another video format through compression technology. Video is a continuous sequence of images, consisting of continuous frames, and one frame is an image. Due to the visual persistence effect of the human eye, when a frame sequence is played at a certain rate, what you see is a video with continuous action. Since the similarity between continuous frames is extremely high, in order to facilitate storage and transmission, the original video can be encoded and compressed to remove the redundancy of spatial and temporal dimensions. The similarity between image frames belonging to the same scene is high. In order to facilitate video compression, scene switching recognition is required.

[0048] That is, when the target video needs to be compressed, the nodes where the scene switching occurs in the target video need to be detected, the image frames corresponding to the target video are reasonably grouped according to the scene switching nodes, and each image group is encoded to obtain the compressed data corresponding to the target video, which can block the error propagation of the video in the encoding prediction process. In the decoding process, the inter-frame prediction method can be used. In the decoding process of the image frames in the same image group, the subsequent frames can refer to the earlier decoded frames. Since the image contents belonging to the same image group are all in the same scene, the current encoded frame can obtain a valid reference image (block) from the earlier decoded frame to ensure the accuracy of the decoded video.

[0049] Please refer to Figure 1 , which shows a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application. The implementation environment includes: a terminal 110, a server 120 and a communication network 130.

[0050] The terminal 110 includes various terminal devices such as mobile phones, tablet computers, desktop computers, portable notebook computers, etc. The terminal 110 is used to provide the target video or the image sequence corresponding to the target video to the server 120 .

[0051] The server 120 includes a scene switching recognition module. The server 120 inputs the target video as a video to be recognized into the recognition module for scene switching recognition, and obtains scene switching recognition results between image frames in the target video. Optionally, the server 120 can provide functions including video interpolation, video compression, and video segmentation to the terminal 110. The recognition module is a module in the above functions. Taking the video interpolation function provided by the server 120 as an example, the terminal 110 requests the server 120 to perform interpolation processing on the target video. After receiving the request, the server 120 determines the scene switching recognition results between image frames in the target video uploaded by the terminal 110 through the recognition module, interpolates the target video based on the scene switching recognition results, generates an interpolated video with a higher frame rate than the target video, and returns the interpolated video to the terminal 110.

[0052] It is worth noting that the above-mentioned server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0053] Among them, cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to realize data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which is used on demand and flexible and convenient. Cloud computing technology will become an important support. The background services of technical network systems require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0054] In some embodiments, the server 120 can also be implemented as a node in a blockchain system. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the blockchain underlying platform, the platform product service layer, and the application service layer.

[0055] The underlying blockchain platform can include processing modules such as user management, basic services, smart contracts, and operation monitoring. Among them, the user management module is responsible for the identity information management of all blockchain participants, including maintaining the generation of public and private keys (account management), key management, and the maintenance of the corresponding relationship between the user's real identity and the blockchain address (authority management), etc., and, under authorization, supervises and audits the transactions of certain real identities and provides risk control rule configuration (risk control audit); the basic service module is deployed on all blockchain node devices to verify the validity of business requests and record the valid requests to the storage after consensus is reached. For a new business request, the basic service first performs interface adaptation analysis and authentication processing (interface adaptation), and then encrypts the business information through the consensus algorithm (consensus management), and transmits it to the shared ledger in a complete and consistent manner after encryption (network communication), and records and stores it; the smart contract module is responsible for the registration and issuance of contracts, contract triggering and contract execution. Developers can define the contract logic in a certain programming language and publish it to the blockchain (contract registration). According to the logic of the contract terms, call the key or other events to trigger the execution and complete the contract logic. At the same time, it also provides the function of contract upgrade and cancellation; the operation monitoring module is mainly responsible for the deployment, configuration modification, contract setting, cloud adaptation and visual output of the real-time status of the product during the product release process.

[0056] Illustratively, the terminal 110 and the server 120 are connected via a communication network 130 .

[0057] Please refer to Figure 2 , which shows a scene switching recognition method shown in an embodiment of the present application. In the embodiment of the present application, the method is applied to Figure 1 In the server shown, the method includes:

[0058] Step 201: Acquire a first image frame and a second image frame.

[0059] The first image frame and the second image frame are two adjacent image frames in the video to be identified. The first image frame corresponds to a first timestamp, and the second image frame corresponds to a second timestamp.

[0060] Optionally, the video to be identified may be a video uploaded by the terminal, or a video obtained from a database.

[0061] Schematically, a video to be identified is obtained, and the video to be identified is subjected to frame processing to obtain an image frame sequence, and a first image frame and a second image frame adjacent in time domain are obtained from the above image frame sequence. The above video to be identified has a frame rate (Frames Per Second, FPS). In some embodiments, frame processing is performed according to the frame rate and video duration corresponding to the video to be identified. For example, if the frame rate corresponding to the video to be identified is 24fps and the video duration corresponding to the video to be identified is 10s, then the video to be identified is subjected to frame processing to obtain an image frame sequence consisting of 240 image frames.

[0062] The first image frame and the second image frame are obtained from the above-mentioned image frame sequence. Optionally, if all nodes where scene switching occurs in the video to be identified are determined, all image frames in the image frame sequence corresponding to the video to be identified are grouped, and each group includes an image pair of <first image frame, second image frame>. For example, the image frame sequence includes n image frames, and the obtained image pair groups are <image frame 1, image frame 2>, <image frame 2, image frame 3>, <image frame 3, image frame 4>, ..., <image frame n-1, image frame n>. Optionally, the first image frame and the second image frame in the image frame sequence can also be determined according to preset requirements. For example, the terminal instructs to perform frame interpolation (interpolation) for the period when the target video object in the video to be identified moves faster. The server identifies the target video object in the video to be identified, determines the image frame including the target video object, and determines whether the displacement of the target video object in the image frame meets the interpolation requirement. If so, the above-mentioned image frame generates an image pair of <first image frame, second image frame> to detect scene switching.

[0063] Step 202: Perform pixel position transformation on the first image frame based on the target timestamp to obtain a first intermediate frame.

[0064] The target timestamp is a timestamp between the first timestamp and the second timestamp. In some embodiments, the target timestamp may be a midpoint timestamp between the first timestamp and the second timestamp, that is, the time length from the target timestamp to the first timestamp is the same as the time length from the target timestamp to the second timestamp. The target timestamp may also be another timestamp between the first timestamp and the second timestamp, which is not limited here.

[0065] In some embodiments, motion estimation is performed on the first image frame and the second image frame respectively based on the target timestamp to determine the change of the first image frame between the first timestamp and the target timestamp, and the change of the second image frame between the second timestamp and the target timestamp. Schematically, the motion estimation method includes a full search method, a predicted motion vector field adaptive fast search method, an enhanced predictive zonal search (EPZS) method, an asymmetric cross multi-layer hexagon search method, an optical flow estimation method, etc., which are not limited here.

[0066] In an embodiment of the present application, the above-mentioned motion estimation process is implemented by an optical flow estimation method based on pixel motion. Schematically, first optical flow information of a first image frame between a first timestamp and a target timestamp is obtained, and the first optical flow information is used to indicate the motion of pixels in the first image frame between the first timestamp and the target timestamp; second optical flow information of a second image frame between a second timestamp and a target timestamp is obtained, and the second optical flow information is used to indicate the motion of pixels in the second image frame between the second timestamp and the target timestamp. Schematically, the above-mentioned optical flow information may include the instantaneous speed (optical flow) of the pixel moving between timestamps, and may also include the displacement of the pixel moving between timestamps, or other data information that can represent the motion of the pixel.

[0067] In some embodiments, the optical flow network U flow Estimate the first optical flow information Flow of the first image frame Frame0 between the first timestamp and the target timestamp 0→t And the second optical flow information Flow of the second image frame Frame1 between the second timestamp and the target time 1→t , as shown in Formula 1.

[0068] Formula 1: Flow 0→t , Flow 1→t =U flow (Frame0, Frame1)

[0069] Optionally, the method for obtaining the above-mentioned optical flow information includes but is not limited to: a gradient-based optical flow estimation method, that is, using the spatiotemporal differential (i.e., the spatiotemporal gradient function) of the time-varying image grayscale (or its filtered form) to calculate the velocity vector of the pixel, for example, the Horn-Schunck algorithm and the Lucas-Kanade (LK) algorithm; a feature-based optical flow estimation method, that is, realizing optical flow estimation by continuously locating and tracking the main features of the target object; a region-based optical flow estimation method, that is, first locating similar regions in the image, and then calculating the optical flow through the displacement of similar regions; a phase-based optical flow estimation method, that is, using phase information to obtain the optical flow field corresponding to the image.

[0070] In some embodiments, after determining the motion of the pixel between the first timestamp and the target timestamp, the pixel position can be aligned according to the motion to obtain the corresponding intermediate frame. Taking the motion represented by optical flow information as an example, the pixel position of the first image frame is transformed based on the first optical flow information to obtain the first intermediate frame.

[0071] Schematically, the above pixel position alignment operation can be an image warping (Warp) operation. Since the optical flow field of the pixel is a two-dimensional vector, it contains the displacement of each pixel in a two-dimensional plane (horizontally in the x direction and vertically in the y direction). The warp alignment operation transforms the coordinates of the pixel points on the image according to the optical flow, thereby translating the pixels to corresponding places in different time domains (adding and subtracting coordinates along the x direction means translating left and right in the horizontal direction, and adding and subtracting coordinates along the y direction means translating up and down in the vertical direction). That is, using the first optical flow information Flow 0→t Align the first image frame Frame0 through warp to get the first intermediate frame Frame 0→t , as shown in Formula 2.

[0072] Formula 2: Frame 0→t =warp(Frame0,Flow 0→t )

[0073] The above pixel position alignment operation may also be implemented through other coordinate transformation or position alignment methods, which are not limited here.

[0074] Step 203: Transform the pixel position of the second image frame based on the target timestamp to obtain a second intermediate frame.

[0075] Schematically, the motion estimation process corresponding to the second image frame is the same as that of the first image frame, and is not described in detail here. In an embodiment of the present application, an optical flow estimation method based on pixel motion is used to implement motion estimation, and second optical flow information of the second image frame between the second timestamp and the target timestamp is obtained as an example. After obtaining the second optical flow information, the pixel position of the second image frame is transformed based on the second optical flow information to obtain a second intermediate frame.

[0076] Taking the pixel position alignment operation as an example of image warping, the second optical flow information Flow 1→t The second image frame Frame1 is aligned by warp to obtain the second intermediate frame Frame 1→t , as shown in Formula 3.

[0077] Formula 3: Frame 1→t =warp(Frame1,Flow 1→t )

[0078] The first intermediate frame and the second intermediate frame are the result of the first image frame and the second image frame moving to the same timestamp, which can eliminate the deviation caused by the movement at different times.

[0079] Step 204 : determining a scene switching recognition result between the first image frame and the second image frame based on a difference between the first intermediate frame and the second intermediate frame.

[0080] In the implementation of the present application, whether a scene switch occurs between the first image frame and the second image frame is determined based on the difference between the first intermediate frame and the second intermediate frame. If a scene switch occurs between the first image frame and the second image frame, there is a large difference between the first image frame and the second image frame; if no scene switch occurs between the first image frame and the second image frame, there is a small difference between the first image frame and the second image frame.

[0081] Illustratively, a first intermediate frame is compared with a second intermediate frame to determine difference information between the first intermediate frame and the second intermediate frame; in response to the difference information reaching a preset difference threshold, the scene switch recognition result between the first image frame and the second image frame is determined to be a first result, and the first result is used to indicate that the scene corresponding to the first image frame is different from the scene corresponding to the second image frame, that is, a scene switch has occurred; or, in response to the difference information failing to match the preset difference threshold, the scene switch recognition result between the first image frame and the second image frame is determined to be a second result, and the second result is used to indicate that the scene corresponding to the first image frame is the same as the scene corresponding to the second image frame, that is, no scene switch has occurred.

[0082] Optionally, the difference information can be determined by a pixel difference method, that is, the pixel difference between each pixel point in the first intermediate frame and the second intermediate frame is determined; the average value of the pixel difference of each image is determined as the difference information. In an example, the width and height of the first intermediate frame and the second intermediate frame are W*H, and the width and height of each pixel point (i, j) in the first intermediate frame Frame are calculated. 0→t and the second intermediate frame 1→t The pixel difference between them is calculated and the mean is taken to obtain the difference information differ, as shown in Formula 4.

[0083] Formula 4:

[0084] Optionally, the difference information may also be determined by histogram statistics, that is, a first histogram corresponding to the first intermediate frame and a second histogram corresponding to the second intermediate frame are obtained; and a difference between the first histogram and the second histogram is determined as the difference information.

[0085] The above difference information can also be determined by feature point matching method, deep learning neural network, etc., which is not limited here.

[0086] In summary, the scene switching recognition method provided by the embodiment of the present application, when identifying scene switching in a video to be identified, performs pixel position transformation on the first image frame and the second image frame in the video to be identified based on the target timestamp, and obtains a first intermediate frame corresponding to the first image frame and a second intermediate frame corresponding to the second image frame, wherein the first intermediate frame can represent the display status of the content in the first image frame at the target timestamp after motion estimation, and the second intermediate frame can represent the display status of the content in the second image frame at the target timestamp after motion estimation, and the scene switching recognition result between the first image frame and the second image frame is determined according to the difference between the first intermediate frame and the second intermediate frame, and the scene switching recognition result will not be misjudged due to the large displacement of the picture content of the first image frame and the second image frame, thereby improving the accuracy of scene switching recognition in the video picture.

[0087] Please refer to Figure 3 , which shows a scene switching recognition method shown in an embodiment of the present application. In the embodiment of the present application, the method is described by taking the optical flow estimation to realize the motion estimation of the pixel as an example. The method includes:

[0088] Step 301: Acquire a first image frame and a second image frame.

[0089] The first image frame and the second image frame are two adjacent image frames in the video to be identified. The first image frame corresponds to a first timestamp, and the second image frame corresponds to a second timestamp.

[0090] In an illustrative manner, a video to be identified is obtained, and the video to be identified is subjected to frame processing to obtain an image frame sequence, and a first image frame and a second image frame adjacent in time domain are obtained from the image frame sequence. The video to be identified has a frame rate (Frames Per Second, FPS) corresponding to the video to be identified. In some embodiments, the frame processing is performed according to the frame rate corresponding to the video to be identified and the video duration. The first image frame and the second image frame are obtained from the image frame sequence.

[0091] Step 302: Acquire first optical flow information of a first image frame between a first timestamp and a target timestamp.

[0092] Optionally, the method for obtaining the above-mentioned optical flow information includes but is not limited to: a gradient-based optical flow estimation method, that is, using the spatiotemporal differential (i.e., the spatiotemporal gradient function) of the time-varying image grayscale (or its filtered form) to calculate the velocity vector of the pixel, for example, the Horn-Schunck algorithm and the Lucas-Kanade (LK) algorithm; a feature-based optical flow estimation method, that is, realizing optical flow estimation by continuously locating and tracking the main features of the target object; a region-based optical flow estimation method, that is, first locating similar regions in the image, and then calculating the optical flow through the displacement of similar regions; a phase-based optical flow estimation method, that is, using phase information to obtain the optical flow field corresponding to the image.

[0093] Step 303: Acquire second optical flow information of the second image frame between the second timestamp and the target timestamp.

[0094] In the embodiment of the present application, through the optical flow network U flow Estimate the first optical flow information Flow of the first image frame Frame0 between the first timestamp and the target timestamp 0→t And the second optical flow information Flow of the second image frame Frame1 between the second timestamp and the target time 1→t .

[0095] Step 304: transform the pixel positions of the first image frame based on the first optical flow information to obtain a first intermediate frame.

[0096] In some embodiments, first coordinate change data of pixels in the first image frame between the first timestamp and the target timestamp is obtained based on the first optical flow information; pixels in the first image frame are displaced based on the first coordinate change data to obtain a first intermediate frame. Schematically, the first coordinate change data is determined by the pixel motion speed indicated by the first optical flow information, that is, the first optical flow information is converted into the first motion speed of the pixels in the first image frame, the first time interval between the first timestamp and the target timestamp is determined, and the first coordinate change data is determined based on the first motion speed and the first time interval.

[0097] In some embodiments, pixel alignment is performed according to the first optical flow information by image twisting to obtain a first intermediate frame, that is, pixels in the first image frame are mapped to new positions according to the first optical flow information to generate the first intermediate frame.

[0098] Step 305: Transform the pixel positions of the second image frame based on the second optical flow information to obtain a second intermediate frame.

[0099] In some embodiments, second coordinate change data of pixels in the second image frame between the second timestamp and the target timestamp is obtained based on the second optical flow information; pixels in the second image frame are displaced based on the second coordinate change data to obtain a second intermediate frame. Schematically, the second coordinate change data is determined by the pixel motion speed indicated by the second optical flow information, that is, the second optical flow information is converted into a second motion speed of pixels in the second image frame, a second time interval between the second timestamp and the target timestamp is determined, and the second coordinate change data is determined based on the second motion speed and the second time interval.

[0100] In some embodiments, pixel alignment is performed according to the first optical flow information by image twisting to obtain a second intermediate frame, that is, pixels in the second image frame are mapped to new positions according to the second optical flow information to generate the second intermediate frame.

[0101] Step 306: compare the first intermediate frame and the second intermediate frame to determine difference information between the first intermediate frame and the second intermediate frame.

[0102] Optionally, the method for determining the above-mentioned difference information includes but is not limited to at least one of the methods such as pixel difference method, histogram statistics method, feature point matching method, deep learning neural network method, etc., which is not limited here.

[0103] In the implementation of the present application, whether a scene switch occurs between the first image frame and the second image frame is determined based on the difference between the first intermediate frame and the second intermediate frame. If a scene switch occurs between the first image frame and the second image frame, there is a large difference between the first image frame and the second image frame; if no scene switch occurs between the first image frame and the second image frame, there is a small difference between the first image frame and the second image frame.

[0104] Step 307 : In response to the difference information reaching a preset difference threshold, determining that the scene switch recognition result between the first image frame and the second image frame is a first result.

[0105] The first result is used to indicate that the scene corresponding to the first image frame is different from the scene corresponding to the second image frame.

[0106] Illustratively, the above difference threshold has a corresponding relationship with the method of obtaining the difference information. For example, when the difference information is obtained by a pixel difference method, the difference threshold corresponds to a preset grayscale threshold.

[0107] Optionally, the above difference threshold may be preset by the system or indicated by the terminal, which is not limited here.

[0108] In the embodiment of the present application, when the difference information reaches a preset threshold, it indicates that there is a scene switch between the first image frame and the second image frame, and the scene switch recognition result between the first image frame and the second image frame is output as the first result.

[0109] Step 308 : In response to the difference information failing to match the preset difference threshold, determining that the scene switch recognition result between the first image frame and the second image frame is a second result.

[0110] The second result is used to indicate that the scene corresponding to the first image frame is the same as the scene corresponding to the second image frame.

[0111] In an embodiment of the present application, when the difference information does not reach a preset threshold, it means that there is no scene switch between the first image frame and the second image frame, that is, the first image frame and the second image frame belong to the same scene, and the scene switch recognition result output between the first image frame and the second image frame is the second result.

[0112] The scene switching recognition method provided in the embodiment of the present application can eliminate the influence of motion conditions on the scene switching judgment between image frames, because the first intermediate frame and the second intermediate frame generated by optical flow information are the result of the original image frame moving to the same target timestamp, wherein the above-mentioned motion conditions include but are not limited to the following conditions: object changes, object movements, lens movements, etc.

[0113] Schematically, the scene switching recognition method provided in the embodiment of the present application uses the intermediate frames after optical flow warp alignment between video image frames to detect scene switching. No additional auxiliary modules are required, and the aligned front and rear frames of the video can effectively eliminate the interference caused by motion displacement, greatly reducing the false alarm rate of scene switching detection. That is, the scene switching recognition method provided in the embodiment of the present application effectively reduces the occurrence of false detection when there is a large displacement in the image based on the original input frame image in the existing method of scene switching detection. The existing method performs detection based on the original input frame, and the error rate between adjacent frames is very high. The present application performs detection based on the aligned image, which can greatly increase the stability of the detection system and reduce the error caused by motion. For example, in a certain motion video, there are actually 5 scene switches, and the detection data is as follows: Figure 4 As shown, detection result A401 is the detection result of the existing method, detection result B402 is the detection result of the present application, the vertical axis is the difference comparison between the current frame and the previous frame, and when the percentage exceeds a certain threshold, it is determined that a scene switch has occurred. It can be seen that detection result A401 cannot identify the true scene switch in some areas (the values ​​are all high), while the corresponding detection result B402 of the present application has obvious peaks at each scene switch, and the corresponding scene switch detection result is more accurate.

[0114] In summary, the scene switching recognition method provided by the embodiment of the present application, when recognizing the scene switching in the video to be recognized, uses the optical flow information to respectively transform the pixel positions of the first image frame and the second image frame in the video to be recognized with the target timestamp as the reference, obtains the first intermediate frame corresponding to the first image frame and the second intermediate frame corresponding to the second image frame, and determines the scene switching recognition result between the first image frame and the second image frame according to the difference between the first intermediate frame and the second intermediate frame. Among them, the first intermediate frame can represent the display of the content in the first image frame at the target timestamp after motion estimation, and the second intermediate frame can represent the display of the content in the second image frame at the target timestamp after motion estimation. The scene switching recognition result determined by the first intermediate frame and the second intermediate frame will not be misjudged due to the large displacement of the screen content of the first image frame and the second image frame, thereby improving the accuracy of scene switching recognition in the video screen.

[0115] Please refer to Figure 5 , which shows a video frame insertion method shown in an embodiment of the present application. In the embodiment of the present application, the above-mentioned scene switching recognition method is applied to the scene of video frame insertion. The method includes:

[0116] Step 501: Obtain a video to be identified.

[0117] Optionally, the video to be identified may be a video uploaded by the terminal, or a video obtained from a database.

[0118] Step 502 : segment the video to be identified according to the original number of frames corresponding to the video to be identified, to obtain an image frame sequence.

[0119] Schematically, the above-mentioned video to be identified has a frame rate, and the original frame number of the video to be identified can be determined according to the frame rate and video duration of the video to be identified. After the original frame number is determined, the video to be identified can be segmented according to the original frame number to obtain the corresponding image frame sequence.

[0120] Step 503: determine a first image frame and a second image frame that are adjacent in time domain from the image frame sequence.

[0121] In an embodiment of the present application, at least one time period to be inserted is determined according to the target frame number requirement corresponding to the video insertion task. The time period to be inserted is the time period in the video to be identified where insertion needs to be performed. The time period to be inserted includes two frames of screen images. The first image frame and the second image frame to be inserted are determined by the time period to be inserted. Schematically, the target frame number requirement is obtained, and the target frame number requirement is used to indicate the number of frames of the interpolated video generated according to the video to be identified; the time period to be inserted in the image frame sequence is determined based on the target frame number requirement; the screen images corresponding to the time period to be inserted are determined as the first image frame and the second image frame; the starting time corresponding to the display duration of the first image frame in the time period to be inserted is determined as the first timestamp; the starting time corresponding to the display duration of the second image frame in the video to be identified is determined as the second timestamp.

[0122] In one example, the frame rate of the video to be identified is 24fps, and the corresponding video duration is 10s. The current video interpolation task indicates the generation of an interpolated video with a frame rate of 60fps. It is determined that 20 frames of images need to be inserted into the video to be identified per second, and there are 23 interpolated frame periods per second. Taking the example of adding one frame to each image pair, 20 interpolated frame periods need to be determined from the 23 interpolated frame periods. Optionally, the above 20 interpolated frame periods can be randomly selected or determined according to preset requirements. Each interpolated frame period corresponds to an image pair <first image frame, second image frame>. The corresponding first image frame, first timestamp, second image frame, and second timestamp can be determined by the above determined interpolated frame period.

[0123] Step 504 , performing pixel position transformation on the first image frame based on the target timestamp to obtain a first intermediate frame.

[0124] The target timestamp is a timestamp between the first timestamp and the second timestamp. In some embodiments, the target timestamp may be a midpoint timestamp between the first timestamp and the second timestamp, that is, the time length from the target timestamp to the first timestamp is the same as the time length from the target timestamp to the second timestamp. The target timestamp may also be another timestamp between the first timestamp and the second timestamp, which is not limited here.

[0125] In the embodiment of the present application, the motion estimation of pixels is realized by optical flow information, that is, the first optical flow information Flow between the first timestamp and the target timestamp of the first image frame Frame0 is estimated by the optical flow network. 0→t , and then warp the first image frame according to the first optical flow information to obtain a first intermediate frame.

[0126] Step 505 , performing pixel position transformation on the second image frame based on the target timestamp to obtain a second intermediate frame.

[0127] In the embodiment of the present application, the second intermediate frame is acquired in the same manner as the first intermediate frame, and will not be described in detail herein.

[0128] Step 506: compare the first intermediate frame and the second intermediate frame to determine difference information between the first intermediate frame and the second intermediate frame.

[0129] Optionally, the method for determining the above-mentioned difference information includes but is not limited to at least one of the methods such as pixel difference method, histogram statistics method, feature point matching method, deep learning neural network method, etc., which is not limited here.

[0130] In the implementation of the present application, whether a scene switch occurs between the first image frame and the second image frame is determined based on the difference between the first intermediate frame and the second intermediate frame. If a scene switch occurs between the first image frame and the second image frame, there is a large difference between the first image frame and the second image frame; if no scene switch occurs between the first image frame and the second image frame, there is a small difference between the first image frame and the second image frame.

[0131] Step 507 : in response to the difference information reaching a preset difference threshold, determining that the scene switching recognition result between the first image frame and the second image frame is a first result.

[0132] The first result is used to indicate that the scene corresponding to the first image frame is different from the scene corresponding to the second image frame.

[0133] Illustratively, the above difference threshold has a corresponding relationship with the method of obtaining the difference information. For example, when the difference information is obtained by a pixel difference method, the difference threshold corresponds to a preset grayscale threshold.

[0134] Optionally, the above difference threshold may be preset by the system or indicated by the terminal, which is not limited here.

[0135] Step 508 : In response to the difference information failing to match the preset difference threshold, determining that the scene switch recognition result between the first image frame and the second image frame is a second result.

[0136] The second result is used to indicate that the scene corresponding to the first image frame is the same as the scene corresponding to the second image frame.

[0137] In the embodiment of the present application, after the scene switching recognition result between the first image frame and the second image frame is determined according to the preset difference threshold, the corresponding target interpolation frame image is acquired based on the scene switching recognition result.

[0138] Step 509 : in response to the scene switching recognition result being the first result, determining the first image frame or the second image frame as a target interpolation frame image.

[0139] In the embodiment of the present application, when there is a scene switch between the first image frame and the second image frame, the first image frame or the second image frame is used as a target interpolation frame image inserted into the target timestamp.

[0140] Step 510 , in response to the scene switching recognition result being the second result, the first intermediate frame and the second intermediate frame are merged to obtain a target interpolated frame image.

[0141] Optionally, when there is no scene switch between the first image frame and the second image frame, the first intermediate frame or the second intermediate frame can be used as a target interpolation frame image inserted into a target timestamp, or the first intermediate frame and the second intermediate frame can be fused through a convolution network to obtain a target interpolation frame image.

[0142] Step 511: insert the target interpolated frame image into the target timestamp in the video to be recognized to generate an interpolated frame video.

[0143] After the target interpolation frame image is determined, the target interpolation frame image is inserted into the target timestamp in the video to be identified to generate an interpolation frame video.

[0144] For an example, see Figure 6, which shows a schematic diagram of the structure of a frame insertion network 600 shown in an exemplary embodiment of the present application. The interpolation network 600 includes an optical flow network 610, a warp processing module 620, a discrimination module 630, a convolutional network 640, and a video synthesis module 650. The first image frame 601 and the second image frame 602 are input into the optical flow network 610 to obtain the first optical flow information 611 and the second optical flow information 612 respectively. After the warp processing module 620, the first intermediate frame 621 and the second intermediate frame 622 can be obtained. The first intermediate frame 621 and the second intermediate frame 622 are input into the discrimination module 630 to determine whether there is a scene switch between the first image frame 601 and the second image frame 602. If so, the first image frame 601 is input into the video synthesis module 650 as the target interpolation video. If not, the first intermediate frame 621 and the second intermediate frame 622 are input into the convolutional network 640 for fusion. The convolutional network 640 inputs the fused target interpolation video into the video synthesis module 650, and the video synthesis module 650 synthesizes the interpolation video.

[0145] In summary, the video interpolation method provided by the embodiment of the present application, when interpolating in the video to be identified, respectively transforms the pixel positions of the first image frame and the second image frame in the video to be identified based on the target timestamp, obtains the first intermediate frame corresponding to the first image frame and the second intermediate frame corresponding to the second image frame, determines the scene switching recognition result between the first image frame and the second image frame according to the difference between the first intermediate frame and the second intermediate frame, and gives different target interpolation images according to different scene switching recognition results. That is, the scene switching recognition result determined by the first intermediate frame and the second intermediate frame will not be misjudged due to the large displacement of the picture content of the first image frame and the second image frame, thereby improving the accuracy of scene switching recognition in the video picture, thereby improving the accuracy of the obtained interpolation video.

[0146] Please refer to Figure 7 , which shows a structural block diagram of a scene switching recognition device provided by an exemplary embodiment of the present application, the device includes the following modules:

[0147] An acquisition module 710 is configured to acquire a first image frame and a second image frame, wherein the first image frame and the second image frame are two adjacent image frames in a video to be identified, the first image frame corresponds to a first timestamp, and the second image frame corresponds to a second timestamp;

[0148] A generating module 720, configured to perform pixel position transformation on the first image frame based on a target timestamp to obtain a first intermediate frame, wherein the target timestamp is a timestamp between the first timestamp and the second timestamp;

[0149] The generating module 720 is further configured to perform pixel position transformation on the second image frame based on the target timestamp to obtain a second intermediate frame;

[0150] The determination module 730 is configured to determine a scene switching recognition result between the first image frame and the second image frame based on a difference between the first intermediate frame and the second intermediate frame.

[0151] In some optional embodiments, such as Figure 8 As shown, the generating module 720 further includes:

[0152] A first acquisition unit 721 is used to acquire first optical flow information of the first image frame between the first timestamp and the target timestamp, where the first optical flow information is used to indicate the motion of pixels in the first image frame between the first timestamp and the target timestamp;

[0153] A generating unit 722, configured to perform pixel position transformation on the first image frame based on the first optical flow information to obtain the first intermediate frame;

[0154] The first acquisition unit 721 is further used to acquire second optical flow information of the second image frame between the second timestamp and the target timestamp, where the second optical flow information is used to indicate the movement of pixels in the second image frame between the second timestamp and the target timestamp;

[0155] The generating unit 722 is further configured to perform pixel position transformation on the second image frame based on the second optical flow information to obtain the second intermediate frame.

[0156] In some optional embodiments, the first acquisition unit 721 is further configured to acquire first coordinate change data of a pixel in the first image frame between the first timestamp and the target timestamp based on the first optical flow information;

[0157] The generating unit 722 is further configured to displace pixels in the first image frame based on the first coordinate change data to obtain the first intermediate frame;

[0158] The first acquisition unit 721 is further configured to acquire second coordinate change data of a pixel in the second image frame between the second timestamp and the target timestamp based on the second optical flow information;

[0159] The generating unit 722 is further configured to shift pixels in the second image frame based on the second coordinate change data to obtain the second intermediate frame.

[0160] In some optional embodiments, the first acquisition unit 721 is further configured to convert the first optical flow information into a first motion speed of pixels in the first image frame;

[0161] The first acquisition unit 721 is further configured to determine a first time interval between the first timestamp and the target timestamp;

[0162] The first acquisition unit 721 is further configured to determine the first coordinate change data based on the first movement speed and the first time interval;

[0163] The first acquisition unit 721 is further configured to convert the second optical flow information into a second motion speed of pixels in the second image frame;

[0164] The first acquisition unit 721 is further configured to determine a second time interval between the second timestamp and the target timestamp;

[0165] The first acquisition unit 721 is further configured to determine the second coordinate change data based on the second movement speed and the second time interval.

[0166] In some optional embodiments, the determining module 730 further includes:

[0167] A comparing unit 731 is used to compare the first intermediate frame with the second intermediate frame to determine difference information between the first intermediate frame and the second intermediate frame;

[0168] The determination unit 732 is used to determine, in response to the difference information reaching a preset difference threshold, that the scene switch recognition result between the first image frame and the second image frame is a first result, the first result being used to indicate that the scene corresponding to the first image frame is different from the scene corresponding to the second image frame; or, in response to the difference information failing to match the preset difference threshold, determine that the scene switch recognition result between the first image frame and the second image frame is a second result, the second result being used to indicate that the scene corresponding to the first image frame is the same as the scene corresponding to the second image frame.

[0169] In some optional embodiments, the comparison unit 731 is further configured to determine a pixel difference between each pixel point in the first intermediate frame and the second intermediate frame;

[0170] The comparison unit 731 is further configured to determine a mean value of pixel differences of each image as the difference information.

[0171] In some optional embodiments, the comparing unit 731 is further configured to obtain a first histogram corresponding to the first intermediate frame and a second histogram corresponding to the second intermediate frame;

[0172] The comparing unit 731 is further configured to determine a difference between the first histogram and the second histogram as the difference information.

[0173] In some optional embodiments, the acquisition module 710 is further configured to acquire a corresponding target interpolated frame image based on the scene switching recognition result;

[0174] The generating module 720 is further configured to insert the target interpolated frame image into the target timestamp in the video to be identified, so as to generate an interpolated frame video.

[0175] In some optional embodiments, the scene switching recognition result includes a first result and a second result, the first result is used to indicate that the scene corresponding to the first image frame is different from the scene corresponding to the second image frame, and the second result is used to indicate that the scene corresponding to the first image frame is the same as the scene corresponding to the second image frame;

[0176] The acquisition module 710 is further configured to, in response to the scene switching recognition result being the first result, determine the first image frame or the second image frame as the target interpolation frame image;

[0177] The acquisition module 710 is further configured to, in response to the scene switching recognition result being the second result, fuse the first intermediate frame with the second intermediate frame to obtain the target interpolated frame image.

[0178] In some optional embodiments, the acquisition module 710 further includes:

[0179] The second acquisition unit 711 is used to acquire the video to be identified;

[0180] A segmentation unit 712 is used to segment the video to be identified according to the number of original frames corresponding to the video to be identified to obtain an image frame sequence;

[0181] The second acquisition unit 711 is further configured to determine, from the image frame sequence, the first image frame and the second image frame that are adjacent in time domain.

[0182] In some optional embodiments, the second acquisition unit 711 is further used to acquire a target frame number requirement, where the target frame number requirement is used to indicate the frame number of the interpolated frame video generated according to the video to be identified;

[0183] The second acquisition unit 711 is further configured to determine a time period to be inserted in the image frame sequence based on the target frame number requirement, wherein the time period to be inserted is a time period in the video to be identified during which frame insertion is required, and the time period to be inserted includes two frames of images;

[0184] The second acquisition unit 711 is further configured to determine the corresponding screen images in the frame-to-be-inserted period as the first image frame and the second image frame;

[0185] The second acquisition unit 711 is further configured to determine a start time corresponding to a display duration of the first image frame in the frame-to-be-inserted period as the first timestamp;

[0186] The second acquisition unit 711 is further configured to determine a start time corresponding to a display duration of the second image frame in the to-be-identified video as the second timestamp.

[0187] In summary, the scene switching recognition device provided by the embodiment of the present application, when recognizing the scene switching in the video to be recognized, respectively transforms the pixel positions of the first image frame and the second image frame in the video to be recognized based on the target timestamp, obtains the first intermediate frame corresponding to the first image frame and the second intermediate frame corresponding to the second image frame, and determines the scene switching recognition result between the first image frame and the second image frame according to the difference between the first intermediate frame and the second intermediate frame. Among them, the first intermediate frame can represent the display status of the content in the first image frame at the target timestamp after motion estimation, and the second intermediate frame can represent the display status of the content in the second image frame at the target timestamp after motion estimation. The scene switching recognition result determined by the first intermediate frame and the second intermediate frame will not be misjudged due to the large displacement of the screen content of the first image frame and the second image frame, thereby improving the accuracy of scene switching recognition in the video screen.

[0188] It should be noted that the scene switching recognition device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the scene switching recognition device provided in the above embodiment and the scene switching recognition method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0189] Fig. 9 The schematic diagram of the structure of a server provided by an exemplary embodiment of the present application is shown, which specifically includes the following structure.

[0190] The server 900 includes a central processing unit (CPU) 901, a system memory 904 including a random access memory (RAM) 902 and a read only memory (ROM) 903, and a system bus 905 connecting the system memory 904 and the central processing unit 901. The server 900 also includes a mass storage device 906 for storing an operating system 913, application programs 914, and other program modules 915.

[0191] The mass storage device 906 is connected to the central processing unit 901 through a mass storage controller (not shown) connected to the system bus 905. The mass storage device 906 and its associated computer readable media provide non-volatile storage for the server 900. That is, the mass storage device 906 may include a computer readable medium (not shown) such as a hard disk or a compact disc read only memory (CD-ROM) drive.

[0192] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other solid-state storage technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that computer storage media are not limited to the above. The above-mentioned system memory 904 and mass storage device 906 can be collectively referred to as memory.

[0193] According to various embodiments of the present application, the server 900 can also be connected to a remote computer on the network through a network such as the Internet. That is, the server 900 can be connected to the network 912 through the network interface unit 911 connected to the system bus 905, or the network interface unit 911 can be used to connect to other types of networks or remote computer systems (not shown).

[0194] The above-mentioned memory also includes one or more programs, and the one or more programs are stored in the memory and configured to be executed by the CPU.

[0195] The embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein at least one instruction, at least one program, code set or instruction set is stored in the memory, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the biometric identification method provided by the above-mentioned method embodiments. Optionally, the computer device can be a terminal or a server.

[0196] An embodiment of the present application also provides a computer-readable storage medium, on which is stored at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the biometric recognition method provided by the above-mentioned method embodiments.

[0197] The embodiments of the present application also provide a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs any of the biometric identification methods described in the above embodiments.

[0198] Optionally, the computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid-state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0199] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0200] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for identifying scene switching, characterized in that: The method comprises: Acquire a first image frame and a second image frame, where the first image frame and the second image frame are two adjacent frames of images in a video to be identified, the first image frame corresponds to a first timestamp, and the second image frame corresponds to a second timestamp; Acquire first optical flow information of the first image frame between the first timestamp and a target timestamp, where the first optical flow information is used to indicate movement of pixels in the first image frame between the first timestamp and the target timestamp, and the target timestamp is a timestamp between the first timestamp and the second timestamp; Performing pixel position transformation on the first image frame based on the first optical flow information to obtain a first intermediate frame; Acquire second optical flow information of the second image frame between the second timestamp and the target timestamp, where the second optical flow information is used to indicate movement of pixels in the second image frame between the second timestamp and the target timestamp; Performing pixel position transformation on the second image frame based on the second optical flow information to obtain a second intermediate frame; Based on the difference between the first intermediate frame and the second intermediate frame, a scene switch recognition result between the first image frame and the second image frame is determined.

2. The method according to claim 1, characterized in that The performing pixel position transformation on the first image frame based on the first optical flow information to obtain the first intermediate frame includes: Acquire first coordinate change data of a pixel in the first image frame between the first timestamp and the target timestamp based on the first optical flow information; Displacing pixels in the first image frame based on the first coordinate change data to obtain the first intermediate frame; The performing pixel position transformation on the second image frame based on the second optical flow information to obtain the second intermediate frame includes: Acquire second coordinate change data of pixels in the second image frame between the second timestamp and the target timestamp based on the second optical flow information; The pixels in the second image frame are displaced based on the second coordinate change data to obtain the second intermediate frame.

3. The method according to claim 2, characterized in that The acquiring first coordinate change data of a pixel in the first image frame between the first timestamp and the target timestamp based on the first optical flow information includes: Converting the first optical flow information into a first motion speed of pixels in the first image frame; determining a first time interval between the first timestamp and the target timestamp; determining the first coordinate change data based on the first movement speed and the first time interval; The acquiring, based on the second optical flow information, second coordinate change data of a pixel in the second image frame between the second timestamp and the target timestamp comprises: Converting the second optical flow information into a second motion speed of pixels in the second image frame; determining a second time interval between the second timestamp and the target timestamp; The second coordinate change data is determined based on the second movement speed and the second time interval.

4. The method according to any one of claims 1 to 3, characterized in that: The determining, based on the difference between the first intermediate frame and the second intermediate frame, a scene switching recognition result between the first image frame and the second image frame comprises: Comparing the first intermediate frame with the second intermediate frame to determine difference information between the first intermediate frame and the second intermediate frame; In response to the difference information reaching a preset difference threshold, the scene switch recognition result between the first image frame and the second image frame is determined to be a first result, and the first result is used to indicate that the scene corresponding to the first image frame is different from the scene corresponding to the second image frame; or, in response to the difference information not reaching the preset difference threshold, the scene switch recognition result between the first image frame and the second image frame is determined to be a second result, and the second result is used to indicate that the scene corresponding to the first image frame is the same as the scene corresponding to the second image frame.

5. The method according to claim 4, characterized in that The comparing the first intermediate frame and the second intermediate frame to determine difference information between the first intermediate frame and the second intermediate frame includes: Determine a pixel difference between the first intermediate frame and the second intermediate frame for each pixel; An average value of pixel differences corresponding to all pixel points included in the first intermediate frame and the second intermediate frame is determined as the difference information.

6. The method according to claim 4, characterized in that The comparing the first intermediate frame and the second intermediate frame to determine difference information between the first intermediate frame and the second intermediate frame includes: Acquire a first histogram corresponding to the first intermediate frame and a second histogram corresponding to the second intermediate frame; A difference between the first histogram and the second histogram is determined as the difference information.

7. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Acquire a corresponding target interpolated frame image based on the scene switching recognition result; The target interpolated frame image is inserted into the target timestamp in the video to be identified to generate an interpolated frame video.

8. The method according to claim 7, characterized in that The scene switching recognition result includes a first result and a second result, the first result is used to indicate that the scene corresponding to the first image frame is different from the scene corresponding to the second image frame, and the second result is used to indicate that the scene corresponding to the first image frame is the same as the scene corresponding to the second image frame; The acquiring a corresponding target interpolated frame image based on the scene switching recognition result includes: In response to the scene switching recognition result being the first result, determining the first image frame or the second image frame as the target interpolation frame image; In response to the scene switching recognition result being the second result, the first intermediate frame and the second intermediate frame are fused to obtain the target interpolated frame image.

9. The method according to any one of claims 1 to 3, characterized in that: The acquiring of the first image frame and the second image frame comprises: Obtaining the video to be identified; Segmenting the video to be identified according to the number of original frames corresponding to the video to be identified to obtain an image frame sequence; The first image frame and the second image frame that are adjacent in time domain are determined from the image frame sequence.

10. The method according to claim 9, characterized in that The determining, from the image frame sequence, the first image frame and the second image frame that are adjacent in time domain comprises: Obtaining a target frame number requirement, where the target frame number requirement is used to indicate the frame number of the interpolated frame video generated according to the video to be identified; Determine a time period to be inserted in the image frame sequence based on the target frame number requirement, wherein the time period to be inserted is a time period in the video to be identified during which frame insertion is required, and the time period to be inserted includes two frames of images; Determine the corresponding screen images in the frame-to-be-inserted period as the first image frame and the second image frame; Determine a start time corresponding to the display duration of the first image frame in the frame-to-be-inserted period as the first timestamp; The starting time corresponding to the display duration of the second image frame in the video to be identified is determined as the second timestamp.

11. A scene switching recognition device, characterized in that: The device comprises: An acquisition module, used to acquire a first image frame and a second image frame, where the first image frame and the second image frame are two adjacent frames of images in a video to be identified, the first image frame corresponds to a first timestamp, and the second image frame corresponds to a second timestamp; A first acquisition unit, configured to acquire first optical flow information of the first image frame between the first timestamp and a target timestamp, wherein the first optical flow information is used to indicate a motion of a pixel in the first image frame between the first timestamp and the target timestamp, and the target timestamp is a timestamp between the first timestamp and the second timestamp; a generating unit, configured to transform pixel positions of the first image frame based on the first optical flow information to obtain a first intermediate frame; The first acquisition unit is further used to acquire second optical flow information of the second image frame between the second timestamp and the target timestamp, where the second optical flow information is used to indicate the movement of pixels in the second image frame between the second timestamp and the target timestamp; The generating unit is further configured to perform pixel position transformation on the second image frame based on the second optical flow information to obtain a second intermediate frame; A determination module is used to determine a scene switching recognition result between the first image frame and the second image frame based on a difference between the first intermediate frame and the second intermediate frame.

12. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the scene switching recognition method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one program code, and the program code is loaded and executed by a processor to implement the scene switching recognition method according to any one of claims 1 to 10.

14. A computer program product, comprising a computer program / instructions, characterized in that: The computer program / instructions are stored in a computer-readable storage medium, and a processor of a computer device reads the computer program / instructions from the computer-readable storage medium. The processor executes the computer program / instructions, so that the computer device executes to implement the scene switching recognition method as described in any one of claims 1 to 10.

Images