Video scene processing method, device, equipment and medium

Abstract

The present application discloses a video scene processing method, device, equipment and medium, the method comprising: obtaining a current video frame sequence from a video stream according to a preset sliding window; performing video lens recognition on the current video frame sequence; when recognizing that the current video frame sequence belongs to the same video lens, performing scene classification recognition on the current video frame sequence; when recognizing that the current video frame sequence contains different scene classification results, smoothing the current video frame sequence according to the scene classification results to obtain a smoothed result. The technical solution provided in the embodiment of the present application effectively solves the problem of visual jumps caused by color adjustment for different video scenes by recognizing different video scenes under the same video lens and then smoothing the video frames.

Description

Technical Field

[0001] The present application relates to the technical field of video image processing, and in particular to a method, device, equipment and medium for processing a video scene. Background Art

[0002] With the development of multimedia technology, a large amount of video data is growing at an explosive rate. Video scene segmentation is the basis for understanding and recognizing multimedia information.

[0003] When performing frame processing on the segmented video scene, the frame processing may not be ideal, causing the processed video to make viewers feel uncomfortable when playing. Summary of the invention

[0004] In view of the above-mentioned defects or deficiencies in the prior art, it is desirable to provide a video scene processing method, apparatus, device and medium to solve the above-mentioned problems caused by frame processing for video scenes.

[0005] On the one hand, an embodiment of the present application provides a video scene processing method, the method comprising:

[0006] Obtain the current video frame sequence from the video stream according to a preset sliding window;

[0007] Perform video shot recognition on the current video frame sequence;

[0008] When it is recognized that the current video frame sequence belongs to the same video shot, the current video frame sequence is subjected to scene classification and recognition;

[0009] When it is recognized that the current video frame sequence contains different scene classification results, the current video frame sequence is smoothed according to the scene classification results to obtain a smoothed result.

[0010] On the one hand, an embodiment of the present application provides a video scene processing device, the device comprising:

[0011] A video frame acquisition module is used to acquire a current video frame sequence from a video stream according to a preset sliding window length;

[0012] A video shot recognition module is used to perform video shot recognition on the current video frame sequence;

[0013] A scene classification and recognition module is used to classify and recognize the scene of the current video frame sequence when it is recognized that the current video frame sequence belongs to the same video shot;

[0014] The fusion processing module is used to perform smoothing processing on the current video frame sequence according to the scene classification results to obtain a smoothed result when recognizing that the current video frame sequence contains different scene classification results.

[0015] On the one hand, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the embodiment of the present application when executing the program.

[0016] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and the computer program is used to implement the method described in the embodiment of the present application.

[0017] The embodiments of the present application provide a video scene processing method, apparatus, device and medium. The method obtains a current video frame sequence from a video stream according to a preset sliding window; then, performs video shot recognition on the current video frame sequence; when it is recognized that the current video frame sequence belongs to the same video shot, the current video frame sequence is subjected to scene classification recognition; when it is recognized that the current video frame sequence contains different scene classification results, the current video frame sequence is smoothed according to the scene classification result to obtain a smoothed result. The technical solution provided by the embodiments of the present application obtains a video frame sequence through a sliding window, then performs shot recognition and video scene recognition on the video frame sequence, and when different video scenes belong to the same video shot, the video frame sequence is smoothed, which can effectively solve the problem of visual jumps caused by color adjustment for different video scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0019] Figure 1 A schematic diagram showing an application scenario of the video scene processing method proposed in an embodiment of the present application is shown;

[0020] Figure 2 A schematic diagram showing a flow chart of a video scene processing method proposed in an embodiment of the present application is shown;

[0021] Figure 3 A schematic diagram showing a flow chart of another video scene processing method provided in an embodiment of the present application is shown;

[0022] Figure 4 A schematic diagram of a process flow of another video scene processing method provided in an embodiment of the present application is shown;

[0023] Figure 5 A schematic diagram of extracting a video frame sequence using a sliding window provided in an embodiment of the present application is shown;

[0024] Figure 6 A schematic diagram showing a smoothing process performed by moving a sliding window according to a preset step length provided in an embodiment of the present application is shown;

[0025] Figure 7 A schematic diagram of the structure of a video scene processing device provided in an embodiment of the present application is shown;

[0026] Figure 8 A schematic diagram of the structure of a processing system of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant disclosure, rather than to limit the disclosure. It should also be noted that, for ease of description, only the parts related to the disclosure are shown in the accompanying drawings.

[0028] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0029] Please refer to Figure 1 , Figure 1 The following is a schematic diagram showing an application scenario of the video scene processing method provided by the embodiment of the present application. Figure 1 As shown, when the image acquisition device 1 acquires the video data, the video data is sent to the image processing device 3 through the network 2. The image processing device 3 performs frame processing on the video data. As the image acquisition device moves, the video data acquired by the image acquisition device will be color-adjusted according to the scene contained in the video data, but this color adjustment will cause the adjusted video data to have a visual jump problem.

[0030] The image acquisition device 1 includes but is not limited to various types of cameras, video cameras, cameras of mobile terminals, etc.

[0031] The network 2 may include a wireless network or a wired network, using standard communication technologies and / or protocols through the wireless network or the wired network. The network is typically the Internet, but may also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network.

[0032] Image processing equipment 3 includes but is not limited to electronic devices and servers. Electronic devices include but are not limited to desktop computers, laptops, or mobile terminals with strong image processing functions. Servers include but are not limited to independent physical servers, server clusters or distributed systems composed of multiple physical servers, cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0033] When performing image processing on video data, if the video data includes a change from an indoor scene to an outdoor scene, the image processing for the video data includes color adjustment for the frame image corresponding to the indoor scene and color adjustment for the frame image corresponding to the outdoor scene. The color adjustment processing for different scenes will cause color jumps. In particular, when different video scenes are switched under the same video lens, the color adjustment processing for different video scenes will make the visual jump problem more obvious.

[0034] "Color adjustment" refers to color enhancement processing of frame images in video data to make the color characteristics of the frame images more prominent, for example, brighter, darker, and more vivid.

[0035] In order to solve this problem, the present application proposes a video scene processing method, which is expected to solve the problem of visual jumps caused by switching between different video scenes under the same video lens.

[0036] Please refer to the following Figure 2 , Figure 2 FIG. 1 is a flow chart of a video scene processing method proposed in an embodiment of the present application. Figure 2 As shown, the method can be performed by a video scene processing device, and the video scene processing device can be pre-configured in an electronic device. The method includes:

[0037] Step 201: Obtain a current video frame sequence from a video stream according to a preset sliding window.

[0038] Step 202, performing video shot recognition on the current video frame sequence;

[0039] Step 203: When it is identified that the current video frame sequence belongs to the same video shot, scene classification and identification are performed on the current video frame sequence.

[0040] Step 204 , when it is identified that the current video frame sequence contains different scene classification results, the current video frame sequence is smoothed according to the scene classification results to obtain a smoothed result.

[0041] In the above steps, the preset sliding window refers to a window used to extract a sequence of video frames to be processed from a video stream. After the video stream is serialized, multiple video frames are extracted from the result of the serialization process by using the sliding window, such as Figure 5 As shown in FIG. 1 , assuming that the length of the sliding window is M and the step size is 1, M consecutive video frames {F0, F1, …, F M-1}.

[0042] Perform video shot recognition on the current video frame sequence. For example, it can be determined whether the video shot corresponding to the current video frame sequence has changed based on the similarity between two adjacent video frames in the current video frame sequence. The current video frame sequence includes multiple video frames. When the similarity indicates that two adjacent video frames in the current video frame sequence are similar, it is determined that the video shots corresponding to the current video frame sequence are the same; when the similarity indicates that two adjacent video frames in the current video frame sequence are not similar, it is determined that the video shots corresponding to the current video frame sequence are different. The similarity can be represented by using histogram pixel difference. Considering that the video shot boundary is a representation of the change in video content, when the image pixel difference between two adjacent frames changes very little, according to the histogram distribution diagram, it can be determined that the pixel difference in the low value area indicates that the video shot has not changed, and the pixel difference in the high value area indicates that the video shot has changed.

[0043] According to the similarity between two adjacent video frames in the current video frame sequence, determining whether a video shot corresponding to the current video frame sequence has changed may include the following steps:

[0044] Determine a histogram pixel difference value between two adjacent video frames in the current video frame sequence, where the histogram pixel difference value is a histogram difference value corresponding to a pixel difference value between the two adjacent video frames;

[0045] When the histogram pixel difference is less than a preset threshold, it is determined that the video shots corresponding to two adjacent video frames in the current video frame sequence are the same;

[0046] When the histogram pixel difference is greater than or equal to a preset threshold, it is determined that the video shots corresponding to two adjacent video frames in the current video frame sequence are different.

[0047] Optionally, determining whether a video shot corresponding to the current video frame sequence has changed according to the similarity between two adjacent video frames in the current video frame sequence includes:

[0048] Determine a histogram pixel difference between two adjacent video frames in the current video frame sequence;

[0049] Constructing a similarity curve according to the mapping relationship between the histogram pixel difference and the time parameter of the current video frame sequence;

[0050] Determine whether there is a stationary point on the similarity curve;

[0051] When it is determined that there is no stationary point in the similarity curve, it is determined that the video shots corresponding to the current video frame sequence are the same;

[0052] When it is determined that there is a stationary point in the similarity curve, it is determined that the video shots corresponding to the current video frame sequence are different.

[0053] When it is recognized that the current video frame sequence belongs to the same video shot, the current video frame sequence is further subjected to scene classification recognition. For example, each video frame in the multiple video frames corresponding to the sliding window can be input into a pre-built scene classification model, and the scene classification result corresponding to each video frame can be output. The scene classification model includes but is not limited to using a visual word bag model to classify the features of the video frame. Alternatively, for each video frame in the current video frame sequence, a plurality of image blocks are obtained by normalization processing, and then the contextual relationship words between adjacent image blocks are input into the word bag model for classification to obtain a scene classification result. Alternatively, each video frame in the current video frame sequence is classified and recognized using a KNN (full name K Nearest Neighbors) classification algorithm to obtain a scene classification result corresponding to each video frame. The scene classification result includes but is not limited to a scene type and a classification probability value corresponding to the scene type. In the embodiment of the present application, when scene recognition is performed on a video frame sequence, it can be determined whether the video frame sequence contains different scene classification results according to the scene types corresponding to two adjacent frames in the video frame sequence. For example, if it is recognized in the current video frame sequence that the scene types corresponding to two adjacent frames are different, it is determined that the current video frame sequence contains two scene types.

[0054] When it is recognized that the current video frame sequence contains different scene classification results, the current video frame sequence is smoothed according to the scene classification results to obtain a smoothed result. Smoothing refers to fusing the color parameters of the video frames in the current video frame sequence. Through smoothing, the color adjustment differences of the video frames of different scenes can be made consistent, thereby solving the visual jump problem caused by color adjustment for different scenes under the same lens.

[0055] Based on the above embodiments, Figure 3 As shown, Figure 3 A schematic flow chart of another video scene processing method provided by an embodiment of the present application is shown. The method can be executed by a video scene processing device, and the video scene processing device can be configured in an electronic device. The method includes:

[0056] Step 301, obtaining a current video frame sequence from a video stream according to a preset sliding window;

[0057] Step 302, performing video shot recognition on the current video frame sequence;

[0058] Step 303, when it is identified that the current video frame sequence belongs to the same video shot, scene classification and recognition are performed on the current video frame sequence to obtain the scene type and the classification probability value corresponding to the scene type;

[0059] Step 304, determining that the current video frame sequence includes at least two scene types, the at least two scene types including a first scene type and a second scene type;

[0060] Step 305, obtaining a first classification probability value corresponding to the first scene type and a second classification probability value corresponding to the second scene type;

[0061] Step 306, performing fusion processing on the target video frame in the current video frame sequence according to the first classification probability value and the second classification probability value to obtain a fusion processing result corresponding to the target video frame;

[0062] Step 307: Use the fusion processing result corresponding to the target video frame as the smoothing result.

[0063] In the above steps, after obtaining the current video frame sequence, video shot recognition is performed on the current video frame sequence. When it is recognized that the current video frame sequence belongs to the same video shot, each frame of the current video frame is input into the scene classification model to obtain the scene type corresponding to each video frame and the classification probability value corresponding to the scene type.

[0064] When smoothing the current video frame sequence according to the scene classification result, the following two methods may be included:

[0065] In the first method, the entire current video frame sequence is treated as a smoothing object. When different video scenes are identified under the same lens, each frame in the current video frame sequence is used as a target video frame. The target video frames in the current video frame sequence are fused according to the first classification probability value and the second classification probability value to obtain a smoothing result. In this way, when moving the sliding window, it moves along the time axis corresponding to the video stream according to the window length of the sliding window. The window length of the sliding window refers to the size of the sliding window. For example, Figure 6As shown, the window length of the sliding window is M, which means that M frames of video frames are read from the video stream. After obtaining the video frame sequence, each frame of the video frame sequence extracted from the video stream is subjected to fusion processing according to the first method. Then, the sliding window is moved according to the window length of the sliding window to obtain a new video frame sequence, and the new video frame sequence is subjected to shot recognition and video scene recognition. If the new video frame sequence belongs to the same video shot as the video frame sequence extracted last time, it is further determined whether the new video frame sequence and the video frame sequence extracted last time belong to the same video scene. If so, there is no need to perform fusion processing on the new video frame sequence, and if not, the new video frame sequence is subjected to fusion processing. If the new video frame sequence and the video frame sequence extracted last time belong to different video shots, there is no need to perform fusion processing on the new video frame sequence. By smoothing the video frame sequence as a whole, it is possible to effectively save the time of fusion processing when multiple video frames in the sliding window are first identified under the same video shot conditions and different video scenes exist, thereby improving the efficiency of fusion processing.

[0066] The second method is to take the first frame in the current video frame sequence as the object for smoothing. When different video scenes are identified under the same lens, the first frame in the current video frame sequence is taken as the target video frame. The target video frames in the current video frame sequence are fused according to the first classification probability value and the second classification probability value to obtain a smooth result. Move the sliding window in the second method and move along the time corresponding to the video stream according to the preset step length. The preset step length refers to the length of each video frame in the video stream. The preset step length includes but is not limited to 1 frame. Figure 6 As shown, the preset step size of the sliding window is 1 frame, M video frames are read from the video stream for the first time, and the starting frame is F0, and M video frames are still read from the video stream for the second time, and the starting frame is F1. The second method of fusion is to fuse the first frame of the video frame sequence extracted from the video stream each time. When it is determined that the current video frame sequence belongs to the same video lens condition and there are different video scenes, the sliding window is moved according to the step size of the sliding window, and the first frame of the video frame sequence in the sliding window moved each time is smoothed, which can effectively improve the consistency of the fusion processing results of the video frame sequences corresponding to adjacent sliding windows. Compared with the first method, its fusion processing result is more accurate and can better avoid the problem of visual jump.

[0067] The fusion processing of the target video frame in the current video frame sequence according to the first classification probability value and the second classification probability value may include:

[0068] Obtain the parameters to be fused corresponding to the target video frame in the current video frame sequence;

[0069] The parameters to be fused are fused with the first classification probability value and the second classification probability value respectively to obtain a fusion processing result corresponding to the target video frame.

[0070] The above parameters to be fused include but are not limited to color adjustment parameters of the video frame, and the color adjustment parameters include but are not limited to image grayscale values.

[0071] For example, the parameters to be fused of the target video frame can be fused using the following fusion processing formula:

[0072]

[0073] Among them I 融合 Indicates the fusion processing result of the target video frame;

[0074] I i Represents the image grayscale value of the i-th target video frame;

[0075] T scenek (w i，k ) represents the classification probability value of the i-th target video frame being classified as scene k;

[0076] T scenej (w i，j ) represents the classification probability value of the i-th target video frame being classified as scene j.

[0077] In the embodiment of the present application, the color adjustment parameters of different scenes are made consistent by fusing the parameters to be fused under the same video lens, thereby achieving a smooth transition of the color adjustment results between different scenes and effectively solving the problem of visual jumps.

[0078] Based on the above embodiments, Figure 4 As shown, Figure 4 A schematic flow chart of another video scene processing method provided by an embodiment of the present application is shown. The method can be executed by a video scene processing device, and the video scene processing device can be configured in an electronic device. The method includes:

[0079] Step 401, obtaining a current video frame sequence from a video stream according to a preset sliding window;

[0080] Step 402, performing video shot recognition on the current video frame sequence;

[0081] Step 403: When it is identified that the current video frame sequence belongs to the same video shot, scene classification and identification are performed on the current video frame sequence.

[0082] Step 404: when it is identified that the current video frame sequence does not belong to the same video shot, the current video frame sequence is not smoothed, but the current video frame sequence needs to be color adjusted. The color adjustment parameter is the grayscale value of the image that has not been smoothed.

[0083] Step 405 , when it is identified that the current video frame sequence belongs to the same video shot and it is identified that the current video frame sequence contains different scene classification results, the current video frame sequence is smoothed according to the scene classification result to obtain a smoothed result.

[0084] Step 406: When it is identified that the current video frame sequence belongs to the same video shot and the current video frame sequence contains the same scene classification result, the current video frame sequence is not smoothed. However, the current video frame sequence needs to be color adjusted. The parameters used for the color adjustment are the grayscale values ​​of the image that have not been smoothed.

[0085] Step 407, moving the sliding window along the time axis corresponding to the video stream according to the preset step size of the sliding window;

[0086] Step 408, obtaining an updated video frame sequence corresponding to the moved sliding window;

[0087] Step 409, updating the current video frame sequence using the updated video frame sequence;

[0088] Return to step 402, continue to perform recognition of the current video frame sequence, and then when it is recognized that the updated video frame sequence and the current video frame sequence belong to the same video shot, continue to perform scene classification recognition on the updated video frame sequence, and when it is recognized that the updated video frame sequence contains different scene classification results, smooth the updated video frame sequence according to the scene classification results.

[0089] In the above steps, the sliding window is moved by a preset step size of the sliding window. After the fusion processing of the current video frame sequence extracted by the sliding window is completed, the updated current video frame sequence extracted after moving the sliding window is fused again until the fusion processing of the previous frame of two adjacent frames corresponding to different scene classification results is completed, which can effectively save the fusion processing time and obtain better fusion processing efficiency.

[0090] Below Figure 6 For example, assuming that the sliding window length is M and the preset sliding step is 1 frame, the first video frame sequence {F0, F1, ..., F M-1}, when it is recognized that the first video frame sequence belongs to the same video shot and the first video frame sequence belongs to the same video scene, no processing is performed on the first video frame sequence; the sliding window is continued to be moved according to the preset sliding step length to obtain the second video frame sequence {F1, F2, ..., F M}, when it is recognized that the second video frame sequence belongs to the same video shot and the second video frame sequence includes different video scenes, the first frame F1 of the second video frame sequence is fused according to the fusion processing formula to obtain a fusion processing result

[0091]

[0092] Then, continue to move the sliding window according to the preset sliding step length to obtain the third video frame sequence {F2, F3, ..., F M+1}, the same processing can be used to identify that the third video frame sequence belongs to the same video shot, and when the third video frame sequence includes different video scenes, the first frame F2 of the third video frame sequence is fused according to the fusion processing formula to obtain the fusion processing result

[0093] Repeatedly move the sliding window according to the preset sliding step size until the M+1th video frame sequence {F M+1 ,F M+2 ,……,F 2M}, after the same video shot recognition process, it is recognized that the third video frame sequence belongs to the same video shot, and the M+1th video frame sequence belongs to the same video scene, then the M+1th video frame sequence {F M+1 ,F M+2 ,……,F 2M}.

[0094] The embodiment of the present application performs smoothing processing frame by frame from the video stream through the above-mentioned sliding window, which can effectively ensure the consistency of color adjustment parameters of video frames in different video scenes, thereby solving the visual jump problem caused by color adjustment for different video scenes in the related art.

[0095] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a different order from the order marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0096] Please refer to the following Figure 7 , Figure 7 The schematic diagram of the structure of the video scene processing device provided by the embodiment of the present application is shown. The device can be configured with a server, pre-downloaded from the server to an electronic device for use, or pre-installed in an electronic device. The device includes:

[0097] The video frame acquisition module 701 is used to acquire the current video frame sequence from the video stream according to a preset sliding window length;

[0098] A video shot recognition module 702 is used to perform video shot recognition on a current video frame sequence;

[0099] The scene classification and recognition module 703 is used to classify and recognize the scene of the current video frame sequence when it is recognized that the current video frame sequence belongs to the same video shot;

[0100] The fusion processing module 704 is used to perform smoothing processing on the current video frame sequence according to the scene classification results to obtain a smoothed result when it is identified that the current video frame sequence contains different scene classification results.

[0101] Optionally, the video frame sequence includes a plurality of video frames corresponding to the sliding window, and the scene classification and recognition module is further used for:

[0102] Each video frame in the multiple video frames is input into a pre-built scene classification model, and the scene classification result corresponding to each video frame is output.

[0103] Optionally, the scene classification result includes a scene type and a classification probability value corresponding to the scene type, and the fusion processing module is further configured to include:

[0104] Determining that the current video frame sequence contains at least two scene types, the at least two scene types comprising a first scene type corresponding to a first frame of two adjacent frames in the current video frame sequence and a second scene type corresponding to a second frame of the two adjacent frames;

[0105] Obtain a first classification probability value corresponding to the first scene type and a second classification probability value corresponding to the second scene type;

[0106] Performing fusion processing on a target video frame in the current video frame sequence according to the first classification probability value and the second classification probability value to obtain a fusion processing result corresponding to the target video frame, where the target video frame is the first frame in the current video frame sequence;

[0107] The fusion processing result corresponding to the target video frame is used as the smoothing result.

[0108] The device also includes:

[0109] A first sliding window moving module, used for moving the sliding window along the time axis corresponding to the video stream according to a preset step length of the sliding window, wherein the preset step length refers to the length of each video frame in the video stream;

[0110] The video frame acquisition module is used to acquire an updated video frame sequence corresponding to the moved sliding window;

[0111] A video shot recognition module, used for performing video shot recognition on the updated video frame sequence;

[0112] A scene classification and recognition module, configured to perform scene classification and recognition on the updated video frame sequence when recognizing that the updated video frame sequence belongs to the same video shot;

[0113] A fusion processing module, configured to perform smoothing processing on the first frame of the updated video frame sequence when it is identified that the updated video frame sequence contains different scene classification results;

[0114] The loop control module is used to return to the step of moving the sliding window until the last video frame of the video stream.

[0115] The scene classification result includes the scene type and the classification probability value corresponding to the scene type, and the fusion processing module is further used for:

[0116] Determining that the current video frame sequence contains at least two scene types, the at least two scene types comprising a first scene type corresponding to a first frame of two adjacent frames in the current video frame sequence and a second scene type corresponding to a second frame of the two adjacent frames;

[0117] Obtain a first classification probability value corresponding to the first scene type and a second classification probability value corresponding to the second scene type;

[0118] Performing fusion processing on a target video frame in the current video frame sequence according to the first classification probability value and the second classification probability value to obtain a fusion processing result corresponding to the target video frame, where the target video frame is each frame in the current video frame sequence;

[0119] The fusion processing result corresponding to the target video frame is used as the smoothing result.

[0120] The device also includes:

[0121] A second sliding window moving module, used for moving the sliding window along the time axis corresponding to the video stream according to the window length of the sliding window, where the window length refers to the size of the sliding window;

[0122] A video frame acquisition module, used to acquire an updated video frame sequence corresponding to the moved sliding window;

[0123] A video shot recognition module, used for performing video shot recognition on the updated video frame sequence;

[0124] A scene classification and recognition module, configured to perform scene classification and recognition on the updated video frame sequence when recognizing that the updated video frame sequence belongs to the same video shot;

[0125] A fusion processing module, configured to perform smoothing processing on each frame of the updated video frame sequence when it is identified that the updated video frame sequence contains different scene classification results;

[0126] The loop control module is used to return to the step of moving the sliding window until the last video frame of the video stream.

[0127] The fusion processing module is also used to:

[0128] Obtain the parameters to be fused corresponding to the target video frame in the current video frame sequence;

[0129] The parameters to be fused are fused with the first classification probability value and the second classification probability value respectively to obtain a fusion processing result corresponding to the target video frame.

[0130] Optionally, the video shot recognition module is further used for:

[0131] Determine whether there is a change in the video shot corresponding to the current video frame sequence according to the similarity between two adjacent video frames in the current video frame sequence;

[0132] When the similarity indicates that two adjacent video frames in the current video frame sequence are similar, it is determined that the video shots corresponding to the current video frame sequence are the same;

[0133] When the similarity indicates that two adjacent video frames in the current video frame sequence are not similar, it is determined that the video shots corresponding to the current video frame sequence are different.

[0134] The embodiment of the present application performs smoothing processing frame by frame from the video stream through the above-mentioned sliding window, which can effectively ensure the consistency of color adjustment parameters of video frames in different video scenes, thereby solving the visual jump problem caused by color adjustment for different video scenes in the related art.

[0135] It should be understood that the units or modules described in the above device are similar to those described in the reference Figure 1 The various steps in the described method correspond to each other. Therefore, the operations and features described above for the method are also applicable to the above-mentioned device and the units contained therein, and will not be repeated here. The above-mentioned device can be pre-implemented in the browser or other security items of the electronic device, and can also be loaded into the browser or its security item of the electronic device by downloading or the like. The corresponding units in the above-mentioned device can cooperate with the units in the electronic device to implement the solution of the embodiment of the present application.

[0136] For the several modules or units mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0137] Reference below Figure 8 , Figure 8 A schematic diagram of the structure of a processing system of an electronic device provided in an embodiment of the present application is shown.

[0138] like Figure 8 As shown, the computer system includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for operation are also stored. The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0139] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0140] In particular, according to an embodiment of the present disclosure, the above reference flow chart Figure 1-2 The described process can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 509, and / or installed from a removable medium 511. When the computer program is executed by a central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present application are executed.

[0141] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0142] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of the code, and the aforementioned module, program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0143] The units or modules involved in the embodiments described in the present application may be implemented by software or hardware. The units or modules described may also be set in a processor. For example, they may be described as: a processor including a video frame acquisition module, a video shot recognition module, a scene classification recognition module, and a fusion processing module. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves. For example, the video frame acquisition module may also be described as "a module for acquiring a current video frame sequence from a video stream according to a preset sliding window length".

[0144] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable storage medium stores one or more programs, and when the above programs are used by one or more processors to execute the method for video scene processing described in the present application.

[0145] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the aforementioned disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A video scene processing method, characterized in that: The method includes: Obtain the current video frame sequence from the video stream according to a preset sliding window; Performing video shot recognition on the current video frame sequence; When it is recognized that the current video frame sequence belongs to the same video shot, performing scene classification and recognition on the current video frame sequence; When it is identified that the current video frame sequence includes different scene classification results, smoothing the current video frame sequence according to the scene classification results to obtain a smoothed result; Wherein, the scene classification result includes a scene type and a classification probability value corresponding to the scene type, and the smoothing process of the current video frame sequence according to the scene classification result includes: Determining that the current video frame sequence contains at least two scene types, the at least two scene types comprising a first scene type corresponding to a first frame of two adjacent frames in the current video frame sequence and a second scene type corresponding to a second frame of the two adjacent frames; Obtaining a first classification probability value corresponding to the first scene type and a second classification probability value corresponding to the second scene type; Performing fusion processing on the target video frame in the current video frame sequence according to the first classification probability value and the second classification probability value to obtain a fusion processing result corresponding to the target video frame When the target video frame is the first frame in the current video frame sequence, taking the fusion processing result corresponding to the target video frame as the smoothing result; Alternatively, when the target video frame is each frame in the current video frame sequence, the fusion processing result is used as the smoothing result.

2. The method according to claim 1, characterized in that The video frame sequence includes a plurality of video frames corresponding to the sliding window, and the performing scene classification and recognition on the current video frame sequence includes: Each video frame in the multiple video frames is input into a pre-built scene classification model, and a scene classification result corresponding to each video frame is output.

3. The method according to claim 1, characterized in that When the target video frame is the first frame in the current video frame sequence, after obtaining the smoothing result, the method further includes: Moving the sliding window along a time axis corresponding to the video stream according to a preset step length of the sliding window, wherein the preset step length refers to the length of each video frame in the video stream; Acquire an updated video frame sequence corresponding to the moved sliding window; Performing video shot recognition on the updated video frame sequence; When it is recognized that the updated video frame sequence belongs to the same video shot, performing scene classification and recognition on the updated video frame sequence; When it is identified that the updated video frame sequence contains different scene classification results, smoothing the first frame of the updated video frame sequence; Return to the above step of moving the sliding window until the last video frame of the video stream.

4. The method according to claim 1, characterized in that: When the target video frame is each frame in the current video frame sequence, after obtaining the smoothing result, the method further includes: Moving the sliding window along a time axis corresponding to the video stream according to a window length of the sliding window, wherein the window length refers to a size of the sliding window; Acquire an updated video frame sequence corresponding to the moved sliding window; Performing video shot recognition on the updated video frame sequence; When it is recognized that the updated video frame sequence belongs to the same video shot, performing scene classification and recognition on the updated video frame sequence; When it is identified that the updated video frame sequence includes different scene classification results, smoothing each frame of the updated video frame sequence; Return to the above step of moving the sliding window until the last video frame of the video stream.

5. The method according to claim 1, characterized in that The performing fusion processing on the target video frame in the current video frame sequence according to the first classification probability value and the second classification probability value includes: Obtaining parameters to be fused corresponding to a target video frame in the current video frame sequence; The parameters to be fused are fused with the first classification probability value and the second classification probability value respectively to obtain a fusion processing result corresponding to the target video frame.

6. The method according to claim 1, characterized in that The performing video shot recognition on the current video frame sequence comprises: Determining whether a video shot corresponding to the current video frame sequence has changed according to a similarity between two adjacent video frames in the current video frame sequence; When the similarity indicates that two adjacent video frames in the current video frame sequence are similar, determining that the video shots corresponding to the current video frame sequence are the same; When the similarity indicates that two adjacent video frames in the current video frame sequence are not similar, it is determined that the video shots corresponding to the current video frame sequence are different.

7. The method according to claim 1, characterized in that The method further includes: When it is identified that the current video frame sequence belongs to different video shots, no smoothing is performed on the current video frame sequence; or, When it is identified that the current video frame sequence belongs to the same video shot and the current video frame sequence belongs to the same scene classification result, no smoothing process is performed on the current video frame sequence.

8. A video scene processing device, characterized in that: The device includes: A video frame acquisition module is used to acquire a current video frame sequence from a video stream according to a preset sliding window length; A video shot recognition module, used for performing video shot recognition on the current video frame sequence; A scene classification and recognition module, configured to perform scene classification and recognition on the current video frame sequence when recognizing that the current video frame sequence belongs to the same video shot; A fusion processing module, configured to, when recognizing that the current video frame sequence contains different scene classification results, perform smoothing processing on the current video frame sequence according to the scene classification results to obtain a smoothing result; Wherein, the scene classification result includes a scene type and a classification probability value corresponding to the scene type, and the smoothing process of the current video frame sequence according to the scene classification result includes: Determining that the current video frame sequence contains at least two scene types, the at least two scene types comprising a first scene type corresponding to a first frame of two adjacent frames in the current video frame sequence and a second scene type corresponding to a second frame of the two adjacent frames; Obtaining a first classification probability value corresponding to the first scene type and a second classification probability value corresponding to the second scene type; Performing fusion processing on the target video frame in the current video frame sequence according to the first classification probability value and the second classification probability value to obtain a fusion processing result corresponding to the target video frame When the target video frame is the first frame in the current video frame sequence, taking the fusion processing result corresponding to the target video frame as the smoothing result; Alternatively, when the target video frame is each frame in the current video frame sequence, the fusion processing result is used as the smoothing result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images