Video processing method, device, equipment, readable storage medium and product
By detecting and comparing the video images, combining the confidence and position of the object, the problem of low accuracy of video stutter detection in the prior art is solved, and more efficient video stutter detection is achieved.
Patent Information
- Application Number
- CN202311271385.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-28
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2043-09-28
AI Technical Summary
The existing video stutter detection methods have low accuracy, mainly because they rely on calculating the pixel value consistency of video images in adjacent frames, and it is impossible to accurately judge video stuttering.
By detecting the object on multi-frame video images, the confidence and position of the object are obtained, and the confidence and position of the object in adjacent frame video images are compared to determine the existence of video stuttering.
It improves the accuracy of video stutter detection, and can more accurately judge the object changes in the video image, thereby handling video stuttering in a timely manner and improving user experience.
Smart Images

Figure CN117440180B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of video processing technology, and in particular to a video processing method, device, equipment, readable storage medium and product. Background Art
[0002] Video freeze is prone to occur during video playback, and video freeze will seriously affect the user experience. Therefore, it is necessary to detect the video freeze during video playback in a timely manner so as to deal with it in a timely manner. The current video freeze detection method is generally to calculate the pixel values of two adjacent frames of video images, and to determine whether there is video freeze by calculating whether the pixel values of the two adjacent frames of video images are consistent. However, when using the pixel values of two adjacent frames of video images for comparison, the histograms of the two frames of video images are generally calculated separately, and the histograms of the two frames of video images are compared to determine whether there is video freeze. However, the histogram reflects the probability distribution of the grayscale values of the image pixels, such as how many pixels have a grayscale value of 100. This method can only roughly determine whether the two frames of video images are consistent, resulting in low accuracy in video freeze detection. Summary of the invention
[0003] The embodiments of the present application provide a video processing method, apparatus, device, readable storage medium and product, which can improve the accuracy of video freeze detection.
[0004] In a first aspect, the present application provides a video processing method, comprising:
[0005] Acquire video data to be processed, where the video data to be processed includes a plurality of continuous frames of video images;
[0006] Performing target detection on the multiple frames of video images to obtain the confidence and position of the objects in the multiple frames of video images;
[0007] Comparing the confidences and positions of objects in adjacent frames of video images in the multiple frames of video images to obtain a comparison result, where the comparison result is used to indicate whether the adjacent frames of video images are consistent;
[0008] Based on the comparison result, it is determined whether the video data to be processed has video freeze.
[0009] In a second aspect, the present application provides a video processing device, including:
[0010] A video acquisition unit, used to acquire video data to be processed, where the video data to be processed includes a plurality of continuous frames of video images;
[0011] An image detection unit, used to perform target detection on the multiple frames of video images to obtain the confidence and position of the object in the multiple frames of video images;
[0012] An image comparison unit, used for comparing the confidence and position of objects in adjacent frames of video images in the multiple frames of video images to obtain a comparison result, wherein the comparison result is used to indicate whether the adjacent frames of video images are consistent;
[0013] A result determination unit is used to determine whether the video data to be processed has video freeze based on the comparison result.
[0014] In a third aspect, the present application provides a computer device, including: a processor, a memory, and a network interface;
[0015] The processor is connected to a memory and a network interface, wherein the network interface is used to provide a data communication function, the memory is used to store computer program codes, and the processor is used to call the computer program codes so that a computer device including the processor executes the video processing method.
[0016] In a fourth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. The computer program is suitable for being loaded and executed by a processor so that a computer device having the processor executes the above-mentioned video processing method.
[0017] In a fifth aspect, the present application provides a computer program product or a computer program, which includes computer instructions. When the computer instructions are executed by a processor, the video processing method provided in various optional methods in the first aspect of the present application is implemented.
[0018] In an embodiment of the present application, by acquiring video data to be processed including multiple consecutive frames of video images, target detection is performed on the multiple frames of video images to obtain the confidence and position of the object in the multiple frames of video images, so that the confidence and position of the object in the adjacent frames of video images in the multiple frames of video images can be compared to obtain a comparison result indicating whether the adjacent frames of video images are consistent, and then it can be determined based on the comparison result whether there is video freeze in the video data to be processed. By comparing the confidence and position of the object in the adjacent frames of video images, it can be determined whether there is a change in the object in the adjacent frames of video images, so that it can be determined whether there is a change in the adjacent frames of video images, and then it can be determined whether there is video freeze in the video data to be processed. Since the confidence and position of the object in the video image are combined to determine whether the object in the video image has changed, the accuracy of video detection can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0020] Figure 1 is a network architecture diagram of a video processing system provided in an embodiment of the present application;
[0021] Figure 2 It is a flowchart of a video processing method provided by an embodiment of the present application;
[0022] Figure 3 is a schematic diagram of a video stagnation provided by an embodiment of the present application;
[0023] Figure 4 It is a schematic diagram of a video that is not stagnant provided by an embodiment of the present application;
[0024] Figure 5 is a flowchart of another video processing method provided in an embodiment of the present application;
[0025] Figure 6 is a schematic diagram of the composition structure of a video processing device provided in an embodiment of the present application;
[0026] Figure 7 It is a schematic diagram of the composition structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0028] The technical solution of the present application can be applied to the scene of detecting video jams. For example, it can be applied to a video playback platform, by detecting the played video data, determining whether the video data has video jams, so that the video jam information can be reported in time, and then the video data can be processed to avoid the video data being played unsmoothly due to video jams, thereby reducing the user experience. The technical solution of the embodiment of the present application can also be applied to any scene that needs to play video data, for example, it can include but is not limited to video playback platforms, video sharing platforms, video editing platforms, etc., and the embodiment of the present application does not limit this. The collection, use and processing of data such as to-be-processed video data and sample video data involved in the embodiment of the present application require the consent of the corresponding personal information subject, and need to comply with the provisions of relevant laws and regulations.
[0029] Please refer to Figure 1 , Figure 1 is a network architecture diagram of a video processing system provided in an embodiment of the present application, such as Figure 1 As shown, the computer device can exchange data with the terminal device, and the number of the terminal devices can be one or at least two. For example, when the number of the terminal devices is multiple, the terminal device may include Figure 1 The terminal device 101a, the terminal device 101b and the terminal device 101c, etc. Among them, taking the terminal device 101a as an example, the computer device 102 can obtain the video data to be processed. Further, the computer device 102 can perform target detection on the multiple frames of video images in the video data to be processed, and obtain the confidence and position of the object in the multiple frames of video images. Further, the computer device 102 can compare the confidence and position of the object in the adjacent frames of video images in the multiple frames of video images to obtain the comparison result, so as to determine whether there is video jamming in the video data to be processed based on the comparison result. Optionally, the computer device 102 can also generate jamming information for the video data to be processed, and report the jamming information for the video data to be processed to the terminal device 101a, so as to prompt relevant personnel to carry out targeted processing on the jamming of the video data to be processed, thereby improving the user experience.
[0030] The computer device mentioned in the embodiments of the present application may refer to a server or a terminal device, or may also be a system composed of a server and a terminal device, which is not limited in the embodiments of the present application. The terminal device may be an electronic device, including but not limited to a mobile phone, a tablet computer, a laptop computer, a vehicle-mounted device, etc. The server mentioned above may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, and big data and artificial intelligence platforms.
[0031] For further information, see Figure 2 , Figure 2 is a flow chart of a video processing method provided by an embodiment of the present application; Figure 2 As shown, the video processing method can be applied to a computer device, and the video processing method includes but is not limited to the following steps:
[0032] S101, obtaining video data to be processed.
[0033] In the embodiment of the present application, the video data to be processed can be obtained by recording with an associated camera device, or obtained from a local database, or can be obtained by uploading with a terminal device. For example, the terminal device can obtain video data by recording any person, animal, landscape, or any element of nature, or the terminal device can obtain video data by recording the video data played on the terminal device by triggering a screen recording operation, or the terminal device can obtain video data composed of multiple video images by continuously capturing the display content on the display interface of the current terminal device by triggering a screenshot operation, etc., which are not limited in the embodiment of the present application.
[0034] The video data to be processed includes multiple continuous video frames. For example, when the video data to be processed is a video, the video data to be processed can be sampled to obtain multiple continuous video frames. Or when the video data to be processed is video data composed of multiple video images obtained by continuous multiple screenshots of a terminal device, each video image can be directly used as each frame of the video image, or the video data to be processed can be sampled to obtain multiple continuous video frames, and so on.
[0035] In some possible scenarios, such as short video scenarios, the video data to be processed is short video data. In short video scenarios, there are situations that affect video jam detection, such as using one image as the content of the entire video. Therefore, the content of the entire video may be basically the same, all of which are the same image, which will result in low accuracy of video jam detection. Or in short video scenarios, there may be a situation where a small number of images such as two images are used as the content of the entire video, so the entire video may be one image for the first few seconds and another image for the last few seconds, which will result in low accuracy of video jam detection. In the embodiment of the present application, when comparing the confidence and position of objects in adjacent frame video images in the video data to be processed, the progress bar can be used as an object in the video image for confidence and position comparison to determine whether the progress bar changes. Since only one image or a small number of images are used as the content of the entire video in the entire video, but the progress bar during video playback is constantly changing, it is possible to determine whether the entire video data is being played based on whether the progress bar changes, thereby determining whether the player is abnormal, etc., and the accuracy of video jam detection can be improved.
[0036] In other possible scenarios, such as live broadcast scenarios, the host's camera may be aimed at a stationary object for a long time, resulting in a situation that affects the video freeze detection result. Through the method in the embodiment of the present application, the likes, comments, favorites, number of viewers and other identifications in the live broadcast scene can be used as objects for video comparison. By detecting whether the confidence and position of these elements in adjacent frame video images have changed, it is possible to determine whether the video is stagnant, such as the live broadcast content stagnating, or the video freezes, such as the terminal device rendering freezes, when freezes occur.
[0037] In an embodiment of the present application, by acquiring multiple consecutive frames of video images, the multiple consecutive frames of video images can be subsequently compared to determine whether objects in the multiple consecutive frames of video images have changed, thereby determining whether there is video freeze in the video data to be processed, which can improve the accuracy of video freeze detection.
[0038] S102, performing target detection on multiple video frames to obtain confidence levels and positions of objects in the multiple video frames.
[0039] In an embodiment of the present application, by performing target detection on multiple frames of video images, the confidence and position of the object in the multiple frames of video images can be obtained, and the confidence and position of the continuous multiple frames of video images can be compared to determine whether there is video jamming. Optionally, the object in the video image in the video data to be processed includes: at least one of the image composition elements and image additional elements in the video image, the image composition elements are image screen elements in the video image, and the image additional elements include at least one of the effect screen elements and progress bars corresponding to the image display effects in the video data to be processed. For example, the image composition elements may include but are not limited to people, animals, plants, electronic products, food, natural landscapes, etc. in the video image. For example, the image additional elements may include but are not limited to likes, comments, number of viewers, barrage information, gift special effects, collections, forwarding and other identifications in the video image, and may also include a progress bar for indicating the current playback progress of the video data to be processed.
[0040] That is to say, the object may include any element of nature, for example, it may include but not limited to physical elements such as people, animals, plants, electronic products, food, natural landscapes, etc., and may also include virtual elements such as image additional elements. The confidence can be used to reflect the possibility of the video image containing the object and the accuracy of predicting the video image containing the object. The greater the confidence of the object, the greater the possibility of the video image containing the object and the greater the accuracy of predicting the video image containing the object. The smaller the confidence of the object, the smaller the possibility of the video image containing the object and the smaller the accuracy of predicting the video image containing the object. Further, the category prediction of each object in the video image can be further performed to determine the object category to which each object in the video image belongs. For example, the object category may include but not limited to people, animals, plants, electronic products, food, natural landscapes, image additional elements, etc. The object category can be further subdivided, for example, the object category of people can be further subdivided into babies, children, the elderly, etc. The object category of animals can be further subdivided into cats, dogs, rabbits, sheep, monkeys, etc. The object category of plants can be further subdivided into flowers, grass, trees, etc. The object category of electronic products can be further subdivided into mobile phones, computers, cameras, voice recorders, etc. The object category of food can be further subdivided into rice, pasta, bread, cake, etc. The object category of natural landscape can be further subdivided into clouds, sky, waterfall, etc.
[0041] Among them, the position of the object in the multi-frame video image may refer to, for example, the center position of the object in the video image, or the center position and width and height of the object in the video image, or the position number of the object in the video image, etc., which is not limited in the embodiments of the present application. The center position of the object in the video image may refer to the coordinate position of the center point of the object mapped on the rectangular coordinate system by establishing a rectangular coordinate system with any point in the video image as the coordinate origin. The center position of the object in the video image can be represented by coordinates. For example, the position of the object in the video image can be selected by a rectangular frame, and the center point coordinates of the rectangular frame are used as the position of the object. Alternatively, multiple edge coordinate points of the object in the video image can be determined, and the average of the multiple edge coordinate points is obtained to obtain the center position coordinates of the object in the video image. The edge coordinate point may refer to the vertex of the object in the video image. For example, when the object is a rectangle, the edge coordinate point may refer to the four vertex coordinates of the object, and the center position of the object in the video image may be the average of the four vertex coordinates of the object.
[0042] In an optional implementation, the object in the multi-frame video image may refer to multiple objects in the video image, or to a specific object in the video image. For example, in order to reduce the amount of calculation, a specific object can be pre-selected, and then the confidence and position of the specific object in the continuous multi-frame video image can be compared to determine whether there is video jamming in the video data to be processed. Since only the confidence and position of the specific object need to be determined, the amount of calculation can be reduced and the efficiency of video jamming detection can be improved. Alternatively, in the scenario of improving detection accuracy, the confidence and position of multiple objects in the continuous multi-frame video image can be compared to determine whether there is video jamming in the video data to be processed. Since the confidence and position of multiple objects in the video image are determined, the accuracy of video jamming detection can be improved. In actual application scenarios, the corresponding processing method can be selected in combination with the video processing requirements to meet the corresponding video processing requirements.
[0043] Optionally, in the embodiment of the present application, a target image detection model can be used to perform target detection on multiple video frames to obtain the confidence and position of the object in the multiple video frames. Among them, the target image detection model can include, but is not limited to, a regression target detection model (You Only Look Once, YOLO), a single shot multi-box detection model (Single Shot MultiBox Detector, SSD), a convolutional neural network model (Convolutional Neural Network, CNN), a recurrent neural network model (Recurrent neural network, RNN), a deep neural network model (Deep Neural Networks, DNN), and the like.
[0044] In a specific implementation, by inputting a video image into a target image detection model, the target image detection model is used to detect the target of the video image. Each detected object in the video image can be marked with a bounding box, which can be used to indicate the position of the object in the video image and the width and height of the object in the video image. At the same time, each detected object's bounding box is also marked with a confidence, which is used to indicate the credibility of the object detected by the target image detection model for the bounding box. Among them, the confidence can refer to the product between the probability that the target image detection model predicts that the bounding box in the video image contains the target and the accuracy of the prediction box. The bounding box can be used to mark the object in the video image. The prediction box can be a plurality of small grids obtained by dividing the video image. An object in the video image can correspond to a plurality of prediction boxes, and each prediction box contains a different part of the object, for example, one prediction box contains the head of the object, and one prediction box contains the body part of the object, and so on. For example, the bounding box can refer to the box with the highest probability of containing the object in the prediction box, or the box with the most object content in the prediction box. The target in the bounding box may refer to an object, that is, the confidence of the object may refer to the product between the probability of predicting that the bounding box in the video image contains the object and the accuracy of the bounding box. The greater the confidence of the object, the greater the product between the probability of predicting that the bounding box in the video image contains the object and the accuracy of the bounding box. The smaller the confidence of the object, the smaller the product between the probability of predicting that the bounding box in the video image contains the object and the accuracy of the bounding box. The confidence can be used to indicate the proportion of the overlapped partial intersection between the size and position of the real target believed by the target image detection model and the size and position of the target predicted by the target image detection model. The larger this proportion is, the more accurate the size and position of the object predicted by the target image detection model is. Furthermore, when the confidence of each bounding box in the video image is detected, the target image detection model can further perform category prediction on the object in each bounding box to determine the object category to which the object in each bounding box in the video image belongs, and then the confidence and object category can be combined to determine whether the two images are consistent.
[0045] In one embodiment, for example, the confidence and position of an object in multiple frames of video images include the confidence and position of any object in any frame of video images. Taking the confidence and position of any object in any frame of video images as an example, the confidence and position of any object in any frame of video images can be obtained in the following manner:
[0046] A target image detection model is used to perform target detection on any object in any frame of video image to obtain an image detection result of any object in any frame of video image. The image detection result is used to indicate the probabilities of multiple prediction boxes associated with any object in any frame of image and the position of any object in any frame of image. The probabilities of multiple prediction boxes associated with any object are used to reflect the credibility of the objects contained in the multiple prediction boxes. The maximum probability among the probabilities of multiple prediction boxes associated with any object in any frame of image is determined as the confidence of any object in any frame of image.
[0047] Among them, since any frame of video image may include one or more objects, and the target image detection model has the same target detection method for each object in the video image, any object can be used as an example for explanation. Since the target image detection model can divide the video image when performing target detection, obtain multiple prediction frames, and detect the probability of containing the object in each prediction frame respectively. Since an object in the video image may have multiple associated prediction frames, that is, when the video image is divided, the same object is divided into multiple prediction frames, and the probability of the prediction frame can be used to reflect the credibility of the object contained in the prediction frame, that is, the credibility of the object detected by the target image detection model for the prediction frame. Therefore, by taking the maximum probability of the probabilities of multiple prediction frames associated with each object in the video image as the confidence of each object in the video image, that is, taking the probability of the prediction frame with the highest credibility of containing the object in the prediction frame as the confidence of the object, the accuracy of target detection can be improved.
[0048] Optionally, the object category to which any object belongs can be further determined based on the confidence of any object in any frame image. For example, when the confidence of any object in any frame image is determined, the category prediction of any object can be further performed to predict the category probability of any object being a plurality of object categories. The category probability is used to indicate the possibility that any object is a plurality of object categories. The object category corresponding to the maximum category probability can be used as the object category to which any object belongs, thereby realizing the prediction of the object category to which the object belongs. For example, the object categories include four object categories: human, cat, dog, and rabbit. The corresponding probabilities of any object in any frame video image being a human, cat, dog, and rabbit are 0.1, 0.2, 0.1, and 0.6, respectively, that is, the probability that any object is a rabbit is the largest. Then, the confidence of any object can be determined to be 0.6, and rabbit can be used as the object category to which any object belongs.
[0049] Optionally, when determining the position of any object in any frame of video image, the target image detection model can also be used to predict the probability of any object being in multiple positions, and the position corresponding to the maximum probability can be determined as the position of the object, or the position of any object in any video image can be determined by other means, which is not limited to this embodiment of the present application.
[0050] It can be understood that in the embodiments of the present application, any object in any frame of video images in the multiple frames of video images contained in the video data to be processed is used as an example to illustrate the confidence and position of any object in any frame of video images. For other objects in any frame of video images in the multiple frames of video images contained in the video data to be processed, or objects in other frames of video images in the multiple frames of video images contained in the video data to be processed, the corresponding confidence and position can be determined by referring to the processing method of any object in any frame of video images. The embodiments of the present application do not elaborate on this.
[0051] Optionally, before using the target image detection model, in order to improve the detection accuracy of the target image detection model, a large amount of sample video data can be used to pre-train the initial image detection model to obtain the target image detection model, so that the target image detection model has the ability to perform target detection on multiple frames of video images in the video data to be processed, and obtain the confidence and position of the objects in the multiple frames of video images. Subsequently, when the target image detection model is used to perform target detection on multiple frames of video images, the confidence and position of the objects in the multiple frames of video images can be obtained.
[0052] In an optional implementation, the process of training a target image detection model can be as follows: obtain sample video data, the sample video data including multiple consecutive frames of sample video images; use the initial image detection model to perform target detection on the multiple frames of sample video images to obtain sample confidence and sample position of objects in the multiple frames of sample video images; perform category prediction on objects in the multiple frames of sample video images to determine sample object categories of objects in the multiple frames of sample video images; obtain sample labels of objects in the multiple frames of sample video images, the sample labels including object category labels and sample position labels of objects in the multiple frames of sample video images; train the initial image detection model based on the sample labels, sample object categories and sample positions of objects in the multiple frames of sample video images to obtain a target image detection model.
[0053] Among them, the method of performing category prediction on objects in multiple frames of sample video images and determining sample object categories of objects in the multiple frames of sample video images may include: performing multiple classifications on each object in the multiple frames of sample video images through a classifier in an initial image detection model, determining the category probability that each object in the multiple frames of sample video images belongs to multiple object categories, and taking the maximum category probability among the category probabilities that each object in the multiple frames of sample video images belongs to multiple object categories as the sample object category of each object in the multiple frames of sample video images.
[0054] In a specific implementation, for example, the sample video data may be sampled to obtain a continuous multi-frame sample video image, and the method of sampling to obtain the continuous multi-frame sample video image may be the same as the method of sampling to obtain the continuous multi-frame video image. For example, the sampling frequency may be the same. The sample label of the object in the multi-frame sample video image refers to the true value of the sample, and for example, it may be obtained by pre-labeling the continuous multi-frame sample video image of the sample video data. The sample label may include an object category label and a sample position label of the object in the multi-frame sample video image. The object category label may be used to indicate the true object category of the object in the multi-frame sample video image, and the sample position label may be used to indicate the true position of the object in the multi-frame sample video image. Therefore, the sample label, sample object category and sample position based on the object in the multi-frame sample video image may refer to: training the initial image detection model based on the object category label and the sample object category of the object in the multi-frame sample video image; and training the initial image detection model based on the sample position label of the object in the multi-frame sample video image and the sample position of the object in the multi-frame sample video image.
[0055] Specifically, based on the object category labels and sample object categories of objects in multiple frames of sample video images, training the initial image detection model may refer to: based on the difference between the object category labels and sample object categories of objects in multiple frames of sample video images, training the initial image detection model. If the object category labels and sample object categories of objects in multiple frames of sample video images are inconsistent, continue to adjust the parameters in the initial image detection model. If the object category labels and sample object categories of objects in multiple frames of sample video images are consistent, indicating that the loss function of the initial image detection model is less than the loss threshold, the initial image detection model at this time may be saved as the target image detection model.
[0056] Further, based on the sample position labels and sample positions of objects in multiple frames of sample video images, training the initial image detection model may refer to: adjusting the parameters in the initial image detection model based on the difference between the sample position labels and the sample positions. The greater the difference between the sample position labels and the sample positions, the greater the loss function of the initial image detection model. When the loss function is greater than or equal to the loss threshold, the parameters in the initial image detection model may continue to be adjusted. The smaller the difference between the sample position labels and the sample positions, the smaller the loss function of the initial image detection model. When the loss function is less than the loss threshold, the initial image detection model at this time may be saved as the target image detection model.
[0057] Optionally, the initial image detection model can be trained based on the two tasks of object category and position, that is, a first loss function is obtained based on the difference between the object category label and the sample object category of the object in multiple frames of sample video images, and a second loss function is obtained based on the difference between the sample position label and the sample position of the object in multiple frames of sample video images, and when the sum of the first loss function and the second loss function is less than the loss threshold, the initial image detection model at this time is saved as the image detection model. The initial image detection model can also be trained based on the two tasks of confidence and position respectively, that is, when the first loss function is less than the first loss threshold and the second loss function is less than the second loss threshold, the initial image detection model at this time is saved as the image detection model, and the embodiments of the present application do not limit this.
[0058] In the embodiment of the present application, the sample object category and sample position of the object in the multi-frame sample video image may refer to the model output value, and the object category label and sample position label of the object in the multi-frame sample video image may refer to the sample true value. By obtaining the difference between the sample true value and the model output value, the parameters in the initial image detection model are adjusted to achieve the purpose of training the initial image detection model. The greater the difference between the sample true value and the model output value, the greater the loss function of the initial image detection model. When the loss function is greater than or equal to the loss threshold, the parameters in the initial image detection model can be further adjusted. The smaller the difference between the sample true value and the model output value, the smaller the loss function of the initial image detection model. When the loss function is less than the loss threshold, the initial image detection model at this time can be saved as the target image detection model. By continuously adjusting the parameters in the initial image detection model, the model output value and the sample true value can be made as consistent as possible, thereby ensuring the stability of the target image detection model, and then improving the accuracy of subsequent target detection. The number of sample video data in the embodiment of the present application can be multiple, and the stability and accuracy of the model can be improved by preparing a large amount of sample video data to train the initial image detection model.
[0059] It can be understood that when the number of iterative training of the initial image detection model is greater than the number threshold, or the loss function of the initial prediction model is less than the loss threshold, or the initial image detection model reaches the convergence condition, it is possible to stop adjusting the parameters in the initial image detection model to obtain the target image detection model.
[0060] In an optional implementation, when the initial image detection model is used to perform target detection on multiple frames of sample video images and the sample confidence of the objects in the multiple frames of sample video images is obtained, multiple prediction boxes associated with the objects in the multiple frames of sample video images can be determined, that is, each object may correspond to multiple prediction boxes, and the initial image detection model can predict the category probability of the object in each prediction box being multiple object categories, and the object category corresponding to the maximum category probability in the category probabilities in the multiple prediction boxes associated with each object can be respectively used as the object category to which the object in the multiple prediction boxes belongs. That is, the sample object category of the object in the multiple frames of sample video images can be determined in combination with the sample confidence of the object in the multiple frames of sample video images. In other words, when the initial image detection model detects multiple prediction boxes associated with any object in the sample video image, it can respectively predict the object category to which the object in each prediction box belongs, and select the final object category in combination with the sample confidence of the object in the multiple frames of sample video images. For example, the number of prediction boxes associated with an object is 3, namely prediction box 1, prediction box 2 and prediction box 3. The object category and sample confidence of prediction box 1 are cat and 0.2, the object category and sample confidence of prediction box 2 are dog and 0.2, and the object category and sample confidence of prediction box 3 are rabbit and 0.6. In this case, prediction box 3 can be used as the bounding box of the object, and 0.6 can be used as the sample confidence of the object.
[0061] In the process of selecting sample video data, sample video data containing various object categories can be obtained, sample video data containing a progress bar for indicating the current video playback progress, and sample video data containing likes, comments, bullet screens, number of viewers, favorites and other identifications can also be obtained. By using sample video data containing multiple object categories to train the initial image detection model, the target image detection model finally obtained can identify the confidence and position of various object categories in the video image contained in the video data to be processed. Optionally, the target image detection model is used to perform statistical analysis on the confidence and position of the objects detected in the video image. When calculating the confidence and position of the objects in the multi-frame video images corresponding to each moment, the more decimal places the confidence retains, the more accurate the video freeze detection result.
[0062] It is understandable that when training the initial image detection model, the corresponding confidences of multiple objects belonging to the same object category may not be equal. When the objects in the sample video images of adjacent frames in the sample video data are moving, the confidences of the objects in the sample video images of adjacent frames obtained through target detection are not equal. When the progress bar in the sample video data is constantly scrolling and changing, the confidence and position of the progress bar in each frame of the sample video image obtained through target detection will change. When the likes, comments, bullet screens, number of viewers, favorites and other identifiers in the sample video images of adjacent frames in the sample video data are constantly changing, the confidences and positions of the corresponding likes, comments, bullet screens, number of viewers, favorites and other identifiers in the sample video images of adjacent frames obtained through target detection may be unequal.
[0063] In an embodiment of the present application, by using sample video images from a large amount of sample video data to train an initial image detection model to obtain a target image detection model, the stability and accuracy of target detection can be improved, thereby improving the accuracy of subsequent target detection on multiple frames of video images, and further improving the confidence and accuracy of determining the position of objects in multiple frames of video images.
[0064] S103, comparing the confidences and positions of objects in adjacent frames of video images in the multiple frames of video images to obtain a comparison result.
[0065] In an embodiment of the present application, by comparing the confidence and position of the object in the adjacent frames of video images in the multi-frame video images, a comparison result is obtained, so that it can be determined whether the adjacent frames of video images are consistent. For example, the comparison result can be used to indicate whether the adjacent frames of video images are consistent. Among them, the consistency of adjacent frames of video images can refer to the same object category, equal confidence of the object, and the same position of the object in the adjacent frames of video images, which means that the image screen corresponding to the adjacent frames of video images has not changed. Adjacent frames of video images can refer to any two consecutive frames of video images in the video data to be processed. Alternatively, adjacent frames can also refer to any two consecutive frames of video images in the continuous multi-frame video images corresponding to a certain moment in the video data to be processed. For example, a moment can refer to 1 second, and the sampling frequency can be less than 1 second, then 1 second can correspond to continuous multi-frame video images, so two frames of video images can be selected from the continuous multi-frame video images corresponding to 1 second as adjacent frames of video images. By dividing the video data to be processed into multiple moments and comparing the adjacent frames of video images corresponding to each moment, the accuracy of video freeze detection can be improved.
[0066] In one embodiment, for example, adjacent video frames may include a first video image and a second video image, and the first video image may be a previous video image or a subsequent video image of the second video image. The comparison result of the adjacent video frames may be determined in the following manner:
[0067] The confidence of each object in the first video image is compared with the confidence of the corresponding object in the second video image, and the position of each object in the first video image is compared with the position of the corresponding object in the second video image; if the confidence of each object in the first video image is equal to the confidence of the corresponding object in the second video image, and the position of each object in the first video image is the same as the position of the corresponding object in the second video image, it is determined that the comparison result indicates that the adjacent frame video images are consistent. If the confidence of at least one object in the first video image is not equal to the confidence of the corresponding object in the second video image, or the position of at least one object in the first video image is not the same as the position of the corresponding object in the second video image, it is determined that the comparison result indicates that the adjacent frame video images are inconsistent.
[0068] Optionally, the confidence of each object in the first video image being equal to the confidence of the corresponding object in the second video image may include but is not limited to: the confidence of each object in the first video image is equal to the confidence of the corresponding object in the second video image, and the object category to which each object in the first video image belongs is the same as the object category to which the corresponding object in the second video image belongs. It can be known from the aforementioned determination of the confidence of the object in the video image that after determining the confidence of the object in the video image, the object category of the object in the video image can be further predicted to determine the object category to which the object in the video image belongs, so that the confidence of the object in the video image, the object category to which the object in the video image belongs, and the position of the object in the video image can be combined to finally determine whether the video images of adjacent frames are consistent.
[0069] Among them, the first video image and the second video image can be any two adjacent frame video images, and the adjacent frame video images can refer to the next frame video image of any frame video image, or can refer to the previous frame video image of any frame video image, and the embodiment of the present application does not limit this. For example, the adjacent frame video image refers to the next frame video image of any frame video image, then when a frame video image is selected from a continuous plurality of frames of video images, the frame video image and the next frame video image constitute adjacent frame video images. Each object in the first video image can refer to one object or multiple objects, for example, each object in the first video image can refer to a specific object in the first video image, or multiple objects, and the embodiment of the present application does not limit this. The corresponding object in the second video image refers to: the same object in the second video image as in the first video image. For example, if an object in the first video image is a cat, then the corresponding object in the second video image is a cat, and the cats in the two frames of video images are the same cat.
[0070] For example, if the confidence of each object in the first video image is equal to the confidence of the corresponding object in the second video image, but the position of each object in the first video image is different from the position of the corresponding object in the second video image, it can be indicated that the position of one or more objects in the first video image has changed, and it can be determined that the comparison result indicates that the adjacent frame video images are inconsistent. Or if the confidence of each object in the first video image is not equal to the confidence of the corresponding object in the second video image, but the position of each object in the first video image is different from the position of the corresponding object in the second video image, it can be indicated that the first video image and the second video image are different images, and the comparison result indicates that the adjacent frame video images are inconsistent. Because if the first video image is unchanged compared with the second video image, the target image detection model predicts that the probability size of the object contained in the bounding box and the accuracy of the bounding box will be completely consistent, and the confidence will also be equal. As long as the probability size of the object contained in the bounding box predicted in the two video images is different, or the accuracy of the bounding box is different, or the confidence is not equal, it means that the two video images are not completely consistent. For example, due to different light intensities in the video image, the color difference of the same object in two frames of video images is different, or replacing an object in the previous frame of video image with another object that is identical except for color may cause the confidence of adjacent frames of video images to be different, but the position of the object is the same. Therefore, comparing the consistency of the video images in combination with the confidence and position of the object in the video image can improve the accuracy of image comparison. By comparing the confidence and position of the corresponding objects in the two frames of video images, it can be determined whether the two frames of video images are consistent, and thus it can be determined whether there is video freeze in the video data to be processed based on the result of whether the video images are consistent.
[0071] In the embodiment of the present application, by comparing the confidence and position of each object in the adjacent frame video images in the processed video data, when the confidence of each object in the adjacent frame video images is equal and the position of each object is the same, it means that the adjacent frame video images are consistent. When the confidence of each object in the adjacent frame video images is not equal, or the position of each object is not the same, it means that the adjacent frame video images are inconsistent. The method of determining whether the adjacent frame video images are consistent by comparing the confidence and position of the objects in the adjacent frame video images is more efficient than the method of calculating whether the pixel values of two adjacent frames of video images are consistent, because the number of objects in the video image is less than the number of pixels in the video image. In addition, since each object in the adjacent frame video images is compared, the consistency judgment method in the embodiment of the present application is more accurate than the probability distribution of the image pixel grayscale values of the two frames of video images.
[0072] S104: Determine whether there is video freeze in the video data to be processed based on the comparison result.
[0073] In the embodiment of the present application, since the comparison result is used to indicate whether adjacent frames of video images are consistent, it is possible to determine whether the video data to be processed has video freezes in combination with the consistency between adjacent frames of video images. Since the video data to be processed includes multiple groups of adjacent frames of video images, each group of adjacent frames of video images can be compared to obtain the comparison result corresponding to each group of adjacent frames of video images. A group of adjacent frames of video images can include two frames of video images, for example, a first video image and a second video image.
[0074] In one embodiment, it is further possible to determine whether there is video freeze in combination with the number of consistent adjacent frame video images. For example, if the comparison result indicates that the adjacent frame video images are consistent, it is determined that there is video freeze in the video image in the video data to be processed; the number of video images with video freeze in the video data to be processed is obtained; if the number of video images with video freeze is greater than a number threshold, it is determined that there is video freeze in the video data to be processed.
[0075] In a possible implementation, if the video images with video freeze are all continuous video frame images, and the number of video images with video freeze is greater than a first number threshold, it is determined that the video data to be processed has video freeze. The number threshold may include the first number threshold.
[0076] In another possible implementation, if the video image with video freeze is not a continuous video frame image, and the number of video images with video freeze is greater than a second number threshold, it is determined that the video data to be processed has video freeze. The number threshold may include the second number threshold, and the first number threshold is less than the second number threshold.
[0077] That is to say, by comparing the consistency of adjacent video images, if the adjacent video images are consistent, and the adjacent video frame images greater than the first quantity threshold are continuous video frame images, it can be determined that the video data to be processed has a jam. For example, the video data to be processed includes 10 consecutive video frames, namely the first video frame, the second video frame, the third video frame, the fourth video frame, the fifth video frame, the sixth video frame, the seventh video frame, the eighth video frame, the ninth video frame, and the tenth video frame. The first quantity threshold is, for example, equal to 4, that is, there are 4 consecutive frames of consistent video images, for example, the first video frame, the second video frame, the third video frame, and the fourth video frame are all consistent, that is, two adjacent frames of video images are continuous, then it is considered that the video data to be processed has a jam. If the adjacent video images are consistent, and the adjacent video frame images greater than the second quantity threshold are not continuous video frame images, it can be determined that the video data to be processed has a jam. For example, if the second quantity threshold is equal to 6, that is, there are 6 consistent frames of video images, for example, the first frame of video image, the second frame of video image, the fifth frame of video image, the sixth frame of video image, the ninth frame of video image, and the tenth frame of video image are all consistent, then it is considered that there is a freeze in the video data to be processed. If there is only the first quantity threshold, such as 4 frames of video images, that is, the first frame of video image, the second frame of video image, the fifth frame of video image, and the sixth frame of video image are consistent, that is, two adjacent frames of video images are not continuous with each other, then it can be considered that there is no freeze in the video data to be processed. Optionally, when there are video images greater than the quantity threshold in the multiple frames of video images corresponding to a certain moment that are consistent, it can be determined that there is video freeze in the video data to be processed at that moment, and so on.
[0078] like Figure 3 As shown, Figure 3 is a schematic diagram of a video stagnation provided by an embodiment of the present application, Figure 3 The two images in the figure are adjacent frame video images. Both images include multiple objects. The following is an explanation of the two object categories of flowers and cats in the two images. Figure 3 The video images 31 and 32 in the video are used for target detection, and the confidence of the flowers in the two video images is 0.66, and the positions of the flowers in the two video images ( Figure 3 ), Figure 3 The confidence score of the cat in video image 31 and video image 32 is equal to 0.61, and the cat is in the same position in the two images, which means Figure 3 Understandably, there is video lag in Figure 3 The following additional image elements are also included, such as progress bar, like, favorite, comment and other logos. Figure 3These additional image elements have not changed, so in order to simplify the description, the process of comparing the additional image elements in the video image 31 and the video image 32 will not be described in detail here.
[0079] like Figure 4 As shown, Figure 4 is a schematic diagram of a video that is not stagnant provided by an embodiment of the present application, Figure 4 The two images in the figure are adjacent frame video images. Both images include multiple objects. The following is an explanation of the two object categories of flowers and cats in the two images. Figure 4 The video images 41 and 42 in the video are used for target detection, and the confidence of the flowers in the two video images is 0.66, and the positions of the flowers in the two video images ( Figure 4 ), Figure 4 The confidence score of the cat in video image 41 (0.61) is not equal to the confidence score of the cat in video image 42 (0.55), and the cat is located in different positions in the two images, which means Figure 4 There is no video lag in the video. Understandably, Figure 4 The following additional elements are also included in the image, such as likes, favorites, comments, etc. Figure 4 These additional image elements in the video image 41 and the video image 42 are unchanged, so in order to simplify the description, the process of comparing the additional image elements in the video image 41 and the video image 42 is not described in detail. Figure 4 The image also includes additional elements such as a progress bar. For example, through target detection, the confidence of the progress bar in the video image 41 is 0.33, and the confidence of the progress bar in the video image 42 is 0.45 ( Figure 4 (not shown in the figure), and the positions of the progress bars in the two images are not equal, so it is also possible to combine the confidence and position comparison results of the progress bars in the two images to determine that the video data to be processed is not stagnant.
[0080] In one embodiment, if it is determined that the video data to be processed has video freeze, the video freeze information can be reported. For example, at least one of the time point, image frame, confidence of the object in the image, and the first video abnormality type corresponding to the adjacent video images with consistency can be obtained, and the first video abnormality type is used to indicate that the video data to be processed has video freeze abnormality; based on at least one of the time point, image frame, confidence of the object in the image, and the first video abnormality type corresponding to the adjacent video images with consistency, the first prompt information for the video data to be processed is generated, and the first prompt information is used to prompt the video data to be processed to perform a video freeze check.
[0081] Among them, the time point may refer to a moment, such as the time point corresponding to the 1st second of the video data to be processed, or the time point corresponding to the Nth second, where N is a positive number greater than 1. Since there is video freeze in the video data to be processed, it is possible to capture an image of the video freeze from the video data to be processed, obtain the time point of the video freeze, the confidence of the object in the freeze video image, and generate a first video anomaly type indicating the video freeze, and report it to the corresponding terminal device for video freeze processing. For example, in the video player stability test scenario, the first prompt information may refer to a log. By reporting the log corresponding to the video freeze to the test terminal, the video player can be quickly processed to improve data processing efficiency.
[0082] In one embodiment, if the number of video images with video freeze is less than or equal to a number threshold, it is determined that there is no video freeze in the video data to be processed; at least one of the time point, image frame, confidence level of the object in the image and the second video abnormality type corresponding to the video image with video freeze is obtained, and the second video abnormality type is used to indicate that the video data to be processed is a video freeze abnormality; based on at least one of the time point, image frame, confidence level of the object in the image and the second video abnormality type corresponding to the video image with video freeze, second prompt information for the video data to be processed is generated, and the second prompt information is used to prompt the video data to be processed to be checked for video freeze.
[0083] The second prompt information is different from the first prompt information, for example, the video anomaly types included in the prompt information may be different, and the number of video images with video stagnation may be different. By reporting different types of prompt information respectively, the relevant terminal device can perform corresponding processing for different video anomalies, for example, manually detect the video images corresponding to the video stagnation to determine whether the video stagnation is a normal situation, such as whether the multiple frames of video images in the video data to be processed are indeed consistent, or the video stagnation is caused by the abnormality of the video player, so as to achieve targeted processing.
[0084] Optionally, since the video data to be processed may include video data at multiple moments, and each moment may correspond to multiple frames of video images, the adjacent frames of video images in the multiple frames of video images corresponding to each moment may be compared respectively to determine whether the video data to be processed has video freeze at each moment. For example, when the video data of one or more moments in multiple moments freezes, it can be considered that the video data to be processed has video freeze. When the video data of each moment in multiple moments is not freezed, it can be considered that the video data to be processed does not have video freeze.
[0085] In the embodiments of the present application, for video processing tasks, such as the automated stability test of a video player, after the R&D process performs technical optimization for the occasional problems of video data being unable to be played and video freezing, a large amount of time is usually required to verify whether these problems have been successfully optimized during the testing process. The reasons why video data cannot be played and video freezing are also affected by many factors such as network quality factors, the player kernel, and upper-layer calls. When troubleshooting problems, there are requirements for the accuracy of video freezing. The technical solution of the present application can realize real-time detection of video freezing and non-playable problems through automated means, and automatically intercept the corresponding log to report video freezing information, which can greatly improve work efficiency.
[0086] In the embodiment of the present application, by judging whether the confidence and position of all objects detected in the adjacent frame video images are consistent, if the confidence and position of all objects in the adjacent frame video images are consistent, it means that the image screen detected in the adjacent frame video images has not changed, and there is video stagnation. For the case where one image is used as the content of the entire video data, the progress bar in the image screen will continue to advance during the video player's playback of the video data. After the progress bar is identified by target detection, the confidence and position of the detected progress bar will continue to change due to the continuous scrolling change of the progress bar, so that it can be determined that the entire image screen has changed and there is no video stagnation. When judging whether the video image is consistent by the current pixel value calculation method, for the scene with one image as the content of the video data as a whole, since the position of the object detected in each frame of the video image will never change, and the confidence of the object will not change, so as to determine that the video playback is normal and there is no jamming, the video jamming detection method in the embodiment of the present application is more accurate.
[0087] Moreover, in the live broadcast scene, the camera device at the host end is aimed at a stationary object for a long time. Due to the scrolling of the comment area, the video image is divided into two parts according to the image screen elements aimed at by the camera device at the host end and the image additional elements composed of the likes, comments, favorites, and the number of viewers. If the target detection results of the image screen elements and the image additional elements in the adjacent frame video images are consistent, such as the confidence and position, it indicates that the image screen of the video player has not changed compared with the previous frame, and there is image screen stagnation, that is, there is video stagnation. If the confidence and position corresponding to the image additional elements in the adjacent frame video images are inconsistent, and the confidence and position corresponding to the image component elements in the adjacent frame video images are consistent, it means that the live broadcast content is stagnant. By reporting the video playback situation during the video playback process, targeted processing can be performed. For example, for the situation of stagnation of the live broadcast content, it can be determined whether the network quality of the host end is poor, or the host end is disconnected, etc. Further, when judging the video freeze, by setting the quantity threshold, it is judged that when the video freezes in multiple consecutive frames of video images, it is determined as video freeze, which can make the video freeze detection result more accurate.
[0088] In an embodiment of the present application, by acquiring video data to be processed including multiple consecutive frames of video images, target detection is performed on the multiple frames of video images to obtain the confidence and position of the object in the multiple frames of video images, so that the confidence and position of the object in the adjacent frames of video images in the multiple frames of video images can be compared to obtain a comparison result indicating whether the adjacent frames of video images are consistent, and then it can be determined based on the comparison result whether there is video freeze in the video data to be processed. By comparing the confidence and position of the object in the adjacent frames of video images, it can be determined whether there is a change in the object in the adjacent frames of video images, so that it can be determined whether there is a change in the adjacent frames of video images, and then it can be determined whether there is video freeze in the video data to be processed. Since the confidence and position of the object in the video image are combined to determine whether the object in the video image has changed, the accuracy of video detection can be improved.
[0089] For further information, see Figure 5 , Figure 5 is a flow chart of another video processing method provided in an embodiment of the present application; Figure 5 As shown, the video processing method can be applied to a computer device, and the video processing method includes but is not limited to the following steps:
[0090] S201, obtaining sample video data, training an initial image detection model based on the sample video data, and obtaining a target image detection model.
[0091] When training the initial image detection model, the sample video data may be sampled to obtain a plurality of continuous sample video images corresponding to the sample video data, thereby training the initial image detection model based on the plurality of continuous sample video images.
[0092] S202, obtaining video data to be processed, using a target image detection model to perform target detection on multiple frames of video images, and obtaining confidence and positions of objects in the multiple frames of video images.
[0093] S203, comparing the confidences and positions of objects in adjacent frames of video images in the multiple frames of video images to obtain a comparison result.
[0094] S204: If the comparison result indicates that the video images of adjacent frames are consistent, and the number of consistent video images is greater than the number threshold, it is determined that video freeze exists in the video data to be processed, and video freeze information is reported.
[0095] It can be understood that if the comparison result indicates that the adjacent frames of video images are consistent, it can mean that the object categories to which the objects in the adjacent frames of video images in the multiple frames of video images belong are the same, and the values corresponding to the confidences of the objects in the adjacent frames of video images in the multiple frames of video images are equal, and the positions of the objects in the adjacent frames of video images in the multiple frames of video images are the same. If the comparison result indicates that the adjacent frames of video images are inconsistent, it can mean that the object categories to which the objects in the adjacent frames of video images in the multiple frames of video images belong are different, or the values corresponding to the confidences of the objects in the adjacent frames of video images in the multiple frames of video images are not equal, or the positions of the objects in the adjacent frames of video images in the multiple frames of video images are different.
[0096] S205: If the comparison result indicates that the video images of adjacent frames are consistent, and the number of consistent video images is less than or equal to the number threshold, it is determined that video stagnation exists in the video data to be processed, and video stagnation information is reported.
[0097] Optionally, for example, the adjacent video frames may include a first video image and a second video image; if it is detected that any object exists in the first video image, and there is no object corresponding to any object in the first video image in the second video image, it is determined that the adjacent video frames are inconsistent, and it is determined that there is no video stagnation in the video data to be processed. If it is detected that the number of objects existing in the first video image is not equal to the number of objects existing in the second video image, it is determined that the adjacent video frames are inconsistent, and it is determined that there is no video stagnation in the video data to be processed.
[0098] S206: If the comparison result indicates that the video images of adjacent frames are inconsistent, it is determined that there is no video freeze in the video data to be processed.
[0099] Here, since the comparison result indicates that the video images of adjacent frames are inconsistent, it can also be determined that there is no video stagnation in the video data to be processed.
[0100] In the embodiment of the present application, the specific implementation of steps S201 to S206 can refer to the implementation of the above-mentioned steps S101 to S104, which will not be repeated here.
[0101] Optionally, in the video processing system of the embodiment of the present application, for example, modules such as an image video acquisition module, a target image detection model, a confidence and position analysis module, a freeze detection module, and a prompt module may be included. For example, the video data to be processed can be acquired based on the image video acquisition module, and target detection can be performed on multiple frames of video images in the image data to be processed based on the target image detection model to obtain the confidence and position of the object in the multiple frames of video images. Further, based on the confidence and position analysis module, the confidence and position of the object in the adjacent frames of video images in the multiple frames of video images corresponding to each moment are compared to determine whether the adjacent frames of video images are consistent. Further, based on the freeze detection module, a quantity threshold is set, and the adjacent frames of video images with consistency are judged in combination with the quantity threshold to determine whether there is video freeze in the multiple frames of video images corresponding to each moment, and then the video freeze information is reported through the prompt module.
[0102] Compared with the video freeze detection method based on frame rate, the video freeze detection method in the embodiment of the present application can more accurately evaluate the video smoothness and improve the video detection efficiency and video quality. Due to the application of the target image detection model, efficient image detection and accurate image classification can be achieved. The embodiment of the present application can be applied to video detection in various complex scenarios. For example, video detection can be performed on real-time video data in a video monitoring device, abnormal conditions of the video monitoring device can be discovered in time, and real-time detection and management of the video monitoring device can be strengthened. In addition, the structure of the embodiment of the present application is simple and easy to implement, which can better ensure the operational safety and stability of the video monitoring system.
[0103] In an embodiment of the present application, by acquiring video data to be processed including multiple consecutive frames of video images, target detection is performed on the multiple frames of video images to obtain the confidence and position of the object in the multiple frames of video images, so that the confidence and position of the object in the adjacent frames of video images in the multiple frames of video images can be compared to obtain a comparison result indicating whether the adjacent frames of video images are consistent, and then it can be determined based on the comparison result whether there is video freeze in the video data to be processed. By comparing the confidence and position of the object in the adjacent frames of video images, it can be determined whether there is a change in the object in the adjacent frames of video images, so that it can be determined whether there is a change in the adjacent frames of video images, and then it can be determined whether there is video freeze in the video data to be processed. Since the confidence and position of the object in the video image are combined to determine whether the object in the video image has changed, the accuracy of video detection can be improved.
[0104] The method of the embodiment of the present application is introduced above, and the device of the embodiment of the present application is introduced below.
[0105] See also Figure 6 , Figure 6 : is a schematic diagram of the composition structure of a video processing device provided in an embodiment of the present application. The above-mentioned video processing device can be deployed on a computer device; the video processing device can be used to execute the corresponding steps in the video processing method provided in an embodiment of the present application. The video processing device 60 includes:
[0106] The video acquisition unit 601 is used to acquire the video data to be processed, where the video data to be processed includes a plurality of continuous video frames;
[0107] The image detection unit 602 is used to perform target detection on the multiple frames of video images to obtain the confidence and position of the object in the multiple frames of video images;
[0108] An image comparison unit 603 is used to compare the confidence and position of the object in the adjacent frames of video images in the multiple frames of video images to obtain a comparison result, and the comparison result is used to indicate whether the adjacent frames of video images are consistent;
[0109] The result determination unit 604 is used to determine whether the video data to be processed has video freeze based on the comparison result.
[0110] Optionally, the adjacent frame video images include a first video image and a second video image; the image comparison unit 603 is specifically used to:
[0111] Comparing the confidence of each object in the first video image with the confidence of the corresponding object in the second video image, and comparing the position of each object in the first video image with the position of the corresponding object in the second video image;
[0112] If the confidence of each object in the first video image is equal to the confidence of the corresponding object in the second video image, and the position of each object in the first video image is the same as the position of the corresponding object in the second video image, then determining that the comparison result indicates that the adjacent frame video images are consistent;
[0113] If the confidence of at least one object in the first video image is not equal to the confidence of the corresponding object in the second video image, or the position of at least one object in the first video image is different from the position of the corresponding object in the second video image, it is determined that the comparison result indicates that the adjacent frame video images are inconsistent.
[0114] Optionally, the objects in the video image in the video data to be processed include: at least one of image composition elements and image additional elements in the video image, the image composition elements are image screen elements in the video image, and the image additional elements include at least one of effect screen elements and progress bars corresponding to the image display effects in the video data to be processed.
[0115] Optionally, the result determination unit 604 is specifically configured to:
[0116] If the comparison result indicates that the adjacent frames of video images are consistent, it is determined that video stagnation exists in the video image in the video data to be processed;
[0117] Obtain the number of video images with video stagnation in the video data to be processed;
[0118] If the number of video images with video stalls is greater than the number threshold, it is determined that the video data to be processed has video stalls.
[0119] Optionally, the video processing device 60 further includes a data reporting unit 605, and the data reporting unit 605 is used to:
[0120] Obtaining at least one of a time point, an image frame, a confidence level of an object in the image, and a first video abnormality type corresponding to adjacent video images having consistency, where the first video abnormality type is used to indicate that the video data to be processed is a video freeze abnormality;
[0121] Based on the time points corresponding to the adjacent video images with consistency, the image frames, the confidence of the objects in the images and at least one of the first video abnormality types, a first prompt information for the video data to be processed is generated, and the first prompt information is used to prompt a video freeze check to be performed on the video data to be processed.
[0122] Optionally, the data reporting unit 605 is further used to:
[0123] If the number of video images with video stalls is less than or equal to the number threshold, it is determined that the video data to be processed does not have video stalls;
[0124] Acquire at least one of a time point, an image frame, a confidence level of an object in the image, and a second video abnormality type corresponding to a video image with video stagnation, the second video abnormality type being used to indicate that the video data to be processed is a video stagnation abnormality;
[0125] Based on at least one of the time point corresponding to the video image with video stagnation, the image frame, the confidence of the object in the image and the second video abnormality type, second prompt information for the video data to be processed is generated, and the second prompt information is used to prompt the video data to be processed to be checked for video stagnation.
[0126] Optionally, the video processing device 60 further includes a model training unit 606, and the model training unit 606 is used to:
[0127] Acquire sample video data, where the sample video data includes a plurality of consecutive frames of sample video images;
[0128] Using the initial image detection model to perform target detection on the multiple frames of sample video images, and obtaining sample confidences and sample positions of objects in the multiple frames of sample video images;
[0129] Predicting the categories of objects in the multiple frames of sample video images to determine the sample object categories of the objects in the multiple frames of sample video images;
[0130] Obtaining sample labels of objects in the multiple frames of sample video images, the sample labels comprising object category labels and sample position labels of the objects in the multiple frames of sample video images;
[0131] Based on the sample labels of the objects in the multiple frames of sample video images, the sample object categories and the sample positions, the initial image detection model is trained to obtain a target image detection model;
[0132] The image detection unit 602 is specifically used for:
[0133] The target image detection model is used to perform target detection on the multi-frame video images to obtain the confidence and position of the object in the multi-frame video images.
[0134] Optionally, the confidence and position of the object in the multiple frames of video images include the confidence and position of any object in any frame of video images; the image detection unit 602 is specifically used to:
[0135] The target image detection model is used to perform target detection on any object in any frame of video image, and an image detection result of any object in any frame of video image is obtained, wherein the image detection result is used to indicate the probabilities of multiple prediction frames associated with any object in any frame of image and the position of any object in any frame of image, and the probabilities of multiple prediction frames associated with any object are used to reflect the credibility of the object contained in the multiple prediction frames;
[0136] The maximum probability among the probabilities of multiple prediction boxes associated with any object in any frame image is determined as the confidence of any object in any frame image.
[0137] It should be noted that Figure 6 For the contents not mentioned in the corresponding embodiments, please refer to the description of the method embodiments, which will not be repeated here.
[0138] In an embodiment of the present application, by acquiring video data to be processed including multiple consecutive frames of video images, target detection is performed on the multiple frames of video images to obtain the confidence and position of the object in the multiple frames of video images, so that the confidence and position of the object in the adjacent frames of video images in the multiple frames of video images can be compared to obtain a comparison result indicating whether the adjacent frames of video images are consistent, and then it can be determined based on the comparison result whether there is video freeze in the video data to be processed. By comparing the confidence and position of the object in the adjacent frames of video images, it can be determined whether there is a change in the object in the adjacent frames of video images, so that it can be determined whether there is a change in the adjacent frames of video images, and then it can be determined whether there is video freeze in the video data to be processed. Since the confidence and position of the object in the video image are combined to determine whether the object in the video image has changed, the accuracy of video detection can be improved.
[0139] See also Figure 7 , Figure 7 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 7As shown, the above-mentioned computer device 70 may include: a processor 701, a network interface 704 and a memory 705. In addition, the above-mentioned computer device 70 may also include: a user interface 703, and at least one communication bus 702. Among them, the communication bus 702 is used to realize the connection and communication between these components. Among them, the user interface 703 may include a display screen (Display), a keyboard (Keyboard), and the optional user interface 703 may also include a standard wired interface and a wireless interface. The network interface 704 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 705 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 705 may optionally also be at least one storage device located away from the aforementioned processor 701. As Figure 7 As shown, the memory 705 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.
[0140] exist Figure 7 In the computer device 70 shown, the network interface 704 can provide a network communication function; the user interface 703 is mainly used to provide an input interface for the user; and the processor 701 can be used to call the device control application stored in the memory 705 to achieve:
[0141] Acquire video data to be processed, where the video data to be processed includes a plurality of continuous frames of video images;
[0142] Performing target detection on the multiple frames of video images to obtain the confidence and position of the objects in the multiple frames of video images;
[0143] Comparing the confidences and positions of objects in adjacent frames of video images in the multiple frames of video images to obtain a comparison result, where the comparison result is used to indicate whether the adjacent frames of video images are consistent;
[0144] Based on the comparison result, it is determined whether the video data to be processed has video freeze.
[0145] It should be understood that the computer device 70 described in the embodiment of the present application can execute the above Figure 2 and Figure 5 The description of the above video processing method in the corresponding embodiment can also be performed as described above. Figure 6 The description of the video processing device in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.
[0146] In an embodiment of the present application, by acquiring video data to be processed including multiple consecutive frames of video images, target detection is performed on the multiple frames of video images to obtain the confidence and position of the object in the multiple frames of video images, so that the confidence and position of the object in the adjacent frames of video images in the multiple frames of video images can be compared to obtain a comparison result indicating whether the adjacent frames of video images are consistent, and then it can be determined based on the comparison result whether there is video freeze in the video data to be processed. By comparing the confidence and position of the object in the adjacent frames of video images, it can be determined whether there is a change in the object in the adjacent frames of video images, so that it can be determined whether there is a change in the adjacent frames of video images, and then it can be determined whether there is video freeze in the video data to be processed. Since the confidence and position of the object in the video image are combined to determine whether the object in the video image has changed, the accuracy of video detection can be improved.
[0147] The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, which includes program instructions, which, when executed by a computer, causes the computer to execute the method of the aforementioned embodiment, and the computer may be part of the computer device mentioned above. For example, the processor 701 mentioned above. As an example, the program instructions may be deployed on a computer device for execution, or deployed on multiple computer devices located at one location for execution, or, alternatively, executed on multiple computer devices distributed at multiple locations and interconnected by a communication network, and multiple computer devices distributed at multiple locations and interconnected by a communication network may constitute a blockchain network.
[0148] Those skilled in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0149] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.
Claims
1. A video processing method, characterized in that: The method comprises: Acquire video data to be processed, wherein the video data to be processed includes a plurality of continuous frames of video images; Performing target detection on the multiple video frames to obtain the confidence and position of the object in the multiple video frames; comparing the confidence and position of the object in adjacent video frames in the multiple video frames to obtain a comparison result, wherein the comparison result is used to indicate whether the adjacent video frames are consistent; Determine whether the video data to be processed has video freeze based on the comparison result; Wherein, the adjacent frames of video images in the continuous multiple frames of video images include a first video image and a second video image; and the confidence and position of objects in the adjacent frames of video images in the multiple frames of video images are compared to obtain a comparison result, including: comparing the confidence of each object in the first video image with the confidence of the corresponding object in the second video image, and comparing the position of each object in the first video image with the position of the corresponding object in the second video image; If the confidence of each object in the first video image is equal to the confidence of the corresponding object in the second video image, and the position of each object in the first video image is the same as the position of the corresponding object in the second video image, then determining that the comparison result indicates that the adjacent frame video images are consistent; If the confidence of at least one object in the first video image is not equal to the confidence of the corresponding object in the second video image, or the position of at least one object in the first video image is different from the position of the corresponding object in the second video image, it is determined that the comparison result indicates that the adjacent frame video images are inconsistent.
2. The method according to claim 1, characterized in that: The objects in the video image in the video data to be processed include: at least one of image composition elements and image additional elements in the video image, the image composition elements are image screen elements in the video image, and the image additional elements include at least one of effect screen elements and progress bars corresponding to the image display effects in the video data to be processed.
3. The method according to claim 1, characterized in that: The determining whether the video data to be processed has video freeze based on the comparison result includes: If the comparison result indicates that the adjacent frame video images are consistent, it is determined that video stagnation exists in the video image in the video data to be processed; Acquire the number of video images with video stagnation in the video data to be processed; If the number of video images with video stalls is greater than the number threshold, it is determined that the video data to be processed has video stalls.
4. The method according to claim 3, characterized in that The method further comprises: Acquire at least one of a time point, an image frame, a confidence level of an object in the image, and a first video abnormality type corresponding to adjacent video images having consistency, wherein the first video abnormality type is used to indicate that the video data to be processed is a video freeze abnormality; Based on the time points, image frames, confidence levels of objects in the images, and at least one of the first video anomaly types corresponding to the adjacent video images that have consistency, first prompt information for the video data to be processed is generated, and the first prompt information is used to prompt a video freeze check to be performed on the video data to be processed.
5. The method according to claim 3, characterized in that: The method further comprises: If the number of video images with video stalls is less than or equal to the number threshold, determining that the video data to be processed does not have video stalls; Acquire at least one of a time point, an image frame, a confidence level of an object in the image, and a second video abnormality type corresponding to a video image with video stagnation, wherein the second video abnormality type is used to indicate that the video data to be processed is a video stagnation abnormality; Based on at least one of the time point, the image frame, the confidence of the object in the image and the second video abnormality type corresponding to the video image with video stagnation, second prompt information for the video data to be processed is generated, and the second prompt information is used to prompt the video data to be processed to be checked for video stagnation.
6. The method according to claim 1, characterized in that The method further comprises: Acquire sample video data, wherein the sample video data includes a plurality of consecutive frames of sample video images; Using an initial image detection model to perform target detection on the multiple frames of sample video images to obtain sample confidences and sample positions of objects in the multiple frames of sample video images; Performing category prediction on objects in the multiple frames of sample video images to determine sample object categories of objects in the multiple frames of sample video images; Acquire sample labels of objects in the multiple frames of sample video images, wherein the sample labels include object category labels and sample position labels of objects in the multiple frames of sample video images; Based on the sample labels of the objects in the multiple frames of sample video images, the sample object categories and the sample positions, the initial image detection model is trained to obtain a target image detection model; The performing target detection on the multiple frames of video images to obtain the confidence and position of the objects in the multiple frames of video images includes: The target image detection model is used to perform target detection on the multiple frames of video images to obtain the confidence and position of the objects in the multiple frames of video images.
7. The method according to any one of claims 1 to 6, characterized in that: The confidence and position of the object in the multiple frames of video images include the confidence and position of any object in any frame of video images; Using a target image detection model to perform target detection on the multiple frames of video images to obtain the confidence and position of the object in the multiple frames of video images, including: Using the target image detection model to perform target detection on any object in any frame of video image, and obtaining an image detection result of any object in any frame of video image, wherein the image detection result is used to indicate the probabilities of multiple prediction boxes associated with any object in any frame of image and the position of any object in any frame of image, and the probabilities of multiple prediction boxes associated with any object are used to reflect the credibility of the object contained in the multiple prediction boxes; The maximum probability among the probabilities of multiple prediction boxes associated with any object in any frame image is determined as the confidence of any object in any frame image.
8. A video processing device, characterized in that: The device comprises: A video acquisition unit, used to acquire video data to be processed, wherein the video data to be processed includes a plurality of continuous video frames; An image detection unit, configured to perform target detection on the multiple frames of video images to obtain the confidence and position of the objects in the multiple frames of video images; An image comparison unit, used for comparing the confidence and position of objects in adjacent frames of video images in the multiple frames of video images to obtain a comparison result, wherein the comparison result is used to indicate whether the adjacent frames of video images are consistent; A result determination unit is used to determine whether the to-be-processed video data has video freeze based on the comparison result; the adjacent frames of video images in the continuous multiple frames of video images include a first video image and a second video image; The image comparison unit is specifically used to compare the confidence of each object in the first video image with the confidence of the corresponding object in the second video image, and to compare the position of each object in the first video image with the position of the corresponding object in the second video image; if the confidence of each object in the first video image is equal to the confidence of the corresponding object in the second video image, and the position of each object in the first video image is the same as the position of the corresponding object in the second video image, then it is determined that the comparison result indicates that the adjacent frames of video images are consistent; if the confidence of at least one object in the first video image is not equal to the confidence of the corresponding object in the second video image, or the position of at least one object in the first video image is different from the position of the corresponding object in the second video image, then it is determined that the comparison result indicates that the adjacent frames of video images are inconsistent.
9. A computer device, characterized in that: include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a data communication function, the memory is used to store program code, and the processor is used to call the program code so that the computer device executes the method described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 7.
11. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.
