Artificial intelligence-based video processing method, apparatus, device, and storage medium

Abstract

The application provides an artificial intelligence-based video processing method and device, equipment and a storage medium. The method comprises: frame extraction and shot processing on a to-be-processed video to obtain at least one to-be-processed image; using a target detection model to perform target detection on a current to-be-processed image in the at least one to-be-processed image to obtain at least one detection region corresponding to at least one type of detection object; determining a subject detection region from the at least one detection region according to the size information of each detection region in the at least one detection region; determining a scene recognition result of the current to-be-processed image based on the subject size information of the subject detection region and the detection object, and then obtaining a scene recognition result of each to-be-processed image in the at least one to-be-processed image; and implementing intelligent processing on the to-be-processed video based on the scene recognition result of each to-be-processed image. Through the application, the efficiency of video processing based on artificial intelligence can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to artificial intelligence technology, and in particular to a video processing method and device based on artificial intelligence, an apparatus, and a storage medium. BACKGROUND

[0002] At present, in the video cover picture generation and highlight video intelligent generation of video processing technology, the distance view recognition function of the video image is crucial. For example, when generating a cover picture, the generation business often needs to provide rich image materials of long shot, medium shot, full shot, close-up, etc., rather than all from large face materials; in video generation, generally, a panoramic or long shot segment is needed as the starting scene, a person close-up as the transition, and a person close-up as the key segment to alternately produce a video with stories and levels. The current common method is to train a regular deep learning distance view recognition model by labeling massive data, including: manually defining different categories of distance views, collecting a large number of images from scratch, manually labeling, cleaning, training and recognizing models, and the like. As can be seen, the current method needs a large amount of manual work to label and collect massive data to support model training to achieve high recognition accuracy, so the workload involved in model training is large, the training time is long, which reduces the efficiency of model training, and further reduces the efficiency of video processing; and when the current method is applied in video analysis tasks, the distance view recognition model needs to analyze a large number of images and has complex analysis tasks, causing the calculation pressure of the video analysis task, and further reducing the efficiency of video processing. SUMMARY

[0003] The embodiments of the present application provide a video processing method and device based on artificial intelligence, an apparatus, and a storage medium, which can improve the efficiency of video processing.

[0004] The technical solutions of the embodiments of the present application are as follows:

[0005] The embodiments of the present application provide a video processing method based on artificial intelligence, comprising:

[0006] Frame extraction and shot processing are performed on the to-be-processed video to obtain at least one to-be-processed image;

[0007] A target detection model is used to perform target detection on a current to-be-processed image in the at least one to-be-processed image to obtain at least one detection region corresponding to at least one type of detection object; the target detection model is used to detect the at least one detection region occupied by the at least one type of detection object from the current to-be-processed image;

[0008] A subject detection region is determined from the at least one detection region according to the size information of each detection region in the at least one detection region;

[0009] determine a scene type recognition result of the current to-be-processed image based on the subject size information of the subject detection region and the detection object, and further obtain a scene type recognition result of each to-be-processed image in the at least one to-be-processed image; the scene type recognition result represents a distance of image content from a visual starting point;

[0010] implement intelligent processing of the to-be-processed video based on the scene type recognition result of each to-be-processed image.

[0011] An embodiment of the present application provides a video processing device based on artificial intelligence, comprising:

[0012] a video frame extraction module configured to perform frame extraction and split shot processing on a to-be-processed video to obtain at least one to-be-processed image;

[0013] a target detection model configured to perform target detection on a current to-be-processed image in the at least one to-be-processed image to obtain at least one detection region corresponding to at least one type of detection object; the target detection model is configured to detect the at least one detection region occupied by the at least one type of detection object from the current to-be-processed image;

[0014] a scene type recognition module configured to determine a subject detection region from the at least one detection region according to size information of each detection region in the at least one detection region, determine a scene type recognition result of the current to-be-processed image based on subject size information of the subject detection region and the detection object, and further obtain a scene type recognition result of each to-be-processed image in the at least one to-be-processed image; the scene type recognition result represents a distance of image content from a visual starting point;

[0015] a video processing module configured to implement intelligent processing of the to-be-processed video based on the scene type recognition result of each to-be-processed image.

[0016] In the apparatus, the main body size information comprises a main body area height and a main body area width; the at least one type of detection object comprises a person object; the scene type identification module is further configured to: when the detection object is the person object, obtain a size ratio of the main body detection area and a first area according to the main body area height and the main body area width; when the first area is greater than or equal to a first preset area threshold, determine the scene type identification result according to the size ratio; or when the first area is less than the first preset area threshold and greater than or equal to a second preset area threshold, determine the scene type identification result based on the main body area height or the main body area width; or the second preset area threshold is less than the first preset area threshold; or when the first area is less than the second preset area threshold, determine the scene type identification result as a long shot and mark the current image to be processed as not meeting a preset result.

[0017] In the apparatus, the size ratio is a ratio of the main body area height to the main body area width; the scene type identification module is further configured to: when the size ratio is greater than a first preset size ratio threshold and less than or equal to a second preset size ratio threshold, determine the scene type identification result as a close-up of a face; or the second preset size ratio threshold is greater than the first preset size ratio threshold; or when the size ratio is greater than the second preset size ratio threshold and less than or equal to a third preset size ratio threshold, determine the scene type identification result as a close-up of a human body; the third preset size ratio threshold is greater than the second preset size ratio threshold; or when the size ratio is greater than the third preset size ratio threshold and less than or equal to a fourth preset size ratio threshold, determine the scene type identification result as a full shot of a human body; the fourth preset size ratio threshold is greater than the third preset size ratio threshold; or when the size ratio is greater than the fourth preset size ratio threshold, determine the scene type identification result as a long shot.

[0018] In the apparatus, the scene type identification module is further configured to: when the main body area height or the main body area width is greater than or equal to a preset first edge length threshold, determine the scene type identification result as a full shot of a human body; or when the main body area height or the main body area width is less than the preset first edge length threshold and greater than or equal to a preset second edge length threshold, determine whether the size ratio is greater than the second preset size ratio threshold; when the size ratio is greater than or equal to the second preset size ratio threshold, determine the scene type identification result as a long shot; or when the size ratio is less than the second preset size ratio threshold, determine the scene type identification result as a long shot; or when the main body area height or the main body area width is less than the preset second edge length threshold, determine the scene type identification result as a long shot and mark the current image to be processed as not meeting a preset result.

[0019] In the device, the main body size information includes a main body area height and a main body area width, and the at least one type of detection object includes an object object. The scene type identification module is further configured to: when the detection object is the object object, obtain a second area of the main body detection area according to the main body area height and the main body area width; when the second area is greater than or equal to a preset third area threshold, determine the scene type identification result as an object close-up; or when the second area is less than the preset third area threshold and greater than or equal to a preset fourth area threshold, determine the scene type identification result as an object close-up; the preset fourth area threshold is less than the preset third area threshold; or when the second area is less than the preset fourth area threshold and greater than or equal to a second preset area threshold, determine the scene type identification result as an object wide shot; the preset fourth area threshold is greater than the second preset area threshold; or when the second area is less than the second preset area threshold, determine the scene type identification result as an object long shot.

[0020] In the device, the target detection model is further configured to: after the frame extraction and the shot processing on the to-be-processed video are performed to obtain at least one to-be-processed image, perform target detection on a current to-be-processed image in the at least one to-be-processed image, and when no detection area is detected, mark the current to-be-processed image as not meeting a preset result, and not use the current to-be-processed image for video processing.

[0021] In the device, the video processing apparatus based on artificial intelligence further includes a model training module. The model training module is configured to: before the target detection model is used to perform target detection on each to-be-processed image in the at least one to-be-processed image to obtain a target detection result, use the initial target detection model to perform target detection on the training sample image set, determine a sample image that does not meet a preset result from the training sample image set; use the sample image that does not meet the preset result as an incremental training sample, obtain a labeling result of the incremental training sample, and then obtain an incremental training sample set; and based on the incremental training sample set and the training sample image set, iteratively train the initial target detection model to obtain the target detection model.

[0022] An electronic device is provided in an embodiment of the present application, and the electronic device includes:

[0023] A memory is configured to store executable instructions.

[0024] A processor is configured to execute the executable instructions stored in the memory, and implement the video processing method based on artificial intelligence provided in the embodiments of the present application.

[0025] The embodiment of the present application provides a storage medium, which stores executable instructions, and is used for causing a processor to execute the method for video processing based on artificial intelligence provided by the embodiment of the present application.

[0026] The embodiment of the present application has the following beneficial effects:

[0027] The size information of the subject detection region in the at least one detection region and the detection object are used to determine the scene type recognition result of the current image to be processed, so that the training workload required for training the scene type recognition model from zero and the image processing workload of the scene type recognition model in the video processing process are reduced, and the efficiency of the video processing is improved. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is an optional structural schematic diagram of the video processing system architecture based on artificial intelligence provided by the embodiment of the present application;

[0029] Figure 2 is an optional structural schematic diagram of the video processing device based on artificial intelligence provided by the embodiment of the present application;

[0030] Figure 3 is an optional flow schematic diagram of the method for video processing based on artificial intelligence provided by the embodiment of the present application;

[0031] Figure 4 is an optional effect schematic diagram of different scene type recognition results provided by the embodiment of the present application;

[0032] Figure 5 is an optional flow schematic diagram of the method for video processing based on artificial intelligence provided by the embodiment of the present application;

[0033] Figure 6 is an effect schematic diagram of the relationship between different scene types of a person and a picture provided by the embodiment of the present application;

[0034] Figure 7 is an optional flow schematic diagram of the method for video processing based on artificial intelligence provided by the embodiment of the present application;

[0035] Figure 8 is an optional flow schematic diagram of the method for video processing based on artificial intelligence provided by the embodiment of the present application;

[0036] Figure 9 is an optional functional module structure diagram of the video processing system based on artificial intelligence provided by the embodiment of the present application;

[0037] Figure 10is an optional flow diagram of training a target detection model provided by an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be described in further detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application. All other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present application.

[0039] In the following description, "some embodiments" are described, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0040] If similar descriptions of "first / second" appear in the application file, the following description is added. In the following description, the terms "first\second\third" referred to are only to distinguish similar objects, and do not represent a specific order of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0042] The relevant data collection and processing in the embodiments of the present application should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0043] Before the embodiments of the present application are described in further detail, the terms and phrases involved in the embodiments of the present application are explained, and the terms and phrases involved in the embodiments of the present application are applicable to the following explanations.

[0044] 1) Deep learning technology: a technology for machine learning using a deep neural network system.

[0045] 2) Annotation: a rectangular annotation box is drawn for an object in an image, and the label corresponding to the box is noted, i.e., the annotation category.

[0046] 3) Recognition model: a mathematical model obtained by machine learning technology after learning the labeled sample data (picture - corresponding relationship of specified label), the parameters of the mathematical model are obtained in the process of learning and training, the parameters of the mathematical model are loaded and the probability that the input sample belongs to a certain physical label within the specified range is calculated in the process of identification and prediction.

[0047] 4) Detection model: a mathematical model obtained by machine learning technology after learning the labeled sample data (picture - corresponding relationship of specified multiple labeled boxes and label pairs), the parameters of the mathematical model are obtained in the process of learning and training, the parameters of the mathematical model are loaded and the probability that the input sample exists a physical label prediction box and the prediction box belongs to a certain physical label within the specified range is calculated in the process of identification and prediction.

[0048] 5) Detection model annotation: the object in the image is drawn with a rectangular annotation box, and the physical label corresponding to the box is noted, that is, the annotation category.

[0049] 6) Artificial intelligence (Artificial Intelligence, AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0050] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning, etc.

[0051] 7) Computer Vision (CV): Computer vision is a science that studies how to make machines "see". More specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further process images to make them more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, trying to establish artificial intelligence systems that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. It also includes common face recognition, fingerprint recognition and other biometric identification technologies.

[0052] 8) Machine Learning (ML): Machine learning is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a specialized study of how computers can simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. It is applied in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rule-based learning.

[0053] With the research and progress of artificial intelligence technology, artificial intelligence technology has been researched and applied in many fields, such as common smart home, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned vehicles, autonomous vehicles, drones, robots, smart medical care, smart customer service, etc. With the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0054] 9) The scope of the scene refers to the difference in the size of the subject presented in the camera recorder due to the different distances between the camera and the subject. The division of the scope of the scene can be divided into five types, from near to far, close-up, close-up, medium shot, full shot, and long shot.

[0055] The scheme provided by the embodiments of the present application relates to image detection and image recognition technologies of artificial intelligence, and is specifically explained as follows: The embodiments of the present application provide a video processing method and device based on artificial intelligence, equipment and storage medium, which can improve the efficiency of video processing. The following describes an exemplary application of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as a notebook computer, a tablet computer, a desktop computer, a set-top box, a mobile device (for example, a mobile phone, a portable music player, a personal digital assistant, a dedicated message device, a portable game device) and various types of user terminals, and can also be implemented as a server. The following describes an exemplary application when the device is implemented as a terminal.

[0056] Referring to Figure 1 , Figure 1 is an optional architecture schematic diagram of the video processing system 100 based on artificial intelligence provided by the embodiments of the present application. To realize a video processing application, the terminal (exemplarily shows the terminal 400-1 and the terminal 400-2) connects the server 200 through the network 300 from the database 500. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0057] The terminal 400-1 is configured to obtain a to-be-processed video from the database 500 through the server 200, perform frame extraction and shot processing on the to-be-processed video to obtain at least one to-be-processed image, use a target detection model to perform target detection on a current to-be-processed image in the at least one to-be-processed image to obtain at least one detection region corresponding to at least one type of detection object, determine a subject detection region from the at least one detection region according to size information of each detection region in the at least one detection region, determine a scene recognition result of the current to-be-processed image based on subject size information of the subject detection region and the detection object, and further obtain a scene recognition result of each to-be-processed image in the at least one to-be-processed image. The scene recognition result represents the distance of the image content from the visual starting point. The intelligent processing of the to-be-processed video is realized based on the scene recognition result of each to-be-processed image, and the processing result of the to-be-processed video, such as a video after intelligent editing or a video cover image, is displayed on the graphical interface 410-1. The terminal 400 is further configured to send the processing result to the server 200 through the network 300, so that the server 200 pushes the processing result to the terminal 400-2, and displays the processing result on the graphical processing interface 410-2 of the terminal 400-2. The server 200 is configured to respond to the obtaining request of the terminal 400-1, and send the to-be-processed video from the database 500 to the terminal 400-1, and receive the processing result of the to-be-processed video of the terminal 400-1, and push the processing result to the terminal 400-2 through the network 300.

[0058] In some embodiments, the server 200 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms. The terminal 400 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, and the like, but is not limited thereto. The terminal and the server can be connected directly or indirectly through wired or wireless communication, which is not limited in the embodiments of the present application.

[0059] Referring to Figure 2 , Figure 2 is a structural schematic diagram of the terminal 400-1 provided by the embodiments of the present application, Figure 2 The terminal 400-1 shown in the figure includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the terminal 400 are coupled together through a bus system 440. It can be understood that the bus system 440 is used to realize the connection communication between the components. In addition to including a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the purpose of clear illustration, all kinds of buses are marked as the bus system 440 in the figure. Figure 2 The processor 410 can be an integrated circuit chip with signal processing capability, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.

[0060] The processor 410 can be an integrated circuit chip with signal processing capability, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.

[0061] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432 that facilitate user input, such as a keyboard, a mouse, a microphone, a touch screen display, a camera, other input buttons and controls.

[0062] The memory 450 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 optionally includes one or more storage devices physically located in proximity to the processor 410.

[0063] The memory 450 includes volatile memory or nonvolatile memory, and can include both volatile and nonvolatile memory. The nonvolatile memory can be read only memory (ROM), and the volatile memory can be random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0064] In some embodiments, the memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, which are exemplarily illustrated below.

[0065] The operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks.

[0066] The network communication module 452 is used to communicate with other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including Bluetooth, wireless fidelity (WiFi), universal serial bus (USB), and the like.

[0067] The presentation module 453 is used to enable the presentation of information via one or more output devices 431 associated with the user interface 430 (e.g., a display screen, a speaker, and the like) (e.g., a user interface for operating peripheral devices and displaying content and information).

[0068] The input processing module 454 is used to detect and interpret one or more user inputs or interactions from one or more input devices 432.

[0069] In some embodiments, the apparatus provided by the embodiments of the present application can be implemented in software, Figure 2 An artificial intelligence-based video processing apparatus 455 stored in the memory 450 is shown, which can be software in the form of programs and plug-ins, including the following software modules: a video frame extraction module 4551, a target detection model 4552, a scene recognition module 4553, and a video processing module 4554. These modules are logical, and thus can be combined or further split according to the functions implemented.

[0070] The functions of the various modules will be described below.

[0071] In other embodiments, the artificial intelligence-based video processing device (hereinafter referred to as the video processing device) provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the artificial intelligence-based video processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0072] The artificial intelligence-based video processing method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the terminal provided in the embodiment of the present application.

[0073] See also Figure 3 , Figure 3 This is an optional flow chart of the video processing method based on artificial intelligence provided by the embodiment of the present application, which will be combined with Figure 3 The steps shown are explained.

[0074] S101, extracting frames and performing storyboard processing on a video to be processed to obtain at least one image to be processed;

[0075] In an embodiment of the present application, a video processing device may perform video storyboarding on a video to be processed. At least one shot is obtained, wherein each shot within the at least one shot contains at least one candidate image. The video processing device extracts a preset number of images from the at least one candidate image contained in each shot as images to be processed, thereby extracting at least one image to be processed from the at least one shot.

[0076] In some embodiments, the video processing device can use SceneDetect v5.0, an open-source Python library for video storyboarding, to storyboard a video to be processed, obtaining multiple shots corresponding to the video to be processed, each shot containing multiple images. Because shots within the same storyboard often depict the same subject, to reduce computational complexity, the video processing device extracts the middle two frames from each shot as the images to be processed, thereby obtaining at least one image to be processed from multiple shots.

[0077] S102, using a target detection model to perform target detection on a current to-be-processed image in the at least one to-be-processed image, to obtain at least one detection region corresponding to at least one type of detection object; the target detection model is used to detect at least one detection region occupied by the at least one type of detection object from the current to-be-processed image.

[0078] In the embodiment of the application, the video processing apparatus can use a multi-class target detection model to perform target detection on each to-be-processed image in the at least one to-be-processed image. For a current to-be-processed image in the at least one to-be-processed image, the video processing apparatus can use the target detection model to extract image features from the current to-be-processed image, and identify and predict an image region containing at least one type of detection object in the current to-be-processed image based on the extracted image features and the at least one type of detection object, and finally output at least one detection region corresponding to the at least one type of detection object through the target detection model, thereby completing target detection on the current to-be-processed image.

[0079] In the embodiment of the application, the region information of the at least one detection region includes the height and width of the detection region, the position coordinates of the detection region in the to-be-processed image, and the confidence of the detection region corresponding to each type of detection object in the at least one type of detection object, that is, the probability of each type of detection object contained in the detection region.

[0080] S103, determining a subject detection region from the at least one detection region according to the size information of each detection region in the at least one detection region.

[0081] In the embodiment of the application, the video processing apparatus can determine a subject detection region in the current to-be-processed image according to the size information of each detection region in the at least one detection region, wherein the image content in the subject detection region represents the subject content of the current to-be-processed image.

[0082] In some embodiments, the size information can be the height and width of the detection region; the video processing apparatus can calculate the area of each detection region according to the height and width of the detection region, and then determine the detection region with the largest area from the at least one detection region as the subject detection region. In other embodiments, the video processing apparatus can determine the detection region with a width-to-height ratio belonging to a preset ratio range as the subject detection region according to the width-to-height ratio of each detection region. The specific selection is selected according to the actual situation, which is not limited in the embodiment of the application.

[0083] S104, determining a scene recognition result of the current to-be-processed image based on the subject size information of the subject detection region and the detection object, and then obtaining a scene recognition result of each to-be-processed image in the at least one to-be-processed image; the scene recognition result represents the distance of the image content from the visual starting point.

[0084] In the embodiments of the present application, since the subject detection region represents the subject content in the current to-be-processed image, the video processing apparatus can determine the scene type recognition result of the current to-be-processed image according to the determined subject detection region size information and in combination with the prior knowledge of the detection object size information. The video processing apparatus performs the same processing on each to-be-processed image in the at least one to-be-processed image, and can obtain the scene type recognition result of each to-be-processed image.

[0085] In the embodiments of the present application, the scene type recognition result can be the size and range of the photographed subject and the picture image presented in the screen frame structure, representing the distance of the image content from the visual starting point. In some embodiments, the scene type recognition result can be long shot, long shot, panorama, close-up and close-up.

[0086] In some embodiments, the long shot, long shot, panorama, close-up and close-up can be as shown in Figure 4

[0087] S105, based on the scene type recognition result of each to-be-processed image, the intelligent processing of the to-be-processed video is realized.

[0088] In the embodiments of the present application, the video processing apparatus can select to-be-processed images of different scene types for intelligent video editing based on the scene type recognition result of each to-be-processed image, and realize intelligent processing of the to-be-processed video; or the video processing apparatus can select a target to-be-processed image suitable for the actual application according to the scene type recognition result of each to-be-processed image, and generate a cover image corresponding to the to-be-processed video by using the target to-be-processed image, so as to realize the combination and push of the cover image and the to-be-processed video.

[0089] In some embodiments, for the intelligent video editing scene, the video processing apparatus can splice and edit to-be-processed images of different scene types by a progressive splicing or a jump splicing method. For example, for progressive splicing, the video processing apparatus can splice to-be-processed images with scene type recognition results of close-up, close-up, medium shot, panorama and long shot in the order from close to far, to obtain a video clip with a far-off effect, or the video processing apparatus can splice to-be-processed images with scene type recognition results of long shot, panorama, medium shot, close-up and close-up in the order from far to close, to obtain a video clip with a close-up effect. For jump splicing, the video processing apparatus can select to-be-processed images of different scene types for splicing in a jump manner according to different editing requirements, to obtain a video clip with obvious visual change characteristics. The specific selection is based on actual conditions, which is not limited in the embodiments of the present application.

[0090] ​It can be understood that, in the embodiment of the present application, the video processing apparatus obtains at least one detection region through the pre-trained target detection model, determines the scene type recognition result of the current image to be processed based on the size information of the subject detection region in the at least one detection region and the detection object, reduces the training workload required for training the scene type recognition model from zero and the image processing workload of the scene type recognition model in the video processing process, thereby improving the efficiency of video processing.

[0091] In some embodiments, referring to Figure 5 , Figure 5 is an optional flowchart of the video processing method based on artificial intelligence provided by the embodiment of the present application, the subject size information includes: subject region height and subject region width; the at least one type of detection object includes: a person object, and the determination of the scene type recognition result of the current image to be processed based on the subject size information of the subject detection region and the detection object in S104 can be implemented by S1041-S1043, which will be described in combination with each step.

[0092] S1041, when the detection object is a person object, the size ratio of the subject detection region and the first area are obtained according to the subject region height and the subject region width.

[0093] In the embodiment of the present application, the video processing apparatus can determine whether the detection object corresponding to the subject detection region is a person object according to the confidence of the subject detection region for each type of detection object in the at least one type of detection object. When it is determined that the detection object is a person object, the video processing apparatus can determine the scene type recognition result of the current image to be processed according to the subject size information of the subject detection region combined with the prior knowledge of the size information of the person object in the video.

[0094] In the embodiment of the present application, when the detection object is a person object, the video processing apparatus can calculate the size ratio of the subject detection region and the first area according to the subject region height and the subject region width, respectively.

[0095] In some embodiments, the video processing apparatus can normalize the size information of the subject detection region to use a unified scene type recognition standard for images to be processed with different resolutions. For example, the video processing apparatus can normalize the size information of the subject detection region to the interval (0, 1), such as for a current image to be processed of 448*448, which includes a subject detection region with a pixel height of 40 and a pixel width of 20, the video processing apparatus takes 40 / 448 as the subject detection region height, 20 / 448 as the subject detection region width, and (40 / 448)*(20 / 448) as the first area of the subject detection region.

[0096] S1042, when the first area is greater than or equal to the first preset area threshold, determining the scene type recognition result according to the size ratio.

[0097] In the embodiments of the present application, when the first area of the subject detection region is greater than or equal to the first preset area threshold, the video processing apparatus can determine the scene type recognition result according to the size ratio, such as the ratio of the subject detection region height to the subject detection region width, or the ratio of the subject detection region width to the subject detection region height.

[0098] In some embodiments, for the subject region height and the subject region width normalized to the (0, 1) interval, the first preset area threshold can be 0.2. Other values can also be selected according to the definition of different scene types, which are selected according to actual conditions, and the embodiments of the present application are not limited.

[0099] In some embodiments, the different scene types of the person object and the picture relationship comparison can be as shown in Figure 6 As can be seen from Figure 6 , for the person object, in some pictures mainly showing the person, the close-up of the face, the close-up of the body, the full view of the body, and the long shot of the body can all occupy a large area in the entire picture; among them, the close-up of the face has a small width-height gap, and as the distance represented by the scene type becomes from near to far, the width-height gap of the close-up of the body, the full view of the body, and the long shot of the body gradually increases. At the same time, in some pictures showing the person or the environment at the same time, the full view of the body and the long shot of the body, including the long shot, can also occupy a small area in the entire picture; at this time, the area occupied by the full view of the body presents the characteristics of a long edge length, the area occupied by the long shot presents the characteristics of a short edge length and a large width-height gap, and the area occupied by the long shot presents the characteristics of a short edge length and a small width-height gap. Therefore, the video processing apparatus can determine the scene type recognition result of the current image to be processed according to the size information of the subject detection region in combination with the characteristics of the person object in different scene types in Figure 6

[0100] In some embodiments, the size ratio is the ratio of the subject region height to the subject region width, and when the first area is greater than or equal to the first preset area threshold, the determination of the scene type recognition result according to the size ratio in S1042 can be realized by executing S201-S204, which will be described in combination with each step.

[0101] S201, when the size ratio is greater than a first preset size ratio threshold and less than or equal to a second preset size ratio threshold, determining the scene type recognition result as a close-up of the face; the second preset size ratio threshold is greater than the first preset size ratio threshold.

[0102] ​In the embodiments of the present application, the first preset size ratio threshold can be a ratio of the minimum height and width of the region occupied by the character object set according to prior knowledge. If it is less than the first preset size ratio threshold, the basic common sense of the character object proportion can be violated. The second preset size ratio threshold can be a threshold set according to the conventional proportion of a close-up of a face (such as the part above the human body shoulders). The second preset size ratio threshold is greater than the first preset size ratio threshold. Therefore, in the case that the first area is greater than or equal to the first preset area threshold, when the size ratio is greater than the first preset size ratio threshold and less than or equal to the second preset size ratio threshold, it is indicated that the subject detection region area is large and the height-width gap is not obvious, which meets the characteristics of the close-up of a face, and the video processing apparatus can determine the scene type recognition result as the close-up of a face.

[0103] In some embodiments, the first preset size ratio threshold can be 1, and the second preset size ratio threshold can be 2. Other values can also be set according to actual conditions, and the specific selection is made according to actual conditions, which is not limited in the embodiments of the present application.

[0104] S202, when the size ratio is greater than the second preset size ratio threshold and less than or equal to a third preset size ratio threshold, the scene type recognition result is determined as a close-up of a human body; the third preset size ratio threshold is greater than the second preset size ratio threshold.

[0105] In the embodiments of the present application, the third preset size ratio threshold is greater than the second preset size ratio threshold, that is, the height-width ratio of the third preset size ratio threshold is greater than the height-width ratio of the character object, which can be considered as the height-width ratio corresponding to the close-up (such as the part above the human body chest). Therefore, in the case that the first area is greater than or equal to the first preset area threshold, when the size ratio is greater than the second preset size ratio threshold and less than or equal to the third preset size ratio threshold, it is indicated that the subject detection region area is large and the height-width gap is relatively obvious, which meets the characteristics of the close-up of a human body, and the video processing apparatus determines the scene type recognition result as the close-up of a human body.

[0106] In some embodiments, the third preset size ratio threshold can be 4, and other values can also be set according to actual conditions, and the specific selection is made according to actual conditions, which is not limited in the embodiments of the present application.

[0107] S203, when the size ratio is greater than the third preset size ratio threshold and less than or equal to a fourth preset size ratio threshold, the scene type recognition result is determined as a full view of a human body.

[0108] In the embodiments of the present application, the fourth preset size ratio threshold value is greater than the third preset size ratio threshold value, which can be considered as that the aspect ratio of the person object is further lengthened along the height direction of the person on the basis of the close-up. In the case that the first area is greater than or equal to the first preset area threshold value, when the size ratio is greater than the third preset size ratio threshold value and less than or equal to the fourth preset size ratio threshold value, it is indicated that the area of the subject detection region is large, and the width-height gap is very obvious, which is consistent with the characteristics of the human full view, and the video processing apparatus determines the scene recognition result as the human full view.

[0109] In some embodiments, the fourth preset size ratio threshold value can be 6, or other values can be set according to actual conditions, and the specific selection is based on actual conditions, which is not limited in the embodiments of the present application.

[0110] S204, when the size ratio is greater than the fourth preset size ratio threshold value, the scene recognition result is determined as the long shot.

[0111] In the embodiments of the present application, when the size ratio is greater than the fourth preset size ratio threshold value, it is indicated that the width-height gap of the subject detection region is further enlarged, and the visual effect of the person object is further away, and accordingly, the video processing apparatus determines the scene recognition result as the long shot.

[0112] S1043, when the first area is less than the first preset area threshold value and greater than or equal to the second preset area threshold value, the scene recognition result is determined based on the subject region height or the subject region width; the second preset area threshold value is less than the first preset area threshold value.

[0113] In the embodiments of the present application, the second preset area threshold value is less than the first preset area threshold value, when the first area of the subject detection region corresponding to the person object is less than the first preset area threshold value and greater than or equal to the second preset area threshold value, it is indicated that the subject detection region occupies a small area in the current image to be processed, and the person object corresponding to the subject detection region is far away from the visual starting point, and the video processing apparatus can further determine the scene recognition result based on the subject region height or the subject region width.

[0114] In some embodiments, for the subject region height and the subject region width normalized to the interval (0, 1), the second preset area threshold value can be 0.01, or other values can be set, and the specific selection is based on actual conditions, which is not limited in the embodiments of the present application.

[0115] In some embodiments, S1043 can be implemented by executing S301-S305, which will be described in combination with each step.

[0116] S301, when the subject region height or the subject region width is greater than or equal to a preset first edge length threshold value, the scene recognition result is determined as the human full view.

[0117] In the embodiment of the present application, when the first area is less than the first preset area threshold and greater than or equal to the second preset area threshold, and when the height of the subject region or the width of the subject region is greater than or equal to the preset first edge length threshold, it is indicated that the area of the subject detection region is small, and the edge length of a certain side is long, which meets the characteristics of the human panoramic view. The video processing apparatus determines the scene recognition result as a human panoramic view.

[0118] In some embodiments, for the height of the subject region and the width of the subject region normalized to the interval (0, 1), the preset first edge length threshold can be set to 0.3, or can be set to other values according to actual conditions. The specific selection is made according to actual conditions, and the embodiment of the present application is not limited.

[0119] S302, when the height of the subject region or the width of the subject region is less than the preset first edge length threshold and greater than or equal to the preset second edge length threshold, it is determined whether the size ratio is greater than the second preset size ratio threshold.

[0120] In the embodiment of the present application, when the first area is less than the first preset area threshold and greater than or equal to the second preset area threshold, and when the height of the subject region or the width of the subject region is less than the preset first edge length threshold and greater than or equal to the preset second edge length threshold, the video processing apparatus can further determine whether the size ratio is greater than the second preset size ratio threshold.

[0121] In some embodiments, for the height of the subject region and the width of the subject region normalized to the interval (0, 1), the preset second edge length threshold can be set to 0.1, or can be set to other values according to actual conditions. The specific selection is made according to actual conditions, and the embodiment of the present application is not limited.

[0122] S303, when the size ratio is greater than or equal to the second preset size ratio threshold, the scene recognition result is determined as a long shot.

[0123] In the embodiment of the present application, when the first area is less than the first preset area threshold and greater than or equal to the second preset area threshold, and the height of the subject region or the width of the subject region is less than the preset first edge length threshold and greater than or equal to the preset second edge length threshold, when the size ratio is greater than or equal to the second preset size ratio threshold, it is indicated that the area of the subject detection region is small, the edge length is short, and the height-width gap is obvious, which meets the characteristics of the long shot. The video processing apparatus determines the scene recognition result as a long shot.

[0124] S304, when the size ratio is less than the second preset size ratio threshold, the scene recognition result is determined as a long shot.

[0125] In the embodiment of the present application, when the first area is smaller than the first preset area threshold and greater than or equal to the second preset area threshold, and the height of the main area or the width of the main area is smaller than the preset first side length threshold and greater than or equal to the preset second side length threshold, when the size ratio is smaller than the second preset size ratio threshold, it indicates that the area of ​​the main detection area is smaller, the side length is shorter, and the difference between width and height is not obvious, which meets the characteristics of a long view, and the video processing device determines the scene recognition result as a long view.

[0126] S305: When the height or width of the subject area is less than the preset second side length threshold, the scene recognition result is determined to be a long-distance scene, and the current image to be processed is marked as not meeting the preset result.

[0127] In an embodiment of the present application, when the first area is less than the first preset area threshold and greater than or equal to the second preset area threshold, and the height of the main body area or the width of the main body area is less than the preset first side length threshold and greater than or equal to the preset second side length threshold, when the height of the main body area or the width of the main body area is less than the preset second side length threshold, it indicates that a side length of the main body detection area is too narrow, and the video processing device can determine the scene recognition result as a long shot. Moreover, since the too narrow side length indicates that the current image to be processed may be at the middle boundary of a long shot or an erroneous detection, the video processing device can further mark the current image to be processed as not meeting the preset result, so as to prompt the subsequent video processing process to further confirm the image that does not meet the preset result, and the image marked as not meeting the preset result can be used in the process of further training the target detection model, which will be explained in the model training part.

[0128] S1044: When the first area is smaller than a second preset area threshold, the scene recognition result is determined to be a long-range scene, and the current image to be processed is marked as not meeting the preset result.

[0129] In an embodiment of the present application, when the first area is smaller than the second preset area threshold, it indicates that the area occupied by the subject detection area in the entire picture is too small. The video processing device can determine the scene recognition result as a long view, and further mark the current image to be processed as not meeting the preset result, so as to prompt the subsequent video processing process to further confirm the image that does not meet the preset result, and the image marked as not meeting the preset result can be used in the process of further training the target detection model, which will be explained in the model training part.

[0130] In some embodiments, after S102, when the video processing device obtains at least one detection area, when the number of detection areas in at least one detection area where the detected objects are human objects is greater than a preset human number threshold, the video processing device can confirm the scene recognition result of the current image to be processed as a human group scene.

[0131] It should be noted that in the embodiments of the present application, the size ratio can also be the ratio of the main body area width to the main body area height, and correspondingly, the first preset size ratio threshold, the second preset size ratio threshold, the third preset size ratio threshold and the fourth preset size ratio threshold can be set based on the predefined ratio of width to height. The specific selection is based on the actual situation, and the embodiments of the present application are not limited.

[0132] It can be understood that in the embodiments of the present application, the video processing apparatus can determine the image foreground and background based on the pre-trained target detection model and the prior knowledge of foreground and background, thereby avoiding directly collecting a large number of labeled samples for recognition, and avoiding introducing an additional deep learning model to cause too long overall calculation time. In addition, the effect is closer to the definition of foreground and background, thereby improving the efficiency and accuracy of the scene recognition result, and further improving the efficiency and accuracy of video processing.

[0133] In some embodiments, referring to Figure 7 , Figure 7 is an optional flowchart of the video processing method based on artificial intelligence provided by the embodiments of the present application. The main body size information includes the main body area height and the main body area width. The at least one type of detection object includes an object object. Exemplarily, the object object can be an image object such as a building, a vehicle, a tree, etc. The process of determining the scene recognition result of the current image to be processed based on the main body size information and the detection object of the main body detection area in S104 can be implemented by S1045-S1049, which will be described in combination with each step.

[0134] S1045, when the detection object is an object object, the second area of the main body detection area is obtained according to the main body area height and the main body area width.

[0135] S1046, when the second area is greater than or equal to a preset third area threshold, the scene recognition result is determined as an object close-up.

[0136] In the embodiments of the present application, when the second area is greater than or equal to the preset third area threshold, it indicates that the proportion of the main body detection area occupied by the object object in the current image to be processed is relatively large, and the distance between the object object and the visual starting point is relatively close. The video processing apparatus determines the scene recognition result as an object close-up.

[0137] In some embodiments, for the main body area height and the main body area width normalized to the (0, 1) interval, the preset third area threshold can be set to 0.3, or can be set to other values according to the actual situation. The specific selection is based on the actual situation, and the embodiments of the present application are not limited.

[0138] S1047, when the second area is less than a preset third area threshold and greater than or equal to a preset fourth area threshold, determining the scene recognition result as a close shot of the object; the preset fourth area threshold is less than the preset third area threshold.

[0139] In the embodiment of the application, the preset fourth area threshold is less than the preset third area threshold. When the second area is less than the preset third area threshold and greater than or equal to the preset fourth area threshold, it indicates that the subject detection area occupied by the object is less than the area corresponding to the close shot and belongs to the area range corresponding to the close shot, and the video processing device determines the scene recognition result as a close shot of the object.

[0140] In some embodiments, for the subject area height and the subject area width normalized to the (0, 1) interval, the preset third area threshold can be set to 0.1, or can be set to other values according to actual conditions, and the specific selection is based on actual conditions, which is not limited in the embodiment of the application.

[0141] S1048, when the second area is less than the preset fourth area threshold and greater than or equal to a second preset area threshold, determining the scene recognition result as a full shot of the object; the preset fourth area threshold is greater than the second preset area threshold.

[0142] In the embodiment of the application, the preset fourth area threshold is greater than the second preset area threshold. When the second area is less than the preset fourth area threshold and greater than or equal to the second preset area threshold, the video processing device determines the scene recognition result as a full shot of the object.

[0143] In some embodiments, for the subject area height and the subject area width normalized to the (0, 1) interval, the preset fourth area threshold can be set to 0.01, or can be set to other values according to actual conditions, and the specific selection is based on actual conditions, which is not limited in the embodiment of the application.

[0144] S1049, when the second area is less than the second preset area threshold, determining the scene recognition result as a long shot of the object.

[0145] In the embodiment of the application, when the second area is less than the second preset area threshold, the video processing device determines the scene recognition result as a long shot of the object.

[0146] It can be understood that in the embodiment of the application, the video processing device can also perform close / long shot recognition on the object of the object category, and the close / long shot recognition of multiple types of detection objects is realized without significantly increasing the model operation amount and processing amount, thereby improving the efficiency of video processing.

[0147] In some embodiments, after S101, S001 can also be included, which will be described in combination with each step.

[0148] S001. When using a target detection model to perform target detection on a current image to be processed in at least one image to be processed, if no detection area is detected, the current image to be processed is marked as not meeting a preset result, and the current image to be processed is not used for video processing.

[0149] In an embodiment of the present application, when a video processing device uses a target detection model to perform target detection on a current image to be processed among at least one of the images to be processed and fails to detect a detection area, it indicates that the current image to be processed contains a target object that the target detection model cannot recognize, and further scene recognition results cannot be inferred based on the recognizable target object. The video processing device marks the current image to be processed as not meeting the preset result and does not use the current image to be processed for video processing. The image marked as not meeting the preset result may also be used in further training of the target detection model, as will be explained in the model training section.

[0150] In some embodiments, see Figure 8 , Figure 8 This is an optional flow chart of the video processing method based on artificial intelligence provided by the embodiment of the present application. Figure 3 Before S102, S401-S403 may also be included, which will be explained in conjunction with each step.

[0151] S401: Use the initial target detection model to perform target detection on the training sample image set, and determine sample images that do not meet the preset results from the training sample image set.

[0152] In an embodiment of the present application, the video processing device can adopt an initial target detection model to perform target detection on the training sample image set based on the default initial training weights of the initial target detection model, and based on the target detection results, combined with the above-mentioned scene recognition method, determine from the training sample image set the training sample images in which no target is detected, and / or the training sample images in which the first area of ​​the subject detection area is less than the second preset area threshold, and / or the training sample images in which the height of the subject area or the width of the subject area is less than the preset second side length threshold as sample images that do not meet the preset results.

[0153] In some embodiments, the initial target detection model can be a yolov5 model or other types of target detection network models. The specific selection is based on actual conditions and is not limited in the embodiments of this application.

[0154] In some embodiments, the training sample image set can use the open source data coco training set as the training sample image set, or other image sets can be used as the training sample image set. The specific selection is made according to the actual situation and is not limited in the embodiments of this application.

[0155] S402, sample images not meeting the preset result are taken as incremental training samples, a labeling result of the incremental training samples is obtained, and then an incremental training sample set is obtained.

[0156] In the embodiments of the present application, the video processing apparatus takes sample images not meeting the preset result as incremental training samples, obtains a labeling result of the incremental training samples, and then obtains an incremental training sample set.

[0157] In the embodiments of the present application, the sample images not meeting the preset result include undetected subject objects, which can be undetected by the initial target detection model or be new type objects not belonging to the preset detection object type of the initial target detection model. The video processing apparatus collects the labeling result of the sample images not meeting the preset result, takes the incremental training samples and the labeling result corresponding to the incremental training samples as the incremental training sample set, and then inputs the incremental training sample set into the training of the initial target detection model to enhance the target detection capability of the initial target detection model.

[0158] In the embodiments of the present application, the labeling result is a labeling region in which a person manually labels the position, size and category of the subject object in the sample image not meeting the preset result.

[0159] S403, based on the enhanced training sample set and the training sample image set, the initial target detection model is iteratively trained to obtain a target detection model.

[0160] In the embodiments of the present application, the video processing apparatus can take the enhanced training sample set and the training sample image set as a full training data set, divide the full training data set into multiple batches to obtain at least one batch of training data sets, and update the network weight by using a standard Stochastic Gradient Descent (SGD) optimization method. Specifically, the video processing apparatus can perform image enhancement preprocessing on each batch of training data sets in the at least one batch of training data sets, and input the preprocessed data into the initial target detection model for forward calculation to obtain a training prediction region set corresponding to each batch of training data sets in the current round of training. The video processing apparatus calculates the confidence loss of the training prediction region set, the classification cross-entropy loss of the positive prediction region in the training prediction region set, the center position coordinate loss, and the width-height loss according to the training prediction region set and the label region set corresponding to each batch of training data sets. Finally, the confidence loss, the classification cross-entropy, the center position coordinate loss, and the width-height loss are added to obtain the total loss of the current round of training. The video processing apparatus obtains the gradient value of the weight adjustment of each layer of neural network of the initial target detection model by the SGD algorithm according to the total loss of the current round of training, updates each layer weight using the gradient value, and performs the next round of training based on the updated weight until the preset training target is reached, such as the training times reaching the preset times or the total loss reaching below the preset loss threshold, and the training is ended to obtain the target detection model.

[0161] In some embodiments, the video processing apparatus can calculate the total loss of each round of training by formula (1) as follows:

[0162]

[0163] wherein S 2 is the position set of the training prediction region in the current training data, obj is the positive detection frame of the training prediction region with an object, and noobj is the negative detection frame of the training prediction region without an object. i and y i are the center coordinates of the label region, and are the center coordinates of the training prediction region, is the center position coordinate loss. w i and h i are the width and height values of the label region, and are the width and height values of the training prediction region,

[0164] is the width-height loss; C i is the confidence score of the training prediction region, is the overlapping part of the training prediction region and the label region,

[0165] for the confidence loss,

[0166] for the prediction class loss.

[0167] It can be understood that in the embodiments of the present application, the video processing apparatus can mine the training difficult examples of target detection by the scene type recognition method, and use the training difficult examples as incremental samples to train the initial target detection model, thereby improving the target detection capability of the target detection model, and further enhancing the accuracy of target detection and the accuracy of video processing.

[0168] In the following, an exemplary application of the embodiments of the present application in an actual application scenario will be described.

[0169] In the embodiments of the present application, referring to Figure 9 , Figure 9 an optional functional module structure diagram of a video processing system based on artificial intelligence is shown. In the system of Figure 9 , the video processing apparatus can apply the far and near scene recognition method in the embodiments of the present application to the video analysis business downstream of the key target detection module originally in the video analysis understanding system, help improve the effect of video understanding and video embedding, and avoid introducing an analysis task complex, far and near scene recognition model with relatively large calculation pressure in the video analysis understanding system, so that the entire video processing system is more lightweight, reduces the time consumption of far and near scene recognition calculation, and improves the video processing efficiency. In the following, taking the intelligent generation of a video cover picture by the video processing system shown in Figure 9 as an example, the exemplary application process of the embodiments of the present application will be further described in combination with Figure 10 . Among them, for the far and near scene recognition module shown in Figure 10 , in the model training process, the module can output the recognition result to the model training module for collecting sample images of missing targets; in the video processing process, the module can output the recognition result to the video processing module for generating a cover picture.

[0170] In the embodiments of the present application, for the model training process, the video processing apparatus can input the original sample set into the target detection model, use the target detection model to perform target detection on the current sample image in the original sample set, and obtain a current sample detection result. The video processing apparatus can analyze the current sample detection result by the far and near scene recognition module in Figure 10 using the method in S501-S504, and obtain a far and near scene recognition result, as follows:

[0171] S501, when the current sample detection result is at least one target detection frame, determining the target detection frame with the largest area from the at least one target detection frame as the main detection region; and normalizing the area, height and width of the main detection region to the value interval of (0, 1), and then obtaining the area a of the main detection region, the height h of the main region, the width w of the main region and the aspect ratio ratio of the main detection region, wherein ratio is equal to h / w.

[0172] S502, when the detection object in the main detection region is a person object, determining the far and near scene recognition result of the current sample image by a person far and near scene judgment method, specifically, the person far and near scene judgment method can be realized by S01-S03, as follows:

[0173] S01, when a is greater than or equal to 0.2, determining the far and near scene recognition result of the current sample image according to the value of ratio.

[0174] In S01, the first preset area threshold is 0.2. S01 can include S01-1 to S01-4, as follows:

[0175] S01-1, when ratio is greater than 1 and less than or equal to 2, the far and near scene recognition result of the current sample image is determined as a close-up of the face.

[0176] In S01-1, the first preset size ratio threshold is 1, and the second preset size ratio threshold is 2.

[0177] S01-2, when ratio is greater than 2 and less than or equal to 4, the far and near scene recognition result of the current sample image is determined as a close-up of the human body.

[0178] In S01-2, the third preset size ratio threshold is 4.

[0179] S01-3, when ratio is greater than 4 and less than or equal to 6, the far and near scene recognition result of the current sample image is determined as a full view of the human body.

[0180] In S01-2, the fourth preset size ratio threshold is 6.

[0181] S01-4, when ratio is greater than 6, the far and near scene recognition result of the current sample image is determined as a far view.

[0182] S02, when a is greater than or equal to 0.01 and less than 0.2, determining the far and near scene recognition result of the current sample image based on the value of w or h.

[0183] In S02, the second preset area threshold is 0.01. S02 can include S02-1 to S02-4, as follows:

[0184] S02-1, when w or h is greater than or equal to 0.3, the far and near scene recognition result of the current sample image is determined as a full view of the human body.

[0185] In S02-1, the first edge length threshold is 0.3.

[0186] S02-2, when w or h is less than 0.3 and greater than or equal to 0.1, if the ratio is greater than or equal to 2, the far and near scene recognition result of the current sample image is determined as a far view.

[0187] In S02-2, the second edge length threshold is 0.1.

[0188] S02-3, when w or h is less than 0.3 and greater than or equal to 0.1, if the ratio is less than 2, the far and near scene recognition result of the current sample image is determined as a far view.

[0189] S02-4, when w or h is less than 0.1, the far and near scene recognition result of the current sample image is determined as a far view, and the current sample image is marked as not meeting the preset result.

[0190] S03, when a is less than 0.01, the far and near scene recognition result of the current sample image is determined as a far view, and the current sample image is marked as not meeting the preset result.

[0191] S503, when the detection object in the subject detection area is an object object, the far and near scene recognition result of the current sample image is determined by the object far and near scene judgment method, specifically, the object far and near scene judgment method can be realized by S11-S14, as follows:

[0192] S11, when a is greater than or equal to 0.3, the far and near scene recognition result of the current sample image is determined as an object close-up.

[0193] In S11, the third area threshold is 0.3.

[0194] S12, when a is less than 0.3 and greater than or equal to 0.1, the far and near scene recognition result of the current sample image is determined as an object close-up.

[0195] In S11, the fourth area threshold is 0.1.

[0196] S13, when a is less than 0.1 and greater than or equal to 0.01, the far and near scene recognition result of the current sample image is determined as an object close-up.

[0197] S14, when a is less than 0.01, the far and near scene recognition result of the current sample image is determined as a far view.

[0198] S504, when the current sample detection result is no target, mark the current sample image as not meeting the preset result.

[0199] In the embodiment of the present application, for the model training process, the video processing apparatus processes each sample image in the original sample set in the same way through the method of S501-S504, obtains all sample images that do not meet the preset result as the sample images missing the target, obtains the label result of the sample images missing the target, and thus obtains the incremental training sample set. The video processing apparatus trains the target detection model using the original sample set and the enhanced training sample set, and uses the trained target detection model to perform the video processing process.

[0200] In the embodiment of the present application, for the video processing process, the video processing apparatus frames and divides the video to be processed to obtain at least one image to be processed, and uses the trained target detection model to perform target detection on the at least one image to be processed to obtain at least one detection region of each image to be processed. The video processing apparatus processes the at least one detection region of each image to be processed in the same way as S501-S504 through the near-far scene recognition module to obtain the near-far scene recognition result of each image to be processed. Finally, the video processing apparatus can determine the target image from the near-far scene recognition result of each image to be processed according to the actual cover picture requirement, and generate a cover picture using the target image. For example, when the actual cover picture requirement is to use a close-up of a person as the cover picture of the video to be processed, the video processing apparatus determines the image to be processed belonging to the close-up of the person as the target image according to the near-far scene recognition result of each image to be processed, and generates a cover picture with a close-up effect using the target image, thereby completing the video processing process.

[0201] It can be understood that in the embodiment of the present application, the video processing apparatus can use the algorithm logic in the near-far scene recognition module to further train the target detection model, mine the missing target sample image from the original sample set as the missing target image, collect the label result of the missing target image to form a labeled sample training set, and use the labeled sample training set to incrementally train and update the target detection neural network model, thereby improving the accuracy of the target detection model and the accuracy of the entire video processing system. Moreover, the video processing apparatus cooperates with the near-far scene recognition logic on the existing target detection model, without increasing the processing workload, and can quickly determine the near-far scene recognition result of the image according to the target detection result of the target detection model, thereby improving the efficiency of video processing.

[0202] The following continues to illustrate an exemplary structure of the video processing apparatus 455 based on artificial intelligence provided in the embodiment of the present application as a software module. In some embodiments, as shown in FIG. 5, the video processing apparatus 455 based on artificial intelligence includes a target detection model 501, a near-far scene recognition module 502, a video processing module 503, and a video processing result output module 504. Figure 2As shown, the software modules stored in the artificial intelligence based video processing apparatus 455 of the memory 450 can include:

[0203] a video frame extraction module 4551 configured to perform frame extraction and shot processing on a to-be-processed video to obtain at least one to-be-processed image;

[0204] a target detection model 4552 configured to perform target detection on a current to-be-processed image in the at least one to-be-processed image to obtain at least one detection region corresponding to at least one type of detection object; the target detection model is configured to detect the at least one detection region occupied by the at least one type of detection object from the current to-be-processed image;

[0205] a scene type identification module 4553 configured to determine a subject detection region from the at least one detection region according to size information of each detection region in the at least one detection region; determine a scene type identification result of the current to-be-processed image based on subject size information of the subject detection region and a detection object, and further obtain a scene type identification result of each to-be-processed image in the at least one to-be-processed image; the scene type identification result represents a distance of image content from a visual starting point;

[0206] a video processing module 4554 configured to implement intelligent processing on the to-be-processed video based on the scene type identification result of each to-be-processed image.

[0207] In some embodiments, the subject size information includes a subject region height and a subject region width; the at least one type of detection object includes a person object; and the scene type identification module 4553 is further configured to, when the detection object is the person object, obtain a size ratio and a first area of the subject detection region according to the subject region height and the subject region width; when the first area is greater than or equal to a first preset area threshold, determine the scene type identification result according to the size ratio; or, when the first area is less than the first preset area threshold and greater than or equal to a second preset area threshold, determine the scene type identification result based on the subject region height or the subject region width; the second preset area threshold is less than the first preset area threshold; or, when the first area is less than the second preset area threshold, determine the scene type identification result as a long shot, and mark the current to-be-processed image as not meeting a preset result.

[0208] In some embodiments, the size ratio is a ratio of the body region height and the body region width; the scene type identification module 4553 is further configured to: when the size ratio is greater than a first preset size ratio threshold and less than or equal to a second preset size ratio threshold, determine the scene type identification result as a close-up of a face; the second preset size ratio threshold is greater than the first preset size ratio threshold; or when the size ratio is greater than the second preset size ratio threshold and less than or equal to a third preset size ratio threshold, determine the scene type identification result as a close-up of a human body; or when the third preset size ratio threshold is greater than the second preset size ratio threshold; when the size ratio is greater than the third preset size ratio threshold and less than or equal to a fourth preset size ratio threshold, determine the scene type identification result as a full view of a human body; the fourth preset size ratio threshold is greater than the third preset size ratio threshold; or when the size ratio is greater than the fourth preset size ratio threshold, determine the scene type identification result as a long shot.

[0209] In some embodiments, the scene type identification module 4553 is further configured to: when the body region height or the body region width is greater than or equal to a first preset edge length threshold, determine the scene type identification result as a full view of a human body; or when the body region height or the body region width is less than the first preset edge length threshold and greater than or equal to a second preset edge length threshold, determine whether the size ratio is greater than the second preset size ratio threshold; when the size ratio is greater than or equal to the second preset size ratio threshold, determine the scene type identification result as a long shot; or when the size ratio is less than the second preset size ratio threshold, determine the scene type identification result as a very long shot; or when the body region height or the body region width is less than the second preset edge length threshold, determine the scene type identification result as a very long shot, and mark the current to-be-processed image as not meeting a preset result.

[0210] In some embodiments, the subject size information comprises a subject region height and a subject region width; the at least one type of detection object comprises an object object; the scene type identification module 4553 is further configured to, when the detection object is the object object, obtain a second area of the subject detection region according to the subject region height and the subject region width; when the second area is greater than or equal to a preset third area threshold, determine the scene type identification result as an object close-up; or, when the second area is less than the preset third area threshold and greater than or equal to a preset fourth area threshold, determine the scene type identification result as an object close-up; the preset fourth area threshold is less than the preset third area threshold; or, when the second area is less than the preset fourth area threshold and greater than or equal to a second preset area threshold, determine the scene type identification result as an object wide shot; the preset fourth area threshold is greater than the second preset area threshold; or, when the second area is less than the second preset area threshold, determine the scene type identification result as an object long shot.

[0211] In some embodiments, the target detection model 4552 is further configured to, after the frame extraction and shot processing of the video to be processed, perform target detection on a current to-be-processed image in the at least one to-be-processed image, and when no detection region is detected, mark the current to-be-processed image as not meeting the preset result, and not use the current to-be-processed image for video processing.

[0212] In some embodiments, the video processing apparatus based on artificial intelligence further comprises a model training module, which is configured to, before the target detection model is used to perform target detection on each to-be-processed image in the at least one to-be-processed image to obtain a target detection result, use the initial target detection model to perform target detection on the training sample image set, determine a sample image that does not meet the preset result from the training sample image set; use the sample image that does not meet the preset result as an incremental training sample, obtain a label result of the incremental training sample, and then obtain an incremental training sample set; based on the incremental training sample set and the training sample image set, iteratively train the initial target detection model to obtain the target detection model.

[0213] It should be noted that the above description of the device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects to the method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0214] The embodiment of the present application provides a computer program product or computer program, the computer program product or computer program comprises computer instructions stored in a computer readable storage medium. The processor of the computer equipment reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer equipment executes the artificial intelligence-based video processing method provided by the embodiment of the present application.

[0215] The embodiment of the present application provides a computer readable storage medium storing executable instructions, wherein the executable instructions are stored, when the executable instructions are executed by a processor, the processor will execute the method provided by the embodiment of the present application, for example, the method shown in Figure 3 、 5 、7、8.

[0216] In some embodiments, the computer readable storage medium can be FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc or CD-ROM and the like memory; it can also be various devices including one or any combination of the above memories.

[0217] In some embodiments, the executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or being deployed as modules, components, subroutines or other units suitable for use in a computing environment.

[0218] As an example, the executable instructions can but not necessarily correspond to files in the file system, can be stored in part of a file storing other programs or data, for example, stored in one or more scripts in a hyper text markup language (HTML, Hyper Text Markup Language) document, stored in a single file dedicated to the program in question, or stored in multiple cooperative files (for example, files storing one or more modules, subroutines or code portions).

[0219] As an example, the executable instructions can be deployed to execute on one computing device, or on multiple computing devices located in one place, or on multiple computing devices distributed in multiple places and interconnected through a communication network.

[0220] In summary, by the embodiments of the present application, the video processing device can determine the far and near scenes of an image based on the pre-trained target detection model and the prior knowledge of the far and near scenes, thereby avoiding directly collecting a large number of labeled samples for recognition, avoiding introducing an additional deep learning model to cause the overall calculation time to be too long, and making the effect closer to the definition of the far and near scenes, thereby improving the efficiency and accuracy of the scene recognition result, and further improving the efficiency and accuracy of the video processing.

[0221] The above merely describes the embodiments of the present application, but is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, and improvement within the spirit and scope of the present application shall be included in the protection scope of the present application.

Claims

1. An artificial intelligence-based video processing method, characterized in that, include: Extract frames and perform segmentation processing on the video to be processed to obtain at least one image to be processed; Using the target detection model, performing target detection on the current image to be processed in the at least one image to be processed to obtain at least one detection area corresponding to at least one type of detection object; The target detection model is used to detect the at least one detection area occupied by the at least one type of detection object from the current image to be processed; Determine a subject detection area from the at least one detection area according to size information of each detection area in the at least one detection area; wherein the subject size information of the subject detection area includes: a subject area height and a subject area width; When the at least one type of detection object is a human object, obtaining a size ratio and a first area of ​​the subject detection area according to the height and width of the subject area; When the first area is greater than or equal to a first preset area threshold, determining a scene recognition result of the current image to be processed according to the size ratio; or When the first area is smaller than the first preset area threshold and greater than or equal to the second preset area threshold, determining the scene recognition result of the current image to be processed based on the height of the main body area or the width of the main body area; the second preset area threshold is smaller than the first preset area threshold; or When the first area is smaller than the second preset area threshold, the scene recognition result is determined to be a long-range scene, and the current image to be processed is marked as not meeting the preset result; thereby obtaining a scene recognition result for each of the at least one image to be processed; the scene recognition result represents the distance between the image content and the visual starting point; Intelligent processing of the video to be processed is achieved based on the scene recognition result of each image to be processed.

2. The method of claim 1, wherein, The dimension ratio is the ratio of the height of the main body region to the width of the main body region; The determining of the scene recognition result of the current image to be processed according to the size ratio includes: When the size ratio is greater than a first preset size ratio threshold and less than or equal to a second preset size ratio threshold, the scene recognition result is determined to be a close-up of a face; the second preset size ratio threshold is greater than the first preset size ratio threshold; or When the size ratio is greater than the second preset size ratio threshold and less than or equal to the third preset size ratio threshold, the scene recognition result is determined as a close-up view of a human body; the third preset size ratio threshold is greater than the second preset size ratio threshold; or When the size ratio is greater than the third preset size ratio threshold and less than or equal to the fourth preset size ratio threshold, the scene recognition result is determined to be a human panorama; the fourth preset size ratio threshold is greater than the third preset size ratio threshold; or When the size ratio is greater than the fourth preset size ratio threshold, the scene recognition result is determined as a distant view.

3. The method of claim 2, wherein, The determining of the scene recognition result of the current image to be processed based on the height of the main body area or the width of the main body area includes: when the subject region height or the subject region width is greater than or equal to a preset first edge length threshold, determining the scene recognition result as a full view of a human body; or when the subject region height or the subject region width is less than the preset first edge length threshold and greater than or equal to a preset second edge length threshold, determining whether the size ratio is greater than a second preset size ratio threshold; when the size ratio is greater than or equal to the second preset size ratio threshold, determining the scene recognition result as a long shot; or when the size ratio is less than the second preset size ratio threshold, determining the scene recognition result as a very long shot; or when the subject region height or the subject region width is less than the preset second edge length threshold, determining the scene recognition result as a very long shot, and marking the current to-be-processed image as not meeting the preset result.

4. The method of claim 1, wherein, The at least one type of detection object includes an object object, and the method further includes: when the detection object is the object object, obtaining a second area of the subject detection region according to the subject region height and the subject region width; when the second area is greater than or equal to a preset third area threshold, determining the scene recognition result of the current to-be-processed image as an object close-up; or when the second area is less than the preset third area threshold and greater than or equal to a preset fourth area threshold, determining the scene recognition result of the current to-be-processed image as an object close-up; the preset fourth area threshold is less than the preset third area threshold; or when the second area is less than the preset fourth area threshold and greater than or equal to a second preset area threshold, determining the scene recognition result of the current to-be-processed image as an object full view; the preset fourth area threshold is greater than the second preset area threshold; or when the second area is less than the second preset area threshold, determining the scene recognition result of the current to-be-processed image as an object long shot.

5. The method of claim 1, wherein, After the frame extraction and shot processing of the to-be-processed video to obtain at least one to-be-processed image, the method further includes: when using a target detection model to perform target detection on a current to-be-processed image in the at least one to-be-processed image, if no detection region is detected, marking the current to-be-processed image as not meeting the preset result, and not using the current to-be-processed image for video processing.

6. The method according to any one of claims 1 to 5, characterized in that, Before using the target detection model to perform target detection on each to-be-processed image in the at least one to-be-processed image to obtain a target detection result, the method further includes: using an initial target detection model to perform target detection on a training sample image set, and determining a sample image that does not meet the preset result from the training sample image set; using the sample image that does not meet the preset result as an incremental training sample, obtaining a labeling result of the incremental training sample, and then obtaining an incremental training sample set; based on the incremental training sample set and the training sample image set, iteratively training the initial target detection model to obtain the target detection model.

7. An artificial intelligence-based video processing apparatus, characterized by comprising: including: a video frame extraction module, configured to perform frame extraction and shot processing on a to-be-processed video to obtain at least one to-be-processed image; a target detection model, configured to perform target detection on a current to-be-processed image in the at least one to-be-processed image, to obtain at least one detection region corresponding to at least one type of detection object; the target detection model is configured to detect, from the current to-be-processed image, the at least one detection region occupied by the at least one type of detection object; a scene type identification module, configured to determine a subject detection region from the at least one detection region according to size information of each detection region in the at least one detection region; wherein subject size information of the subject detection region includes a subject region height and a subject region width; when the at least one type of detection object is a person object, a size ratio of the subject detection region and a first area are obtained according to the subject region height and the subject region width; when the first area is greater than or equal to a first preset area threshold, a scene type identification result of the current to-be-processed image is determined according to the size ratio; or, when the first area is less than the first preset area threshold and greater than or equal to a second preset area threshold, a scene type identification result of the current to-be-processed image is determined based on the subject region height or the subject region width; the second preset area threshold is less than the first preset area threshold; or, when the first area is less than the second preset area threshold, the scene type identification result is determined as a long shot, and the current to-be-processed image is marked as not meeting a preset result; and then a scene type identification result of each to-be-processed image in the at least one to-be-processed image is obtained; the scene type identification result represents a distance of image content from a visual starting point; a video processing module, configured to implement intelligent processing on the to-be-processed video based on the scene type identification result of each to-be-processed image.

8. The apparatus of claim 7, wherein, the size ratio is a ratio of the subject region height to the subject region width; and the scene type identification module is specifically configured to: when the size ratio is greater than a first preset size ratio threshold and less than or equal to a second preset size ratio threshold, the scene type identification result is determined as a face close-up; the second preset size ratio threshold is greater than the first preset size ratio threshold; or, when the size ratio is greater than the second preset size ratio threshold and less than or equal to a third preset size ratio threshold, the scene type identification result is determined as a human body close shot; the third preset size ratio threshold is greater than the second preset size ratio threshold; or, when the size ratio is greater than the third preset size ratio threshold and less than or equal to a fourth preset size ratio threshold, the scene type identification result is determined as a human body full shot; the fourth preset size ratio threshold is greater than the third preset size ratio threshold; or, when the size ratio is greater than the fourth preset size ratio threshold, the scene type identification result is determined as a long shot.

9. The apparatus of claim 8, wherein, the scene type identification module is specifically configured to: when the subject region height or the subject region width is greater than or equal to a preset first edge length threshold, the scene type identification result is determined as a human body full shot; or, when the subject region height or the subject region width is less than the preset first edge length threshold and greater than or equal to a preset second edge length threshold, determining whether the size ratio is greater than a second preset size ratio threshold; when the size ratio is greater than or equal to the second preset size ratio threshold, determining the scene recognition result as a long shot; or, when the size ratio is less than the second preset size ratio threshold, determining the scene recognition result as a very long shot; or, when the subject region height or the subject region width is less than the preset second edge length threshold, determining the scene recognition result as a very long shot, and marking the current to-be-processed image as not meeting the preset result.

10. The apparatus of claim 7, wherein, The at least one type of detection object includes an object object; the scene recognition module is further configured to: when the detection object is the object object, obtaining a second area of the subject detection region according to the subject region height and the subject region width; when the second area is greater than or equal to a preset third area threshold, determining the scene recognition result of the current to-be-processed image as an object close-up; or, when the second area is less than the preset third area threshold and greater than or equal to a preset fourth area threshold, determining the scene recognition result of the current to-be-processed image as an object long shot; the preset fourth area threshold is less than the preset third area threshold; or, when the second area is less than the preset fourth area threshold and greater than or equal to a second preset area threshold, determining the scene recognition result of the current to-be-processed image as an object full shot; the preset fourth area threshold is greater than the second preset area threshold; or, when the second area is less than the second preset area threshold, determining the scene recognition result of the current to-be-processed image as an object long shot.

11. The apparatus of claim 7, wherein, The target detection model is further configured to: when the target detection model is used to perform target detection on a current to-be-processed image in the at least one to-be-processed image, and no detection region is detected, marking the current to-be-processed image as not meeting the preset result, and not using the current to-be-processed image for video processing.

12. The apparatus of claim 7, wherein, The apparatus further includes: a model training module configured to, before performing target detection on each to-be-processed image in the at least one to-be-processed image to obtain a target detection result, perform target detection on a training sample image set using an initial target detection model, and determine a sample image that does not meet the preset result from the training sample image set; use the sample image that does not meet the preset result as an incremental training sample, obtain a labeling result of the incremental training sample, and then obtain an incremental training sample set; based on the incremental training sample set and the training sample image set, iteratively train the initial target detection model to obtain the target detection model.

13. An electronic device, comprising: include: a memory configured to store executable instructions; a processor configured to execute the executable instructions stored in the memory, and implement the method of any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that, executable instructions are stored, and when executed by a processor, implement the method of any one of claims 1 to 6.

15. A computer program product comprising computer-executable instructions or a computer program, characterized in that, The computer executable instructions or computer program, when executed by the processor, implement the method of any one of claims 1 to 6.

Images