Video processing method and device

Abstract

The invention discloses a video processing method and device, and relates to the technical field of multimedia. In the embodiment of the invention, the processing equipment discards the image without the moving object, and since the image without the moving object does not need to be stored, calculation and storage resources occupied by the processing equipment for processing the image without the moving object can be saved, and the time for transmitting the image without the moving object can be shortened. The processing equipment stores the image containing the moving object, so that the information of the moving object contained in the video cannot be lost, and the integrity of the information stored by the processing equipment is ensured. The processing device discards the image not containing the moving object and stores the image containing the moving object, so that the duration occupied by video processing is shortened, and the resource utilization rate of the processing device is improved.

Description

Technical Field

[0001] The present application relates to the field of multimedia technology, and in particular to a method and device for video processing. Background Art

[0002] A video is an image sequence including multiple continuous image frames, and one image frame corresponds to one image. The processing device analyzes and processes the multiple continuous image frames contained in the video in the same way to obtain information about the moving objects contained in the video. In the case where only some of the multiple continuous image frames contained in the video contain moving objects, the image frames contained in the video are analyzed and processed in the same way, which occupies computing and storage resources and takes a long time to process, resulting in low efficiency in analyzing the video. Therefore, it is urgently necessary to provide a video processing method that can save computing and storage resources, shorten processing time, and improve the efficiency of analyzing videos. Summary of the invention

[0003] The present application provides a video processing method and device for solving the problem of occupying computing and storage resources due to processing multiple consecutive frames of images in a video in the same manner.

[0004] In a first aspect, the present application provides a method for video processing. The method is performed by a processing device, and the method includes: the processing device determines whether a moving object is included in a first frame image in a video stream. The first frame image is any frame image in the video stream. The moving object is an object whose state changes between the first frame image and the frame image before the first frame image. Based on the judgment result, the processing device discards the image that does not contain the moving object and stores the image that contains the moving object.

[0005] In the present application, the processing device discards images that do not contain moving objects. Since images that do not contain moving objects do not need to be stored, the computing and storage resources occupied by the processing device for processing images that do not contain moving objects can be saved, and the time for transmitting images that do not contain moving objects can be reduced. The processing device stores images that contain moving objects, and the information of moving objects contained in the video will not be lost, thereby ensuring the integrity of the information stored by the processing device. The processing device discards images that do not contain moving objects and stores images that contain moving objects, which reduces the time occupied by processing videos and improves the resource utilization of the processing device.

[0006] In a possible implementation manner, a processing device analyzes an image containing a moving object.

[0007] In the present application, the processing device analyzes images containing moving objects without analyzing images that do not contain moving objects. In the case where a large number of images that do not contain moving objects are included in a video, the computing and storage resources consumed by the processing device for analyzing the large number of images that do not contain moving objects are saved, and the time for analyzing the large number of images that do not contain moving objects is reduced. The time for the processing device to process the video is shortened, the computing and storage resources consumed for processing the video are saved, and the efficiency of video processing is improved.

[0008] In another possible implementation, the video stream includes a first frame image and a second frame image. The second frame image is a previous adjacent frame of the first frame image, and the processing device may use any of the following possible situations to determine whether the first frame image includes a moving object.

[0009] In a first possible scenario, the first frame image includes the first object, and the second frame image does not include the first object, and it is determined that the first frame image includes a moving object.

[0010] In a second possible scenario, the first frame image does not include the second object, and the second frame image includes the second object, and it is determined that the first frame image includes a moving object.

[0011] In a third possible scenario, the first frame image contains the third object, and the second frame image also contains the third object. The third object contained in the first frame image and the third object contained in the second frame image are in different states, and it is determined that the first frame image contains a moving object.

[0012] In another possible implementation, a processing device analyzes an image containing a moving object, including: the processing device determines a first group of associated images containing a first moving object from stored images containing moving objects. The first moving object is any one of the moving objects contained in the stored images. The first group of associated images contains one or more frames of images to be analyzed. The processing device establishes a first tracking chain of one or more frames of images to be analyzed, and analyzes the first group of associated images based on the first tracking chain. The first tracking chain is used to indicate the behavior of the first moving object in the first group of associated images.

[0013] In another possible implementation, after the processing device establishes a first tracking chain of one or more frames of images to be analyzed, the processing device determines a second group of associated images from the stored images and establishes a second tracking chain of the second group of associated images. The second tracking chain is used to indicate: the behavior of the first moving object in the second group of associated images. The second group of associated images is an image containing the first moving object in the stored images. The shooting time of the last frame of the first group of associated images is the first time, the shooting time of the first frame of the second group of associated images is the second time, and the difference between the first time and the second time is greater than or equal to the time difference threshold.

[0014] In the present application, the processing device uses different tracking chains to analyze the same moving object when the time difference between two adjacent frames is greater than the time difference threshold. This can avoid inaccurate analysis results caused by omissions due to the reappearance of the same moving object after the interval is greater than the time difference threshold, thereby improving the accuracy of video processing.

[0015] In a second aspect, the present application provides a video processing device, which includes modules for executing the video processing method in the first aspect or any possible design of the first aspect.

[0016] In a third aspect, the present application provides an electronic device. The electronic device includes at least one processor and a memory, the memory being used to store a set of computer instructions; when the processor executes the set of computer instructions as an execution device in the first aspect or any possible implementation of the first aspect, the operation steps of the video processing method in the first aspect or any possible implementation of the first aspect are executed.

[0017] In a fourth aspect, the present application provides a video processing system, which includes at least one electronic device described in the fourth aspect.

[0018] In a fifth aspect, the application provides a chip. The chip includes: a processor and a power supply circuit. The power supply circuit is used to supply power to the processor, and the processor is used to execute the operation steps of the video processing method in the first aspect or any possible design of the first aspect.

[0019] In a sixth aspect, the present application provides a computer-readable storage medium. The computer-readable storage medium includes: computer software instructions; when the computer software instructions are executed in an electronic device, the electronic device executes the operation steps of the video processing method in the first aspect or any possible design of the first aspect.

[0020] In a seventh aspect, the present application provides a computer program product. When the computer program product is run on a computer, the computing device is caused to execute the operation steps of the video processing method in the first aspect or any possible implementation of the first aspect.

[0021] The beneficial effects of the second to seventh aspects above can refer to the description of the first aspect or any implementation of the first aspect, and will not be repeated here. Based on the implementations provided by the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of the structure of a video processing system provided in this application;

[0023] Figure 2A A flowchart of a video processing method provided in this application;

[0024] Figure 2B A flowchart of another video processing method provided by the present application;

[0025] Figure 2C A schematic diagram of establishing the first target tracking chain provided in this application;

[0026] Figure 2D A schematic diagram of establishing the second target tracking chain provided by this application;

[0027] Figure 2E A schematic diagram of establishing the third target tracking chain provided for this application;

[0028] Figure 2F A schematic diagram of establishing the fourth target tracking chain provided by this application;

[0029] Figure 2G A schematic diagram of establishing the fifth target tracking chain provided by this application;

[0030] Figure 2H A flowchart of a video processing method is provided;

[0031] Figure 3 A schematic diagram of a process flow of a cutout algorithm provided in this application;

[0032] Figure 4 A schematic diagram of the structure of a video processing device provided in this application;

[0033] Figure 5 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0034] The present application provides a method for video processing, in which a processing device discards images that do not contain moving objects. Since images that do not contain moving objects do not need to be stored, the computing and storage resources occupied by the processing device for processing images that do not contain moving objects can be saved, and the time for transmitting images that do not contain moving objects can be reduced. The processing device stores images that contain moving objects, and the information of moving objects contained in the video will not be lost, thereby ensuring the integrity of the information stored by the processing device. The processing device discards images that do not contain moving objects and stores images that contain moving objects, thereby reducing the time occupied by processing videos and improving the resource utilization of the processing device.

[0035] In order to make the description of the following embodiments clear and concise, a brief introduction to relevant terms is first given.

[0036] Video contains multiple continuous images. When the continuous images change at more than 24 frames per second, according to the principle of visual persistence, the human eye cannot distinguish a single static image, and the multiple images that appear smooth and continuous are called video.

[0037] Figure 1 This is a schematic diagram of the structure of a video processing system provided by the present application. The video processing system 100 may include: a video acquisition device 110, a video processing device 120 and a network. The network may realize the function of video transmission, and the network may include one or more network devices, which may be devices with data forwarding functions such as routers or switches.

[0038] The video acquisition device 110 is used to acquire video. The video acquisition device 110 may be a device that directly acquires video, a computing device that acquires video through computing, or a memory that stores video, which will be described below.

[0039] In the first possible scenario, when the video acquisition device 110 is a device that directly acquires video, the video acquisition device 110 may be a camera device for road monitoring (e.g. Figure 1 The terminal device 111 shown in the figure may also be a terminal device with a video acquisition function. The terminal device may be, but is not limited to, a user equipment (UE), a mobile station (MS), a mobile terminal (MT), etc. The terminal device may be a mobile phone (such as Figure 1 The terminal device 112 shown in FIG. 1 ), a tablet computer (such as Figure 1 ), a computer with wireless transceiver function, a virtual reality (VR) terminal device (such as Figure 1 The terminal device 114 shown in the figure), augmented reality (AR) terminal device, wireless terminal in industrial control, wireless terminal in self driving, wireless terminal in smart city, wireless terminal in smart home, etc.

[0040] In the second possible scenario, when the video acquisition device 110 is a computing device that acquires video by computing, the video acquisition device 110 may be a decoding device (such as a decoder) that decodes a video stream, or a computer (such as a computer) that synthesizes a video. Figure 1 115).

[0041] In a third possible scenario, when the video acquisition device 110 is a storage device for storing video, the video acquisition device 110 may be a memory. The memory may be a cache, a solid state drive (SSD), a hard disk drive (HDD), a storage class memory (SCM), a memory, or other storage medium, exemplarily, a storage particle storing a specific number of bits: a single level cell (SLC), a multi-level cell (MLC), a triple-level cell (TLC), or a quad level cell (QLC), etc.

[0042] The video processing device 120 is used to analyze the video and obtain the information of the moving object contained in the video. The video processing device 120 can be, but is not limited to, a server 121, a data center 122, a computer 123, a computer cluster 124, and the like.

[0043] In a first possible scenario, the video processing device 120 is a server 121. The server 121 may be arranged on a terminal side or on a cloud side.

[0044] In a second possible scenario, the video processing device 120 is a data center 122. The data center 122 may include one or more physical devices with video processing functions, such as a server, a mobile phone, a tablet computer, and the like.

[0045] In a third possible scenario, the video processing device 120 is a computer 123 , which may include a memory, a processor, and one or more interfaces.

[0046] The memory included in the computer 123 can store intermediate data generated during video processing, such as storing multiple frames of images containing moving objects generated during video preprocessing. For the relevant content of the memory, please refer to the above description, which will not be repeated here.

[0047] The processor included in the computer 123 processes the image including the moving object in the video to obtain the information of the moving object contained in the image. The processor may include one or more processor cores. The processor may be a very large-scale integrated circuit. An operating system and other software programs are installed in the processor, so that the processor can access the memory and various peripheral component interconnect (Peripheral Component Interconnect express, PCIe) devices. It is understood that in this embodiment, the core in the processor may be a central processing unit (CPU) or other specific integrated circuits (application specific integrated circuit, ASIC). The processor may also be other general-purpose processors, digital signal processors (digital signal processing, DSP), application-specific integrated circuits (application-specific integrated circuit, ASIC), field programmable gate arrays (field programmable gate array, FPGA), graphics processors (graphics processing unit, GPU), AI chips, systems on chip (system-on-a-chip, SoC) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. In practical applications, the processing device may also include multiple processors.

[0048] The computer 123 includes one or more interfaces that can receive the video sent by the video acquisition device 110 .

[0049] In a fourth possible scenario, the video processing device 120 is a computer cluster 124. The computer cluster 124 refers to a collection of computers connected via a local area network or the Internet (eg, Figure 1Computers 1241 to 1244 are shown in the figure. Each computer can obtain information about the moving object contained in the image in the video according to the received video. Exemplarily, the computer cluster 124 can have a rack, and the rack can establish communication for multiple computers contained in the computer cluster 124 through a wired connection, such as a universal serial bus (USB) or a PCIe high-speed bus. The computer cluster 124 is usually used to perform large tasks (also called jobs). The jobs here are usually large jobs that require more resources to be processed in parallel. This embodiment does not limit the nature and number of jobs. A job may include a task of clipping a moving object contained in an image generated during video processing. The task can be assigned to multiple computing resources for execution. Most tasks are executed concurrently or in parallel, while some tasks need to rely on data generated by other tasks. Each computing device in the computer cluster 124 uses the same hardware and the same operating system; different hardware and different operating systems can also be used in the computers of the computer cluster 124 according to business needs.

[0050] The video processing system 100 may further include: a storage device for storing information of moving objects. The storage device may be a memory. For the relevant contents of the memory, please refer to the relevant description above, which will not be repeated here.

[0051] The video processing system usually analyzes and processes each image contained in the video separately to obtain information about the moving objects contained in the video. In the case where there are a large number of images that do not contain moving objects in the multiple frames of images contained in the video, it is still necessary to process and analyze the images that do not contain moving objects, which wastes computing and storage resources, increases the time required to process and analyze the video, and reduces the efficiency of video processing and analysis. To this end, the present application provides a method for video processing. Figure 2A A schematic diagram of a video processing method provided in the present application, which can be executed by a processing device, which can be Figure 1 The computer. Figure 2A As shown, the method includes the following steps S210 to S220.

[0052] Step S210: The processing device determines whether the first frame image in the video stream contains a moving object.

[0053] Among them, the first frame image is any frame image in the video stream. The moving object is an object whose state changes in the first frame image and the frame image before the first frame image. The object may refer to the content contained in the image, such as vehicles, pedestrians, etc. contained in any frame image in the road monitoring video stream. The present application does not limit the type and number of objects. For example, the object can also be a person, object, pet, etc. in the elevator monitoring video stream. The change in the state of the object may refer to: the object changes from nothing to something, from something to nothing, the position of the object, the angle of rotation of the object, etc. For example, the first frame image includes the first object and the frame image before the first frame image does not include the first object, the first frame image does not include the second object and the frame image before the first frame image includes the second object, and the third object included in the first frame image and the frame image before the first frame image is in different positions, rotation angles, etc.

[0054] The video stream includes multiple frames of continuous images, such as a video including images 1 to n of continuous shooting scene 1. The video stream can be a video stream of road monitoring, a video stream of home monitoring, a video stream of parking lot monitoring, a video stream of elevator monitoring, a video stream of building monitoring, etc., which are shot when the position and orientation of the camera device (such as a camera) are fixed. According to the different sources of the video stream, it can be divided into the following three different situations, which are explained below.

[0055] Scenario 1: The video stream is directly acquired by a video acquisition device, such as a road monitoring device (such as a camera) directly acquiring a road monitoring video stream.

[0056] Case ②: The video stream is calculated by a computing device, such as a decoder decoding the original bit stream to obtain a road monitoring video stream.

[0057] Scenario ③: The video stream is transmitted by other devices, such as a mobile phone transmitting the road monitoring video stream it stores.

[0058] In one possible scenario, the processing device may use any of the following possible scenarios to determine whether any frame image in the video stream is an image including a moving object. The processing device may use an optical flow model to determine whether any frame image in the video stream is an image including a moving object. The optical flow model may include, but is not limited to, a neural network optical flow estimation algorithm (cnns for optical flow using pyramid, warping, and cost volume, PWC-Net), a recursive all-pairs field transforms for optical flow (RAFT) optical flow calculation model, etc.

[0059] Taking an image containing an object as an example, a possible situation in which the processing device determines that the image is an image containing a moving object is described.

[0060] Scenario ①: The image includes a first object, and the previous adjacent frame of the image does not include the first object. The processing device determines that the first object is a moving object and determines that the image is an image containing a moving object. For example, a road monitoring video stream includes image frame 1 and image frame 2, and image frame 1 is the previous adjacent frame of image frame 2. Image frame 2 includes vehicle 1, and image frame 1 does not include vehicle 1. The processing device determines that vehicle 1 is a moving object and image frame 2 is an image containing a moving object.

[0061] Scenario ②: The image does not include the second object, and the previous adjacent frame of the image includes the second object. The processing device determines that the second object is a moving object and determines that the image is an image containing a moving object. For example, a road monitoring video stream includes image frame 1 and image frame 2, and image frame 1 is the previous adjacent frame of image frame 2. Image frame 2 does not include vehicle 1, and image frame 1 includes vehicle 1. The processing device determines that vehicle 1 is a moving object and image frame 2 is an image containing a moving object.

[0062] Scenario ③: The image includes a third object, the previous adjacent frame of the image includes the third object, the image including the third object and the previous adjacent frame of the image including the third object are in different states, and the processing device determines that the third object is a moving object and determines that the image is an image containing a moving object. The state of the object may include, but is not limited to: the rotation angle of the object, the position of the object, etc. For example, a road monitoring video stream includes image frame 1 and image frame 2, and image frame 1 is the previous adjacent frame of image frame 2. Image frame 1 and image frame 2 both contain vehicle 1, vehicle 1 of image frame 1 is located at position 1, and vehicle 2 of image frame 2 is located at position 2, and the processing device determines that image frame 2 is an image containing a moving object.

[0063] When an image contains multiple objects, the processing device uses the above-mentioned situations ① to ③ to judge each of the multiple objects contained in the image. When the image appears in any of the situations described above, the processing device confirms that the image is an image containing a moving object.

[0064] Step S220: The processing device discards the images that do not contain the moving object and stores the images that contain the moving object based on the judgment result.

[0065] The processing device may store all images containing moving objects in the same image set for storage, or may store images containing moving objects in multiple image sets for storage. In the present application, an example is given in which the processing device stores images containing moving objects in the same image set for storage, such as the processing device stores images containing moving objects in image set 1. In the present application, each frame of the image containing a moving object stored in image set 1 may also be referred to as an image to be analyzed. Depending on whether the image frame is the first frame of the video stream, the processing method of the processing device is also different, which is explained in different cases below.

[0066] Case ①, for the first frame image in the video stream.

[0067] The processing device does not judge the first frame image and directly stores the first frame image. For example, if the road monitoring video stream includes image frame 1, image frame 2 to image frame n, the processing device directly stores image frame 1.

[0068] Case ②, for non-first frame images in the video stream.

[0069] The processing device discards images that do not contain moving objects and stores images that contain moving objects. In the present application, the number and type of moving objects contained in the image are not limited, and the image can contain one or more moving objects. For example, a road monitoring video stream includes image frames 1 to n. Image frame 2 contains no moving objects, image frame 3 contains one moving object (such as a vehicle), and image frame 4 contains two moving objects (such as a vehicle and a person). The processing device discards image frame 2 and stores image frame 3 and image frame 4. In the present application, the processing device discards images that do not contain moving objects. Since images that do not contain moving objects do not need to be stored, the computing and storage resources occupied by the processing device for processing images that do not contain moving objects can be saved, and the time for transmitting images that do not contain moving objects can be reduced. The processing device stores images that contain moving objects, and the information of the moving objects contained in the video will not be lost, thereby ensuring the integrity of the information stored by the processing device. Discarding images that do not contain moving objects and storing images that contain moving objects reduces the time occupied by processing videos and improves the resource utilization of the processing device.

[0070] After the processing device discards the image frames that do not contain the moving object in the video and stores the image frames that contain the moving object, as shown in FIG. Figure 2B As shown, the processing device may also perform step S230 to analyze the stored image containing the moving object. Figure 2B A flowchart of another video processing method provided in this application.

[0071] Step S230: The processing device analyzes the image containing the moving object.

[0072] In one possible scenario, a processing device determines a first group of associated images including a first moving object from stored images. The first moving object is any one of the moving objects included in the stored images. The first group of associated images includes one or more frames of images to be analyzed. The processing device establishes a first tracking chain of the first group of associated images, and analyzes the first moving object included in the first group of associated images according to the first tracking chain. The first tracking chain indicates: the behavior of the first moving object in the first group of associated images.

[0073] The following uses ① to describe the process of the processing device establishing a tracking chain, and uses ② to describe the process of the processing device analyzing a moving object.

[0074] ①. The process of establishing a tracking chain for processing equipment.

[0075] According to whether the first moving object appears in the continuous images to be analyzed, there are two different situations as follows: The following takes the case where each frame of the images to be analyzed contains only one identical moving object as an example, and explains them respectively.

[0076] Case ①: The first moving object appears in the continuous images to be analyzed.

[0077] In this case, the first group of associated images includes at least one frame of images to be analyzed that appear continuously. Figure 2C The first target tracking chain establishment schematic diagram provided in this application is as follows: Figure 2C As shown, the processing device establishes a first tracking chain corresponding to the first moving object based on the first frame image of the first group of associated images, and the processing device adds other frame images in the first group of associated images to the first tracking chain.

[0078] Exemplarily, the associated image group 1 includes continuous images 1 to n. The shooting times of images 1 to n are t1 to tn respectively. Images 1 to n all include object 1, and the objects 1 included in images 1 to n are in different states. The process of the processing device establishing the tracking chain 1 corresponding to the object 1 includes the following steps ① to ②.

[0079] Step 1: For the first frame image (image 1).

[0080] The processing device obtains the shooting time t1 of the image 1 , and establishes a tracking chain 1 corresponding to the object 1 .

[0081] Step ②: For other frame images (image 2 to image n).

[0082] The processing device detects and compares the state of the object 1 in the image i and the image i-1, where i is equal to any number from 2 to n. When the state of the object 1 in the image i and the image i-1 is different, the processing device obtains the shooting time ti of the image i and adds the image i to the tracking chain 1. At this time, the tracking chain 1 indicates the behavior of the object 1 in the associated image group 1 (image 1 to image n).

[0083] Case ②: The first moving object appears in discontinuous images to be analyzed.

[0084] In this case, according to the different intervals between discontinuous images to be analyzed, there are two possible examples as follows.

[0085] In a first possible example, the time interval between discontinuous images to be analyzed is smaller than a set threshold.

[0086] In this case, the first group of associated images includes at least one frame of discontinuous image to be analyzed. Figure 2D A schematic diagram of the second target tracking chain provided by this application is shown in FIG. Figure 2D As shown, at this time, the processing device establishes the same tracking chain for the moving object in each frame of the image to be analyzed contained in the first group of associated images. For the relevant content of establishing the tracking chain, please refer to the relevant description above and will not be repeated here.

[0087] Exemplarily, the associated image group 1 includes image 1, image m to image n. Image 1 was shot at time t1, image m was shot at time tm, and image n was shot at time tn. The time difference between tm and t1 is less than the set threshold t0. At this time, the processing device establishes a tracking chain 1 for image 1, image m to image n.

[0088] In a second possible example, the time interval between discontinuous images to be analyzed is greater than or equal to a set threshold.

[0089] Figure 2E The schematic diagram of the third target tracking chain provided by this application is as follows: Figure 2EAs shown, in this case, after the processing device establishes the first tracking chain of the first moving object, the processing device confirms the first moving object from the stored images and establishes the second tracking chain of the first moving object. The second tracking chain is used to indicate: the behavior of the first moving object in the second group of associated images. The second group of associated images is the stored images to be analyzed that contain the first moving object. The shooting time of the last frame image of the first group of associated images is the first time, and the shooting time of the first frame image of the second group of associated images is the second time, and the difference between the first time and the second time is greater than or equal to the time difference threshold. The time difference threshold can be pre-set or can be set by the user in real time as needed, and this application is not limited to this. The above process is illustrated below by example.

[0090] Exemplarily, image set 1 includes continuous images 1 to image k, image k+1 to image m-1, and image m to image n. Image k was shot at time tk, and image m was shot at time tm. The states of object 1 contained in images 1 to image k are different, and the states of object 1 contained in images m to image n are different. Images k+1 to image m-1 may not contain object 1, or may contain object 1. In the case where images k+1 to image m-1 contain object 1, the states of object 1 contained in images k+1 to image m-1 are the same and different from the states of object 1 contained in images k and image m. The difference between tk and tm is greater than or equal to the time difference threshold t0. The process of the processing device establishing a tracking chain for object 1 includes the following steps ① to ②.

[0091] Step 1: For image 1 to image k.

[0092] The processing device uses the tracking chain establishment method described above to establish a tracking chain 10 corresponding to the object 1 included in the images 1 to k.

[0093] Step ②: From image m to image n.

[0094] The processing device adopts the tracking chain establishment method described above to establish a tracking chain 11 corresponding to the object 1 included in the image m to the image n.

[0095] In one possible scenario, Figure 2F A schematic diagram of the fourth target tracking chain provided by this application is shown in FIG. Figure 2F As shown, after the processing device establishes the first tracking chain of the first moving object, the processing device detects that the state of the first moving object included in the third group of images is the same as the state of the first moving object included in the last frame of the first group of associated images, and the processing device ends the first tracking chain. The last frame of the first group of associated images is the preceding adjacent frame of the first frame of the third group of images. The number of images in the third group of images is greater than or equal to the set frame number interval threshold.

[0096] Exemplarily, image set 1 includes continuous images 1 to n. Images 1 to n include object 1, and images 1 to m-1 include object 1 in different states, and images m-1 to n include object 1 in the same state. Images 1 to m-1 form associated image group 1, and images m to n form image group 3. The number of images between image n and image m is greater than or equal to a frame number interval threshold x. At this time, the processing device establishes a tracking chain 1 for object 1 included in images 1 to m-1, and the processing device ends tracking chain 1 at image n.

[0097] ②. The processing device analyzes the moving object based on the tracking chain.

[0098] The processing device analyzes the first moving object, including: the processing device performs behavior analysis on the first object, and the processing device performs attribute analysis on the first object, which are respectively described below.

[0099] Scenario ①: The processing device performs behavioral analysis on the first object.

[0100] In this case, the processing device obtains each frame of the image to be analyzed contained in the first tracking chain, and executes a cutout algorithm on each frame of the image to be analyzed to obtain the first object contained in each frame of the image to be analyzed. The processing device performs behavior analysis based on the obtained first object. For more information about the cutout algorithm, please refer to Figure 3 For the relevant content of behavior analysis, please refer to the general technology, which will not be repeated here.

[0101] Scenario ②: The processing device performs attribute analysis on the first object.

[0102] In this case, the processing device obtains each frame of the image to be analyzed contained in the first tracking chain, and executes a cutout algorithm on each frame of the image to be analyzed to obtain the first object contained in each frame of the image to be analyzed. The processing device compares the first object corresponding to each frame of the image to be analyzed to obtain the first object with the most content. The processing device performs attribute analysis on the first object with the most content obtained. For more information about the cutout algorithm, please refer to Figure 3 For the related content of attribute analysis, please refer to the general technology, which will not be repeated here.

[0103] The above description takes the example that each frame of the image to be analyzed contains a moving object to illustrate how the processing device processes the moving object in each frame of the image to be analyzed. In some possible situations, Figure 2G A schematic diagram of the fifth target tracking chain provided by this application is shown in FIG. Figure 2GAs shown, each frame of the image to be analyzed contained in the image set 1 may contain multiple moving objects. In this case, the processing device can process each of the multiple moving objects one by one using the method described above, which will not be repeated here.

[0104] In this application, the processing device discards images that do not contain moving objects in the video and stores images that contain moving objects in the video before processing and analyzing the video. Figure 2H As shown, Figure 2H This is a flow chart of a video processing method. Compared with the processing device analyzing each frame of the video frame by frame, the present application does not need to process and analyze images that do not contain moving objects, but only needs to process and analyze images that contain moving objects. This can save the computing and storage resources of the processing device for processing and analyzing images that do not contain moving objects, and reduce the time for processing and analyzing images that do not contain moving objects, thereby improving the efficiency of the processing device in processing videos.

[0105] Combination of the above Figure 2A and Figure 2H The video processing method provided by the present application is described below. Figure 3 The cutout algorithm provided in this application is specifically described. Figure 3 A schematic diagram of a process flow of a cutout algorithm provided in this application, the method is executed by a processing device, such as Figure 3 As shown, the method includes the following steps S310 to S340.

[0106] Step S310: The processing device performs target detection on the image to be analyzed to obtain a moving object contained in the image to be analyzed.

[0107] The target detection algorithm adopted by the processing device may include but is not limited to: selective search algorithm, deformable part model algorithm, two-stage algorithm, region convolutional neural network (regions with CNN features, r-cnn) algorithm, spatial pyramid pooling convolutional networks (spp-net) algorithm, fast R-CNN (fastR-CNN) algorithm. For the relevant content of the target detection algorithm, please refer to the description of the general technology, which will not be repeated here.

[0108] After obtaining the moving object contained in the image to be analyzed, the processing device may perform any one or more of the following steps S320 to S340.

[0109] Step S320: The processing device extracts features of the moving object.

[0110] The processing device uses a feature extraction algorithm to extract features from the detected moving object to obtain the content contained in the moving object. The feature extraction algorithm used by the processing device may include but is not limited to: histogram of oriented gradient (HOG), local binary pattern (LBP), HAAR, scale-invariant feature transform (SIFT), Oriented Fast and Rotated Brief (ORB) and other feature extraction algorithms.

[0111] Step S330: The processing device performs a quality score on the moving object.

[0112] The processing device uses a quality scoring algorithm to score the moving object based on the extracted features of the moving object. The quality scoring algorithm used by the processing device may include but is not limited to: a blind / referenceless image spatial quality evaluator (BRISQUE), a neural image assessment algorithm (NIMA), and a learning from rankings for no-reference image quality assessment (RankIQA). For the content of the quality scoring algorithm, please refer to the description of the general technology, which will not be repeated here.

[0113] Step S340: The processing device tracks the moving object.

[0114] The processing device tracks the moving object using a target tracking algorithm based on the extracted features of the moving object. The target tracking algorithm used by the processing device may include, but is not limited to, a sample online realtime tracking (sort) algorithm, a deep sample online realtime tracking (deepsort) and an intersection over union (iou) algorithm. For the target tracking algorithm, please refer to the description of the general technology, which will not be repeated here.

[0115] Combination of the above Figures 2A to 3 The video processing method provided by the present application is described below. Figure 4 The present invention describes a video processing device. Figure 4 A schematic diagram of the structure of a video processing device provided in this application, such as Figure 4 As shown, the video processing device 400 includes: a judgment module 410, a storage module 420 and an analysis module 430.

[0116] The judgment module 410 judges whether the first frame image in the video stream contains a moving object. The first frame image is any frame image in the video stream. The moving object is an object whose state changes between the first frame image and the frame image before the first frame image. The storage module 420 discards the image that does not contain the moving object and stores the image that contains the moving object based on the judgment result of the judgment module 410.

[0117] In a possible scenario, the analysis module 430 analyzes the image containing the moving object stored in the storage module 420 .

[0118] In one possible scenario, the analysis module 430 is specifically used to: determine a first group of associated images containing a first moving object from stored images containing moving objects, and establish a first tracking chain for the first group of associated images. The first moving object is any one of the moving objects contained in the stored images. The first group of associated images contains one or more frames of images to be analyzed. The first tracking chain is used to indicate the behavior of the first moving object in the first group of associated images. The analysis module 430 is also specifically used to: establish a first tracking chain based on one or more frames of images to be analyzed, and analyze the first group of associated images based on the first tracking chain.

[0119] In a possible scenario, after the analysis module 430 establishes the first tracking chain of the first moving object, the analysis module 430 is further specifically used to: determine a second group of associated images containing the first moving object from the stored images, and establish a second tracking chain of the second group of associated images. The second tracking chain is used to indicate: the behavior of the first moving object in the second group of associated images. The shooting time of the last frame of the first group of associated images is the first time, the shooting time of the first frame of the second group of associated images is the second time, and the difference between the first time and the second time is greater than or equal to the set time difference threshold.

[0120] In a possible scenario, any one or more of the following possible scenarios occur, and the determination module 420 determines that the image is an image containing a moving object.

[0121] Scenario ①: the image includes a first moving object, and the previous adjacent frame of the image does not include the first moving object, and the judgment module 420 determines that the image is an image to be analyzed.

[0122] Case ②: the image does not include the second moving object, and the previous adjacent frame of the image includes the second moving object, and the judgment module 420 determines that the image is an image to be analyzed.

[0123] Case ③: the image includes a third moving object, the previous adjacent frame of the image includes the third moving object, the image including the third moving object and the previous adjacent frame of the image including the third moving object are in different states, and the judgment module 420 determines that the image is an image to be analyzed.

[0124] Combination of the above Figure 4 The video processing device provided by the present application is described below. Figure 5 An electronic device provided by the present application is described. Figure 5 A schematic diagram of the structure of an electronic device provided in this application, such as Figure 5 As shown, the electronic device 500 includes: a memory 510 and at least one processor 520. The processor 520 can implement Figures 2A to 3 The memory 510 is used to store software instructions corresponding to the video processing method described above.

[0125] As an optional implementation, in hardware implementation, the electronic device 500 may refer to a chip or a chip system encapsulating one or more processors 520 .

[0126] For example, when the electronic device 500 is used to implement the method steps in the above embodiment, the processor 520 included in the electronic device 500 executes the steps of the above method and possible sub-steps thereof.

[0127] In an optional scenario, the electronic device 500 may further include a communication interface 530, which may be used to send and receive data.

[0128] In the embodiment of the present application, the communication interface 530, the processor 520 and the memory 510 can be connected via a bus 540, and the bus 540 can be divided into an address bus, a data bus, a control bus, etc. The bus 540 can be a peripheral component interconnect express (PCIe) bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a computer express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc.

[0129] The electronic device 500 can also implement Figure 4 The functions of the video processing device shown are not described in detail here.

[0130] The method steps in the embodiments of the present application can also be implemented by the processor 520 executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor 520 so that the processor 520 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 520. The processor 520 and the storage medium can be located in an ASIC. In addition, the ASIC can be located in an electronic device. Of course, the processor 520 and the storage medium can also exist in an electronic device as discrete components.

[0131] The present application also provides a chip, which includes: a processor and a power supply circuit. The power supply circuit is used to supply power to the processor. The processor is used to implement Figures 2A to 3 A method of video processing is described.

[0132] In one possible design, the chip also includes a memory for storing program instructions and / or data.

[0133] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD).

[0134] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A video processing method, characterized in that: The method comprises: For a first frame image in a video stream, determining whether the first frame image contains a moving object, the first frame image being any frame image in the video stream, and the moving object being an object whose state changes between the first frame image and a frame image before the first frame image; Based on the determination result, images not containing moving objects are discarded, and images containing moving objects are stored.

2. The method according to claim 1, characterized in that The method further comprises: The image containing the moving object is analyzed.

3. The method according to claim 1 or 2, characterized in that: The video stream includes the first frame image and the second frame image, wherein the second frame image is a previous adjacent frame of the first frame image. The determining whether the first frame image contains a moving object comprises: The first frame image includes a first object, and the second frame image does not include the first object, and determining that the first frame image includes a moving object; And / or, the first frame image does not include the second object, and the second frame image includes the second object, and it is determined that the first frame image includes a moving object; And / or, the first frame image contains a third object, the second frame image contains the third object, the third object contained in the first frame image and the third object contained in the second frame image are in different states, and it is determined that the first frame image contains a moving object.

4. The method according to claim 2 or 3, characterized in that: The analyzing the image containing the moving object comprises: Determining a first group of associated images including a first moving object from the stored images including moving objects; the first moving object is any one of the moving objects included in the stored images, and the first group of associated images includes one or more frames of images to be analyzed; A first tracking chain of the one or more frames of images to be analyzed is established, and the first group of associated images is analyzed based on the first tracking chain, where the first tracking chain is used to indicate the behavior of the first moving object in the first group of associated images.

5. The method according to claim 4, characterized in that After establishing the first tracking chain of the one or more frames of images to be analyzed, the method further includes: A second group of associated images including a first moving object is determined from the stored images, and a second tracking chain of the second group of associated images is established; the second tracking chain is used to indicate: the behavior of the first moving object in the second group of associated images, the shooting time of the last frame of the first group of associated images is a first time, the shooting time of the first frame of the second group of associated images is a second time, and the difference between the first time and the second time is greater than or equal to a time difference threshold.

6. A video processing device, characterized in that: The device comprises: A judging module, configured to: for a first frame image in a video stream, judge whether the first frame image contains a moving object, wherein the first frame image is any frame image in the video stream, and the moving object is an object whose state changes between the first frame image and a frame image before the first frame image; The storage module is used to: based on the judgment result, discard the image that does not contain the moving object and store the image that contains the moving object.

7. The device according to claim 6, characterized in that The device also includes: The analysis module is used to analyze the image containing the moving object.

8. The device according to claim 6 or 7, characterized in that The video stream includes the first frame image and the second frame image, wherein the second frame image is a previous adjacent frame of the first frame image. The judgment module is specifically used to: if the first frame image includes a first object and the second frame image does not include the first object, determine that the first frame image includes a moving object; And / or, the first frame image does not include the second object, and the second frame image includes the second object, and it is determined that the first frame image includes a moving object; And / or, the first frame image contains a third object, the second frame image contains the third object, the third object contained in the first frame image and the third object contained in the second frame image are in different states, and it is determined that the first frame image contains a moving object.

9. The device according to claim 7 or 8, characterized in that The analysis module is specifically used to: determine a first group of associated images containing a first moving object from the stored images containing moving objects; the first moving object is any one of the moving objects contained in the stored images, and the first group of associated images contains one or more frames of images to be analyzed; The analysis module is further specifically used to: establish a first tracking chain for the one or more frames of images to be analyzed, and analyze the first group of associated images based on the first tracking chain, wherein the first tracking chain is used to indicate the behavior of the first moving object in the first group of associated images.

10. The device according to claim 9, characterized in that The analysis module is also used to: determine a second group of associated images containing a first moving object from the stored images, and establish a second tracking chain of the second group of associated images; the second tracking chain is used to indicate: the behavior of the first moving object in the second group of associated images, the shooting time of the last frame of the first group of associated images is a first time, the shooting time of the first frame of the second group of associated images is a second time, and the difference between the first time and the second time is greater than or equal to a time difference threshold.

Images