Video Processing

Abstract

Apparatus, methods and computer program products related to video processing are disclosed. The apparatus may include means for receiving data representing a first frame of video content including a plurality of frames and, in the first frame, for at least one object in the first frame, determining an object type and position. The apparatus may also include means for determining a number of frames N to skip based on a comparison of the type and position of the object in the first frame with the type and position of one or more objects in one or more previous frames and for providing the N+1th frame instead of the skipped frame to means for applying the frame to an image model.

Description

Technical Field

[0001] Exemplary embodiments relate to video processing, such as processing of video frames for provision to computational analysis models. Background Art

[0002] Machine or computer vision is an interesting and important aspect of many emerging applications. Computer vision may involve receiving video frames (which may, but need not, be real-time video frames) and applying them to a computational analysis model that may, for example, identify specific objects or situations based on the received video, and may even make certain inferences and / or decisions based on the analysis. Summary of the invention

[0003] The scope of protection sought for the various embodiments of the invention is set out by the independent claims.

[0004] The embodiments and features described in this specification that do not fall within the scope of the independent claims, if any, are to be construed as examples useful for understanding the various embodiments of the present invention.

[0005] According to a first aspect, a device is provided, comprising a component for performing the following operations: receiving data of a first frame representing video content including multiple frames; in the first frame, determining an object type and a position for at least one object in the first frame; determining a number of frames N to skip based on a comparison of the type and position of the object in the first frame with the type and position of one or more objects in one or more previous frames; and providing the N+1th frame instead of the skipped frame to a component for applying the frame to an image model.

[0006] Determining object types and positions in a first frame may include identifying a plurality of objects in the first frame, determining, for each object, the object type and position in the first frame, and wherein determining the number of frames N to skip is based on a comparison of the corresponding types and positions of all objects in the first frame with the types and positions of one or more objects in one or more previous frames.

[0007] The component may be further configured to: receive a subsequent frame of the plurality of frames of video content, and repeat the determining operation and the providing operation for the subsequent frame.

[0008] The component may be configured to determine the number of frames N to skip using a policy model that receives as input a state parameter S representing the respective type and position of one or more objects in the first frame or any subsequent frame.

[0009] The policy model may be a pre-trained policy model.

[0010] The policy model can be pre-trained using reinforcement learning methods.

[0011] The policy model can be trained by using a reinforcement learning method and an object detection model, wherein the reinforcement learning method takes multiple training images as input and the object detection model is used to determine key frames in the multiple training images, wherein the reinforcement learning method uses a reward function that rewards an increase in the number of skipped frames N for a given state while penalizing the skipping of the determined key frames.

[0012] The object detection model may determine key frames as frames where new objects appear or object transitions occur.

[0013] The image model processing means may be remote from the device.

[0014] The image model processing component may be a computational model for analyzing a provided frame and for generating an inference output.

[0015] A means as defined herein may include: at least one processor; and at least one memory connected to the at least one processor.

[0016] According to another aspect, a method may be provided, comprising: receiving data of a first frame representing video content comprising multiple frames; in the first frame, determining an object type and position for at least one object in the first frame; determining a number of frames N to skip based on a comparison of the type and position of the object in the first frame with the type and position of one or more objects in one or more previous frames; and providing the N+1th frame instead of the skipped frame to a visual model processor.

[0017] According to another aspect, a non-transitory computer-readable medium may be provided, comprising program instructions stored thereon for executing a method, the method comprising: receiving data representing a first frame of video content comprising multiple frames; in the first frame, determining an object type and position for at least one object in the first frame; determining a number of frames N to skip based on a comparison of the type and position of the object in the first frame with the type and position of one or more objects in one or more previous frames; and providing the N+1th frame to a visual model processor instead of the skipped frame.

[0018] According to another aspect, a device may be provided, comprising components for performing the following operations: receiving data representing multiple training images, the training images representing one or more video clips; determining key frames in the multiple training images; and generating a trained policy model using a reinforcement learning method for determining the number of frames N to skip in response to a detected state S, the reinforcement learning method taking the multiple training images as input and using a reward function that rewards an increase in the number of skipped frames N for a given state while penalizing the skipping of the determined key frames to achieve the trained policy model.

[0019] According to another aspect, a method may be provided, comprising: receiving data representing a plurality of training images, the training images representing one or more video clips; determining key frames in the plurality of training images; and generating a trained policy model using a reinforcement learning method for determining a number of frames N to skip in response to a detected state S, the reinforcement learning method taking the plurality of training images as input and using a reward function that rewards an increase in the number of skipped frames N for a given state while penalizing the skipping of determined key frames to achieve the trained policy model.

[0020] According to another aspect, a device may be provided, comprising at least one processor, at least one memory directly connected to the at least one processor, the at least one memory comprising computer program code, the at least one processor and the at least one memory and the computer program code being configured to execute a method comprising: receiving data of a first frame representing video content comprising a plurality of frames; in the first frame, determining an object type and position for at least one object in the first frame; determining a number of frames N to be skipped based on a comparison of the type and position of the object in the first frame with the type and position of one or more objects in one or more previous frames; and providing the N+1th frame to the visual model processor instead of the skipped frame.

[0021] According to another aspect, a device may be provided, comprising at least one processor, at least one memory directly connected to the at least one processor, the at least one memory comprising computer program code, the at least one processor and the at least one memory and the computer program code being configured to execute a method, the method comprising: receiving data representing a plurality of training images, the training images representing one or more video clips; determining key frames in the plurality of training images; generating a trained policy model using a reinforcement learning method for determining the number of frames N to be skipped in response to a detected state S, the reinforcement learning method taking the plurality of training images as input and using a reward function that rewards an increase in the number of frames N skipped for a given state while penalizing the skipping of the determined key frames to achieve the trained policy model.

[0022] According to another aspect, a non-transitory computer-readable medium may be provided, comprising program instructions stored thereon for executing a method, the method comprising: receiving data representing a first frame of video content comprising multiple frames; in the first frame, determining an object type and position for at least one object in the first frame; determining a number of frames N to skip based on a comparison of the type and position of the object in the first frame with the type and position of one or more objects in one or more previous frames; and providing the N+1th frame to a visual model processor instead of the skipped frame.

[0023] According to another aspect, a non-transitory computer-readable medium may be provided, comprising program instructions stored thereon for executing a method, the method comprising: receiving data representing a plurality of training images, the training images representing one or more video clips; determining key frames in the plurality of training images; generating a trained policy model using a reinforcement learning method for determining the number of frames N to skip in response to a detected state S, the reinforcement learning method taking the plurality of training images as input and using a reward function that rewards an increase in the number of frames N skipped for a given state while penalizing the skipping of the determined key frames to achieve the trained policy model.

[0024] According to another aspect, a device may be provided, comprising at least one processor; and at least one memory comprising computer program code, which, when executed by the at least one processor, causes the device to: receive data representing a first frame of video content comprising multiple frames; in the first frame, determine an object type and position for at least one object in the first frame; determine a number of frames N to skip based on a comparison of the type and position of the object in the first frame with the type and position of one or more objects in one or more previous frames; and provide the N+1th frame to the visual model processor instead of the skipped frame.

[0025] According to another aspect, a device may be provided, comprising at least one processor; and at least one memory comprising computer program code, which, when executed by the at least one processor, causes the device to: receive data representing a plurality of training images, the training images representing one or more video clips; determine key frames in the plurality of training images; and generate a trained policy model using a reinforcement learning method for determining the number of frames N to skip in response to a detected state S, the reinforcement learning method taking the plurality of training images as input and using a reward function that rewards an increase in the number of frames N skipped for a given state while penalizing the skipping of the determined key frames to achieve the trained policy model. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Exemplary embodiments will now be described, by way of non-limiting examples, with reference to the accompanying drawings, in which:

[0027] Figure 1 is a top view of an object using one or more image capture devices and a memory management unit that may be configured according to some example embodiments;

[0028] Figure 2 yes Figure 1 A schematic diagram of a memory management unit shown in ;

[0029] Figure 3is a front view of a mobile terminal that utilizes one or more image capture devices and that may be configured in accordance with some example embodiments;

[0030] Figure 4 is a block diagram of a virtual reality system that may be configured according to some exemplary embodiments;

[0031] Figure 5 is a flow chart illustrating processing operations that may include an exemplary embodiment;

[0032] Figure 6 is a flow chart that more generally illustrates processing operations that may include exemplary embodiments;

[0033] Figure 7 is a flow chart illustrating processing operations that may include exemplary embodiments related to training a policy model;

[0034] Fig. 8A and Figure 8B are respectively a frame sequence of video content, each frame sequence showing a recognized object, which can be used to understand the exemplary embodiment;

[0035] Fig. 9 is a schematic diagram of components that may include an exemplary embodiment;

[0036] Fig.10 is a plan view of a non-transitory medium that can store computer code that, when executed on one or more processors, causes the process operations described herein to be performed. DETAILED DESCRIPTION

[0037] Exemplary embodiments relate to video processing, which may involve receiving data, hereinafter referred to as "video data," which may include multiple frames of one or more video clips or video programs. The video data may be received directly from a capture source, such as from one or more cameras, or may be received from some form of providing system such as a media server. The video data may be received as a streaming service or in one or more download batches. The video data may represent a live video feed; it is data received from a capture source in real time or near real time, although this need not be the case.

[0038] Exemplary embodiments may involve processing the video data before it is transmitted to some other video processing system or module, such as providing or implementing a computational analysis model, which receives the video data and performs some analysis to generate an inference output and / or make a decision that can be applied to some electronic, computerized or other technical system. For example, the analysis may involve one or more of the following: identifying one or more objects in the video, determining specifically what they are in the "real world", tracking their movement, predicting future movement, detecting a particular situation or event (e.g., a dangerous or suspicious event), and / or taking some form of action.

[0039] refer to Figure 1 For example, in the field of autonomous vehicles, one or more cameras 10 may be mounted on the body of a vehicle 11, each capturing a corresponding viewport 12 and feeding the generated video data, which represents, for example, captured video at 60 frames per second, to one or more management units 13. The one or more management units 13 may be one or more computerized systems that analyze the received video data to take actions based on computerized models that may be stored locally or possibly remotely in the form of data. For example, these actions may include moving the vehicle forward or backward at a specific speed, applying the brakes, deciding how to apply the brakes urgently, deciding in which direction to turn the wheels, etc. The management unit 13 may receive other data, such as position data from an onboard GNSS receiver, traffic data from a remote system, map data, etc. Basically, the computerized model may determine what happened in real time or near real time as a result of detecting a specific situation contained in the video data. The situation may include one or more sub-states, such as the detection of an object 14 at a given position relative to the current speed and motion vector of the vehicle, the speed and motion vector of the object, and the type of object. The computerized model can use all of these sub-states to, for example, determine whether to apply emergency brakes, e.g., if object 14 is a person and could cause a collision; determine whether to change the speed and / or direction of the vehicle if this would help avoid a collision; or, determine whether to do nothing, e.g., if the object is trash.

[0040] Figure 2is a schematic diagram of a management unit 13 according to an exemplary embodiment. The management unit 13 may include a pre-processing module 20 for receiving video data from one or more cameras 10. The purpose of the pre-processing module 20 will be explained later, but is generally configured to select or filter which video frames are to be provided to a visual computing analysis model 21 (hereinafter referred to as a visual model), which performs more processing-intensive tasks involved in analyzing the provided video frames. The visual model 21 may generate an inference and / or decision output, which may be displayed, for example, to a driver or passenger information system, and / or may be input to a control unit 22, in which case the control unit 22 controls one or more electrical and / or mechanical aspects of the vehicle 11. For example, the control unit 22 may issue a signal to control the application of a brake 23. The same control unit 22 or other control units may control other aspects, such as steering, acceleration, indication, lighting, etc.

[0041] exist Figure 3 Another example is shown in FIG, which relates to an augmented reality (AR) application. A mobile device 30, such as a smart phone or tablet computer, may include a display screen 31 and one or more cameras 32 for capturing video data representing video frames. Note that capturing may not mean storing the video on the memory of the mobile device, but may mean displaying it to the display screen 31. When a user of the mobile device 30 captures one or more objects 33 within the viewport of the camera, the video data may be analyzed, for example, to identify objects that can be used for specific purposes, such as people, barcodes, retail items, buildings, etc. For example, a visual model can be used to analyze the detected person, perform facial recognition, and display the person's name 34 on the display screen 31.

[0042] AR is one of many technologies that fall under the umbrella term of virtual reality (VR). Any VR-related technology can use the methods and devices disclosed herein. For example, Figure 4A generalized VR system 40 is shown, including a media player 41 and a VR display device 42. The media player 41 may include a personal computer, a laptop computer, or a game console. In some cases, the media player 41 may include a portion of the VR display device 42. The VR display device 42 may include headphones, glasses, goggles, or a mobile phone. A remote content provider 43 may store and send streaming video data, which in the context of an embodiment is VR content data for display to the VR display device 42. The content provider 43 streams the VR content data over a data network 44, which may be any network, for example, an IP network such as an air network, a mobile IP network such as 3G, 4G, or 5G, a multicast network, or a broadcast network. If the data network 44 is unidirectional, a return channel for providing feedback from the VR display device 42 to the content provider 43 may be provided by another data network. The VR display device 42 may include two display screens, and possibly speakers or headphones for outputting corresponding audio data. The VR display device 42 may also include one or more cameras for capturing the user's surrounding space. Video data captured by one or more cameras may be analyzed in real time by a vision model, for example to warn a user about particular types of surrounding objects, pause the VR content data, and / or display a picture-in-picture view of surrounding objects.

[0043] In all examples, the visual model used to analyze the video frames may be stored locally in the user device, whether in the vehicle 11, the mobile device 30 or the VR system 40, or in some cases may be provided on an edge device, such as at a router or network, or even on a remote server, such as in a cloud-based system. The visual model may be generated based on one or more deep learning methods.

[0044] Given the potential complexity or the models generated using deep learning, executing such vision models can be computationally expensive. For real-time or near real-time applications, vision models can be computationally very expensive, especially since video data can be received at, for example, 60 frames per second. Some known vision models execute at a speed of approximately 5 trillion operations per second (TOPS), which gives an idea of ​​the magnitude of processing involved when executing a computational model with input video frames. Therefore, this involves a lot of energy consumption. An exemplary real-time object detection vision model is called YOLO (You Only Look Once), which processes images at 30 frames per second.

[0045] The exemplary embodiments herein can reduce energy consumption by reducing the number of frames provided to the visual model analysis stage, which can use a deep learning computational model. This achieves lower energy consumption without, for example, compressing the visual model, which can result in reduced performance and accuracy, which can be critical for certain real-time applications such as autonomous vehicles. If the visual model is remote from the relevant device, such as in the cloud, the exemplary embodiments also reduce the number of frames that need to be transmitted over the relevant network. Therefore, the exemplary embodiments can provide bandwidth savings.

[0046] An exemplary embodiment involves receiving a frame of video data, identifying one or more objects therein, and determining a number of frames N to skip based on a type and position of the one or more objects in the frame compared to a type and position of one or more object frames in one or more previous frames. The skipped frames are not provided to a visual model for image analysis. Instead, a subsequent frame N+1 is the next frame provided, and the process may be repeated for further received frames, whereby the number of frames N to skip may change dynamically based on the type and spatiotemporal locality observations.

[0047] The policy model may be used to determine the number of frames N to skip. The policy model may be a pre-trained policy model provided as a separate data set or structure on a relevant device (mobile device or vehicle, or any other form of device involved). The policy model may be pre-trained using a reinforcement learning (RL) method, such as by using a Markov decision process (MDP) to generate a Q network. The policy model may receive as input a state parameter S representing the spatiotemporal locality of one or more objects in the current frame relative to one or more previous frames.

[0048] An object type may refer to a class of objects, e.g., to distinguish humans from animals, and both humans and animals from vehicles, etc. Different subcategories may even be classified within each category, e.g., within the human category, there may be gender subcategories (male and female), short human and tall human subcategories, etc. Similarly, subcategories may be provided for animals (e.g., cheetahs, cats) and vehicles (e.g., cars, ships, airplanes). The type of an object may be generated as an inference from an object recognition model that has been pre-trained to classify received image data as including one or more recognized object types and / or subcategories.

[0049] Spatiotemporal locality refers to the extent to which an identified object changes its spatial position over time. An object may include a patch of spatially adjacent pixels in a given frame (segment), or may be determined by identifying an area of ​​interest (e.g., within a rectangular bounding box compared to non-objects such as background terrain). This may give some indication of what the object is (e.g., a slow or fast moving object), which in some exemplary embodiments may be used to determine what action to perform, e.g., how many frames to skip. In some embodiments, the type of object identified, e.g., the category and / or subcategory to which it belongs, is computationally determined using a trained computational model, and the temporal locality of the object type across multiple frames may provide additional information that provides a better estimate of the number of frames N to skip. Thus, for example, a policy model may treat an object identified as a first type differently from an object identified as a second type. For example, the first type may be a car and the second type may be a boat. In some embodiments, multiple objects in a received frame are evaluated using a policy model.

[0050] In some embodiments, all pixels or objects in a received frame are evaluated using a policy model, whereby some pixels or objects may represent one or more objects of particular interest, and other pixels may represent non-objects, such as background. Thus, the evaluation may be performed based on the entire frame, whereby the policy model determines the number of frames to skip based not on just one object, but on all objects in the frame.

[0051] Thus, by reducing the number of frames provided to subsequent processing-heavy stages of video analysis, energy consumption can be reduced. This is achieved through a data-driven (as opposed to, e.g., heuristic) approach, where the accuracy of inferences or decisions made by subsequent video analysis stages should be maintained or at least not significantly degraded.

[0052] Figure 5 is a flow chart of operations performed on a device on which an exemplary embodiment is provided or performed. For example, the device may be Figure 1-4Any of the systems shown in . The device receives an input video frame 50, which can be received in real time or near real time, so that a visual model 52 generates an inference output 53 based on a process or skip controller 56, which determines which frames are passed for processing by the visual model 52 and which frames are skipped. The decision of the process or skip controller 56 is made based on a policy model 54, which informs the process or skip controller 56 how many frames N to skip. Nevertheless, the skipped frames N can still be output to a display screen, for example, but are not passed to the visual model 52 for analysis processing. A feature extraction stage 58 can be provided to extract and determine the type of objects, identify their coordinates according to known methods, and provide this information as input to the policy model 54. The feature extraction stage 58 may include a pre-trained model for determining the type of object, which may include one or more layers of sub-types. Exemplary pre-trained models may include one or more of the following: YOLO, TinyYOLO, and a feature extractor based on a convolutional neural network (CNN).

[0053] Alternatively or additionally, as indicated by dashed line 57, vision model 52 may be used, at least in part, to determine object type and / or subtype and provide inferred output to policy model 54. It is assumed that vision model 52 is configured to perform some further visual processing tasks in addition to determining object type. Policy model 54 may compare each object type and coordinate to the type and coordinates (and possibly other parameters) of a corresponding object in one or more previous frames. Policy model 54 may access data store 59 or cache that maintains a predefined or variable length buffer representing objects and coordinates in previous frames. Policy model 54 generates the next value of N, and the process may repeat.

[0054] Figure 6 is a flow chart of a more general form of operations that may be performed by hardware, software, or a combination of both.

[0055] The first operation 6.1 may include receiving data representing a first frame of video content including a plurality of frames.

[0056] The second operation 6.2 may include: in the first frame, for at least one object in the first frame, determining an object type and a position.

[0057] The third operation 6.3 may include determining a number of frames N to skip based on a comparison of the type and location of the object in the first frame with the type and location of one or more objects in one or more previous frames.

[0058] A fourth operation 6.4 may include providing the N+1th frame instead of the skipped frame to a vision model for processing the frame and, for example, providing an inference output.

[0059] The policy model 54 mentioned above can be pre-trained and provided as an independent model to be stored on a relevant device. It can be pre-trained by an external system and deployed or distributed on a network.

[0060] Regarding the training phase, this may involve training a reinforcement learning (RL) model using unlabeled video clips; this is a potentially large number of video frames from different sources that have not been classified or labeled in any particular way for use in a machine learning approach. Such unlabeled videos may be freely available from various providers and / or may be obtained from the Internet. The policy model 54 may be trained using a Markov decision process (MDP), in which there are multiple elements, i.e., states, actions, and rewards. In other embodiments, the training frames may be labeled.

[0061] For example, state S can be defined as video frame F n The current state of the video frame F n Objects may be included that have an inferred type and / or subtype and have pixels at certain coordinates and some historical information about the object (eg, coordinates in one or more previous frames).

[0062] For example, action A may provide an update to state S. Action A may correspond to the number of frames N that should be skipped. Thus, knowledge of both the input (the coordinates in that frame) and the output (what the type and spatiotemporal locality information indicates about one or more objects) in a given state S may result in operation A updating that state, e.g., skipping N frames. For example, stationary or slowly moving object types, such as a resting cat, may result in skipping a greater number of frames than fast moving object types, such as a moving car. Therefore, during training, the semantics of the object type are considered to provide additional information useful for generating an appropriate action A. Taking the above example, without knowing whether the object type is "cat" or "car", if the two object bounding boxes are located at the same location (x 1 ,y 1 ,z 1 ), then their states S are the same. The merge object type distinguishes the two objects as having corresponding different states, for example:

[0063] Status S A =(x 1 ,y 1 ,z 1 )+(cat)

[0064] Status S B =(x 1 ,y 1 ,z 1 )+(Car)

[0065] The corresponding state S A , S B A training phase can be used to specify more accurate and fine-grained skipping strategies suitable for their semantics.

[0066] For example, for each action A performed in state S, a reward R may be assigned to the agent. The reward may depend on the input state S, the action A, and the output state S' reached after performing action A. The reward R may be designed during the training phase to increase or maximize the number of skipped frames (i.e., N should be large). In this way, if more frames N are skipped, a larger reward R may be given to the agent. The purpose of this is to reduce the number of frames that need to be processed in the subsequent video analysis phase using the visual model. However, in order to avoid missing any important objects or events in the video analysis phase, for example by skipping important frames, the training also penalizes the reward R if any so-called key frames are lost due to skipping. Therefore, in some embodiments, the key frames can be determined by applying the training frames to a pre-trained object detection model to identify frames in which new objects appear or existing objects are transformed. For example, during training, for each frame skipped, the reward R may be increased by a value x, but for each key frame skipped, it is reduced by a value y. The values ​​of x and y may be the same or different.

[0067] Therefore, the optimal policy model can be: maximize the reward over the entire video segment while avoiding missing key frames.

[0068] After sufficient training, e.g., after a large but feasible number of training frames have been processed, the RL model is saved and can be deployed as a policy model to a server such as the reference Figure 1-4 Such an apparatus is shown and described. Thus, while generating the RL model itself may be computationally intensive, it need only be performed once before being deployed to other apparatuses for implementation.

[0069] Figure 7 is a flow chart of a more general form of training operations that can be performed by hardware, software, or a combination of both.

[0070] The first operation 7.1 may include receiving data representing a plurality of training frames. The training frames may be unlabeled video frames.

[0071] The second operation 7.2 may include determining key frames in the training image. Key frames may be frames in which new objects or object transitions are detected. Key frames may be detected using a pre-trained object detection model.

[0072] The third operation 7.3 may include generating a trained policy model using a reinforcement learning (RL) method for determining the number of frames N to skip based on the current state S. The state S may take into account the type and / or subtype of the object.

[0073] Fig. 8A and Figure 8B A plurality of consecutive frames of a respective video clip are each shown, by way of example, to illustrate how the frames may be processed using a pre-trained policy model 54 .

[0074] Fig. 8A Successive frames 80A-80E are shown, depicting a relatively stationary or slowly moving foreground object 81. Upon receiving the first frame 80A, the method may identify the foreground object 81 and the background object 84, and extract features, such as the coordinates of the spots of pixels that make up the foreground object and the coordinates of the spots of pixels that make up the background object. Other pixels corresponding to other objects (not shown) may also be considered. In other embodiments, feature extraction may involve identifying candidate objects within a bounding box. The spots / candidate objects may be passed to a computational model, for example, within the feature extraction stage 58 or as part of the visual model 52, to determine the object type and / or subtype, for example, "cat". The policy model 54 may compare the type and coordinates with the types and coordinates of one or more previous frames (not shown) to identify that the foreground object 81 is a cat and has not moved or has barely moved. Therefore, the policy model 54 may determine that this frame and the next three frames (N=4) may be skipped, and the subsequent frame 80E (the N+1th frame) is provided to the visual model 52.

[0075] Figure 8B Successive frames 82A-82E are shown depicting a relatively fast-moving foreground object 83, such as a moving vehicle. Upon receiving the first frame 82A, the method may identify the foreground object 83 and the background object 85 and extract features, such as the coordinates of the blobs of pixels that make up the foreground object. In other embodiments, feature extraction may involve identifying candidate objects within a bounding box. The blobs / candidate objects may be passed to a computational model, for example, within a feature extraction stage 58 or as part of a visual model 52, to determine an object type and / or subtype, e.g., "car." The policy model 54 may compare the type and coordinates with the types and coordinates of one or more previous frames (not shown) to identify that the foreground object 83 is a car and has moved a relatively large amount. Therefore, the policy model 54 may determine that the frame and the next frame (N=2) are to be skipped, and a subsequent frame 82C (the N+1th frame) is provided to the visual model 52. This process may be repeated for the next frame 82D based on the spatiotemporal locality of that frame relative to one or more previous frames (eg, 82A-82C), which may result in a different number of skipped frames, eg, N=1.

[0076] Thus, in some examples, substantially the entire frame is segmented into multiple objects, or multiple objects are extracted, and then processed by the policy model 54. In this way, an enhanced frame skip decision is made based on the entire frame (rather than a sub-region) or all objects in the frame to skip subsequent processing of one or more complete frames.

[0077] Fig. 9 Is executable Figure 6 or Figure 7 Schematic diagram of components of the device 90 for operation of the present invention. The device 90 may have a controller 91 tightly coupled to a memory 94, as well as optional hardware keys 96, a display 98, and a camera 99. The device 90 may include at least one network interface 110 for connecting to a network or other data network, for example, which may be a wired or wireless modem. For example, the network interface 110 can be used to download the policy model 54 and / or the visual model 52 from a remote server, or can be used to provide the number of filtered frames to the visual model if it is remote from the device 90.

[0078] The memory 94 may include a RAM 92 and a ROM 93 .

[0079] The processor 91 may be connected to each of the other components to control the operation thereof.

[0080] The memory 94 may include a non-volatile memory, a hard disk drive (HDD), or a solid state drive (SSD). The ROM 93 of the memory 94 may store, among other things, an operating system 112, and may store one or more software applications 114. The RAM 92 of the memory 94 may be used by the processor 91 to temporarily store data. The operating system 112 may include code that, when executed by the processor, executes Figure 6 or Figure 7 operations and their variations.

[0081] The processor 91 may take any suitable form. For example, it may be a microcontroller, multiple microcontrollers, a processor, or multiple processors.

[0082] The policy model 54 and / or the vision model may be stored in the ROM 93 or in a separate storage module, for example on a memory card.

[0083] Fig.10 A non-transitory medium 120 according to some embodiments is shown. The non-transitory medium 120 is a computer readable storage medium. It may be, for example, a CD, a DVD, a USB stick, a Blu-ray disc, etc. The non-transitory medium 120 stores computer program code so that when the computer program code is stored by, for example, Fig. 9 When the processor 91 of the processor executes, the device executes Figure 6 and / or Figure 7Any of the methods.

[0084] In the above, the names of network elements, protocols and methods may be based on current standards. In other versions or other technologies, the names of these network elements and / or protocols and / or methods may be different, as long as they provide corresponding functions. For example, embodiments may be deployed in 2G / 3G / 4G / 5G networks and more generations of 3GPP, but may also be deployed in non-3GPP radio networks such as WiFi.

[0085] The memory may be volatile or non-volatile. It may be, for example, RAM, SRAM, flash memory, FPGA block RAM, DCD, CD, USB stick, and Blu-ray disc.

[0086] If not stated otherwise or clearly stated from the context, a statement that two entities are different means that they perform different functions. This does not mean that they are based on different hardware. That is, each entity described in this specification can be based on different hardware, or some or all entities can be based on the same hardware. This does not mean that they are based on different software. That is, each entity described in this specification can be based on different software, or some or all entities can be based on the same software. Each entity described in this specification can be embodied in the cloud.

[0087] As non-limiting examples, implementation of any of the blocks, devices, systems, techniques, or methods described above includes implementation as hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof. Some embodiments may be implemented in the cloud.

[0088] Reference to "component" may mean any hardware or software or a combination thereof. For example, a component may refer to at least one processor; and at least one memory connected to at least one processor.

[0089] It should be understood that what has been described above is what is presently considered to be the preferred embodiment. However, it should be noted that the description of the preferred embodiment has been given by way of example only and that various modifications may be made without departing from the scope defined in the appended claims.

Claims

1. An apparatus for video processing, comprising means for performing the following operations: receiving data representing a first frame of video content comprising a plurality of frames; In the first frame, for at least one object in the first frame, determining an object type and a location; determining, using a policy model, a number N of frames to skip among the plurality of frames based on a comparison of the type and position of the at least one object in the first frame with the type and position of one or more objects in one or more previous frames of the video content, wherein the policy model has been trained using a reinforcement learning method that uses a reward function that rewards an increase in the number N of skipped frames while penalizing the skipping of determined key frames for a given state indicating the type and position of one or more objects in training frames, wherein the key frames are frames in which new objects appear or object transitions occur; and The N+1th frame of the plurality of frames, instead of the skipped frame, is provided to means for applying the frame to a vision model for analyzing the provided N+1th frame.

2. The device according to claim 1, wherein: Determining the object type and position in the first frame includes: identifying multiple objects in the first frame, for each object, determining the object type and position in the first frame, and wherein determining the number of frames N to skip is based on a comparison of the corresponding types and positions of all objects in the first frame with the types and positions of one or more objects in one or more previous frames.

3. The device according to claim 1 or claim 2, wherein: The means is further configured to receive a subsequent frame of the plurality of frames and to repeat the determining and providing operations for the subsequent frame.

4. The device according to claim 1 or claim 2, wherein: The policy model receives as input a state parameter S representing a corresponding type and position of at least one object in the first frame or any subsequent frame.

5. The device according to claim 1, wherein: The reinforcement learning method takes a plurality of training images as input, and the object detection model is used to determine key frames in the plurality of training images.

6. The device according to claim 5, wherein: The object detection model has been pre-trained to classify received image data as comprising one or more identified object types.

7. The device according to claim 1 or claim 2, wherein: The visual model processing component is remote from the device.

8. The device according to claim 1 or claim 2, wherein: The vision model processing component is a computational model used to analyze the provided frames and to produce inference output.

9. The device according to claim 1 or claim 2, wherein: The components include: at least one processor; and At least one memory is connected to the at least one processor.

10. A method for video processing, comprising: receiving data representing a first frame of video content comprising a plurality of frames; In the first frame, for at least one object in the first frame, determining an object type and a location; determining, using a policy model, a number N of frames to skip among the plurality of frames based on a comparison of the type and position of the at least one object in the first frame with the type and position of one or more objects in one or more previous frames of the video content, wherein the policy model has been trained using a reinforcement learning method that uses a reward function that rewards an increase in the number N of skipped frames while penalizing the skipping of determined key frames for a given state indicating the type and position of one or more objects in training frames, wherein the key frames are frames in which new objects appear or object transitions occur; and The N+1th frame of the plurality of frames is provided to the visual model processor instead of the skipped frame for analyzing the provided N+1th frame.

11. A non-transitory computer readable medium comprising program instructions stored thereon for performing a method comprising: receiving data representing a first frame of video content comprising a plurality of frames; In the first frame, for at least one object in the first frame, determining an object type and a location; determining, using a policy model, a number N of frames to skip based on a comparison of the type and position of the at least one object in the first frame with the type and position of one or more objects in one or more previous frames of the video content, wherein the policy model has been trained using a reinforcement learning method that uses a reward function that rewards an increase in the number N of skipped frames while penalizing the skipping of determined key frames for a given state indicating the type and position of one or more objects in training frames, wherein the key frames are frames in which new objects appear or object transitions occur; and The N+1th frame of the plurality of frames is provided to the visual model processor instead of the skipped frame for analyzing the provided N+1th frame.

12. An apparatus for video processing, comprising means for performing the following operations: receiving data representing a plurality of training images, the training images representing one or more video clips; Determine key frames in the plurality of training images, wherein: The key frame is a frame in which a new object appears or an object transition occurs; as well as A trained policy model is generated using a reinforcement learning method for determining the number of frames N to skip in response to a detected state S indicating the corresponding type and position of at least one object in the frame, the reinforcement learning method taking the plurality of training images as input and using a reward function that rewards an increase in the number of skipped frames N for a given state while penalizing the skipping of key frames determined to achieve the trained policy model.

13. A method for video processing, comprising: receiving data representing a plurality of training images, the training images representing one or more video clips; determining key frames in the plurality of training images, wherein the key frames are frames in which new objects appear or object transitions occur; A trained policy model is generated using a reinforcement learning method for determining the number of frames N to skip in response to a detected state S indicating the corresponding type and position of at least one object in the frame, the reinforcement learning method taking the plurality of training images as input and using a reward function that rewards an increase in the number of skipped frames N for a given state while penalizing the skipping of key frames determined to achieve the trained policy model.

Images