A method, device, electronic device and storage medium for content perception and tracking

Abstract

The present invention discloses a method, device, electronic device and storage medium for content perception and tracking, the method comprising: identifying perception elements in video frames through a feature recognition neural network model and outputting feature information. Then, the center position of the area to be cropped of all video frames in the video slice is determined based on the feature information of the perception elements, and then the video frame is cropped based on the center position and a given target cropping size. The technical solution provided by the present invention can accurately and automatically perceive the content of interest to the user, and track and crop the content of interest to the user, which solves a series of shortcomings of traditional methods such as large investment, high labor cost, low efficiency, and complex operation, avoids viewing discomfort caused by abrupt lens switching, and greatly improves the user's viewing experience.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a method, device, electronic device and storage medium for content perception and tracking. Background Art

[0002] In streaming media application scenarios such as live sports events, the content acquisition end uses video acquisition equipment to produce video content and distribute it to the network. The content receiving end is often interested in a specific area, such as faces, bodies, moving objects, etc., and this area will continue to move and change. The content in other areas not only affects the audience's intake of the area of ​​interest, but also wastes a lot of computing power and bandwidth during video compression and network transmission. Therefore, it is crucial to intelligently perceive and track video content and effectively distribute it to the streaming media system.

[0003] At present, the commonly used methods of video content perception and tracking are: 1. At the content collection site, multiple video collection devices are deployed in areas where the content of interest to the audience may appear, and the content is collected and connected to the director's desk. Professionals watch the content of each lens and manually switch the content of a certain lens to connect to the streaming media distribution system. 2. Deploy video collection equipment with movable camera positions at the content collection site, and professionals operate the equipment to move and track the content of interest.

[0004] The above traditional methods can solve the problem of content perception and tracking, but they are costly, complex and inefficient. In addition, abrupt camera switching will give viewers a bad viewing experience. How to perceive and track video content efficiently, simply and at low cost is the key. Summary of the invention

[0005] Since the existing methods have the above-mentioned problems, the embodiments of the present invention provide a method, device, electronic device and storage medium for content perception and tracking.

[0006] Specifically, the embodiment of the present invention provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present invention provides a method for content perception and tracking, including:

[0008] Step 1: Slice the original video to obtain video slices.

[0009] Step 2, downsampling the frame rate of the video slice to obtain sampled frames and unsampled frames; wherein the sampling rate ranges from (0, 100%].

[0010] Step 3: input the sampled frame into a feature recognition neural network model, identify the perceptual elements in the sampled frame, output feature information of the perceptual elements, and mark the video frame containing the perceptual elements as the first video frame.

[0011] The feature information includes one or more of the following information: the type of the perception element, the position information of the perception element, and the edge information of the perception element.

[0012] Step 4: determine the center position of the to-be-cropped area of ​​all video frames in the video slice, and then crop the video frames according to the center position and a given target cropping size.

[0013] In a second aspect, an embodiment of the present invention provides a content sensing and tracking device, including:

[0014] The slicing module is used to slice the original video to obtain video slices.

[0015] The downsampling module is used to downsample the frame rate of the video slice to obtain sampled frames and unsampled frames; wherein the sampling rate ranges from (0, 100%].

[0016] The feature recognition module is used to input the sampling frame into a feature recognition neural network model, identify the perceptual elements in the sampling frame, output feature information of the perceptual elements, and mark the video frame containing the perceptual elements as the first video frame.

[0017] The feature information includes one or more of the following information: the type of the perception element, the position information of the perception element, and the edge information of the perception element.

[0018] The cropping module is used to determine the center position of the to-be-cropped area of ​​all video frames in the video slice, and then crop the video frame according to the center position and a given target cropping size.

[0019] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: at least one processor, and a memory communicatively linked to the at least one processor.

[0020] The memory contains instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the methods described in any one of the above aspects.

[0021] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to use a computer to execute any of the methods described in any of the above aspects.

[0022] It can be seen from the above technical solutions that the technical solutions provided by the embodiments of the present invention have the following beneficial effects: the technical solutions provided by the present invention can accurately and automatically sense the content that the user is interested in, and track and crop the content that the user is interested in, which solves a series of shortcomings of traditional methods such as large investment, high labor cost, low efficiency, complex operation, etc., while avoiding viewing discomfort caused by abrupt lens switching, greatly improving the user's viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A flowchart of a content sensing and tracking method provided by an embodiment of the present invention.

[0025] Figure 2 A schematic diagram of determining the center position of a to-be-cropped area of ​​a first video frame in a video slice provided by an embodiment of the present invention.

[0026] Figure 3 A flowchart of a content sensing and tracking method provided by another embodiment of the present invention.

[0027] Figure 4 A flowchart of a content sensing and tracking method provided by another embodiment of the present invention.

[0028] Figure 5 A schematic diagram of the structure of a content sensing and tracking device provided by an embodiment of the present invention.

[0029] Figure 6 A schematic structural diagram of a content sensing and tracking device provided by another embodiment of the present invention.

[0030] Figure 7 A schematic structural diagram of a content sensing and tracking device provided by another embodiment of the present invention.

[0031] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0033] Embodiment 1

[0034] Figure 1 A flow chart of a method for content perception and tracking provided by an embodiment of the present invention is shown. Figure 1 As shown, the content sensing and tracking method provided by the embodiment of the present invention specifically includes the following contents:

[0035] The original video is sliced ​​to obtain video slices.

[0036] The video slice is input into a feature recognition neural network model to identify the perceptual elements in the video frame, and the feature information of the perceptual elements is output, and the video frame containing the perceptual elements is marked as the first video frame.

[0037] The perceptual elements include but are not limited to human faces, human bodies, animals, moving objects, content that needs to be displayed in a focused manner, etc.

[0038] The characteristic information of the perception element includes: the type of the perception element and the position information of the perception element.

[0039] In this step, the training method of the feature recognition neural network model is as follows:

[0040] S11, selecting an image data set containing various types of perception elements, and marking feature information of the image perception elements.

[0041] S12, inputting the image data set and the position information of the sensing elements into the feature recognition neural network model to be trained for learning, so as to obtain a preliminarily trained feature recognition neural network model.

[0042] S13, inputting the test image containing various types of perception elements into the preliminarily trained feature recognition neural network model for recognition, and outputting the estimated position information of the perception elements.

[0043] S14, calculating loss by combining the feature information of the estimated perception element and the feature information of the actual perception element.

[0044] S15, modifying the parameters of the feature recognition neural network model according to the loss, and repeating S13-S15 until the feature recognition neural network model converges.

[0045] The center position of the to-be-cropped area of ​​all video frames in the video slice is determined, and then the video frames are cropped according to the center position and a given target cropping size.

[0046] The determining the center position of the area to be cropped of all video frames in the video slice includes: determining the center position of the area to be cropped of the first video frame in the video slice and determining the center position of the area to be cropped of non-first video frames in the video slice. Figure 2A schematic diagram of determining the center position of a to-be-cropped area of ​​a first video frame in a video slice provided by an embodiment of the present invention is shown.

[0047] The determining the center position of the to-be-cropped area of ​​the first video frame in the video slice includes:

[0048] Preset weight data for each sensor element type.

[0049] According to the type of perception elements output by the feature recognition neural network model, the position information of the perception elements, and the weight data of each type of perception elements, the center position of the to-be-cropped area of ​​the first video frame is determined by a weighted average method.

[0050] In this step, it can be understood that the setting of the weight data of each type of perception element is related to the actual application scenario. In actual applications, the corresponding type of weight parameters can be set in advance according to the different perception types as needed. The embodiment of the present invention does not limit the weight setting of the perception element type.

[0051] The determining the center position of the non-first video frame to be cropped area in the video slice includes:

[0052] The first video frame in the current video slice is used as a reference frame.

[0053] According to the determined center position of the to-be-cropped region in the reference frame, a smooth curve of the movement of the center position of the to-be-cropped region in the current video slice is fitted.

[0054] The center position of the to-be-cropped area of ​​the non-first video frame in the current video slice is estimated according to the smooth curve.

[0055] Embodiment 2

[0056] Figure 3 A flowchart of a content sensing and tracking method provided by another embodiment of the present invention is shown. Figure 3 As shown, the content sensing and tracking method provided by the embodiment of the present invention specifically includes the following contents:

[0057] The original video is sliced ​​to obtain video slices.

[0058] The frame rate of the video slice is downsampled to obtain sampled frames and unsampled frames.

[0059] In this step, since the amount of calculation for processing the original video slice frame by frame is large, it not only occupies a large amount of computing resources, but also consumes a lot of time. Therefore, in order to improve processing efficiency, the original video slice is downsampled. However, it should be noted that the number of downsampled samples is not limited in the embodiment of the present invention and is set according to actual conditions. It can be understood that the larger the number of samples, the greater the pressure on computing power, but the higher the accuracy. When selecting the number of samples in actual application scenarios, it is necessary to consider the balance between computing power and accuracy.

[0060] The sampling frame is input into a feature recognition neural network model to identify the perceptual elements in the sampling frame, output feature information of the perceptual elements, and mark the video frame containing the perceptual elements as the first video frame.

[0061] The perceptual elements include but are not limited to human faces, human bodies, animals, moving objects, content that needs to be displayed in a focused manner, etc.

[0062] The characteristic information of the perception element includes: the type of the perception element and the position information of the perception element.

[0063] In this step, the training method of the feature recognition neural network model has been described in Example 1 and will not be described in detail here.

[0064] The center position of the to-be-cropped area of ​​all video frames in the video slice is determined, and then the video frames are cropped according to the center position and a given target cropping size.

[0065] The determining the center position of the area to be cropped of all video frames in the video slice includes: determining the center position of the area to be cropped of the first video frame in the video slice and determining the center position of the area to be cropped of non-first video frames in the video slice.

[0066] The non-first video frames include non-sampled frames and video frames in which no perceptual elements exist in sampled frames.

[0067] The determining the center position of the to-be-cropped area of ​​the first video frame in the video slice includes:

[0068] Preset weight data for each sensor element type.

[0069] According to the type of perception elements output by the feature recognition neural network model, the position information of the perception elements, and the weight data of each type of perception elements, the center position of the to-be-cropped area of ​​the first video frame is determined by a weighted average method.

[0070] In this step, it can be understood that the setting of the weight data of each type of perception element is related to the actual application scenario. In actual applications, the corresponding type of weight parameters can be set in advance according to the different perception types as needed. The embodiment of the present invention does not limit the weight setting of the perception element type.

[0071] The determining the center position of the non-first video frame to be cropped area in the video slice includes:

[0072] The first video frame in the current video slice is used as a reference frame.

[0073] According to the determined center position of the to-be-cropped region in the reference frame, a smooth curve of the movement of the center position of the to-be-cropped region in the current video slice is fitted.

[0074] The center position of the to-be-cropped area of ​​the non-first video frame in the current video slice is estimated according to the smooth curve.

[0075] Furthermore, in another embodiment, in order to enable the cropped shot image to smoothly track the perceived elements and avoid the jumping feeling of the shot image, when slicing the original video, there are N repeated video frames in adjacent video slices, that is, the last N video frames of the previous video slice are the same as the first N video frames in the current video slice.

[0076] The specific number of repeated frames can be set according to actual conditions and is not limited in the embodiment of the present invention.

[0077] At this time, determining the center position of the non-first video frame to be cropped area in the video slice includes:

[0078] The first video frame in the current video slice and the first N video frames in the current video slice whose center position of the to-be-cropped area has been determined are used as reference frames.

[0079] In this step, it should be noted that since the first N video frames of the current video slice are the last N frames of the previous video slice, the center position of the area to be cropped of the first N video frames of the current video slice has been determined in the previous video slice, and can therefore be used as a reference frame for the current video slice.

[0080] According to the determined center position of the to-be-cropped region in the reference frame, a smooth curve of the movement of the center position of the to-be-cropped region in the current video slice is fitted.

[0081] The center position of the to-be-cropped region of non-first video frames excluding the first N video frames in the current video slice is estimated according to the smooth curve.

[0082] Embodiment 3

[0083] Figure 4 A flowchart of a content sensing and tracking method provided by another embodiment of the present invention is shown. Figure 4 As shown, the content sensing and tracking method provided by the embodiment of the present invention specifically includes the following contents:

[0084] The original video is sliced ​​to obtain video slices.

[0085] The frame rate of the video slice is downsampled to obtain sampled frames and unsampled frames.

[0086] In this step, since the amount of calculation for processing the original video slice frame by frame is large, it not only occupies a large amount of computing resources, but also consumes a lot of time. Therefore, in order to improve processing efficiency, the original video slice is downsampled. However, it should be noted that the number of downsampled samples is not limited in the embodiment of the present invention and is set according to actual conditions. It can be understood that the larger the number of samples, the greater the pressure on computing power, but the higher the accuracy. When selecting the number of samples in actual application scenarios, it is necessary to consider the balance between computing power and accuracy.

[0087] The sampling frame is input into a feature recognition neural network model to identify the perceptual elements in the sampling frame, output feature information of the perceptual elements, and mark the video frame containing the perceptual elements as the first video frame.

[0088] The perceptual elements include but are not limited to human faces, human bodies, animals, moving objects, content that needs to be displayed in a focused manner, etc.

[0089] The characteristic information of the perception element includes: the type of the perception element, the position information of the perception element, and the edge information of the perception element.

[0090] In this step, the training method of the feature recognition neural network model has been described in Example 1 and will not be described in detail here.

[0091] The center position of the to-be-cropped area of ​​all video frames in the video slice is determined, and then the video frames are cropped according to the center position and a given target cropping size.

[0092] The determining the center position of the area to be cropped of all video frames in the video slice includes: determining the center position of the area to be cropped of the first video frame in the video slice and determining the center position of the area to be cropped of non-first video frames in the video slice.

[0093] The non-first video frames include non-sampled frames and video frames in which no perceptual elements exist in sampled frames.

[0094] The determining the center position of the to-be-cropped area of ​​the first video frame in the video slice includes:

[0095] Preset weight data for each sensor element type.

[0096] According to the type of perception elements output by the feature recognition neural network model, the position information of the perception elements, and the weight data of each type of perception elements, the center position of the to-be-cropped area of ​​the first video frame is determined by a weighted average method.

[0097] In this step, it can be understood that the setting of the weight data of each type of perception element is related to the actual application scenario. In actual applications, the corresponding type of weight parameters can be set in advance according to the different perception types as needed. The embodiment of the present invention does not limit the weight setting of the perception element type.

[0098] The determining the center position of the non-first video frame to be cropped area in the video slice includes:

[0099] The first video frame in the current video slice is used as a reference frame.

[0100] According to the determined center position of the to-be-cropped region in the reference frame, a smooth curve of the movement of the center position of the to-be-cropped region in the current video slice is fitted.

[0101] The center position of the to-be-cropped area of ​​the non-first video frame in the current video slice is estimated according to the smooth curve.

[0102] Furthermore, in order to present the perceptual element in the picture of the cropped video frame as much as possible, the to-be-cropped area of ​​the video frame in the video slice is selected according to the edge information of the picture perceptual element before cropping, and the video frame is cropped according to the center position and the given target cropping size, including:

[0103] Determine a region to be cropped of the first video frame.

[0104] The determining of the area to be cropped of the first video frame includes:

[0105] The area to be cropped of the current first video frame is determined according to the edge information of the perceptual element of the current first video frame, the center position of the area to be cropped, and the specified aspect ratio.

[0106] The size of the area to be cropped is a size that is proportionally enlarged s times the minimum size that can contain all the perceptual elements in the current first video frame, and s ≥1.

[0107] The specified aspect ratio is the aspect ratio of the target crop size.

[0108] It should be noted that the specific value of s can be set according to actual conditions, and the present invention does not limit this. Since the area ratio of the perceptual element in the picture directly affects the audience's viewing experience, it is recommended that the value of s does not exceed 2 based on experience.

[0109] The area to be cropped of the non-first video frame is determined according to the size of the area to be cropped of the first video frame.

[0110] The determining the to-be-cropped area of ​​the non-first video frame according to the to-be-cropped area of ​​the first video frame includes:

[0111] If the size of the area to be cropped of the current first video frame is larger than the target cropping size, then in the current video slice, the non-first video frames between the current first video frame and the previous first video frame use the size of the area to be cropped of the current first video frame as a reference, and reduce the area to be cropped proportionally frame by frame until the size of the area to be cropped is consistent with the target cropping size.

[0112] If the size of the area to be cropped of the current first video frame is smaller than the target cropping size, then in the current video slice, the non-first video frame between the current first video frame and the previous first video frame uses the size of the area to be cropped of the current first video frame as a reference, and increases the area to be cropped proportionally frame by frame until the size of the area to be cropped is consistent with the target cropping size.

[0113] If the size of the to-be-cropped area of ​​the current first video frame is equal to the target cropping size, then the size of the to-be-cropped area of ​​the non-first video frames between the current first video frame and the previous first video frame in the current video slice is the target cropping size.

[0114] The size of the area to be cropped is compared with the target cropping size, and the size of the area to be cropped of all video frames is reduced or enlarged to the target cropping size.

[0115] Among them, if the size of the area to be cropped is equal to the target cropping size, no scaling is performed; if the size of the area to be cropped is larger than the target cropping size, the size of the area to be cropped is reduced to the target cropping size; if the size of the area to be cropped is smaller than the target cropping size, the size of the area to be cropped is enlarged to the target cropping size.

[0116] Embodiment 4

[0117] Figure 5 A content sensing and tracking device provided by an embodiment of the present invention is shown. Figure 5 As shown, the content sensing and tracking device provided by the embodiment of the present invention specifically includes the following contents:

[0118] The slicing module is used to slice the original video to obtain video slices.

[0119] The feature recognition module is used to input the video slice into the feature recognition neural network model, identify the perceptual elements in the video frame, output the feature information of the perceptual elements, and mark the video frame containing the perceptual elements as the first video frame.

[0120] In this step, the training method of the discriminant network model has been recorded in Example 1 and will not be repeated here.

[0121] The perceptual elements include but are not limited to human faces, human bodies, animals, moving objects, content that needs to be displayed in a focused manner, etc.

[0122] The characteristic information of the perception element includes: the type of the perception element and the position information of the perception element.

[0123] The cropping module is used to determine the center position of the to-be-cropped area of ​​all video frames in the video slice, and then crop the video frame according to the center position and a given target cropping size.

[0124] The determining the center position of the area to be cropped of all video frames in the video slice includes: determining the center position of the area to be cropped of the first video frame in the video slice and determining the center position of the area to be cropped of non-first video frames in the video slice.

[0125] The determining the center position of the to-be-cropped area of ​​the first video frame in the video slice includes:

[0126] Preset weight data for each sensor element type.

[0127] According to the type of perception elements output by the feature recognition neural network model, the position information of the perception elements, and the weight data of each type of perception elements, the center position of the to-be-cropped area of ​​the first video frame is determined by a weighted average method.

[0128] In this step, it can be understood that the setting of the weight data of each type of perception element is related to the actual application scenario. In actual applications, the corresponding type of weight parameters can be set in advance according to the different perception types as needed. The embodiment of the present invention does not limit the weight setting of the perception element type.

[0129] The determining the center position of the non-first video frame to be cropped area in the video slice includes:

[0130] The first video frame in the current video slice is used as a reference frame.

[0131] According to the determined center position of the to-be-cropped region in the reference frame, a smooth curve of the movement of the center position of the to-be-cropped region in the current video slice is fitted.

[0132] The center position of the to-be-cropped area of ​​the non-first video frame in the current video slice is estimated according to the smooth curve.

[0133] Embodiment 5

[0134] Figure 6 A content sensing and tracking device provided by another embodiment of the present invention is shown. Figure 6 As shown, the content sensing and tracking device provided by the embodiment of the present invention specifically includes the following contents:

[0135] The slicing module is used to slice the original video to obtain video slices.

[0136] The down-sampling module is used to down-sample the frame rate of the video slice to obtain sampled frames and unsampled frames.

[0137] In this step, since the amount of calculation for processing the original video slice frame by frame is large, it not only occupies a large amount of computing resources, but also consumes a lot of time. Therefore, in order to improve processing efficiency, the original video slice is downsampled. However, it should be noted that the number of downsampled samples is not limited in the embodiment of the present invention and is set according to actual conditions. It can be understood that the larger the number of samples, the greater the pressure on computing power, but the higher the accuracy. When selecting the number of samples in actual application scenarios, it is necessary to consider the balance between computing power and accuracy.

[0138] The feature recognition module is used to input the sampling frame into a feature recognition neural network model, identify the perceptual elements in the sampling frame, output feature information of the perceptual elements, and mark the video frame containing the perceptual elements as the first video frame.

[0139] The perceptual elements include but are not limited to human faces, human bodies, animals, moving objects, content that needs to be displayed in a focused manner, etc.

[0140] The characteristic information of the perception element includes: the type of the perception element and the position information of the perception element.

[0141] In this step, the training method of the feature recognition neural network model has been described in Example 1 and will not be described in detail here.

[0142] The cropping module is used to determine the center position of the to-be-cropped area of ​​all video frames in the video slice, and then crop the video frame according to the center position and a given target cropping size.

[0143] The determining the center position of the area to be cropped of all video frames in the video slice includes: determining the center position of the area to be cropped of the first video frame in the video slice and determining the center position of the area to be cropped of non-first video frames in the video slice.

[0144] The non-first video frames include non-sampled frames and video frames in which no perceptual elements exist in sampled frames.

[0145] The determining the center position of the to-be-cropped area of ​​the first video frame in the video slice includes:

[0146] Preset weight data for each sensor element type.

[0147] According to the type of perception elements output by the feature recognition neural network model, the position information of the perception elements, and the weight data of each type of perception elements, the center position of the to-be-cropped area of ​​the first video frame is determined by a weighted average method.

[0148] In this step, it can be understood that the setting of the weight data of each type of perception element is related to the actual application scenario. In actual applications, the corresponding type of weight parameters can be set in advance according to the different perception types as needed. The embodiment of the present invention does not limit the weight setting of the perception element type.

[0149] The determining the center position of the non-first video frame to be cropped area in the video slice includes:

[0150] The first video frame in the current video slice is used as a reference frame.

[0151] According to the determined center position of the to-be-cropped region in the reference frame, a smooth curve of the movement of the center position of the to-be-cropped region in the current video slice is fitted.

[0152] The center position of the to-be-cropped area of ​​the non-first video frame in the current video slice is estimated according to the smooth curve.

[0153] Furthermore, in another embodiment, in order to enable the cropped shot image to smoothly track the perceived elements and avoid the jumping feeling of the shot image, when slicing the original video, there are N repeated video frames in adjacent video slices, that is, the last N video frames of the previous video slice are the same as the first N video frames in the current video slice.

[0154] The specific number of repeated frames can be set according to actual conditions and is not limited in the embodiment of the present invention.

[0155] At this time, determining the center position of the non-first video frame to be cropped area in the video slice includes:

[0156] The first video frame in the current video slice and the first N video frames in the current video slice whose center position of the to-be-cropped area has been determined are used as reference frames.

[0157] In this step, it should be noted that since the first N video frames of the current video slice are the last N frames of the previous video slice, the center position of the area to be cropped of the first N video frames of the current video slice has been determined in the previous video slice, and can therefore be used as a reference frame for the current video slice.

[0158] According to the determined center position of the to-be-cropped region in the reference frame, a smooth curve of the movement of the center position of the to-be-cropped region in the current video slice is fitted.

[0159] The center position of the to-be-cropped region of non-first video frames excluding the first N video frames in the current video slice is estimated according to the smooth curve.

[0160] Embodiment 6

[0161] Figure 7 A content sensing and tracking device provided by another embodiment of the present invention is shown. Figure 7 As shown, the content sensing and tracking device provided by the embodiment of the present invention specifically includes the following contents:

[0162] The slicing module is used to slice the original video to obtain video slices.

[0163] The down-sampling module is used to down-sample the frame rate of the video slice to obtain sampled frames and unsampled frames.

[0164] The feature recognition module is used to input the sampling frame into a feature recognition neural network model, identify the perceptual elements in the sampling frame, output feature information of the perceptual elements, and mark the video frame containing the perceptual elements as the first video frame.

[0165] The perceptual elements include but are not limited to human faces, human bodies, animals, moving objects, content that needs to be displayed in a focused manner, etc.

[0166] The characteristic information of the perception element includes: the type of the perception element, the position information of the perception element, and the edge information of the perception element.

[0167] In this step, the training method of the feature recognition neural network model has been described in Example 1 and will not be described in detail here.

[0168] The cropping module is used to determine the center position of the to-be-cropped area of ​​all video frames in the video slice, and then crop the video frame according to the center position and a given target cropping size.

[0169] The determining the center position of the area to be cropped of all video frames in the video slice includes: determining the center position of the area to be cropped of the first video frame in the video slice and determining the center position of the area to be cropped of non-first video frames in the video slice.

[0170] The non-first video frames include non-sampled frames and video frames in which no perceptual elements exist in sampled frames.

[0171] The determining the center position of the to-be-cropped area of ​​the first video frame in the video slice includes:

[0172] Preset weight data for each sensor element type.

[0173] According to the type of perception elements output by the feature recognition neural network model, the position information of the perception elements, and the weight data of each type of perception elements, the center position of the to-be-cropped area of ​​the first video frame is determined by a weighted average method.

[0174] In this step, it can be understood that the setting of the weight data of each type of perception element is related to the actual application scenario. In actual applications, the corresponding type of weight parameters can be set in advance according to the different perception types as needed. The embodiment of the present invention does not limit the weight setting of the perception element type.

[0175] The determining the center position of the non-first video frame to be cropped area in the video slice includes:

[0176] The first video frame in the current video slice is used as a reference frame.

[0177] According to the determined center position of the to-be-cropped region in the reference frame, a smooth curve of the movement of the center position of the to-be-cropped region in the current video slice is fitted.

[0178] The center position of the to-be-cropped area of ​​the non-first video frame in the current video slice is estimated according to the smooth curve.

[0179] Furthermore, in order to present the perceptual element in the picture of the cropped video frame as much as possible, the to-be-cropped area of ​​the video frame in the video slice is selected according to the edge information of the picture perceptual element before cropping, and the cropping module includes:

[0180] The first cropping area selection module is used to determine the to-be-cropped area of ​​the first video frame.

[0181] The determining the to-be-cropped area of ​​the first video frame includes:

[0182] The area to be cropped of the current first video frame is determined according to the edge information of the perceptual element of the current first video frame, the center position of the area to be cropped, and the specified aspect ratio.

[0183] The size of the area to be cropped is a size that is proportionally enlarged s times the minimum size that can contain all the perceptual elements in the current first video frame, and s ≥1.

[0184] The specified aspect ratio is the aspect ratio of the target crop size.

[0185] It should be noted that the specific value of s can be set according to actual conditions, and the present invention does not limit this. Since the area ratio of the perceptual element in the picture directly affects the audience's viewing experience, it is recommended that the value of s does not exceed 2 based on experience.

[0186] The second cropping area selection module is used to determine the to-be-cropped area of ​​the non-first video frame according to the size of the to-be-cropped area of ​​the first video frame.

[0187] The determining the to-be-cropped area of ​​the non-first video frame according to the to-be-cropped area of ​​the first video frame includes:

[0188] If the size of the area to be cropped of the current first video frame is larger than the target cropping size, then in the current video slice, the non-first video frames between the current first video frame and the previous first video frame use the size of the area to be cropped of the current first video frame as a reference, and reduce the area to be cropped proportionally frame by frame until the size of the area to be cropped is consistent with the target cropping size.

[0189] If the size of the area to be cropped of the current first video frame is smaller than the target cropping size, then in the current video slice, the non-first video frame between the current first video frame and the previous first video frame uses the size of the area to be cropped of the current first video frame as a reference, and increases the area to be cropped proportionally frame by frame until the size of the area to be cropped is consistent with the target cropping size.

[0190] If the size of the to-be-cropped area of ​​the current first video frame is equal to the target cropping size, then the size of the to-be-cropped area of ​​the non-first video frames between the current first video frame and the previous first video frame in the current video slice is the target cropping size.

[0191] The cropping area scaling module is used to compare the size of the to-be-cropped area with the target cropping size, and reduce or enlarge the size of the to-be-cropped area of ​​all video frames to the target cropping size.

[0192] Among them, if the size of the area to be cropped is equal to the target cropping size, no scaling is performed; if the size of the area to be cropped is larger than the target cropping size, the size of the area to be cropped is reduced to the target cropping size; if the size of the area to be cropped is smaller than the target cropping size, the size of the area to be cropped is enlarged to the target cropping size.

[0193] Embodiment 7

[0194] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, such as Figure 8 As shown, the electronic device specifically includes the following contents: a processor 801 , a memory 802 , a communication interface 803 and a communication bus 804 .

[0195] The processor 801, the memory 802, and the communication interface 803 communicate with each other via the communication bus 804; the communication interface 803 is used to realize information transmission between various devices.

[0196] The processor 801 is used to call the computer program in the memory 802, and the processor implements all steps of the above-mentioned content perception and tracking method when executing the computer program.

[0197] Embodiment 8

[0198] Based on the same inventive concept, another embodiment of the present invention provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, all the steps of the above-mentioned content perception and tracking method are implemented.

[0199] In the embodiments of the present invention, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. In the description of this specification, the description of "in one embodiment", "in another embodiment", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment are included in at least one embodiment of the present invention. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments described in this specification and the features of the different embodiments without contradiction.

[0200] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for content perception and tracking, characterized in that: include: Slice the original video to obtain video slices; The frame rate of the video slice is downsampled to obtain a sampled frame and a non-sampled frame; wherein the sampling rate ranges from (0, 100%]; Inputting the sampled frame into a feature recognition neural network model, identifying the perceptual element in the sampled frame, outputting feature information of the perceptual element, and marking the video frame containing the perceptual element as the first video frame; The feature information includes one or more of the following information: the type of the sensing element, the position information of the sensing element, and the edge information of the sensing element; Determine the center position of the first video frame and the non-first video frame to be cropped area in the video slice, and then crop the video frame according to the center position and a given target cropping size; The non-first video frames include non-sampled frames and video frames in which no perceptual elements exist in the sampled frames; The determining the center position of the to-be-cropped area of ​​the first video frame in the video slice includes: Preset weight data for each type of perception element; According to the type of the perception element output by the feature recognition neural network model, the position information of the perception element, and the weight data of each type of the perception element, the center position of the area to be cropped of the first video frame is determined by a weighted average method.

2. The method for content perception and tracking according to claim 1, characterized in that: The determining the center position of the non-first video frame to be cropped area in the video slice includes: Taking the first video frame in the current video slice as a reference frame; Fitting a smooth curve of the movement of the center position of the to-be-cropped area in the current video slice according to the center position of the to-be-cropped area determined in the reference frame; The center position of the to-be-cropped area of ​​the non-first video frame in the current video slice is estimated according to the smooth curve.

3. The method for content perception and tracking according to claim 1, characterized in that: There are N repeated video frames in the adjacent video slices, that is, the last N video frames of the previous video slice are the same as the first N video frames in the current video slice.

4. The method for content perception and tracking according to claim 3, characterized in that: The determining the center position of the non-first video frame to be cropped area in the video slice includes: The first video frame in the current video slice and the first N video frames in the current video slice whose center positions of the to-be-cropped regions have been determined are used as reference frames; Fitting a smooth curve of the movement of the center position of the to-be-cropped area in the current video slice according to the center position of the to-be-cropped area determined in the reference frame; The center position of the to-be-cropped region of non-first video frames excluding the first N video frames in the current video slice is estimated according to the smooth curve.

5. The method for content sensing and tracking according to claim 1, characterized in that: The step of cropping the video frame according to the center position and a given target cropping size includes: Determining a to-be-cropped area of ​​the first video frame; Determine the area to be cropped of the non-first video frame according to the area to be cropped of the first video frame; Comparing the size of the area to be cropped with the target cropping size, and reducing or enlarging the size of the area to be cropped of all video frames to the target cropping size; Among them, if the size of the area to be cropped is equal to the target cropping size, no scaling is performed; if the size of the area to be cropped is larger than the target cropping size, the size of the area to be cropped is reduced to the target cropping size; if the size of the area to be cropped is smaller than the target cropping size, the size of the area to be cropped is enlarged to the target cropping size.

6. The method for content sensing and tracking according to claim 5, characterized in that: The determining the to-be-cropped area of ​​the first video frame includes: Determine the to-be-cropped area of ​​the current first video frame according to edge information of the perceptual element of the current first video frame, a center position of the to-be-cropped area, and a specified aspect ratio; The size of the area to be cropped is the size of the minimum size that can contain all the perceptual elements in the current first video frame, which is proportionally enlarged s times, and s ≥ 1; The specified aspect ratio is the aspect ratio of the target crop size.

7. The method for content sensing and tracking according to claim 5, characterized in that: The determining the to-be-cropped area of ​​the non-first video frame according to the to-be-cropped area of ​​the first video frame includes: If the size of the area to be cropped of the current first video frame is larger than the target cropping size, then in the current video slice, the non-first video frames between the current first video frame and the previous first video frame use the size of the area to be cropped of the current first video frame as a reference, and proportionally reduce the area to be cropped frame by frame until the size of the area to be cropped is consistent with the target cropping size; If the size of the area to be cropped of the current first video frame is smaller than the target cropping size, the non-first video frames between the current first video frame and the previous first video frame in the current video slice use the size of the area to be cropped of the current first video frame as a reference, and increase the area to be cropped in equal proportion frame by frame until the size of the area to be cropped is consistent with the target cropping size; If the size of the to-be-cropped area of ​​the current first video frame is equal to the target cropping size, then the size of the to-be-cropped area of ​​the non-first video frames between the current first video frame and the previous first video frame in the current video slice is the target cropping size.

8. A content sensing and tracking device, characterized in that: include: A slicing module is used to slice the original video to obtain video slices; A downsampling module is used to downsample the frame rate of the video slice to obtain a sampled frame and a non-sampled frame; wherein the sampling rate has a value range of (0, 100%]; A feature recognition module, used for inputting the sample frame into a feature recognition neural network model, identifying the perceptual element in the sample frame, outputting feature information of the perceptual element, and marking the video frame containing the perceptual element as the first video frame; The feature information includes one or more of the following information: the type of the sensing element, the position information of the sensing element, and the edge information of the sensing element; A cropping module, used to determine the center position of the first video frame and the non-first video frame to be cropped in the video slice, and then crop the video frame according to the center position and a given target cropping size; The non-first video frames include non-sampled frames and video frames in which no perceptual elements exist in the sampled frames; The determining the center position of the to-be-cropped area of ​​the first video frame in the video slice includes: Preset weight data for each type of perception element; According to the type of the perception element output by the feature recognition neural network model, the position information of the perception element, and the weight data of each type of the perception element, the center position of the area to be cropped of the first video frame is determined by a weighted average method.

9. The content sensing and tracking device according to claim 8, characterized in that: The determining the center position of the non-first video frame to be cropped area in the video slice includes: Taking the first video frame in the current video slice as a reference frame; Fitting a smooth curve of the movement of the center position of the to-be-cropped area in the current video slice according to the center position of the to-be-cropped area determined in the reference frame; The center position of the to-be-cropped area of ​​the non-first video frame in the current video slice is estimated according to the smooth curve.

10. The content sensing and tracking device according to claim 8, characterized in that: There are N repeated video frames in the adjacent video slices, that is, the last N video frames of the previous video slice are the same as the first N video frames in the current video slice.

11. The content sensing and tracking device according to claim 10, characterized in that: The determining the center position of the non-first video frame to be cropped area in the video slice includes: The first video frame in the current video slice and the first N video frames in the current video slice whose center positions of the to-be-cropped regions have been determined are used as reference frames; Fitting a smooth curve of the movement of the center position of the to-be-cropped area in the current video slice according to the center position of the to-be-cropped area determined in the reference frame; The center position of the to-be-cropped region of non-first video frames excluding the first N video frames in the current video slice is estimated according to the smooth curve.

12. The content sensing and tracking device according to claim 8, characterized in that: The cutting module comprises: A first cropping region selection module, used to determine a to-be-cropped region of the first video frame; A second cropping region selection module, configured to determine the to-be-cropped region of the non-first video frame according to the to-be-cropped region of the first video frame; A cropping region scaling module, used for comparing the size of the to-be-cropped region with the target cropping size, and reducing or enlarging the size of the to-be-cropped region of all video frames to the target cropping size; Among them, if the size of the area to be cropped is equal to the target cropping size, no scaling is performed; if the size of the area to be cropped is larger than the target cropping size, the size of the area to be cropped is reduced to the target cropping size; if the size of the area to be cropped is smaller than the target cropping size, the size of the area to be cropped is enlarged to the target cropping size.

13. The content sensing and tracking device according to claim 12, characterized in that: The determining the to-be-cropped area of ​​the first video frame includes: Determine the to-be-cropped area of ​​the current first video frame according to edge information of the perceptual element of the current first video frame, a center position of the to-be-cropped area, and a specified aspect ratio; The size of the area to be cropped is the size of the minimum size that can contain all the perceptual elements in the current first video frame, which is proportionally enlarged s times, and s ≥ 1; The specified aspect ratio is the aspect ratio of the target crop size.

14. The content sensing and tracking device according to claim 12, characterized in that: The determining the to-be-cropped area of ​​the non-first video frame according to the to-be-cropped area of ​​the first video frame includes: If the size of the area to be cropped of the current first video frame is larger than the target cropping size, then in the current video slice, the non-first video frames between the current first video frame and the previous first video frame use the size of the area to be cropped of the current first video frame as a reference, and proportionally reduce the area to be cropped frame by frame until the size of the area to be cropped is consistent with the target cropping size; If the size of the area to be cropped of the current first video frame is smaller than the target cropping size, the non-first video frames between the current first video frame and the previous first video frame in the current video slice use the size of the area to be cropped of the current first video frame as a reference, and increase the area to be cropped in equal proportion frame by frame until the size of the area to be cropped is consistent with the target cropping size; If the size of the to-be-cropped area of ​​the current first video frame is equal to the target cropping size, then the size of the to-be-cropped area of ​​the non-first video frames between the current first video frame and the previous first video frame in the current video slice is the target cropping size.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the content sensing and tracking method according to any one of claims 1 to 7 is implemented.

16. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the content sensing and tracking method as claimed in any one of claims 1 to 7 is implemented.

Images