Video and image content identifying and capturing method

Abstract

The invention discloses a video and image content identifying and capturing method, which relates to the technical field of image content identification, performs multi-dimensional partitioning on each frame of image, does not perform high-resolution processing on all regions, only keeps high resolution in a high-priority region, reduces irrelevant feature interference, and improves the recognition efficiency. The method comprises the following steps: carrying out secondary partitioning on a key target in a high-priority region, capturing edges, dynamic change regions and fine feature details of a moving object in combination with a bidirectional pyramid attention BiFPN mechanism, and meanwhile, capturing information details on different levels more meticulously by adopting a multi-scale feature extraction and fusion strategy in combination with the bidirectional pyramid attention BiFPN mechanism. Parameters are automatically optimized according to the feature density and the motion state in each frame of image, the adaptability to a complex environment is enhanced, optimization can be performed according to the actual demand of the current frame feature in a static scene or a rapidly changing dynamic scene, and the method has high robustness and scene adaptability.

Description

Technical Field

[0001] The present invention relates to the technical field of image content recognition, and in particular to a video and image content recognition and capture method. Background Art

[0002] Most of the current video and image content recognition methods continuously capture images from video streams for image content analysis in order to identify key targets. These methods rely on technologies such as convolutional neural networks (CNNs), key frame extraction, and fixed-resolution sampling. They perform well when processing static scenes and single-target recognition, but perform poorly in multi-target and dynamic scenes.

[0003] Specifically, traditional solutions will perform unified high-resolution processing on the entire frame of the image. Even if only a few areas contain key information, a global analysis of the entire image will be performed. This not only wastes a lot of computing resources, but also when the background and static areas occupy a large proportion of the picture, the global analysis will lead to a waste of computing power, reducing processing efficiency and real-time response capabilities.

[0004] In order to solve the above problems, some solutions add key frame extraction technology on the basis of frame-level processing, and only sample key frames to reduce the amount of data and improve the overall processing speed. In addition, some solutions also use CNN-based local feature detection technology to predefine specific areas, such as faces or target objects, so as to automatically capture targets in large-scale images. However, most of these solutions rely on preset feature templates or fixed resolutions for processing, lack of adaptive capabilities, and it is difficult to effectively capture key targets in multi-target or complex dynamic scenes. Therefore, a video and image content recognition and capture method is urgently needed to solve such problems. Summary of the invention

[0005] In view of the above existing problems, the present invention is proposed.

[0006] The present invention provides a video and image content recognition and capture method to solve the problem that the fixed resolution processing method adopted by the traditional capture scheme lacks flexibility in multi-target scenarios and cannot dynamically adjust the resolution according to the target detail requirements. Secondly, the traditional local detection scheme relies on predefined areas, cannot perform effective feature hierarchical analysis on scenes with complex backgrounds and multiple targets, and is difficult to perform refined and hierarchical extraction of the content of each area.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] The present invention provides a video and image content recognition and capture method, which comprises:

[0009] Step S1, partitioning and priority evaluation,

[0010] Each frame of video or image is preliminarily scanned and partitioned, and the MobileNet feature detection model is used to evaluate the feature density, motion state, and content complexity of each region;

[0011] Step S2, multi-scale feature extraction,

[0012] Perform deep feature extraction on the high priority areas evaluated in step S1, dynamically adjust the resolution of each partition and combine multi-scale features:

[0013] The resolution of each partition is dynamically adjusted according to the regional priority. High-priority areas are processed at higher resolutions, while medium and low-priority areas have lower resolutions, thereby improving computing efficiency.

[0014] It also fuses features of different resolutions to ensure that more details are captured in high-priority areas and improve recognition accuracy;

[0015] Step S3, key target segmentation,

[0016] For high-priority areas, key targets, i.e., motion and salient objects, are segmented and densely extracted:

[0017] Perform secondary partitioning of key targets in high-priority areas to identify dense sub-areas containing key information, i.e., the edges and dynamically changing areas of key targets;

[0018] In dense sub-regions, the bidirectional pyramid attention BiFPN mechanism is combined to deeply capture the dynamic changes and small-scale details of key targets;

[0019] Step S4, dynamic aggregation of multi-resolution data,

[0020] All captured data are uniformly summarized and dynamically adjusted.

[0021] Furthermore, in step S1, the evaluation process of each region includes:

[0022] Detect the key features and motion trends of each partition and filter out information-intensive areas. Key features include object edges, textures, and color changes.

[0023] Based on feature density and motion status, each area is divided into three priorities: high, medium, and low.

[0024] Furthermore, in step S1, preliminary scanning, partitioning and feature density evaluation are performed:

[0025] Each frame image is partitioned, and MobileNet is used to extract the feature vector of each region to quantify the feature density, motion state and content complexity. Assume that the input image size is H×W, divided into N×M small grids, and each grid size is h×w, then the number of partitions is K=N×M;

[0026] For each region R i,j , where i, j are the row and column indices of the partition, extract the eigenvector:

[0027] f i,j =MobileNet(R i,j ), where f i,j Indicates region R i,j The characteristic vector of is used to calculate the characteristic density and motion state, R i,j represents the region at the partition index (i, j), and MobileNet represents the feature vector process of the feature detection model;

[0028] Calculate the feature density in the region and define the feature density as the feature vector f i,j The root mean square value of is:

[0029] Among them, D i,j For region R i,j The characteristic density of f indicates the information complexity of the region. i,j,k is the feature vector f i,j The kth characteristic component in f i,j feature dimension.

[0030] Furthermore, in step S1, the method of performing preliminary scanning, partitioning and feature density evaluation also includes:

[0031] Differentiate the feature vectors of the corresponding regions in two consecutive frames, evaluate the motion trend, and define the region R i,j The motion state of is the mean square value of the feature vector difference between frames:

[0032] Among them, M i,j For region R i,j The motion state of the region reflects the change range of the region between two frames. and Respectively represent the region R in the tth frame and the t-1th frame i,j The kth characteristic component of ;

[0033] Based on the feature density D i,j and motion state M i,j , assign priorities to each area and set thresholds to distinguish high, medium, and low priorities:

[0034] If Di,j >D th And M i,j >M th , then P i,j = high,

[0035] like or Then P i,j =Medium,

[0036] In other cases, P i,j = low, where P i,j For region R i,j Priority, D th is the high and low distinguishing threshold of feature density, M th A threshold is created to distinguish between high and low motion states; each frame is scanned initially, feature density and motion state are calculated, and judgment is made based on the threshold to automatically divide high, medium, and low priority areas.

[0037] Furthermore, in step S2, deep feature extraction and multi-scale resolution fusion are performed on the high priority area. Specifically:

[0038] According to the feature density D of each high priority area i,j and motion state M i,j , dynamically allocate different resolutions, assuming that the original image resolution is R orig , then the high priority area R i,j The adjusted resolution is defined as:

[0039] in, For high priority area R i,j The dynamically adjusted resolution, R orig is the resolution of the original image, α and β are control weights used to balance the influence of feature density and motion state, D max and M max It is the maximum value of feature density and motion state in all high priority areas of the current frame, which is used for normalization;

[0040] For each adjusted high priority area Extract features at different scales. Let the feature scale be the set S {s1,s2,…,s n}, then at each scale s k The following features are expressed as:

[0041] in, Indicates region R i,j In scale s kFeatureExtract is a feature extraction function that uses convolution operation to extract regional features at a specific scale.

[0042] Furthermore, in step S2, after multi-scale feature extraction, features of different scales are weighted fused to define a high priority region R i,j The fused feature vector is:

[0043] Among them, is the final fusion feature of the high priority area, w k For scale s k The weight is adaptively determined based on the characteristic response strength of each scale, w k is the weight, which is the response strength r of each scale feature k Sure:

[0044] and, Among them, r k For scale s k The characteristic response intensity, |R i,j | is region R i,j The total number of pixels, is the pixel x in the region at scale s k The eigenvalues ​​below.

[0045] Furthermore, in step S3, key target segmentation and BiFPN dense feature extraction are performed in the high priority area:

[0046] In the high-priority area, the key targets are secondary partitioned, focusing on the feature-intensive area, and setting the high-priority area R i,j Contains key objectives T i,j , then T i,j is subdivided into a grid of p×q subregions, where each subregion is represented by Where m and n are partition indexes,

[0047] The characteristic density of each sub-region is defined as:

[0048] in, Indicates sub-area The feature density is used to reflect edge and texture changes. is the total number of pixels in the sub-region, I(x) is the image intensity of pixel x, is the modulus of the pixel gradient, used to capture local edges;

[0049] Based on feature density And the gradient similarity of adjacent sub-regions, select dense sub-regions for further analysis, and define the gradient similarity between adjacent sub-regions as:

[0050] in, Indicates sub-area With its neighborhood N m,n Gradient similarity, N m,n express The set of neighborhood sub-regions of Represents the target sub-region The adjacent sub-region, located in N m,n In the equation, a and b represent the sub-region indexes in the neighborhood, σ is a parameter for adjusting the similarity sensitivity, For sub-region The average gradient within reflects the consistency between regions and meets the similarity threshold The sub-region of is determined as a dense sub-region containing key details.

[0051] Furthermore, in step S3, the bidirectional pyramid attention BiFPN mechanism is combined to perform multi-scale feature fusion:

[0052] In the selected dense sub-regions, BiFPN is used to fuse multi-scale features. The features at scale s are set as The BiFPN fusion feature is defined as:

[0053] in, For dense sub-region The multi-scale fusion features of L represent the number of multi-scale layers, w s is the adaptive weight of scale s, satisfying Represents the sub-region features extracted at scale s, weight w s Determined by calculation based on the response strength of the scale.

[0054] Furthermore, in step S4, the unified aggregation and dynamic adjustment method is:

[0055] Dynamically integrate the feature results of different priority areas and resolutions and compare them globally;

[0056] When a change in feature density is detected in a high-priority area, the resolution and feature extraction strategy of that area are automatically adjusted to respond to the scene dynamics.

[0057] Furthermore, in step S4, feature integration of different priority areas is performed:

[0058] For the high, medium, and low priority regional features extracted in steps S1, S2, and S3, a weighted method is used to generate global features. Let the high priority regional feature be F high , the medium priority area feature is F mid, the low priority area feature is F low , the global feature is expressed as:

[0059] F global =w high ·F high +w mid ·F mid +w low ·F low , where F global is a global feature representation used for global comparison, w high ,w mid ,w low are weighted coefficients for high, medium, and low priority areas, satisfying w high +w mid +w low =1, F high ,F mid ,F low They are the feature representations of different priority areas, and the weights w high ,w mid ,w low Dynamic calculation based on the characteristic response strength of each priority area;

[0060] In generating the global feature representation F global After that, with the target feature F target Compare and determine whether they match. The similarity is: Among them, S represents the similarity between the global feature and the target feature, · is the vector dot product operation, ‖F global ‖ and ‖F target ‖ is the Euclidean norm of the vector. When the similarity S exceeds the set threshold τ, the recognition is considered successful, otherwise it enters the dynamic adjustment strategy;

[0061] During the processing, the feature density changes of the high priority area are continuously monitored. Let the feature density of the high priority area of ​​the current frame be D current , the feature density of the previous frame is D prev , the characteristic density change rate is:

[0062] Where ΔD represents the relative rate of change of feature density. If ΔD exceeds the preset threshold δ, the resolution R of the area is automatically increased and the feature extraction strategy is refined. The adjusted resolution R new The expression is: new =R orig ×(1+γ·ΔD), where R new is the resolution after adaptive adjustment, R orig is the original resolution, and γ is the adjustment sensitivity coefficient.

[0063] The beneficial effects of the present invention are:

[0064] The present invention performs multi-dimensional partitioning on each frame of the image, does not perform high-resolution processing on all areas, maintains high resolution only in high-priority areas, and reduces resolution in medium and low priority areas, thereby greatly reducing the consumption of computing resources for irrelevant areas, and improves resource utilization efficiency through adaptive weighting based on multi-scale feature fusion, and can accurately extract features at different scales in high-priority areas to reduce interference from irrelevant features.

[0065] The present invention performs secondary partitioning of key targets in high-priority areas and combines the bidirectional pyramid attention BiFPN mechanism to capture the edges, dynamically changing areas and small feature details of moving objects, greatly enhancing the adaptability to dynamic changes of multiple targets.

[0066] The present invention adopts an adaptive dynamic adjustment strategy to timely adjust the resolution and feature extraction strategy by monitoring the changes in feature density in high-priority areas. When significant feature density changes are detected in key areas, the resolution is automatically improved and the feature extraction strategy is strengthened to maintain recognition accuracy and real-time performance in rapidly changing scenes.

[0067] The present invention adopts a multi-scale feature extraction and fusion strategy. In complex dynamic scenes, combined with the bidirectional pyramid attention mechanism BiFPN, it can capture information details at different levels more meticulously. At the same time, it can automatically optimize parameters according to the feature density and motion state in each frame image, and further enhance the adaptability to complex environments through partition priority and resolution adjustment. Whether it is a static scene or a rapidly changing dynamic scene, it can be optimized according to the actual needs of the current frame features, and has a high degree of robustness and scene adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0069] Figure 1 The figure is a flow chart of the video and image content recognition and capture method of the present invention. DETAILED DESCRIPTION

[0070] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0071] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0072] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0073] Example 1, reference Figure 1 This embodiment provides a method for identifying and capturing video and image content, comprising the following steps:

[0074] Step S1, partitioning and priority evaluation,

[0075] Each frame of video or image is preliminarily scanned and partitioned, and the MobileNet feature detection model is used to evaluate the feature density, motion state, and content complexity of each region;

[0076] In step S1, the evaluation process of each region includes:

[0077] Detect the key features and motion trends of each partition and filter out information-intensive areas. Key features include object edges, textures, and color changes.

[0078] Based on feature density and motion status, each area is classified into three priorities: high, medium, and low;

[0079] In step S1, preliminary scanning, partitioning and feature density evaluation are performed:

[0080] Each frame image is partitioned, and MobileNet is used to extract the feature vector of each region to quantify the feature density, motion state and content complexity. Assume that the input image size is H×W, divided into N×M small grids, and each grid size is h×w, then the number of partitions is K=N×M;

[0081] For each region R i,j , where i, j are the row and column indices of the partition, extract the eigenvector:

[0082] f i,j =MobileNet(R i,j ), where f i,j Indicates region R i,j The characteristic vector of is used to calculate the characteristic density and motion state, R i,jrepresents the region at the partition index (i, j), and MobileNet represents the feature vector process of the feature detection model;

[0083] Calculate the feature density in the region and define the feature density as the feature vector f i,j The root mean square value of is:

[0084] Among them, D i,j For region R i,j The characteristic density of f indicates the information complexity of the region. i,j,k is the feature vector f i,j The kth characteristic component in f i,j The characteristic dimension of

[0085] In step S1, the method of performing preliminary scanning, partitioning and feature density evaluation also includes:

[0086] Differentiate the feature vectors of the corresponding regions in two consecutive frames, evaluate the motion trend, and define the region R i,j The motion state of is the mean square value of the feature vector difference between frames:

[0087] Among them, M i,j For region R i,j The motion state of the region reflects the change range of the region between two frames. and Respectively represent the region R in the tth frame and the t-1th frame i,j The kth characteristic component of ;

[0088] Based on the feature density D i,j and motion state M i,j , assign priorities to each area and set thresholds to distinguish high, medium, and low priorities:

[0089] If D i,j >D th And M i,j >M th , then P i,j = high,

[0090] like or Then P i,j =Medium,

[0091] In other cases, P i,j = low, where P i,j For region R i,j Priority, D th is the high and low distinguishing threshold of feature density, M thDistinguish the threshold for high and low motion status; perform a preliminary scan on each frame, calculate the feature density and motion status, make judgments based on the threshold, and automatically divide the high, medium, and low priority areas;

[0092] Specifically, multi-dimensional partitioning is performed on each frame of the image, and the MobileNet model is used to quickly extract regional feature density and motion status, and the regions are automatically divided into high, medium, and low priorities based on feature complexity, ensuring that the system can concentrate resources on key areas and avoid excessive calculation of background or low-dynamic areas.

[0093] Step S2, multi-scale feature extraction,

[0094] Perform deep feature extraction on the high priority areas evaluated in step S1, dynamically adjust the resolution of each partition and combine multi-scale features:

[0095] The resolution of each partition is dynamically adjusted according to the regional priority. High-priority areas are processed at higher resolutions, while medium and low-priority areas have lower resolutions, thereby improving computing efficiency.

[0096] It also fuses features of different resolutions to ensure that more details are captured in high-priority areas and improve recognition accuracy;

[0097] In step S2, deep feature extraction and multi-scale resolution fusion are performed on the high priority area. Specifically:

[0098] According to the feature density D of each high priority area i,j and motion state M i,j , dynamically allocate different resolutions, assuming that the original image resolution is R orig , then the high priority area R i,j The adjusted resolution is defined as:

[0099] in, For high priority area R i,j The dynamically adjusted resolution, R orig is the resolution of the original image, α and β are control weights used to balance the influence of feature density and motion state, D max and M max It is the maximum value of feature density and motion state in all high priority areas of the current frame, which is used for normalization;

[0100] For each adjusted high priority area Extract features at different scales. Let the feature scale be the set S {s1,s2,…,s n}, then at each scale s k The following features are expressed as:

[0101] in, Indicates region R i,j In scale s k The feature vector under, FeatureExtract is the feature extraction function, which uses convolution operation to extract regional features at a specific scale;

[0102] In step S2, after multi-scale feature extraction, features of different scales are weighted and fused to define the high priority region R i,j The fused feature vector is:

[0103] Among them, is the final fusion feature of the high priority area, w k For scale s k The weight is adaptively determined based on the characteristic response strength of each scale, w k is the weight, which is the response strength r of each scale feature k Sure:

[0104] and, Among them, r k For scale s k The characteristic response intensity, |R i,j | is region R i,j The total number of pixels, is the pixel x in the region at scale s k The eigenvalues ​​of

[0105] Specifically, in high-priority areas, multi-scale feature extraction and adaptive fusion strategies are used to enhance the ability to capture complex details. Feature extraction is performed on high-priority areas at different scales, and an adaptive weight fusion strategy is adopted to make feature fusion more layered, which helps to capture finer edges, textures and dynamic changes in the scene.

[0106] Step S3, key target segmentation,

[0107] For high-priority areas, key targets, i.e., motion and salient objects, are segmented and densely extracted:

[0108] Perform secondary partitioning of key targets in high-priority areas to identify dense sub-areas containing key information, i.e., the edges and dynamically changing areas of key targets;

[0109] In dense sub-regions, the bidirectional pyramid attention BiFPN mechanism is combined to deeply capture the dynamic changes and small-scale details of key targets;

[0110] In step S3, key target segmentation and BiFPN dense feature extraction are performed in the high priority area:

[0111] In the high-priority area, the key targets are secondary partitioned, focusing on the feature-intensive area, and setting the high-priority area R i,j Contains key objectives T i,j , then T i,j is subdivided into a grid of p×q subregions, where each subregion is represented by Where m and n are partition indexes,

[0112] The characteristic density of each sub-region is defined as:

[0113] in, Indicates sub-area The feature density is used to reflect edge and texture changes. is the total number of pixels in the sub-region, I(x) is the image intensity of pixel x, is the modulus of the pixel gradient, used to capture local edges;

[0114] Based on feature density And the gradient similarity of adjacent sub-regions, select dense sub-regions for further analysis, and define the gradient similarity between adjacent sub-regions as:

[0115] in, Indicates sub-area With its neighborhood N m,n Gradient similarity, N m,n express The set of neighborhood sub-regions of Represents the target sub-region The adjacent sub-region, located in N m,n In the equation, a and b represent the sub-region indexes in the neighborhood, σ is a parameter for adjusting the similarity sensitivity, For sub-region The average gradient within reflects the consistency between regions and meets the similarity threshold The sub-region is determined as a dense sub-region containing key details;

[0116] In step S3, the bidirectional pyramid attention BiFPN mechanism is combined to perform multi-scale feature fusion:

[0117] In the selected dense sub-regions, BiFPN is used to fuse multi-scale features. The features at scale s are set as The BiFPN fusion feature is defined as:

[0118] in, For dense sub-region The multi-scale fusion features of L represent the number of multi-scale layers, w s is the adaptive weight of scale s, satisfying Represents the sub-region features extracted at scale s, weight w s Determined by calculation based on the response strength of the scale;

[0119] Specifically, in the process of key target segmentation, the moving or salient targets in the high-priority areas are further partitioned and densely featured, the feature-dense areas are refined and analyzed, the edges and dynamically changing areas of the key targets are identified, and the bidirectional pyramid attention BiFPN mechanism is combined to capture small-scale details and dynamic change information in specific areas, thereby improving the recognition accuracy in complex multi-target scenarios.

[0120] Step S4, dynamic aggregation of multi-resolution data,

[0121] All captured data are summarized and dynamically adjusted;

[0122] In step S4, the unified aggregation and dynamic adjustment method is:

[0123] Dynamically integrate the feature results of different priority areas and resolutions and compare them globally;

[0124] When a change in feature density is detected in a high-priority area, the resolution and feature extraction strategy of that area are automatically adjusted to respond to the scene dynamics;

[0125] In step S4, feature integration of different priority areas is performed:

[0126] For the high, medium, and low priority regional features extracted in steps S1, S2, and S3, a weighted method is used to generate global features. Let the high priority regional feature be F high , the medium priority area feature is F mid , the low priority area feature is F low , the global feature is expressed as:

[0127] F global =w high ·F high +w mid ·F mid +w low ·F low , where F global is a global feature representation used for global comparison, w high ,w mid ,w low are weighted coefficients for high, medium, and low priority areas, satisfying w high +w mid +w low=1, F high ,F mid ,F low They are the feature representations of different priority areas, and the weights w high ,w mid ,w low Dynamic calculation based on the characteristic response strength of each priority area;

[0128] In generating the global feature representation F global After that, with the target feature F target Compare and determine whether they match. The similarity is: Among them, S represents the similarity between the global feature and the target feature, · is the vector dot product operation, ‖F global ‖ and ‖F target ‖ is the Euclidean norm of the vector. When the similarity S exceeds the set threshold τ, the recognition is considered successful, otherwise it enters the dynamic adjustment strategy;

[0129] During the processing, the feature density changes of the high priority area are continuously monitored. Let the feature density of the high priority area of ​​the current frame be D current , the feature density of the previous frame is D prev , the characteristic density change rate is:

[0130] Where ΔD represents the relative rate of change of feature density. If ΔD exceeds the preset threshold δ, the resolution R of the area is automatically increased and the feature extraction strategy is refined. The adjusted resolution R new The expression is: new =R orig ×(1+γ·ΔD), where R new is the resolution after adaptive adjustment, R orig is the original resolution, γ is the adjusted sensitivity coefficient;

[0131] Specifically, it provides a dynamic adaptive adjustment strategy to continuously monitor changes in feature density in high-priority areas and optimize the resolution and feature extraction strategy in real time. When changes in feature density in high-priority areas are detected, the system automatically increases the resolution of the area, enhances feature extraction, and adapts to rapidly changing scenarios.

[0132] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A video and image content recognition and crawling method, characterized in that: include, Step S1, partitioning and priority evaluation, Each frame of video or image is preliminarily scanned and partitioned, and the MobileNet feature detection model is used to evaluate the feature density, motion state, and content complexity of each region; Step S2, multi-scale feature extraction, Perform deep feature extraction on the high priority areas evaluated in step S1, dynamically adjust the resolution of each partition and combine multi-scale features: Dynamically adjust the resolution of each partition based on regional priority; And fuse features of different resolutions; Step S3, key target segmentation, For high-priority areas, key targets, i.e., motion and salient objects, are segmented and densely extracted: Perform secondary partitioning of key targets in high-priority areas to identify dense sub-areas containing key information, i.e., the edges and dynamically changing areas of key targets; In dense sub-regions, the bidirectional pyramid attention BiFPN mechanism is combined to deeply capture the dynamic changes and small-scale details of key targets; Step S4, dynamic aggregation of multi-resolution data, All captured data are uniformly summarized and dynamically adjusted.

2. A video and image content recognition and capture method according to claim 1, characterized in that: In step S1, the evaluation process of each region includes: Detect the key features and motion trends of each partition and filter out information-intensive areas. Key features include object edges, textures, and color changes. Based on feature density and motion status, each area is divided into three priorities: high, medium, and low.

3. A video and image content recognition and capture method according to claim 2, characterized in that: In step S1, preliminary scanning, partitioning and feature density evaluation are performed: Each frame image is partitioned, and MobileNet is used to extract the feature vector of each region to quantify the feature density, motion state and content complexity. Assume that the input image size is H×W, divided into N×M small grids, and each grid size is h×w, then the number of partitions is K=N×M; For each region R i,j , where i, j are the row and column indices of the partition, extract the eigenvector: f i,j =MobileNet(R i,j ), where f i,j Indicates region R i,j The characteristic vector of is used to calculate the characteristic density and motion state, R i,j represents the region at the partition index (i, j), and MobileNet represents the feature vector process of the feature detection model; Calculate the feature density in the region and define the feature density as the feature vector f i,j The root mean square value of is: Among them, D i,j For region R i,j The characteristic density of f indicates the information complexity of the region. i,j,k is the feature vector f i,j The kth characteristic component in f i,j feature dimension.

4. A video and image content recognition and capture method according to claim 3, characterized in that: In step S1, the method of performing preliminary scanning, partitioning and feature density evaluation also includes: Differentiate the feature vectors of the corresponding regions in two consecutive frames, evaluate the motion trend, and define the region R i,j The motion state of is the mean square value of the feature vector difference between frames: Among them, M i,j For region R i,j The motion state of the region reflects the change range of the region between two frames. and Respectively represent the region R in the tth frame and the t-1th frame i,j The kth characteristic component of Based on the feature density D i,j and motion state M i,j , assign priorities to each area and set thresholds to distinguish high, medium, and low priorities: If D i,j >D th And M i,j >M th , then P i,j = high, like or Then P i,j =Medium, In other cases, P i,j = low, where P i,j For region R i,j Priority, D th is the high and low distinguishing threshold of feature density, M th The threshold for distinguishing between high and low motion states.

5. A video and image content recognition and capture method according to claim 4, characterized in that: In step S2, deep feature extraction and multi-scale resolution fusion are performed on the high priority area. Specifically: According to the feature density D of each high priority area i,j and motion state M i,j , dynamically allocate different resolutions, assuming that the original image resolution is R orig , then the high priority area R i,j The adjusted resolution is defined as: in, For high priority area R i,j The dynamically adjusted resolution, R orig is the resolution of the original image, α and β are control weights used to balance the influence of feature density and motion state, D max and M max It is the maximum value of feature density and motion state in all high priority areas of the current frame, which is used for normalization; For each adjusted high priority area Extract features at different scales. Let the feature scale be the set S {s1,s2,…,s n }, then at each scale s k The following features are expressed as: in, Indicates region R i,j In scale s k FeatureExtract is a feature extraction function that uses convolution operation to extract regional features at a specific scale.

6. A video and image content recognition and capture method according to claim 5, characterized in that: In step S2, after multi-scale feature extraction, features of different scales are weighted and fused to define the high priority region R i,j The fused feature vector is: Among them, is the final fusion feature of the high priority area, w k For scale s k The weight is adaptively determined based on the characteristic response strength of each scale, w k is the weight, which is the response strength r of each scale feature k Sure: and, Among them, r k For scale s k The characteristic response intensity, |R i,j | is region R i,j The total number of pixels, is the pixel x in the region at scale s k The eigenvalues ​​below.

7. A video and image content recognition and capture method according to claim 6, characterized in that: In step S3, key target segmentation and BiFPN dense feature extraction are performed in the high priority area: In the high-priority area, the key targets are secondary partitioned, focusing on the feature-intensive area, and setting the high-priority area R i,j Contains key objectives T i,j , then T i,j is subdivided into a grid of p×q subregions, where each subregion is represented by Where m and n are partition indexes, The characteristic density of each sub-region is defined as: in, Indicates sub-area The feature density is used to reflect edge and texture changes. is the total number of pixels in the sub-region, I(x) is the image intensity of pixel x, is the modulus of the pixel gradient, used to capture local edges; Based on feature density And the gradient similarity of adjacent sub-regions, select dense sub-regions for further analysis, and define the gradient similarity between adjacent sub-regions as: in, Indicates sub-area With its neighborhood N m,n Gradient similarity, N m,n express The set of neighborhood sub-regions of Represents the target sub-region The adjacent sub-region, located in N m,n In the equation, a and b represent the sub-region indexes in the neighborhood, σ is a parameter for adjusting the similarity sensitivity, For sub-region The average gradient within reflects the consistency between regions and meets the similarity threshold The sub-region of is determined as a dense sub-region containing key details.

8. A video and image content recognition and capture method according to claim 7, characterized in that: In step S3, the bidirectional pyramid attention BiFPN mechanism is combined to perform multi-scale feature fusion: In the selected dense sub-regions, BiFPN is used to fuse multi-scale features. The features at scale s are set as The BiFPN fusion feature is defined as: in, For dense sub-region The multi-scale fusion features of L represent the number of multi-scale layers, w s is the adaptive weight of scale s, satisfying Represents the sub-region features extracted at scale s, weight w s Determined by calculation based on the response strength of the scale.

9. A video and image content recognition and capture method according to claim 8, characterized in that: In step S4, the unified aggregation and dynamic adjustment method is: Dynamically integrate the feature results of different priority areas and resolutions and compare them globally; When a change in feature density is detected in a high-priority area, the resolution and feature extraction strategy of that area are automatically adjusted to respond to the scene dynamics.

10. A video and image content recognition and capture method according to claim 9, characterized in that: In step S4, feature integration of different priority areas is performed: For the high, medium, and low priority regional features extracted in steps S1, S2, and S3, a weighted method is used to generate global features. Let the high priority regional feature be F high , the medium priority area feature is F mid , the low priority region feature is f low , the global feature is expressed as: F global =w high ·F high +w mid ·F mid +w low ·F low , where F global is a global feature representation used for global comparison, w high ,w mid ,w low are weighted coefficients for high, medium, and low priority areas, satisfying w high +w mid +w low =1, F high ,F mid ,F low They are the feature representations of different priority areas, and the weights w high ,w mid ,w low Dynamic calculation based on the characteristic response strength of each priority area; In generating the global feature representation F global After that, with the target feature F target Compare and determine whether they match. The similarity is: Among them, S represents the similarity between the global feature and the target feature, · is the vector dot product operation, ‖F global ‖ and ‖F target ‖ is the Euclidean norm of the vector. When the similarity S exceeds the set threshold τ, the recognition is considered successful, otherwise it enters the dynamic adjustment strategy; During the processing, the feature density changes of the high priority area are continuously monitored. Let the feature density of the high priority area of ​​the current frame be D current , the feature density of the previous frame is D prev , the characteristic density change rate is: Where ΔD represents the relative rate of change of feature density. If ΔD exceeds the preset threshold δ, the resolution R of the area is automatically increased and the feature extraction strategy is refined. The adjusted resolution R new The expression is: new =R orig ×(1+γ·ΔD), where R new is the resolution after adaptive adjustment, R orig is the original resolution, and γ is the adjustment sensitivity coefficient.

Images