Data processing method and device, electronic equipment, storage medium and product

Abstract

The invention discloses a data processing method and device, electronic equipment, a storage medium and a product. Comprising the following steps: acquiring video stream data, and extracting a plurality of effective video frames from the video stream data; inputting the effective video frame into a pre-trained object classification model to obtain a target classification result corresponding to the effective video frame; determining a target object type included in the effective video frame according to the target classification result; wherein the object classification model comprises a feature extraction module, a feature classification module and a result output module, the feature classification module comprises a low-dimensional feature capsule layer, a high-dimensional feature capsule layer and a classification capsule layer, and the output of the feature extraction module is the input of the low-dimensional feature capsule layer. The output of the low-dimensional feature capsule layer is the input of the high-dimensional feature capsule layer, the output of the high-dimensional feature capsule layer is the input of the classification capsule layer, and the result output module is used for outputting the output result of the classification capsule layer. And the accuracy of video stream data analysis is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device, electronic device, storage medium and product. Background Art

[0002] With the development of deep learning technology, convolutional neural networks (CNN) have achieved great success in the field of computer vision and have been widely used in target detection, image classification, image segmentation, medical image analysis, video analysis and other fields.

[0003] Currently, IoT platforms primarily use the YOLO family of algorithms for video analysis. These algorithms consist of a network architecture, a base network, a feature extraction layer, grid division, bounding box prediction, non-maximum suppression, and a loss function, enabling efficient and accurate real-time object detection. YOLO's network architecture is based on a CNN, but CNNs have certain limitations, such as their inability to identify spatial features between features. Therefore, capsule networks were introduced. However, capsule networks currently have slow processing speeds and poor performance on complex visual tasks, and they still struggle with video analysis. Summary of the Invention

[0004] The present invention provides a data processing method, device, electronic device, storage medium and product to solve the problem of poor video analysis and processing effect.

[0005] According to one aspect of the present invention, there is provided a data processing method, comprising:

[0006] Acquire video stream data, and extract multiple valid video frames from the video stream data;

[0007] For multiple valid video frames, input the valid video frames into a pre-trained object classification model to obtain target classification results corresponding to the valid video frames; wherein the target classification results are used to characterize the types of objects included in the valid video frames;

[0008] Determine the target object type included in the valid video frame according to the target classification result;

[0009] Among them, the object classification model includes a feature extraction module, a feature classification module and a result output module. The feature classification module includes a low-dimensional feature capsule layer, a high-dimensional feature capsule layer and a classification capsule layer. The output of the feature extraction module is the input of the low-dimensional feature capsule layer, the output of the low-dimensional feature capsule layer is the input of the high-dimensional feature capsule layer, and the output of the high-dimensional feature capsule layer is the input of the classification capsule layer. The result output module is used to output the output result of the classification capsule layer.

[0010] Optionally, the valid video frame is input into a pre-trained object classification model to obtain a target classification result corresponding to the valid video frame, including: extracting feature information in the valid video frame based on the feature extraction module in the object classification model; processing the feature information based on the low-dimensional feature capsule layer to obtain a first feature vector, wherein the first feature vector includes multiple capsule vectors, each capsule vector includes a spatial posture feature and a first activation value; inputting the first feature vector output by the low-dimensional feature capsule layer into the high-dimensional feature capsule layer to obtain a second feature vector; inputting the second feature vector into the classification capsule layer to obtain a target classification result; and outputting the target classification result based on the result output module.

[0011] Optionally, the low-dimensional feature capsule layer includes a spatial posture extraction unit and an activation function value determination unit, the spatial posture extraction unit is used to determine the first spatial posture information based on the feature information; the activation function value determination unit is used to determine the first activation value of the first spatial posture information.

[0012] Optionally, the high-dimensional feature capsule layer includes a data enhancement processing unit, at least two filtering units, a reverse attention unit, a frequency domain attention unit and a routing unit, and the at least two filtering units are divided into a first filtering unit and a second filtering unit; the first feature vector output by the low-dimensional feature capsule layer is input into the high-dimensional feature capsule layer to obtain a second feature vector, including: receiving the first feature vector based on the high-dimensional feature capsule layer; performing data enhancement processing on the first feature vector based on the data enhancement processing unit to obtain a feature enhancement vector and a second activation value; processing the feature enhancement vector based on the first filtering unit and the second filtering unit respectively to obtain a first filter vector and a second filter vector; processing the first filter vector based on the reverse attention unit to obtain a first attention feature, and processing the second filter vector based on the frequency domain attention unit to obtain a second attention feature; processing the first attention feature, the second attention feature, the first filter vector, the second filter vector, the feature enhancement vector and the second activation value based on the routing unit to obtain a second feature vector; wherein the second feature vector includes at least the object feature in the valid video frame, the second spatial posture information of the object feature and the third activation value that the object feature belongs to a preset category.

[0013] Optionally, the second spatial posture information and the third activation value in the second feature vector are determined in the following manner: determining the coupling coefficient based on the first attention feature and the first filter vector; fusing the second attention feature and the second filter vector to obtain the third attention feature; determining the second spatial posture information and the third activation value based on the third attention feature, the coupling coefficient, the second activation value, the activation value determination function and the posture matrix determination function.

[0014] Optionally, determining the target object type included in the valid video frame according to the target classification result includes: determining the target object type included in the valid video frame according to a category identifier in the target classification result.

[0015] According to another aspect of the present invention, there is provided a data processing apparatus, comprising:

[0016] A video frame extraction module is used to obtain video stream data and extract multiple valid video frames from the video stream data;

[0017] The target classification result determination module is configured to input the valid video frames into a pre-trained object classification model for a plurality of valid video frames to obtain target classification results corresponding to the valid video frames; wherein the target classification results are used to characterize the types of objects included in the valid video frames;

[0018] A target object type determination module, configured to determine the target object type included in the valid video frame according to the target classification result;

[0019] Among them, the object classification model includes a feature extraction module, a feature classification module and a result output module. The feature classification module includes a low-dimensional feature capsule layer, a high-dimensional feature capsule layer and a classification capsule layer. The output of the feature extraction module is the input of the low-dimensional feature capsule layer, the output of the low-dimensional feature capsule layer is the input of the high-dimensional feature capsule layer, and the output of the high-dimensional feature capsule layer is the input of the classification capsule layer. The result output module is used to output the output result of the classification capsule layer.

[0020] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0021] at least one processor; and

[0022] a memory communicatively connected to at least one processor; wherein,

[0023] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by the at least one processor so that the at least one processor can perform the data processing method of any embodiment of the present invention.

[0024] According to another aspect of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data processing method of any embodiment of the present invention when executed.

[0025] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, wherein the computer program implements the data processing method according to any embodiment of the present invention when executed by a processor.

[0026] The technical solution of the embodiment of the present invention obtains video stream data and extracts multiple valid video frames from the video stream data, thereby realizing processing by extracting valid video frames from the video stream, which can reduce the amount of data processing and improve processing efficiency; for multiple valid video frames, the valid video frames are input into a pre-trained object classification model to obtain target classification results corresponding to the valid video frames; wherein the target classification results are used to characterize the object types included in the valid video frames; based on the target classification results, the target object types included in the valid video frames are determined, and the object classification model including multiple capsule layers is adopted to more effectively extract and process features. The capsule network has stronger robustness to changes in the position and posture of the object and can more accurately identify the object type; wherein the object classification model includes a feature extraction module, a feature classification module and a result output module. The feature classification module includes a low-dimensional feature capsule layer, a high-dimensional feature capsule layer and a classification capsule layer. The output of the feature extraction module is the input of the low-dimensional feature capsule layer, the output of the low-dimensional feature capsule layer is the input of the high-dimensional feature capsule layer, and the output of the high-dimensional feature capsule layer is the input of the classification capsule layer. The result output module is used to output the output result of the classification capsule layer. This solution obtains video stream data and extracts multiple valid video frames from it, and then classifies and processes these frames with the help of a pre-trained object classification model. It can accurately identify the object types in the video, realize dynamic and continuous analysis of video content, effectively cover different scenes and time periods in the video, avoid the one-sidedness of single frame analysis, and improve the comprehensive understanding of the overall content of the video; the object classification model adopts a structural design including a feature extraction module, a feature classification module and a result output module. Through the hierarchical feature extraction and processing of the multi-layer capsule network, it can more effectively capture the detailed features and spatial relationships of objects in the video frame, improve the accuracy and reliability of classification, and thus accurately determine the target object type in the valid video frame, providing better basic data and decision support for various application scenarios such as video content analysis, target detection, and intelligent monitoring, thereby enhancing the intelligence level and practical application value of the system.

[0027] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 This is a flow chart of a data processing method provided by the first embodiment of the present invention;

[0030] Figure 2 is a schematic structural diagram of an object classification model applicable to an embodiment of the present invention;

[0031] Figure 3 is a flow chart of a data processing method provided by Embodiment 2 of the present invention;

[0032] Figure 4 is a schematic structural diagram of a high-dimensional feature capsule layer applicable to an embodiment of the present invention;

[0033] Figure 5 Schematic diagram of a multi-branch routing mechanism applicable to an embodiment of the present invention;

[0034] Figure 6 is a structural diagram of a data processing device provided by Embodiment 3 of the present invention;

[0035] Figure 7 It is a structural diagram of an electronic device for implementing the data processing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0037] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] Example 1

[0039] Figure 1This is a flow chart of a data processing method provided by the first embodiment of the present invention. This embodiment is applicable to the case of analyzing and processing video stream data. The method can be executed by a data processing device. The data processing device can be implemented in the form of hardware and / or software. The data processing device can be configured in electronic devices such as computers and servers. Figure 1 As shown, the method includes:

[0040] S110 , acquiring video stream data, and extracting multiple valid video frames from the video stream data.

[0041] The term "video stream data" refers to video content transmitted in the form of a continuous data stream. It can be captured by a camera and then transmitted to a server or data processing terminal via a network transmission method for storage and data processing. A valid data frame specifically represents an image frame with actual analytical value that is screened out from the video stream data. The video stream data can be sampled using a preset valid frame extraction method to select valid video frames that represent key scenes, object motion states, or feature changes in the video stream data, while eliminating redundant or meaningless video frames. This reduces the amount of data processing while retaining the core information required for subsequent analysis and processing such as target recognition and behavior judgment. The preset valid frame extraction method includes, but is not limited to, a time interval sampling method, a key frame detection method, and a method for extracting video frames based on preset screening rules. The time interval sampling method refers to sampling and processing according to a preset time interval or a preset number of image frame intervals to obtain valid video frames. The key frame detection method refers to extracting key frames from the video stream data based on features such as motion changes or scene switching to obtain valid video frames. The method for extracting video frames based on preset screening rules refers to setting corresponding thresholds based on the required image clarity and information richness to obtain valid video frames. The effective frame extraction method can be set and called according to actual needs and is not limited here.

[0042] Specifically, when extracting multiple valid video frames after acquiring video stream data, a continuous video stream can be acquired through an acquisition device, which reaches the processing end after encoding compression and network transmission. Alternatively, video stream data can be acquired from a server, and then a frame sampling method is used to perform frame screening on the video stream data based on a fixed time interval and / or content change to obtain multiple video frames. The obtained multiple video frames can be determined as valid video frames. For example, for a fixed time interval, 10 frames can be extracted per second; for frame screening based on content change, one or more video frames can be continuously extracted when a sudden change in picture motion is detected. Optionally, after obtaining multiple video frames, the multiple video frames can be evaluated in combination with an image quality assessment algorithm to eliminate invalid frames that are blurred, noisy, or repetitive, and retain valid frames with high clarity and obvious features.

[0043] In some embodiments, frame screening based on content changes can be performed when a sudden change in picture motion is detected. Specifically, video frames are first extracted from the video stream data at a preset time interval, and then the similarity of adjacent video frames is calculated. Any video frame among the adjacent video frames whose similarity meets a preset similarity threshold is determined as a valid video frame, and adjacent video frames whose similarity does not meet the preset similarity threshold are all determined as valid video frames.

[0044] In this embodiment, by extracting multiple valid video frames from the video stream data, the amount of data processing is reduced, which helps to improve the efficiency of subsequent analysis, avoids the waste of computing resources caused by full-frame processing, and ensures the validity and pertinence of the data; by determining the valid video frames, it helps to ensure the accuracy of subsequent tasks such as object recognition and classification, provides high-quality frame data support for video intelligent analysis, and at the same time reduces storage costs and realizes efficient use of data.

[0045] S120. For multiple valid video frames, input the valid video frames into a pre-trained object classification model to obtain target classification results corresponding to the valid video frames; wherein the target classification results are used to characterize the object types included in the valid video frames.

[0046] Among them, Figure 2 The structure of an object classification model is shown in FIG. The object classification model includes a feature extraction module, a feature classification module, and a result output module. The feature classification module includes a low-dimensional feature capsule layer, a high-dimensional feature capsule layer, and a classification capsule layer. The output of the feature extraction module is the input of the low-dimensional feature capsule layer, the output of the low-dimensional feature capsule layer is the input of the high-dimensional feature capsule layer, and the output of the high-dimensional feature capsule layer is the input of the classification capsule layer. The result output module is used to output the output of the classification capsule layer.

[0047] Specifically, multiple valid video frames are respectively input into a pre-trained object classification model, and features of the video frames are extracted through the feature extraction module to output feature information. The feature information output by the feature extraction module is transmitted to the low-dimensional feature capsule layer of the feature classification module. The low-dimensional feature capsule layer performs preliminary structured processing on the feature information and then passes it to the high-dimensional feature capsule layer, which further integrates the spatial hierarchical relationship to form more abstract semantic features. The feature information is then input into the classification capsule layer to determine the probability distribution of the object type through vector operations. Finally, the target classification result is output by the result output module to characterize the object type in the valid video frame.

[0048] In this embodiment, the capsule network hierarchical structure is utilized, and the local details and overall semantic associations of objects in the effective video frame can be captured layer by layer from the low-dimensional feature capsule layer to the high-dimensional feature capsule layer, which can better preserve the feature space position relationship than the traditional convolutional neural network; the modular design makes the model well interpretable, and the feature extraction and classification processes are clearly layered, which is suitable for scenarios such as video surveillance and autonomous driving that have high requirements for real-time and accuracy, and provides efficient object recognition capabilities for video content understanding.

[0049] S130: Determine the target object type included in the valid video frame according to the target classification result.

[0050] Among them, the target object type refers to the specific category of the object contained in the video frame determined after analyzing the valid video frame through the object classification model. It maps the visual objects in the video frame to a pre-defined category system based on the target classification results, such as cars, pedestrians or bicycles, to characterize the attribute characteristics of the object at the semantic level, and provide clear semantic labels for the understanding, retrieval and subsequent decision-making of the video content. The judgment process can be combined with the classification probability threshold and the temporal correlation information of the context frame to improve the accuracy and reliability of object recognition in complex scenarios.

[0051] Specifically, based on the target classification results output by the object classification model, the classification probability distribution of each valid video frame is analyzed, and the category with the highest probability value is selected as the target object type of the frame; if there are multiple high-probability categories, a comprehensive judgment can be made in combination with the preset threshold or the classification results of the context frame to finally determine the object type actually contained in the video frame. For example, when there are multiple high-probability categories in the target classification results, the candidate categories that meet the conditions can be first screened out through the preset probability threshold. If there are multiple candidates, the category with the highest probability is selected or weighted fusion is performed in combination with the category semantic similarity and preset weights; at the same time, the current frame can also be associated with the classification results of several previous and next frames, and a comprehensive judgment can be made through majority voting, sliding window smoothing or temporal logic constraints, and finally the target object type in the video frame can be determined by combining threshold screening and context analysis strategies.

[0052] In this embodiment, the target object type included in the valid video frame is determined based on the target classification result, and the abstract classification result can be converted into an understandable object type label in real time, providing direct semantic information support for applications such as video content retrieval and event warning, so that the abstract classification result has actual business value. In addition, the computational complexity of this process is low, and it can be efficiently connected to the real-time video analysis system to improve the accuracy of the classification results.

[0053] Optionally, determining the target object type included in the valid video frame according to the target classification result includes: determining the target object type included in the valid video frame according to a category identifier in the target classification result.

[0054] Specifically, a category identifier is extracted from the target classification result. The category identifier corresponds to a predefined object type. For example, the identifier "01" is set to represent a car, and the identifier "02" is set to represent a pedestrian. By querying the mapping relationship table between the category identifier and the object type, the target object type in the valid video frame is directly matched. For example, if the classification result outputs the category identifier "03", and "03" in the mapping table corresponds to "bicycle", then it is determined that the target object type of the frame is a bicycle.

[0055] In this embodiment, through the direct mapping of category identifiers and object types, rapid conversion of classification results to object types is achieved. The processing flow is concise and efficient, and can meet the performance requirements of real-time video analysis. The pre-defined mapping relationship is clear and unified, avoiding the subjectivity of manual labeling and ensuring the consistency of object type determination. It is suitable for scenarios with high real-time requirements, and can quickly output understandable semantic labels, providing direct support for subsequent data statistics, event warnings and other applications, reducing the processing complexity of the system.

[0056] The technical solution of this embodiment is to extract multiple valid video frames from the video stream data by acquiring video stream data; for multiple valid video frames, the valid video frames are input into a pre-trained object classification model to obtain target classification results corresponding to the valid video frames; wherein the target classification results are used to characterize the object types included in the valid video frames; according to the target classification results, the target object types included in the valid video frames are determined; wherein the object classification model includes a feature extraction module, a feature classification module and a result output module, the feature classification module includes a low-dimensional feature capsule layer, a high-dimensional feature capsule layer and a classification capsule layer, the output of the feature extraction module is the input of the low-dimensional feature capsule layer, the output of the low-dimensional feature capsule layer is the input of the high-dimensional feature capsule layer, the output of the high-dimensional feature capsule layer is the input of the classification capsule layer, and the result output module is used to output the output result of the classification capsule layer. This solution obtains video stream data containing multiple valid video frames and inputs it into an object classification model consisting of a feature extraction module, a low-dimensional feature capsule layer, a high-dimensional feature capsule layer, a classification capsule layer, a feature classification module, and a result output module. After processing at each layer, the classification results are output to determine the type of target object in the valid video frame. This solves the problem of low classification accuracy caused by the traditional method's insufficient capture of feature space hierarchical relationships and semantic information, improves classification accuracy, and can realize real-time recognition and analysis of video objects.

[0057] Example 2

[0058] Figure 3 This is a flow chart of a data processing method provided by the second embodiment of the present invention. The method of this embodiment is a further optimization of the method of the above embodiment. Optionally, feature information in the valid video frame is extracted based on the feature extraction module in the object classification model; the feature information is processed based on the low-dimensional feature capsule layer to obtain a first feature vector, wherein the first feature vector includes multiple capsule vectors, each capsule vector includes a spatial posture feature and a first activation value; the first feature vector output by the low-dimensional feature capsule layer is input into the high-dimensional feature capsule layer to obtain a second feature vector; the second feature vector is input into the classification capsule layer to obtain a target classification result; the target classification result is output based on the result output module. Figure 3 As shown, the method includes:

[0059] S310: Acquire video stream data, and extract multiple valid video frames from the video stream data.

[0060] S320 . For multiple valid video frames, extract feature information from the valid video frames based on a feature extraction module in the object classification model.

[0061] Among them, the feature extraction module can be specifically understood as a module that integrates a feature extraction model constructed using a residual network as the basic network, which is used to extract features of different levels corresponding to valid video frames, including shallow features, middle features and deep features. For example, the shallow layer captures basic features such as edges and colors, the middle layer is integrated into features such as components and structures, and the deep layer forms features such as object categories and scene understanding, ultimately providing mathematical semantic support for video analysis and intelligent decision-making.

[0062] Specifically, after inputting each valid video frame into the object classification model, the feature extraction module first performs layer-by-layer convolution operations on the video frame through the convolution layer to extract the underlying visual features such as edges and textures; then the pooling layer reduces the dimension and retains the key features, and then through a combination of multi-layer convolution and nonlinear activation functions, gradually extracts mid-level features such as shape and structure, as well as high-level abstract features with semantic information, and finally forms feature information containing multi-dimensional visual information, completing the extraction of feature information in the valid video frame. Optionally, the feature information can also be normalized to obtain normalized feature information, eliminating the differences in numerical scale and dimension between different features, avoiding slow convergence or falling into local optimality during model training due to large feature numerical spans, ensuring that the contribution weight of each feature to the model is balanced, and improving the stability and generalization ability of the algorithm.

[0063] In this embodiment, the effective video frames are processed by the feature extraction module, and feature representations at different levels can be automatically learned from the video frames, providing rich feature information for subsequent classification, target detection and other tasks. The feature extraction and classification tasks are deeply coupled, which can improve the feature expression ability and task execution efficiency of the overall model, reduce the interference of noise and redundant information, and lay a high-quality data foundation for the subsequent capsule layer to analyze the hierarchical relationship of the feature space, thereby improving the accuracy and stability of object classification.

[0064] S330 : Process the feature information based on the low-dimensional feature capsule layer to obtain a first feature vector.

[0065] Among them, the first feature vector includes multiple capsule vectors, each of which includes a spatial posture feature and a first activation value. The low-dimensional feature capsule layer can be specifically understood as being used to convert convolution features into capsule inputs, that is, to reshape the feature information output by the feature extraction module into a vector sequence, namely the first feature vector. The first feature vector includes multiple capsule vectors. The spatial posture feature of each capsule vector specifically represents the direction and position relationship of the local parts of the object. The first activation value represents the existence probability of the feature represented by the capsule vector. The activation value can be calculated by the squash function. For example, the feature information is processed by a 9×9×256×8 convolution kernel, the step size is set to 2, and each 20×20 spatial position is converted to an 8-dimensional vector. Assuming that the spatial size is 10×10 after convolution, the final output is 1152 capsules. The 8-dimensional vector of each capsule is the underlying posture representation, and the activation value corresponding to the 8-dimensional vector is calculated by the squash function.

[0066] Specifically, the feature information is input into the low-dimensional feature capsule layer, and the feature information is processed by the low-dimensional feature capsule layer. First, the underlying features are clustered into multiple capsule units. Each capsule encodes the spatial transformation relationship through the posture matrix and outputs a scalar activation value to represent the probability of feature existence. Then, the weights between capsules are optimized through an iterative routing algorithm, so that the connection weights between related capsules are enhanced and the weights of irrelevant connections are weakened. Finally, the first feature vector containing multiple capsule vectors is aggregated to generate, where the scalar activation value is the first activation value.

[0067] In this embodiment, the low-dimensional feature capsule layer simultaneously encodes spatial posture features and activation values ​​through capsule vectors, breaking through the limitation of traditional neurons that only output scalars, and can more completely retain the spatial relationship information of objects, especially suitable for processing objects with posture changes; the dynamic routing mechanism iteratively optimizes the coupling coefficient to make the information transmission between capsules more targeted and enhance the ability to capture key features; the generated first feature vector represents the underlying features in a structured manner, providing a basic representation containing spatial semantics for the hierarchical feature extraction of subsequent high-dimensional feature capsule layers, effectively improving the model's robustness to object deformation, occlusion, etc., and laying a solid foundation for feature recognition in complex scenes.

[0068] Optionally, the low-dimensional feature capsule layer includes a spatial posture extraction unit and an activation function value determination unit, the spatial posture extraction unit is used to determine the first spatial posture information based on the feature information; the activation function value determination unit is used to determine the first activation value of the first spatial posture information.

[0069] In this embodiment, the low-dimensional feature capsule layer includes a spatial posture extraction unit and an activation function value determination unit, wherein the spatial posture extraction unit determines the corresponding first spatial posture information based on the input feature information by parsing and encoding the geometric attributes in the feature, and represents the spatial relationship of the feature in the form of a matrix or vector, wherein the geometric attributes include but are not limited to position, direction and scale; the activation function value determination unit quantizes the first spatial posture information, calculates its modulus through a nonlinear activation function and maps it to a specific interval, thereby determining the first activation value that represents the probability and intensity of the existence of the posture information, thereby realizing the measurement of the feature effectiveness, wherein the nonlinear activation function includes but is not limited to the squash function.

[0070] In this embodiment, the low-dimensional feature capsule layer processes the input feature information through a spatial posture extraction unit. The unit uses a preset weight matrix to perform a convolution operation with the feature information to extract spatial posture information such as edge direction and local component position to form a preliminary feature map; then the activation function value determination unit performs a nonlinear transformation on the features with spatial posture information, and calculates the first activation value through an activation function such as a Squash function. This value is used to characterize the probability of the existence of the feature, and integrates it with the spatial posture information to finally generate multiple capsule vectors containing spatial posture features and the first activation value, which together constitute the first feature vector.

[0071] In this embodiment, the spatial posture extraction unit can accurately capture the spatial position relationship of the local features of the object in the video frame, so that the model can perceive the relative position and orientation of the object parts, effectively improving the robustness to the rotation and deformation of the object; the activation function value determination unit clarifies the validity of the feature by introducing the activation value, avoiding invalid features from interfering with subsequent processing; the synergistic effect of the two units enables the first feature vector to have both geometric spatial information and semantic confidence, providing structured input for the high-dimensional feature capsule layer to integrate global features. Compared with traditional feature extraction methods, it can more completely retain the spatial hierarchical features of the object, thereby improving the accuracy of object classification and adaptability to complex scenes.

[0072] S340: Input the first feature vector output by the low-dimensional feature capsule layer into the high-dimensional feature capsule layer to obtain a second feature vector.

[0073] Among them, the high-dimensional feature capsule layer is the key structure in the capsule network that inherits low-dimensional features and realizes high-level semantic abstraction. It receives the feature vectors output by the low-dimensional capsule through a multi-branch routing mechanism, and after weighted aggregation of the weight matrix transformation and prediction vector, combines the spatial posture information of the underlying features layer by layer into a more complex structured representation. The second feature vector refers to the output of the high-dimensional feature capsule layer, which is used to perform high-level semantic encoding of the input feature vector. It encapsulates more abstract semantic information in vector form, such as the overall structure of the object or category-related features. Its vector modulus represents the probability of the corresponding entity's existence, and the vector direction retains the spatial relationship of the features. This representation method not only avoids the loss of spatial information of traditional convolutional scalar features, but also realizes feature combination from local to global through hierarchical mapping, enabling the model to more efficiently process visual tasks with spatial hierarchies and providing a more robust representation basis for complex visual tasks.

[0074] Specifically, after the first feature vector output by the low-dimensional feature capsule layer is input into the high-dimensional feature capsule layer, the first feature vector is processed by the high-dimensional feature capsule layer to output a second feature vector. In this embodiment, the high-dimensional feature capsule layer processes multiple capsule vectors in the first feature vector through a multi-branch routing mechanism, uses a multi-branch structure to split the number of capsules, and each capsule is used as an independent branch for protocol routing, and activates the capsules of the next layer, reducing the amount of computation between capsules and the inference time of the model. The use of a multi-branch structure can ensure that each capsule branch is independent of each other, ensuring that each capsule branch is not interfered with by other branches, and can obtain more effective feature information. For each branch, the initial coupling coefficient between the low-dimensional capsule and the high-dimensional capsule is first calculated. Then, based on the spatial posture features and first activation values ​​of each capsule vector, the coupling coefficient is iteratively updated to enhance the connection weights of the relevant features, so that the high-dimensional capsule can integrate the spatial hierarchical relationship of local features. For example, local features such as edges and textures are combined into the overall structure of the object parts, and finally a second feature vector containing more abstract semantic information is generated. The second feature vector includes multiple high-level capsule vectors, each of which represents a more complex combination of features of the object and a corresponding second activation value. That is, the multiple capsule vectors in the input first feature vector are processed by the high-dimensional feature capsule layer to obtain a prediction vector, which is then weighted by the iteratively updated routing coefficient in the high-dimensional feature capsule layer. The weighted prediction vectors are accumulated to obtain the input of the high-dimensional capsule. Finally, the squash activation function is used to compress the corresponding feature vector to obtain the second activation value, thereby obtaining multiple high-level capsule vectors, and the multiple high-level capsule vectors form the second feature vector.

[0075] In this embodiment, the spatial correlation of low-dimensional features is integrated through a multi-branch routing mechanism. The multi-branch structure ensures that each capsule branch is independent of each other, which can ensure that each capsule branch is not interfered with by other branches, and can obtain more effective feature information, so that the second feature vector can capture the hierarchical structure between object components, such as the positional relationship between wheels and body. Compared with traditional pooling operations, it can better retain the spatial position information of features and enhance the model's ability to represent the overall structure of the object; local features are mapped to more abstract semantic concepts through high-dimensional capsule layers, thereby enhancing the model's ability to distinguish similar objects, for example, similar objects are cars and trucks; the second activation value is further filtered through hierarchical transmission to reduce noise interference, providing a more reliable feature representation for the final decision of the classification capsule layer, thereby improving the accuracy of object classification and adaptability to complex postures and partially occluded scenes.

[0076] Optionally, the high-dimensional feature capsule layer includes a data enhancement processing unit, at least two filtering units, a reverse attention unit, a frequency domain attention unit and a routing unit, and the at least two filtering units are divided into a first filtering unit and a second filtering unit; the first feature vector output by the low-dimensional feature capsule layer is input into the high-dimensional feature capsule layer to obtain a second feature vector, including: receiving the first feature vector based on the high-dimensional feature capsule layer; performing data enhancement processing on the first feature vector based on the data enhancement processing unit to obtain a feature enhancement vector and a second activation value; processing the feature enhancement vector based on the first filtering unit and the second filtering unit respectively to obtain a first filter vector and a second filter vector; processing the first filter vector based on the reverse attention unit to obtain a first attention feature, and processing the second filter vector based on the frequency domain attention unit to obtain a second attention feature; processing the first attention feature, the second attention feature, the first filter vector, the second filter vector, the feature enhancement vector and the second activation value based on the routing unit to obtain a second feature vector; wherein the second feature vector includes at least the object feature in the valid video frame, the second spatial posture information of the object feature and the third activation value that the object feature belongs to a preset category.

[0077] Among them, Figure 4 The schematic diagram of a high-dimensional feature capsule layer structure is shown, comprising an enhancement processing unit, at least two filtering units, a reverse attention unit, a frequency domain attention unit, and a routing unit. In this embodiment, the at least two filtering units are divided into a first filtering unit and a second filtering unit. The first filtering unit includes at least one filtering unit, and the second filtering unit includes at least one filtering unit. The enhancement processing unit is used to perform affine transformation on the input feature vector to obtain various transformed poses of the input feature data, thereby achieving data enhancement. The filtering unit is used to filter the input features of this unit using a channel filtering mechanism to eliminate invalid feature information. Using a learnable global attention coefficient, it selectively discards some capsules to remove invalid feature information. The reverse attention unit includes two branches: one for learning the probability that a pixel unit belongs to each category, and the other for learning the probability distribution that a pixel unit does not belong to each category. Finally, the prediction results of the two branches are combined and output using a learnable attention weight to capture overlooked detailed features, avoid feature information loss, and enhance the spatial position relationship between components. The frequency domain attention unit uses a discrete Fourier transform to convert the second filter vector of a single capsule to the frequency domain. A learnable attention weight matrix is ​​used to capture more detailed feature information.

[0078] The second activation value represents the activation state after feature enhancement, such as feature validity or confidence, providing a confidence reference for the previous processing for the feature fusion of subsequent routing units, and affecting the weight distribution of feature transfer. The first filter vector represents the feature vector obtained after the feature enhancement vector is filtered by the first filter unit. It usually retains basic semantic or local structural information and filters out some redundant or noise features. It is used as a basic feature branch to provide original feature input for the reverse attention unit and participate in the coupling coefficient calculation and subsequent feature fusion. The second filter vector refers to the feature vector obtained by the second filter unit performing another path filtering on the feature enhancement vector. It forms a parallel feature branch with the first filter vector and is used to provide input for the frequency domain attention unit. It is fused with the second attention feature to generate a more comprehensive third attention feature. The first attention feature refers to the feature obtained after the reverse attention unit processes the first filter vector. It focuses on the local key area in space, suppresses background or irrelevant information, is used to calculate the coupling coefficient, strengthen the correlation strength between features, and participates in the feature integration of the routing unit to improve the feature expression ability of the target object. Second-level attention features refer to features obtained after processing the second filter vector by the frequency-domain attention unit. They filter key information from the frequency dimension, such as high-frequency details or low-frequency contours, enhancing the feature's robustness to transformations like rotation and scaling. They are then fused with the second filter vector to generate third-level attention features, supplementing the semantic information in the frequency domain and improving the feature's multi-dimensional representation capabilities. Second-level spatial pose information refers to features calculated using a pose matrix determination function. They represent the spatial geometric properties of objects in the valid video frame, including but not limited to three-dimensional pose parameters such as position, orientation, scale, and rotation angle. They are used to determine the spatial location of object features in the video frame and, combined with category information, provide spatial representation for tasks such as video understanding and object detection. Third-level activation values ​​refer to activation values ​​calculated using an activation value determination function. They represent the confidence level, such as a probability value, that an object feature belongs to a predefined category. This represents the model's semantic judgment of the object and serves as one of the criteria for final classification. Together with spatial pose information, they form a complete object representation, improving the model's classification and localization accuracy.

[0079] Specifically, after the high-dimensional feature capsule layer receives the first feature vector output by the low-dimensional feature capsule layer, it first performs affine transformation processing on the first feature vector through the data enhancement processing unit, that is, it processes each capsule vector of the first feature vector to generate a feature enhancement vector corresponding to the spatial posture feature and a second activation value after the affine transformation of the first activation value; then, the first filtering unit and the second filtering unit are used to perform channel filtering processing on the feature enhancement vector respectively to obtain the first filter vector and the second filter vector, wherein the first filter vector r i The method of determining is through r i =V·A r +br The feature enhancement vector V is processed to obtain, where i is the serial number of the capsule vector corresponding to the feature enhancement vector, V is the feature enhancement vector, and A r is the learnable weight coefficient, b r is the bias term, the second filter vector The method of determining The feature enhancement vector is processed to obtain, where V is the feature enhancement vector, A p is the learnable weight coefficient, b p is the bias term; then the reverse attention unit extracts features such as object edges that are ignored by traditional attention from the first filter vector to form the first attention feature. The frequency domain attention unit analyzes the second filter vector in the frequency domain, enhances frequency domain features such as texture and structure, and obtains the second attention feature. Finally, the routing unit fuses multi-source features and activation values, and generates the second feature vector through dynamic weight allocation. The second feature vector contains object features, spatial posture matrix and classification activation value.

[0080] In this embodiment, the data enhancement processing unit improves the robustness of the model to multiple object postures by expanding feature diversity; the filtering unit eliminates invalid feature information to ensure feature purity; the combination of reverse attention and frequency domain attention can not only capture easily ignored features such as object edges, but also enhance structural information in the frequency domain, making up for the limitations of traditional attention; the routing unit dynamically weights to achieve adaptive fusion of features, so that the second feature vector has spatial position relationship, semantic activation confidence and multi-dimensional feature representation at the same time. Compared with the traditional capsule layer, the object classification accuracy in complex scenes is higher, and the integrity and anti-interference ability of feature representation are significantly enhanced.

[0081] Optionally, the second spatial posture information and the third activation value in the second feature vector are determined in the following manner: determining the coupling coefficient based on the first attention feature and the first filter vector; fusing the second attention feature and the second filter vector to obtain the third attention feature; determining the second spatial posture information and the third activation value based on the third attention feature, the coupling coefficient, the second activation value, the activation value determination function and the posture matrix determination function.

[0082] Among them, the coupling coefficient is used to measure the correlation strength between the bottom-level capsule and the high-level capsule. Its essence is an attention weight. The feature transfer path is adaptively adjusted through the iterative update mechanism, so that the network can more accurately capture the hierarchical relationship between features. ij Represents the connection weight between the bottom-level capsule i and the high-level capsule j. The larger its value, the greater the contribution of the output of the bottom-level capsule i to the high-level capsule j. The coupling coefficient represents the contribution of each voting capsule to the activation of the high-level capsule. The coupling coefficient c ij The calculation formula is:

[0083]

[0084] Among them, i represents the serial number of the bottom-level capsule, j represents the serial number of the high-level capsule, n represents the number of bottom-level capsules, and r ij is the fused feature vector of the first attention feature and the first filter vector. The activation value determination function refers to a function that calculates the activation probability based on weighted sum, and the posture matrix determination function refers to integrating multi-dimensional posture parameters to generate a spatial transformation matrix.

[0085] The expression of the activation value determination function is:

[0086]

[0087] Among them, 1≤k≤n, is the second activation value corresponding to the i-th capsule vector, a j is the third activation value.

[0088] The expression of the posture matrix determination function is:

[0089]

[0090] Among them, 1≤k≤n, represents the second filter vector, f i Represents the second attention feature.

[0091] Specifically, the coupling coefficient is calculated based on the semantic correlation between the first attention feature and the first filter vector to characterize the connection strength between the features. Specifically, the first attention feature and the first filter vector are fused to obtain the fused feature vector r ij , and then calculate the fusion feature vector according to the calculation formula of the coupling coefficient to obtain the corresponding coupling coefficient; sum the second attention feature and the second filter vector to obtain the third attention feature; then use the fusion feature vector r ij Substitute it into the activation value for calculation to obtain the second activation value, and determine the function according to the second posture matrix. Combined with the third attention feature and the coupling coefficient, finally derive the spatial posture information that can characterize the spatial position relationship of the object parts, and the second activation value representing that the feature belongs to the preset category.

[0092] In this embodiment, the introduction of the coupling coefficient quantifies the degree of correlation between features, making the construction of spatial posture information more structured; by fusing the second posture matrix and the activation value processing results, the second eigenvector retains both the spatial transformation information after data enhancement and the confidence level of feature validity; the synergistic effect of the activation value determination function and the posture matrix determination function realizes the integrated representation of semantic activation value and spatial posture information, which can effectively improve the classification accuracy of the model for complex posture objects, and enhances the anti-interference ability and semantic interpretability of the features through multi-source information fusion.

[0093] Optionally, the second spatial posture information and the third activation value are determined based on the third attention feature, the coupling coefficient, the second activation value, the activation value determination function and the posture matrix determination function, including: processing the coupling coefficient and the second activation value based on the activation value determination function to obtain the third activation value; processing the third activation value, the coupling coefficient and the third attention feature based on the posture matrix determination function to obtain the second spatial posture information.

[0094] Specifically, the coupling coefficient and the second activation value are substituted into the activation value determination function to calculate the third activation value that can characterize the current feature activation level; then, with the help of the posture matrix determination function, the third activation value, the coupling coefficient and the third attention feature are used as inputs of the posture matrix determination function to derive the second spatial posture information used to describe the object spatial information.

[0095] In this embodiment, by processing the coupling coefficient and activation value through the activation value determination function, the feature activation strength can be dynamically adjusted, the sensitivity to key features can be enhanced, and the interference of irrelevant information can be suppressed; the posture matrix determination function is used to fuse multi-dimensional information, including activation value, coupling coefficient and attention feature, which can more comprehensively capture the spatial relationship and posture characteristics of the object and improve the accuracy of posture estimation; this process realizes adaptive weighting of features and precise modeling of spatial information through the combination of attention mechanism and matrix operation, which is particularly suitable for scenes with changeable object postures, provides reliable spatial posture data support for subsequent target tracking and detection, and enhances the system's understanding of dynamic scenes.

[0096] In a specific embodiment, Figure 5The structural diagram of a multi-branch routing mechanism shown in the figure is that the inter-capsule agreement routing mechanism consists of a reverse attention unit, a frequency domain attention unit and a routing unit, which is called straight routing (SR), and the routing unit is a multi-branch routing unit. Reverse Attention Unit (RA): In order to further enhance the robustness of the capsule network. Straight Routing uses the RA unit to strengthen the spatial relationship between the voting capsule and the high-level capsule, that is, the spatial relationship between the part and the whole. The voting capsule here can be understood as the feature obtained after processing by the enhanced processing unit. The RA unit is divided into two branches. The two branches learn different feature information respectively, and then use the attention weight parameters to combine the learning results of the two branches and output the final result, that is, the first attention feature. Frequency Domain Attention Unit (FA): In order to obtain fine-grained feature information, improve the classification performance of the capsule network on complex data sets. As Figure 5 As shown in the figure, SR divides the capsule layer into multiple capsule branches along the dimension of the capsule channel. The voting capsules of each branch are sent to the FA module, which uses 1DDFT (One-Dimensional Discrete Fourier Transform) to convert the feature information into the frequency domain. Then, the FA module sends the frequency domain feature information to two branches. In the channel dimension, the two branches use convolution operations with different convolution kernel sizes. The purpose of this operation is to find the frequency features with higher importance in the pose. Finally, the features are reconverted to the spatial domain through 1DIDFT. Multi-branch routing unit: The multi-branch unit is proposed to maximize the advantages of the capsule network and minimize the interference of other factors. The multi-branch structure ensures that each capsule branch is independent of each other, ensuring that each capsule branch is not interfered with by other branches and can obtain more effective feature information. In fact, when the opinions of the vast majority of voting capsules are consistent, the higher-level capsules will be activated, avoiding the negative effects caused by different branches as much as possible. There are two important parameters involved in the routing unit, namely the coupling coefficient c. ij and the activation value a of the high-level capsule j Each layer of capsule is r ij Generate the coupling coefficient c through the softmax function ij The coupling coefficient represents the contribution of each voting capsule to the activation of the higher-level capsule. The coupling coefficient is related to the activation value after convolution transformation. Multiply together to generate a weighted vote. The activation value a of the high-level capsule j is the sum of weighted votes of lower-level capsules. The pose matrix p of the next layer of capsulesj , by the prediction matrix Determined by weighted average.

[0097] S350: Input the second feature vector into the classification capsule layer to obtain the target classification result.

[0098] Specifically, after the second feature vector is input into the classification capsule layer, the layer processes the object features, spatial posture information and second activation value contained therein through a dynamic routing mechanism. It first calculates the initial coupling coefficient between the high-dimensional feature capsule and the classification capsule, and then iteratively updates the coupling coefficient based on the spatial posture information and the second activation value in the second feature vector to enhance the feature connection weights related to the preset category; subsequently, the classification capsule integrates the global features through vector operations to generate a capsule vector for each preset category, whose modulus corresponds to the activation probability of the category in the video frame. Finally, by selecting the category capsule vector with the largest modulus, the target classification result representing the object type of the video frame is output.

[0099] In this embodiment, the classification capsule layer integrates the multi-dimensional information of the second eigenvector through dynamic routing, which can capture the semantic association between the overall structure of the object and the preset category, accurately distinguish similar categories, and improve classification accuracy; the processing process combines spatial posture information and activation values, so that the target classification results have both geometric position description and semantic confidence, providing a more reliable decision-making basis for video content understanding, and is suitable for scenarios with high requirements for real-time classification accuracy.

[0100] S360: Output the target classification result based on the result output module.

[0101] Specifically, after the result output module receives the target classification result generated by the classification capsule layer, it first normalizes the modulus probability of the category capsule vector, for example, by processing it through the Softmax function, to ensure that the probability value is within a reasonable range; then, according to the preset output format, such as JSON, XML, the classification result is encapsulated, including category identification, probability value and corresponding object type description and other information; if the system supports multimodal output, the classification result will also be associated and integrated with the video frame timestamp, scene metadata, etc., and finally the target classification result will be output to the external system or user end through the API interface, database storage or visualization interface.

[0102] S370: Determine the target object type included in the valid video frame according to the target classification result.

[0103] The technical solution of this embodiment obtains video stream data and extracts multiple valid video frames from the video stream data. For each of the multiple valid video frames, feature information in the valid video frames is extracted based on a feature extraction module in an object classification model. The feature information is processed based on a low-dimensional feature capsule layer to obtain a first feature vector, wherein the first feature vector includes multiple capsule vectors, each of which includes a spatial posture feature and a first activation value. The first feature vector output by the low-dimensional feature capsule layer is input into a high-dimensional feature capsule layer to obtain a second feature vector. The second feature vector is input into a classification capsule layer to obtain a target classification result. The target classification result is output based on a result output module. Based on the target classification result, the target object type included in the valid video frame is determined. This solution obtains video stream data containing multiple valid video frames and inputs it into an object classification model consisting of a feature extraction module, a low-dimensional feature capsule layer, a high-dimensional feature capsule layer, and a feature classification module in a classification capsule layer, and a result output module. After processing by each layer, the classification result is output to determine the target object type in the valid video frame. This solution solves the problem of low classification accuracy caused by traditional methods that fail to capture feature spatial hierarchical relationships and semantic information, thereby improving classification accuracy and enabling real-time video object recognition and analysis.

[0104] Example 3

[0105] Figure 6 This is a structural diagram of a data processing device provided by the third embodiment of the present invention. Figure 6 As shown, the device includes:

[0106] The video frame extraction module 610 is used to obtain video stream data and extract multiple valid video frames from the video stream data;

[0107] The target classification result determination module 620 is configured to input the valid video frames into a pre-trained object classification model to obtain target classification results corresponding to the valid video frames; wherein the target classification results are used to characterize the types of objects included in the valid video frames;

[0108] A target object type determination module 630 is configured to determine the target object type included in the valid video frame according to the target classification result;

[0109] Among them, the object classification model includes a feature extraction module, a feature classification module and a result output module. The feature classification module includes a low-dimensional feature capsule layer, a high-dimensional feature capsule layer and a classification capsule layer. The output of the feature extraction module is the input of the low-dimensional feature capsule layer, the output of the low-dimensional feature capsule layer is the input of the high-dimensional feature capsule layer, and the output of the high-dimensional feature capsule layer is the input of the classification capsule layer. The result output module is used to output the output result of the classification capsule layer.

[0110] The technical solution of this embodiment is to obtain video stream data through a video frame extraction module and extract multiple valid video frames from the video stream data; the target classification result determination module inputs the valid video frames into a pre-trained object classification model for the multiple valid video frames to obtain target classification results corresponding to the valid video frames; wherein the target classification results are used to characterize the object types included in the valid video frames; the target object type determination module determines the target object types included in the valid video frames according to the target classification results; wherein the object classification model includes a feature extraction module, a feature classification module and a result output module, the feature classification module includes a low-dimensional feature capsule layer, a high-dimensional feature capsule layer and a classification capsule layer, the output of the feature extraction module is the input of the low-dimensional feature capsule layer, the output of the low-dimensional feature capsule layer is the input of the high-dimensional feature capsule layer, the output of the high-dimensional feature capsule layer is the input of the classification capsule layer, and the result output module is used to output the output result of the classification capsule layer. This solution obtains video stream data containing multiple valid video frames and inputs it into an object classification model consisting of a feature extraction module, a low-dimensional feature capsule layer, a high-dimensional feature capsule layer, a classification capsule layer, a feature classification module, and a result output module. After processing at each layer, the classification results are output to determine the type of target object in the valid video frame. This solves the problem of low classification accuracy caused by the traditional method's insufficient capture of feature space hierarchical relationships and semantic information, improves classification accuracy, and can realize real-time recognition and analysis of video objects.

[0111] Based on the above embodiment, optionally, a target classification result determination module 620 is specifically used to extract feature information in a valid video frame based on a feature extraction module in an object classification model; process the feature information based on a low-dimensional feature capsule layer to obtain a first feature vector, wherein the first feature vector includes multiple capsule vectors, each capsule vector includes a spatial posture feature and a first activation value; input the first feature vector output by the low-dimensional feature capsule layer into a high-dimensional feature capsule layer to obtain a second feature vector; input the second feature vector into a classification capsule layer to obtain a target classification result; and output the target classification result based on a result output module.

[0112] Optionally, the low-dimensional feature capsule layer includes a spatial posture extraction unit and an activation function value determination unit, wherein the spatial posture extraction unit is configured to determine first spatial posture information based on the feature information; and the activation function value determination unit is configured to determine a first activation value for the first spatial posture information.

[0113] Optionally, the high-dimensional feature capsule layer includes a data enhancement processing unit, at least two filtering units, a reverse attention unit, a frequency domain attention unit and a routing unit, and the at least two filtering units are divided into a first filtering unit and a second filtering unit; the target classification result determination module 620 is specifically used to receive a first feature vector based on the high-dimensional feature capsule layer; perform data enhancement processing on the first feature vector based on the data enhancement processing unit to obtain a feature enhancement vector and a second activation value; process the feature enhancement vector based on the first filtering unit and the second filtering unit respectively to obtain a first filter vector and a second filter vector; process the first filter vector based on the reverse attention unit to obtain a first attention feature, and process the second filter vector based on the frequency domain attention unit to obtain a second attention feature; process the first attention feature, the second attention feature, the first filter vector, the second filter vector, the feature enhancement vector and the second activation value based on the routing unit to obtain a second feature vector; wherein the second feature vector includes at least the object feature in the valid video frame, the second spatial posture information of the object feature and the third activation value of the object feature belonging to a preset category.

[0114] Optionally, the target classification result determination module 620 is specifically used to determine the coupling coefficient based on the first attention feature and the feature enhancement vector; obtain the third attention feature based on the second attention feature, the second posture matrix output by the data enhancement processing unit, and the output result after processing the first activation value; determine the spatial posture information and the second activation value based on the third attention feature, the coupling coefficient, the activation value determination function and the posture matrix determination function.

[0115] Optionally, the target object type determination module 630 is specifically configured to determine the target object type included in the valid video frame according to the category identifier in the target classification result.

[0116] The data processing device provided by the embodiment of the present invention can execute the data processing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0117] Example 4

[0118] Figure 71 is a structural diagram of an electronic device provided in Embodiment 4 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0119] like Figure 7 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0120] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0121] The processor 11 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the data processing method.

[0122] In some embodiments, the data processing method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data processing method in any other suitable manner (e.g., by means of firmware).

[0123] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0124] Computer programs for implementing the data processing methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0125] Example 5

[0126] Embodiment 5 of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a processor to execute a data processing method, the method comprising:

[0127] Acquire video stream data, and extract multiple valid video frames from the video stream data;

[0128] For multiple valid video frames, input the valid video frames into a pre-trained object classification model to obtain target classification results corresponding to the valid video frames; wherein the target classification results are used to characterize the types of objects included in the valid video frames;

[0129] Determine the target object type included in the valid video frame according to the target classification result;

[0130] Among them, the object classification model includes a feature extraction module, a feature classification module and a result output module. The feature classification module includes a low-dimensional feature capsule layer, a high-dimensional feature capsule layer and a classification capsule layer. The output of the feature extraction module is the input of the low-dimensional feature capsule layer, the output of the low-dimensional feature capsule layer is the input of the high-dimensional feature capsule layer, and the output of the high-dimensional feature capsule layer is the input of the classification capsule layer. The result output module is used to output the output result of the classification capsule layer.

[0131] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0132] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0133] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0134] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0135] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0136] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: include: Acquire video stream data, and extract multiple valid video frames from the video stream data; For the multiple valid video frames, input the valid video frames into a pre-trained object classification model to obtain object classification results corresponding to the valid video frames; wherein the object classification results are used to characterize the types of objects included in the valid video frames; Determining the target object type included in the valid video frame according to the target classification result; Among them, the object classification model includes a feature extraction module, a feature classification module and a result output module. The feature classification module includes a low-dimensional feature capsule layer, a high-dimensional feature capsule layer and a classification capsule layer. The output of the feature extraction module is the input of the low-dimensional feature capsule layer, the output of the low-dimensional feature capsule layer is the input of the high-dimensional feature capsule layer, and the output of the high-dimensional feature capsule layer is the input of the classification capsule layer. The result output module is used to output the output result of the classification capsule layer.

2. The method according to claim 1, characterized in that The step of inputting the valid video frame into a pre-trained object classification model to obtain a target classification result corresponding to the valid video frame includes: Extracting feature information from the valid video frame based on a feature extraction module in the object classification model; Processing the feature information based on the low-dimensional feature capsule layer to obtain a first feature vector, wherein the first feature vector includes a plurality of capsule vectors, each of the capsule vectors including a spatial posture feature and a first activation value; Inputting the first feature vector output by the low-dimensional feature capsule layer into the high-dimensional feature capsule layer to obtain a second feature vector; Inputting the second feature vector into the classification capsule layer to obtain a target classification result; The target classification result is output based on the result output module.

3. The method according to claim 2, characterized in that The low-dimensional feature capsule layer includes a spatial posture extraction unit and an activation function value determination unit. The spatial posture extraction unit is used to determine first spatial posture information based on the feature information; the activation function value determination unit is used to determine a first activation value of the first spatial posture information.

4. The method according to claim 2, characterized in that The high-dimensional feature capsule layer includes a data enhancement processing unit, at least two filtering units, a reverse attention unit, a frequency domain attention unit, and a routing unit, wherein the at least two filtering units are divided into a first filtering unit and a second filtering unit; The step of inputting the first feature vector output by the low-dimensional feature capsule layer into the high-dimensional feature capsule layer to obtain a second feature vector includes: receiving the first feature vector based on the high-dimensional feature capsule layer; Performing data enhancement processing on the first feature vector based on the data enhancement processing unit to obtain a feature enhancement vector and a second activation value; Processing the feature enhancement vector based on the first filtering unit and the second filtering unit respectively to obtain a first filtering vector and a second filtering vector; Processing the first filter vector based on the reverse attention unit to obtain a first attention feature, and processing the second filter vector based on the frequency domain attention unit to obtain a second attention feature; Processing the first attention feature, the second attention feature, the first filter vector, the second filter vector, the feature enhancement vector, and the second activation value based on the routing unit to obtain the second feature vector; The second feature vector includes at least the object feature in the valid video frame, the second spatial posture information of the object feature, and the third activation value of the object feature belonging to a preset category.

5. The method according to claim 4, characterized in that The second spatial posture information and the third activation value in the second feature vector are determined in the following manner: determining a coupling coefficient based on the first attention feature and the first filter vector; fusing the second attention feature and the second filter vector to obtain a third attention feature; The second spatial posture information and the third activation value are determined according to the third attention feature, the coupling coefficient, the second activation value, the activation value determination function and the posture matrix determination function.

6. The method according to claim 1, characterized in that The determining, according to the target classification result, the target object type included in the valid video frame comprises: The target object type included in the valid video frame is determined according to the category identifier in the target classification result.

7. A data processing device, characterized in that: include: A video frame extraction module is used to obtain video stream data and extract multiple valid video frames from the video stream data; an object classification result determination module, configured to input the plurality of valid video frames into a pre-trained object classification model to obtain object classification results corresponding to the valid video frames; wherein the object classification results are used to characterize the types of objects included in the valid video frames; a target object type determination module, configured to determine the target object type included in the valid video frame according to the target classification result; Among them, the object classification model includes a feature extraction module, a feature classification module and a result output module. The feature classification module includes a low-dimensional feature capsule layer, a high-dimensional feature capsule layer and a classification capsule layer. The output of the feature extraction module is the input of the low-dimensional feature capsule layer, the output of the low-dimensional feature capsule layer is the input of the high-dimensional feature capsule layer, and the output of the high-dimensional feature capsule layer is the input of the classification capsule layer. The result output module is used to output the output result of the classification capsule layer.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the data processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data processing method according to any one of claims 1 to 6 when executed.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the data processing method according to any one of claims 1 to 6.

Images