Video stream identification method based on task layering, distributed system and storage medium

Abstract

The invention provides a video stream identification method based on task layering, a distributed system and a storage medium, and relates to the field of computer vision based on artificial intelligence. The method comprises the following steps: extracting a target frame image in a target video stream and preprocessing the target frame image to obtain a preprocessed image; analyzing and describing the preprocessed image, and extracting an analysis result including object types, object relationships and background description information in the image; establishing a model library comprising an indoor scene recognition model, a handwritten number recognition model and an outdoor scene recognition model; the analysis results are correspondingly divided into different data sets; performing targeted training on different models by adopting the corresponding data sets to obtain an optimization model; and deploying the optimization model into a distributed system.

Description

Technical Field

[0001] The present invention relates to the field of computer vision based on artificial intelligence, and in particular to a video stream recognition method based on task layering, a distributed system and a storage medium. Background Art

[0002] Computer vision, as an important branch of artificial intelligence, is dedicated to enabling machines to "see" and understand the world. Its core lies in the analysis, recognition and understanding of images and videos. Advances in this field, such as the emergence of deep learning models, have greatly improved recognition accuracy, but have also brought about a high demand for computing resources.

[0003] In image processing, from raw image preprocessing, feature extraction to advanced semantic understanding, each step places high demands on algorithm efficiency and accuracy. Especially in practical applications, how to achieve fast and accurate image recognition with limited computing resources has become an urgent problem to be solved. Therefore, combining the advantages of computer vision and distributed learning, exploring efficient task scheduling and resource allocation strategies has become the key to improving the performance of image processing systems. Summary of the invention

[0004] Purpose of the invention: To propose a video stream recognition method, a distributed system and a storage medium based on task layering to solve the above-mentioned problems existing in the prior art.

[0005] First, a video stream recognition method based on task layering is proposed, and the steps are as follows:

[0006] Get the target video stream;

[0007] Extracting a target frame image from a target video stream, and preprocessing the target frame image to obtain a preprocessed image;

[0008] Analyzing and describing the preprocessed image to extract analysis results; the analysis results include object categories, object relationships and background description information in the preprocessed image;

[0009] Building a model library; the model library includes an indoor scene recognition model, a handwritten digit recognition model, and an outdoor scene recognition model;

[0010] The analysis results are divided into different data sets; the corresponding data sets are used to perform targeted training on different models to obtain optimized models;

[0011] The optimization model is deployed into a distributed system.

[0012] In a further embodiment of the first aspect, the target video stream is obtained by: capturing the video stream on the display device through a camera, or directly importing a video file and decoding it locally into a video stream.

[0013] In a further embodiment of the first aspect, the target frame image is preprocessed, including noise reduction filtering and binarization.

[0014] In a further embodiment of the first aspect, the pre-processed image is analyzed and described by constructing an encoder-decoder framework;

[0015] The encoder-decoder framework includes a pre-trained CNN classification network, an SGTR scene graph generation transformer, and a support vector machine;

[0016] The CNN classification network identifies the object categories in the preprocessed image;

[0017] The object relationships between different objects in the preprocessed image are analyzed by the SGTR scene graph generation transformer.

[0018] Object relationships include spatial relationships and / or interaction relationships, which are represented by triple relationships:<subject,predicate,object> ; Wherein, subject represents the subject in the image, object represents the object in the image, and predicate represents the spatial relationship and / or interaction relationship between the subject and the object;

[0019] Analyzing the background description information of the pre-processed image by a support vector machine to classify the environment where the object in the pre-processed image is located, including indoor or outdoor;

[0020] Summarize object categories, object relationships, and background description information to form analysis results.

[0021] In a further embodiment of the first aspect, the output expression of the convolutional layer in the CNN classification network is as follows:

[0022]

[0023] Where b env is the environment-related bias term; is the environment-related convolution kernel; ReLU(*) represents the linear rectification function; f env (x) represents the output of the convolutional layer with environmental awareness; I i Represents a region or feature map in the input image; n represents the number of convolution kernels.

[0024] The expression of the attention mechanism in the SGTR scene graph generation converter is as follows:

[0025]

[0026] In the formula, Q obj , K obj 、V obj Represent the query vector, key vector, and value vector of the object in the predetermined environment respectively; d k represents the dimension of the key vector; V obj A vector of values ​​representing the object; represents the transpose of the key vector; Softmax(*) represents the normalized exponential function.

[0027] When the support vector machine processes the background description information of the preprocessed image, the influence of the environment type is considered and the optimization problem is expressed as:

[0028]

[0029] sty i (w T φ env (x i )+b)1-ξ i ,ξ i ≥0

[0030] In the formula, y i Indicates the environment type label, which can be divided into indoor or outdoor; φ env (x i ) represents the environmental feature mapping function, which is used to transform the input feature x i Mapped to a new feature space; w represents the norm of the weight vector w, which is used to control the complexity of the model; b represents the bias term; C represents the regularization parameter, which is used to balance the maximization of the classification interval and the minimization of the number of misclassified points; m represents the number of samples in the training data set; ξ i represents a slack variable, which is used to allow some samples to be within the interval or misclassified; b represents a bias term.

[0031] In a further embodiment of the first aspect, according to the analysis results, a decision tree algorithm is used to select a corresponding model in a model library, and different data sets are used to perform targeted training on the model to obtain an optimized model;

[0032] The optimization model is packaged into a Docker image and deployed to a distributed system.

[0033] In a further embodiment of the first aspect, the execution process of the decision tree algorithm includes:

[0034] Define the input attributes of the decision tree, including object categories, object relationships, and background descriptions;

[0035] Define the target output of the decision tree, including indoor scene recognition model, handwritten digit recognition model, and outdoor scene recognition model;

[0036] Use the training dataset to build a decision tree model; the training process splits the dataset based on the value of the attribute and recursively creates child nodes for each split until a preset stopping condition is reached;

[0037] Each internal node of the decision tree represents an attribute test; each leaf node represents a category, that is, the selected model;

[0038] When classifying new data, start from the root node and move down the tree according to the attribute test results on the node until you reach a leaf node, and then output the category corresponding to the leaf node;

[0039] The node splitting in the decision tree is evaluated by information gain, which is calculated as:

[0040]

[0041] In the formula, Child j It represents the subset obtained by segmentation through feature attributes; Parent represents the state of the current node before segmentation; Entropy(*) is the calculation formula of entropy;

[0042]

[0043] In the formula, S is the sample set; c is the number of categories; p i is the proportion of samples of the i-th category to the total samples.

[0044] The second aspect of the present invention proposes a distributed video stream recognition system based on task layering. The video stream recognition method based on task layering disclosed in the first aspect above divides the image recognition task into two layers: the first layer uses a large model on an edge high-performance computer to perform an overall analysis to determine the object categories, object relationships, and background descriptions in the environment; the second layer deploys a lightweight model on distributed devices based on the results of the first layer.

[0045] According to a third aspect of the present invention, a computer-readable storage medium is provided, wherein at least one executable instruction is stored in the storage medium. When the executable instruction is executed on an electronic device, the electronic device executes the task-layered video stream recognition method disclosed in the first aspect.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] (1) By dividing the image recognition task into two levels, the present invention can utilize computing resources more efficiently. The first level uses a larger model for global analysis, quickly filters and classifies basic information, and reduces the complexity of subsequent processing. The second level uses a lightweight model for detailed recognition based on the results of the first level, which not only reduces unnecessary full-image fine analysis, but also speeds up the recognition speed and improves the system response efficiency.

[0048] (2) The hierarchical strategy allows the system to focus on different recognition targets at different levels. The large model in the first layer can better understand the overall context of the image, such as object categories and the relationships between them, which is crucial for accurately grasping the meaning of the image. The lightweight model in the second layer focuses on the detailed feature recognition of specific objects. This refined processing helps to improve the recognition accuracy of object details and ensure that even tiny features in complex scenes are not ignored.

[0049] (3) The layered architecture design of the present invention provides higher flexibility and scalability for distributed systems. As technology advances or requirements change, the models of each layer can be independently upgraded or replaced without rebuilding the entire system. In addition, this modular design facilitates adding new recognition tasks or optimizing the performance of a specific layer to adapt to more diverse application scenarios and future technological developments, thereby maintaining the competitiveness and effectiveness of the system in the long term. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic diagram of the architecture of a distributed system for video stream recognition based on task layering in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features known in the art are not described.

[0052] Embodiment 1:

[0053] This embodiment discloses a video stream recognition method based on task layering, the steps are as follows:

[0054] Capture the video stream on the display device through the camera, or directly import the video file and decode it into a video stream locally.

[0055] The target frame image is extracted from the target video stream, and denoising filtering and binarization operations are performed on the target frame image to obtain a preprocessed image.

[0056] An encoder-decoder framework is constructed to analyze and describe the preprocessed images and extract the analysis results.

[0057] Among them, the encoder-decoder framework consists of three parts: pre-trained CNN classification network, SGTR scene graph generation converter, and support vector machine.

[0058] Formula (1) is the output expression of the convolutional layer in the CNN classification network.

[0059]

[0060] b env is the environment-related bias term; is the environment-related convolution kernel; ReLU(*) represents the linear rectification function.

[0061] Formula (2) is the expression of the attention mechanism in the SGTR scene graph generation transformer.

[0062]

[0063] Q obj , K obj 、V obj Respectively represent the query vector, key vector, and value vector of the object in the predetermined environment; represents the transpose of the key vector; Softmax(*) represents the normalized exponential function.

[0064] When the support vector machine (SVM) processes the background description information of the preprocessed image, the influence of the environment type is considered and the optimization problem is expressed as Equation (3) and Equation (4).

[0065]

[0066] st y i (w T φ env (x i )+b)1-ξ i ,ξ i ≥0 (4)

[0067] y i Indicates the environment type label (indoor or outdoor); φ env (x i ) represents the environmental feature mapping function, which is used to transform the input feature x i Mapped to a new feature space.

[0068] The CNN classification network identifies the object categories in the preprocessed image; the SGTR scene graph generation converter analyzes the object relationships between different objects in the preprocessed image; the object relationships include spatial relationships and / or interaction relationships, which are represented by triple relationships as follows:<subject,predicate,object> ; Wherein, subject represents the subject in the image, object represents the object in the image, and predicate represents the spatial relationship and / or interaction relationship between the subject and the object; the support vector machine analyzes the background description information of the preprocessed image and classifies the environment where the object in the preprocessed image is located, including indoor or outdoor. The object category, object relationship and background description information are summarized to form the analysis result.

[0069] Build a model library including indoor scene recognition model, handwritten digit recognition model, and outdoor scene recognition model.

[0070] The analysis results are divided into different data sets, and the decision tree algorithm is used to select the corresponding model in the model library. Different data sets are used to conduct targeted training on the model to obtain the optimized model.

[0071] The process of building a decision tree algorithm is as follows:

[0072] 1. Define the input attributes (features) of the decision tree, such as object categories, object relationships, background descriptions, etc.

[0073] 2. Define the target output (label) of the decision tree, which is the selected lightweight model.

[0074] 3. Use the training data set to build a decision tree model. The training process splits the data set according to the value of the attribute and recursively creates child nodes for each split until the stopping condition is reached (all instances belong to the same class or the maximum tree depth is reached, etc.).

[0075] 4. Each internal node of the decision tree represents an attribute test, and each leaf node represents a category (here is a model choice).

[0076] 5. When classifying new data, start from the root node and move down the tree according to the attribute test results on the node until you reach a leaf node, and then output the category corresponding to the leaf node (model selection).

[0077] Mathematically, node splitting in a decision tree is evaluated by information gain. Information gain measures the reduction in entropy after splitting by a feature, and is used to determine the importance of a feature. The greater the information gain, the more helpful this feature is for classification. The information gain calculation formula is shown in formula (5).

[0078]

[0079] Finally, the optimized model is packaged into a Docker image and deployed to a distributed system.

[0080] Embodiment 2:

[0081] This embodiment discloses a distributed system for video stream recognition based on task stratification, which can be used to implement the specific process of video stream recognition disclosed in the above embodiment 1. Figure 1 .

[0082] First layer end-to-end image holistic analysis

[0083] Video stream access: The robot captures the video stream on the big screen through the camera.

[0084] Image preprocessing: Preprocess the images in the video stream, including noise reduction filtering and binarization, to improve image quality and reduce noise and interference.

[0085] Noise reduction filter: Gaussian filter kernel size: 5x5

[0086] Standard deviation σ: 1.0

[0087] Binarization: Threshold: 128

[0088] Multimodal large model analysis: Use an encoder-decoder framework to analyze and describe the image. This process extracts object categories, object relationships, and background description information.

[0089] Learning rate: 0.001

[0090] Batch Size: 32

[0091] Hidden layer size: 512

[0092] Attention Heads: 8

[0093] Object category recognition: Based on the features of the image, the pre-trained CNN classification model is used to automatically identify the object categories in the image.

[0094] Input image size: 416x416

[0095] Confidence threshold: 0.5

[0096] Non-maximum suppression (NMS) threshold: 0.4

[0097] Object relationship analysis: Use SGTR (Scene Graph Generation Transformer) to further analyze the spatial relationship and interaction relationship recognition algorithm between objects. The results are represented by triple relationships, namely<subject,predicate,object> These triplets capture the complex relationship between objects in the image, where subject and object represent the objects in the image, and predicate describes the relationship between the two objects. (For example, output <book, on..., table>: directly indicates that the book is placed on the table).

[0098] Background description: Combine the overall information of the image and use support vector machines to classify the environment in which the object is located.

[0099] Background description vocabulary size: 2, select the environment type (indoor / outdoor) according to the vocabulary.

[0100] Second layer lightweight deployment

[0101] Model optimization: Based on the analysis results obtained in the first layer, the decision tree algorithm is used to select different models in the model library (yolo-v4-coco-indoor / svm-Lenet-digital / yolo-v7-ImageNet-outdoor) and train them with different data sets. The selected models are further optimized, such as pruning and quantization, to adapt to the edge computing environment, reduce computing load and improve performance.

[0102] Pruning rate: 0.500 (removing 50% of the model parameters or neurons)

[0103] Pruning strategy: global pruning, pruning based on the absolute value of weights

[0104] Post-pruning fine-tuning rounds: 10 rounds (fine-tune the model after pruning to restore performance)

[0105] Learning rate (fine-tuning): initial learning rate 0.001, using learning rate decay strategy (cosine annealing)

[0106] Quantization bit number: 8 bits (from floating point to 8-bit integer)

[0107] Distributed deployment: Package the optimized model into docker and upload it to the distributed system.

[0108] Embodiment 3:

[0109] This embodiment discloses an example of an indoor robot fixed-point strike platform using the above-mentioned embodiments 1 and 2.

[0110] 1. Hardware parameters

[0111] Jetson Robotics Platform: Adoption Jetson AGX Xavier TM As the main controller, it has powerful computing power and high energy efficiency. Jetson Nano TM The remote-controlled robots serve as the video acquisition end and the turret operation end respectively.

[0112] Camera: A 1080p high-resolution, high-frame-rate wide-angle camera is used to obtain video streams for identifying the environment on the large screen.

[0113] Storage device: Equipped with 128G storage device to store video streams, algorithm operation results and other related data.

[0114] Power supply equipment: Provides stable power supply for the robot platform and other hardware equipment.

[0115] Network equipment: Indoor WLAN used to connect the robot platform with other devices or systems for data transmission and communication.

[0116] Strike function turret: A turret with three rotating strike functions, used to strike targets.

[0117] 2. Software Environment

[0118] Operating system: Ubuntu operating system is used to support the operation and development environment of the Jetson robot platform.

[0119] Programming language: Python programming language is used to write and debug the programs and algorithms in this technical solution.

[0120] Development Tools: Use the TensorFlow deep learning framework and library to implement algorithms and techniques such as image encoders, decoders, and pre-trained models.

[0121] Debugging tool: Use the GDB debugging tool to debug and optimize the programs and algorithms in this technical solution.

[0122] Network communication library: Use the ROS robot operating system library to achieve data transmission and communication between the robot platform and other devices or systems.

[0123] Data repository: Use the SQLite database library to store and manage video streams, algorithm operation results, and other related data.

[0124] Other related software libraries and tools: Use image processing libraries and tools such as OpenCV to support algorithms and techniques such as image encoders, decoders, and pre-trained models.

[0125] 3. Experimental Content

[0126] The robot uses the camera to recognize the simulated environment on the big screen: the robot obtains the video stream on the big screen through the camera and uploads it to Jetson AGX Xavier TM The edge server performs the first level of analysis and description of the environment. During the analysis process, the encoder-decoder framework is used for image analysis and description. This stage obtains the object categories (threat / non-threat), object relationships (<subject,predicate,object> ) and background description (indoor / outdoor).

[0127] Switching algorithm: The analysis result obtained in the first layer is [threat, <human, behind..., wall>, outdoor], and yolo-v7-ImageNet-outdoor is automatically selected from the model library (yolo-v4-coco-indoor / svm-Lenet-digital / yolo-v7-ImageNet-outdoor).

[0128] The optimized model is packaged, uploaded and deployed to the distributed system. Jetson Nano TM When the turret car identifies the target object, it manually confirms the target that needs to be tracked and hit. The turret car aims according to the received coordinates and prepares to hit the target.

[0129] In practical applications, the operation logic of the video stream identification method based on task layering disclosed in the above embodiment can also be written into a set of executable instructions, which are written into a storage medium and run on an electronic device. More specific examples of the computer-readable storage medium mentioned in this embodiment may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program used by or in combination with an instruction execution system, an apparatus, or a device.

[0130] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes may be made to it in form and detail without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A video stream recognition method based on task layering, characterized in that: The steps include: Get the target video stream; Extracting a target frame image from a target video stream, and preprocessing the target frame image to obtain a preprocessed image; Analyzing and describing the preprocessed image to extract analysis results; the analysis results include object categories, object relationships and background description information in the preprocessed image; Building a model library; the model library includes an indoor scene recognition model, a handwritten digit recognition model, and an outdoor scene recognition model; The analysis results are divided into different data sets; the corresponding data sets are used to perform targeted training on different models to obtain optimized models; The optimization model is deployed into a distributed system.

2. The video stream recognition method based on task layering according to claim 1 is characterized in that: The target video stream is obtained by: capturing the video stream on the display device through a camera, or directly importing a video file and decoding it into a video stream locally; and preprocessing the target frame image, including noise reduction filtering and binarization.

3. The video stream recognition method based on task layering according to claim 1 is characterized in that: Analyzing and describing the preprocessed image by constructing an encoder-decoder framework; The encoder-decoder framework includes a pre-trained CNN classification network, an SGTR scene graph generation converter, and a support vector machine; The CNN classification network identifies the object category in the preprocessed image; The SGTR scene graph generator analyzes the object relationship between different objects in the preprocessed image; the object relationship includes a spatial relationship and / or an interaction relationship, which is represented by a triple relationship as<subject,predicate,object> ; Wherein, subject represents the subject in the image, object represents the object in the image, and predicate represents the spatial relationship and / or interaction relationship between the subject and the object; The support vector machine analyzes the background description information of the pre-processed image to classify the environment where the object in the pre-processed image is located, including indoor or outdoor; The object categories, object relationships and background description information are summarized to form an analysis result.

4. The video stream recognition method based on task layering according to claim 3 is characterized in that: The output expression of the convolutional layer in the CNN classification network is as follows: Where b env is the environment-related bias term; is the environment-related convolution kernel; ReLU(*) represents the linear rectification function; f env (x) represents the output of the convolutional layer with environmental awareness; I i Represents a region or feature map in the input image; n represents the number of convolution kernels; The expression of the attention mechanism in the SGTR scene graph generation converter is as follows: In the formula, Q obj , K obj 、V obj Represent the query vector, key vector, and value vector of the object in the predetermined environment respectively; d k represents the dimension of the key vector; V obj A vector of values ​​representing the object; represents the transpose of the key vector; Softmax(*) represents the normalized exponential function.

5. The video stream recognition method based on task layering according to claim 4 is characterized in that: When the support vector machine processes the background description information of the preprocessed image, the influence of the environment type is considered and the optimization problem is expressed as: style i (w T f env (x i )+b)1-ξ i ,x i ≥0 In the formula, y i Indicates the environment type label, which can be divided into indoor or outdoor; φ env (x i ) represents the environmental feature mapping function, which is used to transform the input feature x i Mapped to a new feature space; w represents the norm of the weight vector w, which is used to control the complexity of the model; b represents the bias term; C represents the regularization parameter, which is used to balance the maximization of the classification interval and the minimization of the number of misclassified points; m represents the number of samples in the training data set; ξ i represents a slack variable, which is used to allow some samples to be within the interval or misclassified; b represents a bias term.

6. The video stream recognition method based on task layering according to claim 3 is characterized in that: According to the analysis results, a decision tree algorithm is used to select a corresponding model in the model library, and different data sets are used to perform targeted training on the model to obtain an optimized model; The optimization model is packaged into a Docker image and deployed to a distributed system.

7. The video stream recognition method based on task layering according to claim 6 is characterized in that: The execution process of the decision tree algorithm includes: Define the input attributes of the decision tree, including object categories, object relationships, and background descriptions; Define the target output of the decision tree, including indoor scene recognition model, handwritten digit recognition model, and outdoor scene recognition model; Use the training dataset to build a decision tree model; the training process splits the dataset based on the value of the attribute and recursively creates child nodes for each split until a preset stopping condition is reached; Each internal node of the decision tree represents an attribute test; each leaf node represents a category, that is, the selected model; When classifying new data, start from the root node and move down the tree according to the attribute test results on the node until you reach a leaf node, and then output the category corresponding to the leaf node, that is, the selected model.

8. The video stream recognition method based on task layering according to claim 7 is characterized in that: The node splitting in the decision tree is evaluated by information gain, which is calculated as: In the formula, Child j It represents the subset obtained by segmentation through feature attributes; Parent represents the state of the current node before segmentation; Entropy(*) is the calculation formula of entropy; In the formula, S is the sample set; c is the number of categories; p i is the proportion of samples of the i-th category to the total samples.

9. A distributed system for video stream recognition based on task stratification, characterized in that: By using the task-layered video stream recognition method according to any one of claims 1 to 8, the image recognition task is divided into two layers: The first layer uses a large model on edge high-performance computers to perform overall analysis to determine the object categories, object relationships, and background descriptions in the environment; The second layer deploys a lightweight model based on the results of the first layer.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one executable instruction. When the executable instruction is executed on the electronic device, the electronic device executes the task-layered video stream recognition method according to any one of claims 1 to 8.

Images