A method and system for object recognition processing based on video images

Abstract

The present application discloses a method and system for object recognition processing based on video images, the method comprising: acquiring a series of frames from a video in chronological order and treating the series of frames as a sequence of images; selecting a predetermined image from the sequence of images, and selecting an object to be recognized from the predetermined image; tracking the object in the sequence of images, marking the center point of the object to be recognized in each image in the sequence of images; segmenting the object to be recognized from the image according to the center point in each image, spatially reconstructing the object to be recognized based on the segmentation result to obtain three-dimensional information of the object to be recognized; and obtaining the posture of the object to be recognized based on the three-dimensional information. This application solves the problem of low accuracy in the prior art for the posture recognition of objects in videos, thereby improving the recognition accuracy of the object posture in video images to a certain extent.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method and system for object recognition processing based on video images. Background Art

[0002] The patent application document with application number 201911066497.6 discloses a method, device, medium and equipment for determining the posture of a target object. The method includes: determining historical posture data of the target object from a previous frame image including the target object; obtaining historical posture distribution information based on the historical posture data of the target object; determining historical posture data that meets preset requirements as standard posture data based on the historical posture distribution information; and determining the target posture of the target object in the current frame image based on the standard posture data.

[0003] The background technology of this patent application discloses the following: "While driving a vehicle, in order to ensure the safety of the driver and passengers, a vehicle-mounted camera can be set to monitor the driver's driving behavior for warning and correction. Among them, many illegal driving behaviors can be identified by estimating facial features, such as smoking, making phone calls, and fatigue driving, etc." It can be seen that the technology of this patent application is mainly used for the identification of drivers. Drivers are usually fixed, so it is easy to accumulate data, so that historical posture data can be used for posture recognition. However, in many cases, there is no historical posture data, such as video surveillance in public places. In this case, the method of this patent application cannot be applied, and other existing posture recognition methods have the problem of low accuracy. Summary of the Invention

[0004] The embodiments of the present application provide a method and system for object recognition processing based on video images, so as to at least solve the problem of low accuracy in posture recognition of objects in videos in the prior art.

[0005] According to one aspect of the present application, a method for target recognition processing based on video images is provided, comprising: acquiring a series of frames from a video according to a time sequence and using the series of frames as a sequence of images; selecting a predetermined image from the sequence of images, and selecting a target to be recognized from the predetermined image; tracking the target in the sequence of images, marking the center point of the target to be recognized in each image in the sequence of images; segmenting the target to be recognized from the image according to the center point in each image, and spatially reconstructing the target to be recognized according to the segmentation result to obtain three-dimensional information of the target to be recognized; and obtaining the posture of the target to be recognized based on the three-dimensional information.

[0006] Furthermore, the first image is selected from the sequence of images as the predetermined image.

[0007] Furthermore, selecting the target to be identified from the predetermined image includes: encoding the predetermined image to obtain an encoding of the entire image; obtaining a mask of the target to be selected in the image based on the identified target to be identified; segmenting the target to be identified from the image according to the center point in each image includes: obtaining a mask of the target to be identified in the image according to the center point of the target to be identified in each image, and segmenting the target to be identified from the image according to the mask.

[0008] Furthermore, obtaining the mask of the target to be identified in each image includes: clustering the pixel points in each image using a clustering algorithm in a neural network model according to the center point in each image and the identification information for identifying the target to be identified in the predetermined image, to obtain pixel points belonging to the target to be identified and pixel points that do not belong to the target to be identified; and generating a mask of the target to be identified in the image according to the pixel points belonging to the target to be identified.

[0009] Furthermore, a ratio of the mask of the target to be identified in the image to the overall mask of the image is lower than a threshold.

[0010] According to another aspect of the present application, a target recognition and processing system based on video images is also provided, including: an acquisition module for acquiring a series of frames from a video according to a time sequence and using the series of frames as a sequence of images; a selection module for selecting a predetermined image from the sequence of images and selecting a target to be recognized from the predetermined image; a marking module for tracking the target in the sequence of images and marking the center point of the target to be recognized in each image in the sequence of images; a segmentation module for segmenting the target to be recognized from the image according to the center point in each image, and spatially reconstructing the target to be recognized according to the segmentation result to obtain three-dimensional information of the target to be recognized; and a processing module for obtaining the posture of the target to be recognized based on the three-dimensional information.

[0011] Furthermore, the selection module is used to: encode the predetermined image to obtain the encoding of the entire image; obtain the mask of the target to be selected in the image based on the identified target to be identified; the segmentation module is used to: obtain the mask of the target to be identified in the image based on the center point of the target to be identified in each image, and segment the target to be identified from the image according to the mask.

[0012] Furthermore, the selection module is used to: cluster the pixels in each image using a clustering algorithm in a neural network model according to the center point in each image and the identification information for identifying the target to be identified in the predetermined image, to obtain pixel points belonging to the target to be identified and pixel points that do not belong to the target to be identified; and generate a mask of the target to be identified in the image according to the pixel points belonging to the target to be identified.

[0013] According to another aspect of the present application, an electronic device is also provided, comprising a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the above-mentioned method steps.

[0014] According to another aspect of the present application, a readable storage medium is provided, on which computer instructions are stored, wherein the computer instructions implement the above method steps when executed by a processor.

[0015] In an embodiment of the present application, a series of frames are obtained from a video in chronological order and used as a sequence of images; a predetermined image is selected from the sequence of images, and a target to be identified is selected from the predetermined image; the target is tracked in the sequence of images, and the center point of the target to be identified in each image in the sequence of images is marked; in each image, the target to be identified is segmented from the image according to the center point, and the target to be identified is spatially reconstructed based on the segmentation result to obtain three-dimensional information of the target to be identified; and the posture of the target to be identified is obtained based on the three-dimensional information. This application solves the problem of low accuracy in the posture recognition of targets in videos in the prior art, thereby improving the recognition accuracy of target postures in video images to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0017] Figure 1 is a flowchart of a method for object recognition processing based on video images according to an embodiment of the present application;

[0018] Figure 2 This is a flowchart of a method for calculating the posture of a small target at a distance based on computer vision according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0020] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0021] In the following implementation, masks in image processing are involved, so we will first explain masks. In image processing, a mask is a special image used to specify the area to be operated on the original image. Masks are usually binary images (i.e., each pixel on the image has only two possible values, usually 0 and 255, representing black and white, respectively), but can also be grayscale images or multi-channel images. However, binary masks are the most common in general applications. Masks can be applied in the following aspects:

[0022] Extracting Region of Interest (ROI): Through masking, specific regions can be extracted from the original image for further analysis or processing. In the mask, the region of interest is set to white (or a higher grayscale value), while the region of no interest remains black (or a lower grayscale value).

[0023] Removing unwanted parts: In contrast to extracting regions of interest, masks can also be used to remove parts of an image. These areas can be removed by setting the corresponding areas in the mask to black and then applying the mask to the original image.

[0024] Transparency processing: When processing multi-channel images (such as RGB images), the mask can also be used as a transparency channel to control the visibility of various parts of the image.

[0025] Image fusion: Through different masks, different parts of multiple images can be fused together to create images with special effects.

[0026] Image segmentation also involves image encoding. For example, image segmentation can be achieved through three interactive modes: click, box selection, and automatic recognition (the constraints on the objects to be segmented used in these interactive modes can be understood as hint encoding). Image segmentation can be achieved using neural network models. For example, after being trained on millions of images and over a billion masks, these neural network models can return valid segmentation masks for any hint. In this case, the hint is the segmentation task, which can be foreground / background points, a rough bounding box or mask, a click, text, or generally any information indicating the content to be segmented in the image. This task also serves as a pre-training objective for the model. The neural network model architecture can include three components that work together to return valid segmentation masks: an image encoder, which generates a one-time image embedding; a hint encoder, which generates a hint embedding (the hint can be a point, box, or text); and a lightweight mask decoder that combines the hint and image encoder embeddings. The image encoder generates a one-time image embedding that can be applied before the hint model. The hint encoder encodes background points, masks, bounding boxes, or text into an embedding vector in real time. Points and boxes are represented by positional encodings, with learned embeddings added for each hint type. Free-form text hints are represented by an off-the-shelf text encoder from CLIP. Dense hints, such as masks, are embedded in convolutions and summed element-wise with the image embedding. The mask decoder predicts a segmentation mask based on the embeddings from the image and hint encoders. It maps the image embedding, hint embedding, and output token to the mask. All embeddings are updated by the decoder block, which uses hint self-attention and cross-attention in both directions (from hint to image embedding and back).

[0027] The following implementation also utilizes a video tracking algorithm. Video object tracking is widely used in fields such as surveillance, autonomous driving, and human-computer interaction. There are many video tracking algorithms. For example, OpenCV provides several object tracking algorithms, among which MeanShift and CamShift are two classic and commonly used algorithms. The MeanShift algorithm is a density-based, non-parametric clustering algorithm originally used for image segmentation and later introduced to the field of object tracking. Its core concept is to track the target by iteratively calculating the centroid of the target region and shifting the window center to the centroid. The basic steps of the MeanShift algorithm are as follows: Initialize the window: In the first frame of the video, manually or automatically select a target region as the initial window. Calculate the centroid: In the current window, calculate the centroid of the target region (i.e., the mean of the pixels). Move the window: Move the window center to the centroid. Iterate: Repeat steps 2 and 3 until the window center stops changing or the maximum number of iterations is reached. The CamShift (Continuously Adaptive MeanShift) algorithm is an improved version of MeanShift that better tracks targets by adaptively adjusting the window size. The CamShift algorithm builds on MeanShift by adding window size and orientation adjustments, allowing it to adapt to changes in the target's size and rotation within the video. The basic steps of the CamShift algorithm are as follows: Initialize the window: Similar to MeanShift, select an initial window in the first frame of the video. Calculate the center of mass: Calculate the center of mass of the target area within the current window. Move the window: Move the window center to the center of mass. Adjust the window size and orientation: Adjust the window based on the target's size and orientation. Iterate: Repeat steps 2 through 4 until the window center stops changing or the maximum number of iterations is reached. Of course, there are many other video tracking algorithms, all of which can be applied to the following examples and will not be detailed here.

[0028] The following embodiments also involve neural network models, such as the k-means clustering algorithm, which is explained below. The k-means clustering algorithm is an iterative clustering analysis algorithm, whose steps are: pre-dividing the data into K groups, then randomly selecting K objects as initial cluster centers, then calculating the distance between each object and each seed cluster center, and assigning each object to the cluster center closest to it. The cluster centers and the objects assigned to them represent a cluster. Each time a sample is assigned, the cluster center of the cluster is recalculated based on the existing objects in the cluster. This process will be repeated until a certain termination condition is met. The termination condition can be that no (or a minimum number of) objects are reassigned to different clusters, no (or a minimum number of) cluster centers change again, and the sum of squared errors is locally minimized. Other clustering algorithms or neural network models can also be applied to the following embodiments, and will not be described in detail here.

[0029] It should be noted that the algorithms in the following embodiments can be applied to target recognition in various situations, especially for the pose calculation of small targets at a distance. The methods in the following embodiments will achieve better results. In the following embodiments, a method for calculating the pose of small targets at a distance based on computer vision is provided. This algorithm can accurately calculate the six-degree-of-freedom (6D) pose of small targets at a distance and weakly textured targets (of course, it can also be used to identify other types of targets). The following example uses the calculation of small targets at a distance to illustrate. The small targets mentioned here can be understood as targets whose proportion in the image (video frame) is less than a threshold.

[0030] The following embodiment provides a method for object recognition processing based on video images. Figure 1 is a flow chart of a method for object recognition processing based on video images according to an embodiment of the present application. Figure 1 As shown below, Figure 1 The steps involved in the method are described.

[0031] Step S102 : acquiring a series of frames from the video in time sequence and using the series of frames as a sequence of images.

[0032] Step S104 : selecting a predetermined image from the sequence of images, and selecting a target to be identified from the predetermined image; for example, selecting the first image from the sequence of images as the predetermined image.

[0033] Step S106 : Tracking the target in the sequence of images, and marking the center point of the target to be identified in each image in the sequence of images.

[0034] Step S108 : Segment the target to be identified from each image according to the center point, and spatially reconstruct the target to be identified according to the segmentation result to obtain three-dimensional information of the target to be identified.

[0035] Step S110 , obtaining the posture of the target to be identified according to the three-dimensional information.

[0036] The above steps solve the problem of low accuracy in the prior art in recognizing the posture of a target in a video, thereby improving the recognition accuracy of the posture of a target in a video image to a certain extent.

[0037] In the above steps, the target to be identified can be selected using a mask. For example, selecting the target to be identified from the predetermined image can include the following steps: encoding the predetermined image to obtain an encoding of the entire image; and obtaining a mask of the target to be identified within the image based on the identified target to be identified. As an optional embodiment, a determination is made as to whether the ratio of the mask of the target to be identified within the image to the overall mask of the image is below a threshold. If so, the segmentation, reconstruction, and pose recognition steps are continued. Otherwise, another pre-configured algorithm is selected for target recognition.

[0038] Other algorithms can be used in this context. For example, patent application No. 202110979155.4 provides a method, apparatus, computing device, and storage medium for inverting the attitude of a space target. The method includes: obtaining the principal axis vector of the space target corresponding to the ISAR image; generating a parameter vector corresponding to each particle in a particle swarm; the parameter vector being a vector composed of preset attitude angle parameters; calculating, for each particle, the two-dimensional projection vector of the parameter vector corresponding to the particle on the ISAR imaging plane; and constructing a cost function for the particle based on the two-dimensional projection vector and the principal axis vector of the particle; cyclically updating the parameter vector of each particle according to the particle swarm optimization algorithm and the cost function of each particle until a pre-set loop exit condition is met, at which point the loop update ends; and determining the attitude angle of the space target based on the currently updated parameter vector of each particle in the particle swarm. This solution has technical drawbacks and deficiencies: the particle swarm optimization (PSO) algorithm may fall into a local optimum during the solution process, especially when the objective function and cost function have complex forms or multiple local optima. If the particle swarm lacks an effective global search strategy, the resulting attitude angle may not be the globally optimal solution, thus affecting the accuracy of the inversion results. The cost function construction of this method relies on the two-dimensional projection vector and principal axis vector of the particle. If the cost function does not well capture the characteristics of the space target's attitude or performs poorly under complex ISAR image conditions, the accuracy of the final attitude angle estimation may be affected. In addition, the preset attitude angle parameters may not be sufficient to fully describe the attitude changes of the space target, limiting the applicability of the method. Therefore, this method is not recommended for identifying the attitude of small targets, as its accuracy is affected when identifying the attitude of small targets.

[0039] After obtaining the mask, segmenting the target to be identified from the image according to the center point in each image includes: obtaining a mask of the target to be identified in the image according to the center point of the target to be identified in each image, and segmenting the target to be identified from the image according to the mask.

[0040] In order to make the acquisition of the mask more accurate, a neural network model can be used to further process the mask, that is, obtaining the mask of the target to be identified in each image can include the following steps: clustering the pixels in each image using the clustering algorithm in the neural network model according to the center point in each image and the identification information for identifying the target to be identified in the predetermined image, and obtaining the pixel points belonging to the target to be identified and the pixel points that do not belong to the target to be identified; generating the mask of the target to be identified in the image according to the pixel points belonging to the target to be identified.

[0041] In one optional embodiment, two clustering algorithms are used to cluster the pixels in each image. A first clustering algorithm is used to cluster the pixels to obtain a first pixel belonging to the target to be identified; a second clustering algorithm is used to cluster the pixels to obtain a second pixel belonging to the target to be identified. The pixels that are identical between the first and second pixels are considered to be pixels belonging to the target to be identified. This optional embodiment can improve the accuracy of the mask of the target to be identified in the image.

[0042] The above embodiment can be used for identifying small targets, that is, the ratio of the mask of the target to be identified in the image to the overall mask of the image is lower than a threshold.

[0043] As an optional embodiment, segmenting the target to be identified from each image according to the center point includes: after obtaining the mask of the target to be identified in each image, obtaining feature points in the mask that can reflect the characteristics of the target to be identified, and matching the feature points in each image; segmenting the target to be identified from the image in each image according to the matching result and the center point.

[0044] Figure 2 FIG. 1 is a flow chart of a method for calculating the posture of a small target at a distance based on computer vision according to an embodiment of the present application. Figure 2 As shown in the figure, in this process, after selecting the first image, the target is selected in the first image, and then the target is tracked based on the sequence effect. After the target is finely segmented, the target feature points are matched, and then the spatial coordinates of the matching points are restored to calculate the target posture. Figure 2 It can be seen from the process shown in that the process involves the following aspects: 1. Interactive selection of targets; 2. Tracking the selected targets in the sequence images and performing secondary fine segmentation. The fine segmentation here is relative to the target selection. The accuracy used in selecting the target in the first step is less than the accuracy of segmentation in the sequence images in the second step. Therefore, it is called fine segmentation in the second step, that is, the fine segmentation here is a relative concept; 3. Matching target feature points on the image after the secondary fine segmentation; 4. Restoring the three-dimensional spatial position of the feature matching points; 5. Calculating the target posture transformation of the previous and next frames.

[0045] pass Figure 2The process shown in Figure 1 performs a preliminary extraction of the selected target and automatically tracks the target in the sequence of images, ensuring that the target is accurately located in each frame and its center point is marked. During the tracking process, the target is subjected to a secondary fine extraction to improve the completeness and accuracy of the target features. Subsequently, a deep learning method is used to match the target features in adjacent frames, and the three-dimensional information of the target is restored through spatial reconstruction of the matching points. Finally, the 6D pose of the target is calculated based on the spatial restoration results, ensuring the accuracy and robustness of the pose estimation. This method has broad application prospects in the detection and pose calculation of small targets at long distances, and is particularly suitable for complex scenes with weakly textured targets.

[0046] Below Figure 2 The algorithm characteristics of the steps in are described. The algorithm has the following characteristics.

[0047] Interactive target selection. Because it is a small target at a distance, conventional automatic detection algorithms will face problems such as the inability to conduct targeted training and poor generalization, which will lead to the inability to accurately identify key targets. The interactive target selection method first obtains the first image in the sequence and encodes it using the ViT image encoder to obtain the encoding of the entire image. A point selection method is used to select a point in a certain image in the sequence. By selecting the foreground and background, a list of point selection coordinates is obtained. Based on the selected coordinate list, a hint encoder is used to obtain a hint code. Then, a mask decoder is used to jointly decode the image code and hint code information to obtain the target mask after decoding.

[0048] The Vision Transformer (ViT) represents a paradigm shift in image processing by applying the Transformer architecture, originally designed for sequential data in NLP, to image data. It works by dividing the image into fixed-size patches, flattening them into vectors, and treating them as tokens, similar to words in a sentence. These tokens are then processed by multiple Transformer encoder layers, which include a multi-head self-attention mechanism to capture relationships between patches, a feedforward neural network for reinforcement learning, and normalization of residual connections to stabilize training. A special class tag is used for classification, which is processed through a fully connected layer to output a prediction. ViT's advantages include global context understanding, flexibility in handling different input sizes and tasks, and scalability to large datasets. However, it also has limitations, such as high data requirements for optimal performance and computational cost due to the quadratic complexity of the self-attention mechanism.

[0049] (2) Based on the target mask of the first frame image in the sequence, the video target tracking algorithm is used to locate and track the target in the subsequent frames, thereby obtaining the step-by-step mask information of the target in each frame. Through this process, the temporal and spatial changes of the target can be effectively tracked, ensuring the consistent calibration of the target in dynamic scenes. In order to further improve the accuracy of target extraction, the k-means clustering algorithm is applied for secondary refinement processing to address the errors that may exist in the initial mask extraction process. That is, the clustering algorithm is used to cluster the pixels in each image to obtain the pixels belonging to the target to be identified and the pixels that do not belong to the target to be identified. The mask of the target to be identified in the image is generated based on the pixels belonging to the target to be identified; this step can optimize the contour and morphology of the target and eliminate the inaccuracies in the initial extraction, thereby providing more accurate mask data for subsequent feature matching and target recognition, ensuring the accuracy and robustness of downstream tasks.

[0050] (3) After the target mask undergoes secondary refinement, the target's salient feature points (such as corners, edges, and textures) are extracted. These feature points uniquely identify the target across different frames. Using the LightGlue feature matching algorithm, the feature points in the secondary segmented target image are matched with the corresponding feature points in other frames. Feature point matching not only facilitates target tracking but also resolves issues such as partial occlusion and motion blur in complex scenes.

[0051] In the field of image processing, feature matching is a common approach. With the advancement of deep learning technology, neural networks are increasingly being utilized to address this problem. LightGlue emerged in this context. Its unique algorithmic design quickly attracted industry attention. This lightweight model not only delivers excellent performance across a wide range of application scenarios, but also greatly simplifies integration, making it easy for even beginners to get started. Over time, LightGlue has undergone continuous iterations and updates, becoming the tool of choice for many computer vision projects. LightGlue's most compelling feature is its self-evaluation mechanism. This allows the model to dynamically adjust its computing resources based on the complexity of the task, achieving both accuracy and efficiency. Specifically, when LightGlue determines that all predictions have reached a sufficiently high confidence level, it terminates the computation early, avoiding unnecessary computation. Furthermore, LightGlue decisively excludes feature points deemed mismatched, further reducing unnecessary computation. These design features ensure that LightGlue maintains high performance while also being easy to integrate and use. LightGlue uses an advanced graph neural network architecture that can automatically learn more robust feature representations and maintain high matching accuracy even under extreme conditions. Secondly, by introducing a self-evaluation mechanism, LightGlue can flexibly adjust the calculation process according to the specific circumstances of the current task. When the model determines that all prediction results have reached a sufficiently high confidence level, it will end the calculation in advance, significantly improving the processing speed. In addition, LightGlue will also actively eliminate feature points that are considered unmatched, further reducing unnecessary computational burden.

[0052] (4) Based on the matching results of the feature points, the three-dimensional spatial position of the matching points is reconstructed using multi-view geometry methods. By fusing the camera's internal parameters (such as focal length, optical center offset, distortion coefficient) and external parameters (such as pose matrix), combined with the motion information between adjacent frames, an accurate spatial mapping relationship is established. This process projects the target's feature points from two-dimensional image coordinates to a three-dimensional world coordinate system through triangulation, structural optics principles, or optimization algorithms, achieving precise positioning of the target in real physical space. This three-dimensional reconstruction process not only provides spatial distribution information of the target, but also lays a solid geometric foundation for subsequent target pose estimation, motion trajectory analysis, and three-dimensional scene understanding.

[0053] There are many ways to reconstruct the three-dimensional space of matching points, for example, the process of determining the affine transformation parameters such as the position, orientation, and scale of feature points. Existing feature point detection algorithms include the Laplacian detection algorithm, the Difference of Gaussian (DOG) algorithm, and the * detection algorithm that detects the position of corner points in an image based on the gradient covariance matrix. Among them, the DOG-based feature point detection method - the scale-invariant feature transform (SIFT) - is currently the best performing feature point detection algorithm. The SIFT process: DOG filtering is performed on the input image, and all maxima and minima in the filtered image are searched. The pixel coordinates corresponding to these extreme values ​​are the feature point coordinates. When the size of the feature area is roughly equivalent to the DOG filter width, an extreme value is present.

[0054] SIFT extracts an area around a feature point and describes it using a feature vector as a feature descriptor. The SIFT descriptor uses image gradients to calculate a histogram of local image gradient directions. This histogram is then created around the feature point in a 4x4 histogram grid. Each histogram contains eight gradient directions, resulting in a 128-dimensional feature vector.

[0055] Image feature point matching: One method is relatively simple, comparing the Euclidean distance between descriptors and setting a threshold for determination. Another method is to search for the nearest neighbor in the feature space to determine the matching relationship. The threshold can be learned, and a distance ratio test can also be used to eliminate false matches. Another indexing structure is a multidimensional search tree, which is based on the priority search strategy of the hierarchical K-means tree. In high-dimensional approximate nearest neighbor (ANN) search problems, multi-random KD trees perform optimally. To further eliminate false matches, geometric verification is required. The Random Sampling Consensus Algorithm (RANSAC) algorithm is typically used to estimate the fundamental matrix between the two views. False matches are then eliminated by calculating the degree of fit between the image matching and the fundamental matrix. Available fundamental matrix estimation methods include the normalized eight-point method and the gold standard method.

[0056] Of course, other algorithms for reconstructing the three-dimensional spatial positions of matching points can also be applied to this embodiment, and will not be described in detail here.

[0057] (5) By accurately analyzing the three-dimensional spatial position of feature points and combining the geometric transformation relationship between consecutive frames of the target in the video sequence, its posture changes at different time steps can be calculated, including key motion parameters such as rotation, translation, and scale transformation. This process relies on the posture estimation algorithm in computer vision and derives the target's motion trajectory and posture transformation matrix in three-dimensional space based on the optimized pose estimation method.

[0058] There are many pose estimation algorithms. For example, keypoint detection is a common pose estimation method that estimates pose by detecting the locations of keypoints on a person or object. Common keypoint detection algorithms include machine learning-based methods such as random forests, support vector machines, and deep learning methods such as convolutional neural networks (CNNs). These algorithms train models to accurately detect keypoint locations. Pose regression: Pose regression transforms the pose estimation problem into a regression problem. It maps the input image to the target pose and learns this mapping through training models. Common pose regression algorithms include linear regression, support vector regression, and decision tree regression. These algorithms predict the target pose by learning the relationship between input image features and the target pose. Deep learning methods have made significant progress in the field of pose estimation. Deep learning-based methods build deep neural network models to learn pose information directly from input images. Convolutional neural networks (CNNs) are a commonly used deep learning model that automatically extracts features from images for pose estimation. In addition, some image-based keypoint detection methods also use deep learning techniques, such as hourglass networks and ResNet. 3D pose estimation: In addition to 2D pose estimation, there are also methods for estimating the 3D pose of objects or people. These methods perform pose estimation based on information such as multi-view geometry, depth sensors, or multi-camera systems. 3D pose estimation methods can provide richer pose information and are important for application scenarios that require accurate 3D pose. The choice of pose estimation method depends on the specific application requirements and data characteristics. For accurate pose estimation, deep learning methods generally achieve better results. For applications with higher real-time requirements, some fast key point detection algorithms may be more suitable. In addition, combining multiple methods and algorithms, such as cascade methods and iterative optimization, can also improve the accuracy and stability of pose estimation.

[0059] As can be seen from the above description, the above embodiment can calculate the pose of small targets with weak textures at long distances. This technology can be applied to other complex scenes without excessive human intervention, reducing labor costs.

[0060] In this embodiment, an electronic device is provided, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to execute the method in the above embodiment.

[0061] The above program can be executed in a processor or stored in a memory (or computer-readable medium). Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and can be implemented using any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0062] These computer programs can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps of the functions specified in one or more blocks can be implemented by different modules corresponding to different steps.

[0063] This embodiment provides such a device or system. The system is called a target recognition processing system based on video images, and includes: an acquisition module for acquiring a series of frames from a video according to a time sequence and using the series of frames as a sequence of images; a selection module for selecting a predetermined image from the sequence of images and selecting a target to be recognized from the predetermined image; a marking module for tracking the target in the sequence of images and marking the center point of the target to be recognized in each image in the sequence of images; a segmentation module for segmenting the target to be recognized from the image according to the center point in each image, and performing spatial reconstruction of the target to be recognized based on the segmentation result to obtain three-dimensional information of the target to be recognized; and a processing module for obtaining the posture of the target to be recognized based on the three-dimensional information.

[0064] The system or device is used to implement the functions of the method in the above-mentioned embodiment. Each module in the system or device corresponds to each step in the method, which has been explained in the method and will not be repeated here.

[0065] Optionally, the selection module is used to: encode the predetermined image to obtain the encoding of the entire image; obtain the mask of the target to be selected in the image based on the identified target to be identified; the segmentation module is used to: obtain the mask of the target to be identified in the image based on the center point of the target to be identified in each image, and segment the target to be identified from the image according to the mask.

[0066] Optionally, the selection module is used to: cluster the pixels in each image using a clustering algorithm in a neural network model according to the center point in each image and the identification information for identifying the target to be identified in the predetermined image, to obtain pixel points belonging to the target to be identified and pixel points that do not belong to the target to be identified; and generate a mask of the target to be identified in the image based on the pixel points belonging to the target to be identified.

[0067] The above-mentioned implementation solves the problem of low accuracy in the prior art in recognizing the posture of a target in a video, thereby improving the recognition accuracy of the posture of a target in a video image to a certain extent.

[0068] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A method for target recognition processing based on video images, characterized in that: include: Acquire a series of frames from the video in time sequence and use the series of frames as a sequence of images; Selecting a predetermined image from the sequence of images, and selecting a target to be identified from the predetermined image; Tracking the target in the sequence of images, and marking the center point of the target to be identified in each image in the sequence of images; Segmenting the target to be identified from each image according to the center point, and spatially reconstructing the target to be identified based on the segmentation result to obtain three-dimensional information of the target to be identified; Obtaining the posture of the target to be identified according to the three-dimensional information; wherein, encoding the predetermined image obtains an encoding of the entire image; obtaining a mask of the target to be selected in the image based on the identified target to be identified; obtaining a mask of the target to be identified in each image based on the center point of the target to be identified in the image, and segmenting the target to be identified from the image based on the mask; Among them, according to the center point in each image and the identification information for identifying the target to be identified in the predetermined image, the pixel points in each image are clustered using the clustering algorithm in the neural network model to obtain pixel points belonging to the target to be identified and pixel points that do not belong to the target to be identified; according to the pixel points belonging to the target to be identified, a mask of the target to be identified in the image is generated; wherein, two clustering algorithms are used to cluster the pixel points in each image, and the first clustering algorithm is used to cluster to obtain the first pixel point belonging to the target to be identified; the second clustering algorithm is used to obtain the second pixel point belonging to the target to be identified, and the same pixel points in the first pixel point and the second pixel point are used as the pixel points belonging to the target to be identified.

2. The method according to claim 1, characterized in that A first image is selected from the sequence of images as the predetermined image.

3. The method according to claim 1, characterized in that A ratio of the mask of the target to be identified in the image to the overall mask of the image is lower than a threshold.

4. A target recognition and processing system based on video images, characterized in that: include: An acquisition module, configured to acquire a series of frames from a video according to a time sequence and use the series of frames as a sequence of images; A selection module, configured to select a predetermined image from the sequence of images and select a target to be identified from the predetermined image; A marking module, configured to track the target in the sequence of images and mark a center point of the target to be identified in each image in the sequence of images; a segmentation module, configured to segment the target to be identified from the image according to the center point in each image, and spatially reconstruct the target to be identified based on the segmentation result to obtain three-dimensional information of the target to be identified; A processing module, configured to obtain the posture of the target to be identified based on the three-dimensional information; wherein, encoding the predetermined image obtains an encoding of the entire image; obtaining a mask of the target to be selected in the image based on the identified target to be identified; obtaining a mask of the target to be identified in each image based on the center point of the target to be identified in the image, and segmenting the target to be identified from the image based on the mask; Among them, according to the center point in each image and the identification information for identifying the target to be identified in the predetermined image, the pixel points in each image are clustered using the clustering algorithm in the neural network model to obtain pixel points belonging to the target to be identified and pixel points that do not belong to the target to be identified; according to the pixel points belonging to the target to be identified, a mask of the target to be identified in the image is generated; wherein, two clustering algorithms are used to cluster the pixel points in each image, and the first clustering algorithm is used to cluster to obtain the first pixel point belonging to the target to be identified; the second clustering algorithm is used to obtain the second pixel point belonging to the target to be identified, and the same pixel points in the first pixel point and the second pixel point are used as the pixel points belonging to the target to be identified.

5. An electronic device comprising a memory and a processor; wherein: The memory is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method steps according to any one of claims 1 to 3.

6. A readable storage medium having computer instructions stored thereon, wherein: When the computer instructions are executed by a processor, the method steps described in any one of claims 1 to 3 are implemented.

Images