Data processing method, device, electronic device and storage medium

Abstract

An embodiment of the present application provides a data processing method, apparatus, electronic device, and storage medium, the method comprising: acquiring a frame image of video data, and determining a feature vector and a frame image combination of a target object within a detection frame of the frame image; determining a feature map based on the similarity between target objects in the frame image combination, wherein the similarity includes feature similarity and / or spatial similarity; determining a classification result of each target object based on the feature map of each target object, so as to determine a movement trajectory of the target object in the video data; and improving the accuracy of movement trajectory recognition.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, a data processing device, an electronic device, and a storage medium. Background Art

[0002] Video multi-target trajectory recognition is a computer vision task that is mainly used to determine the trajectories of different targets in a video. It has important applications in security, autonomous driving and other fields.

[0003] However, the current methods for identifying target movement trajectories have low recognition accuracy. Summary of the Invention

[0004] The embodiments of the present application provide a data processing method that can improve the accuracy of movement trajectory analysis.

[0005] Correspondingly, an embodiment of the present application also provides a data processing device, an electronic device and a storage medium to ensure the implementation and application of the above system.

[0006] In order to solve the above problems, an embodiment of the present application discloses a data processing method, which includes: obtaining a frame image of video data, and determining a feature vector and a frame image combination of a target object within a detection frame of the frame image; determining a feature map based on the similarity between each target object in the frame image combination, wherein the similarity includes feature similarity and / or spatial similarity; determining a classification result of each target object based on the feature map of each target object to determine the movement trajectory of the target object in the video data.

[0007] In order to solve the above problems, an embodiment of the present application discloses a data processing method, which includes: obtaining a frame image of road video data, and determining a feature vector and a frame image combination of a target object within a detection frame of the frame image; determining a feature map based on the similarity between each target object in the frame image combination, wherein the similarity includes feature similarity and / or spatial similarity; determining a classification result of each target object based on the feature map of each target object to determine a movement trajectory of the target object in the road video data; determining a behavior type of the target object based on the movement trajectory to determine a corresponding behavior analysis result.

[0008] In order to solve the above problems, an embodiment of the present application discloses a data processing method, which includes: obtaining a frame image of live video data, and determining a feature vector and a frame image combination of a target object within a detection frame of the frame image; determining a feature map based on the similarity between each target object in the frame image combination, wherein the similarity includes feature similarity and / or spatial similarity; determining a classification result of each target object based on the feature map of each target object to determine the movement trajectory of the target object in the live video data; and adding virtual information to the target object in the live video data based on the movement trajectory.

[0009] In order to solve the above problems, an embodiment of the present application discloses a data processing method, which includes: obtaining frame images of driving video data, and determining the feature vector and frame image combination of the target object within the detection frame of the frame image; determining a feature map based on the similarity between the target objects in the frame image combination, the similarity including feature similarity and / or spatial similarity; determining the classification result of each target object based on the feature map of each target object to determine the movement trajectory of the target object in the driving video data; and determining driving instructions based on the movement trajectory to control vehicle driving.

[0010] In order to solve the above problems, an embodiment of the present application discloses a data processing method, which includes: providing an interactive page to obtain video data to be processed based on the interactive page; obtaining a frame image of the video data, and determining the feature vector and frame image combination of the target object in the detection frame of the frame image; determining a feature map based on the similarity between the target objects in the frame image combination, the similarity including feature similarity and / or spatial similarity; determining the classification result of each target object based on the feature map of each target object, so as to determine the movement trajectory of the target object in the video data as an analysis result, and issuing the analysis result.

[0011] In order to solve the above problems, an embodiment of the present application discloses a data processing device, which includes: a video data acquisition module, used to acquire frame images of video data and determine the feature vector and frame image combination of the target object in the detection frame of the frame image; a feature map acquisition module, used to determine the feature map based on the similarity between the target objects in the frame image combination, wherein the similarity includes feature similarity and / or spatial similarity; a movement trajectory acquisition module, used to determine the classification result of each target object based on the feature map of each target object, so as to determine the movement trajectory of the target object in the video data.

[0012] In order to solve the above problems, an embodiment of the present application discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes one or more methods described in the above embodiments.

[0013] In order to solve the above problems, embodiments of the present application disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes one or more methods described in the above embodiments.

[0014] Compared with the prior art, the embodiments of the present application have the following advantages:

[0015] In an embodiment of the present application, a frame image of video data can be obtained, and the feature vector of the target object in the detection frame of the frame image and the frame image combination can be determined; then, based on the feature vector of the target object in the frame image combination, the feature similarity between each target object in the frame image combination can be determined; based on the detection frame, the spatial similarity between each target object can be determined; then, based on the feature similarity and spatial similarity between each target object, an association can be established for the same target object in different frame images to form a corresponding feature map. Then, based on the feature map, the classification result corresponding to the target object can be determined, and the movement trajectory of the target object in the video data can be obtained. In the process of data association, the embodiment of the present application can perform data association based on the feature similarity and spatial similarity between the target objects, so that the association of features and the association of spatial positions between the target objects can be considered, and a more accurate classification can be obtained, thereby improving the accuracy of the analysis of the movement trajectory of the target object in the video data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of a data processing method according to an embodiment of the present application;

[0017] Figure 2 is a flowchart of a data processing method according to another embodiment of the present application;

[0018] Figure 3 is a flowchart of a data processing method according to another embodiment of the present application;

[0019] Figure 4 is a flowchart of a data processing method according to another embodiment of the present application;

[0020] Figure 5 is a flowchart of a data processing method according to another embodiment of the present application;

[0021] Figure 6A is a flowchart of a data processing method according to another embodiment of the present application;

[0022] Figure 6B is a flowchart of a data processing method according to another embodiment of the present application;

[0023] Figure 7Ais a flowchart of a data processing method according to another embodiment of the present application;

[0024] Figure 7B is a flowchart of a data processing method according to another embodiment of the present application;

[0025] Figure 8 is a structural diagram of a data processing device according to an embodiment of the present application;

[0026] Figure 9 is a structural diagram of a data processing device according to another embodiment of the present application;

[0027] Figure 10 is a structural diagram of a data processing device according to another embodiment of the present application;

[0028] Figure 11 is a structural diagram of a data processing device according to another embodiment of the present application;

[0029] Figure 12 is a structural diagram of a data processing device according to another embodiment of the present application;

[0030] Figure 13 It is a structural diagram of an exemplary device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0032] The embodiments of the present application can be applied to the field of target trajectory recognition in video data, which refers to the identification of the trajectory of a target in video data. The embodiments of the present application can analyze the characteristics and positions of targets (or target objects) in different frames of video data, thereby performing data association between target objects in different frames, thereby obtaining the corresponding classification of the target objects in the frame images, and then obtaining the movement trajectory of the target objects in the video data.

[0033] Specifically, such as Figure 1As shown, the embodiment of the present application can obtain multiple frame images of video data, and then, on the one hand, a detection frame for locating a target object in the frame image can be determined, and the target object in the detection frame can be feature identified to obtain a feature vector; on the other hand, a target number of continuous frame images can be obtained to form a frame image combination, and the frame image combination can also be called a graph. After determining the feature vectors of the frame image combination and the target object in the frame image, the relevant objects corresponding to the target object in the target frame image can be screened out based on the feature vectors of the target object in the target frame image in the frame image combination, and the corresponding feature vectors can be extracted to form nodes, wherein the target frame image is a frame image in the frame image combination, and the relevant objects of the target object in the target frame image include a first object and a second object, the first object refers to an object whose feature vector is similar to the feature vector of the target object, and the second object refers to an object whose feature vector is similar to the feature vector of the first object; the node includes a node feature, and the node feature characterizes the feature of the object in the video data. After determining the node, the adjacent nodes can be determined based on the feature vector of the target object, and a first connecting edge can be established between the nodes; the nodes that meet the overlap condition can also be determined based on the overlap between the detection frames of the target object and a second connecting edge can be added to form a feature graph. Afterwards, the classification result of the target object can be determined based on the node features of the nodes and the edge features of the connecting edges in the feature graph. After the classification result is determined, the target object in each frame image can be connected based on the classification result of the target object to obtain the movement trajectory of the target object in the video data. It should be noted that Figure 1 The trajectory diagram in the example shown is only for ease of understanding and cannot be regarded as a limitation of the movement trajectory.

[0034] The inventors of the present application embodiment have adopted a variety of methods to identify the movement trajectory, but found in the actual application process that the recognition accuracy of some recognition methods is not high. For example, a method for identifying the target trajectory is to extract the feature vectors of the target in different frame images in the video data, and then compare the feature similarity between the target objects based on the feature vectors, and perform data association on the targets in different frame images in the video according to the feature similarity, so as to determine the classification corresponding to the target, and then determine the trajectory of the target in the video. However, the inventors found in the actual use process that in the process of data association, if the feature matching method is adopted, the position of the target in the video is prone to jump, resulting in a poor recognition effect of the target trajectory. Therefore, the inventors proposed a new target trajectory recognition method. In the process of data association (establishing associations between target objects), data association analysis can be performed based on the feature similarity and spatial similarity of the target objects in different frame images, so that the association of features between target objects and the association of spatial positions can be considered, and then a more accurate classification is obtained, thereby improving the accuracy of the target object movement trajectory analysis in the video data.

[0035] The embodiments of the present application can be applied to the scene of target trajectory recognition of the target object of the video data, for example, it can be applied in the fields of security, autonomous driving, live video and the like. For example, the embodiments of the present application can be applied in the scene of security, and the security video can be analyzed to determine the feature vector and detection frame of the target object, and the feature correlation and spatial correlation between the target objects in different frame images can be analyzed to obtain the classification result of the target object, determine the trajectory of the target object in the video, and thus determine whether the behavior of the target object complies with the specification (such as mistakenly entering a prohibited area, etc.). For another example, the embodiments of the present application can be applied in the scene of autonomous driving, and the moving trajectory of the target object (such as a vehicle, pedestrian, obstacle) in the driving video data can be analyzed to determine the moving trajectory of the target object (such as a vehicle, pedestrian, obstacle) in the driving video data, so as to determine the corresponding driving instructions to control the driving of the vehicle. For another example, the embodiments of the present application can also be applied in the analysis scene of live video, and the moving trajectory of the host's face can be identified, and then virtual information (such as glasses special effects) can be added to the host's face to improve the live broadcast effect.

[0036] The embodiment of the present application provides a data processing method that can be applied to a processing end, which can be understood as a device for acquiring and analyzing video data. The method can analyze target objects in different frame images in the video data, determine the detection frame and feature vector of the target object, and then perform data association on the target objects in different frame images, thereby determining the classification result of the target object and determining the movement trajectory of the target object in the video data. Specifically, Figure 2 As shown, the method includes:

[0037] Step 202: Obtain a frame image of the video data and determine the feature vector and frame image combination of the target object within the detection frame of the frame image. The video data can be a road video captured by a road camera, a video captured by a community camera, a live video, a vehicle driving video, etc. After obtaining the frame image of the video data, the embodiment of the present application can, on the one hand, extract the features of the target object, and on the other hand, filter out some continuous frame images to form a frame image combination. Specifically, on the one hand, the embodiment can identify the target object, determine the number of target objects in the frame image and determine the corresponding detection frame, and then extract the feature vector of the target object within the detection frame. Specifically, as an optional embodiment, the feature vector of the target object within the detection frame of the frame image includes: determining the detection frame within the frame image and extracting the detection image within the detection frame; extracting the features of the target object within the detection image to obtain a feature vector. The embodiment of the present application can identify the target object in the frame image and use a bounding box regression method to determine the detection frame corresponding to the target object, and the detection frame is used to locate the target object. Then, the detection image within the detection frame is extracted, and then the features corresponding to the target object are extracted to obtain a feature vector. In an optional example, an embodiment of the present application can use a pre-trained convolutional neural network (CNN) model based on a residual network (Residual Network, ResNet) as a network architecture (such as ResNet50) to extract the features of the target object, wherein the residual network is easy to optimize and can improve the accuracy by increasing the depth. The residual block inside it uses a skip connection to alleviate the gradient vanishing problem caused by increasing the depth in the deep neural network. The CNN model is a type of feedforward neural network (Feedforward Neural Networks) that includes convolution calculations and has a deep structure. It is one of the representative algorithms of deep learning.

[0038] On the other hand, the embodiment of the present application can adopt a method of dynamically updating the frame image combination, continuously updating the frame images in the frame image combination, so that different frame images can be continuously analyzed. Specifically, as an optional embodiment, the step of determining the frame image combination includes: obtaining a target number of continuous frame images as the frame image combination, and after determining the feature map corresponding to the frame image combination, updating the frame images in the frame image combination. The embodiment of the present application can pre-set the target number of frame images contained in the frame image combination, and establish an empty image group queue to add a target number of continuous frame images to the empty image group queue to form a frame image combination for analysis. Specifically, as an optional embodiment, the acquisition of the target number of continuous frame images as the frame image combination includes: obtaining a target number of continuous frame images; determining the image group queue, and adding the target number of continuous frame images to the image group queue to form a frame image combination. The image group queue can be understood as a task queue to be analyzed. The embodiment of the present application can add a target number of frame images to the task queue to be analyzed, so as to analyze the target number of frame images and determine the corresponding feature map. In addition, the embodiment of the present application can adopt a dynamic evolutionary approach to continuously update the frame images in the frame image combination, thereby forming multiple groups of frame image combinations to analyze the frame images of the video data. Specifically, as an optional embodiment, the updating of the frame images in the frame image combination includes: deleting at least one frame image from the image group queue and adding a corresponding number of frame images to the image group queue to form an updated frame image combination. This embodiment can adopt a dynamic evolutionary approach to continuously delete frame images from the image group queue and add a corresponding number of frame images to form a frame image combination containing a target number of continuous frame images, and the embodiment of the present application can delete the nodes corresponding to the deleted frame images and reuse the nodes corresponding to the non-deleted frame images. Compared with the method of reconstructing the nodes of each frame image, the method of this embodiment can improve data processing efficiency.

[0039] After determining the frame image combination and the feature vector of the frame image, a feature map can be determined in step 204 based on the similarity between the target objects in the frame image combination, wherein the similarity includes feature similarity and / or spatial similarity, and the feature map is used to associate the same target object in different frame images. The feature map includes nodes and connecting edges, and the nodes include node features, and the node features represent the feature vectors of the target objects; the connecting edges represent the associations between the nodes. The embodiment of the present application can establish associations between target objects based on at least one of the feature similarity and spatial similarity between the target objects. The embodiment of the present application can determine the feature similarity between the target objects in the frame image combination based on the feature vectors of the target objects in the frame image combination; determine the spatial similarity between the target objects based on the overlap between the detection frames of the target objects; and then establish associations for the same target object based on the feature similarity and / or spatial similarity between the target objects to form a corresponding feature map. Specifically, as an optional embodiment, the feature map is determined based on the similarity between the target objects in the frame image combination, including: based on the feature vector of the target object in the frame image combination, screening out the related objects corresponding to the target object in the target frame image, and extracting the corresponding feature vector to form a node; based on the similarity between the nodes, adding connecting edges between the nodes to form a feature map of the target object in the corresponding target frame image.

[0040] The embodiment of the present application can filter out the related objects corresponding to the target object in the target frame image based on the feature vector of the target object in the frame image combination, and extract the corresponding feature vector to form a node. Wherein, the target frame image is a frame image in the frame image combination, and the related objects of the target object in the target frame image include a first object and a second object, the first object refers to an object whose feature vector is similar to the feature vector of the target object, and the second object refers to an object whose feature vector is similar to the feature vector of the first object. Specifically, as an optional embodiment, the method of filtering out the related objects corresponding to the target object in the target frame image based on the feature vector of the target object in the frame image combination, and extracting the corresponding feature vector to form a node, includes: filtering out the first object related to the target object of the target frame image based on the feature vector of the target object in the frame image combination, and filtering out the second object related to the first object; determining the feature vectors of the first object and the second object to form a node.

[0041] In an embodiment of the present application, a preset number of first objects and second objects can be pre-set, and based on the similarity of the feature vectors between the target objects, the targets similar to the target object in the target frame image are sorted by similarity, thereby screening out a first preset number of first objects, and using each first object as a two-hop node (or two-hop neighbor), sorting the target objects similar to the first object, thereby screening out a second preset number of second objects, and then obtaining the feature vectors of the target object in the target frame image, the first object of the target object, and the second object of the target object, to form multiple nodes corresponding to the target object in the target frame image. In an optional example, an embodiment of the present application can use a K-Nearest Neighbor (KNN) algorithm to determine the first object and the second object to obtain corresponding nodes. The KNN algorithm is a method for classifying each record in a data set.

[0042] After determining the multiple nodes corresponding to the target object in the target frame image, the multiple nodes corresponding to the target object in the target frame image may include nodes corresponding to the target object, and may also include nodes that do not correspond to the target object. Therefore, the embodiment of the present application can define the degree of association between nodes by adding connecting edges between nodes. Specifically, the embodiment of the present application can add connecting edges between nodes based on whether the feature vectors of each target object are adjacent, and can add connecting edges between nodes based on whether the overlap between the detection frames of each target object meets the overlap requirement. Specifically, as an optional embodiment, the adding of connecting edges between nodes based on the similarity between each node includes: determining the feature similarity between each node based on the feature vector of each node, and determining the nodes with adjacent feature vectors based on the feature similarity to add a first connecting edge; determining the spatial similarity between each node based on the overlap between the detection frames of each node, and determining the nodes that meet the overlap condition to add a second connecting edge.

[0043] Nodes with adjacent feature vectors can be understood as the two nodes with the highest feature similarity. The embodiment of the present application can obtain the feature vectors of each node, and then determine the feature similarity between the feature vectors of each node, and add a first connection edge between the nodes with the highest feature similarity. The overlap can also be called the Intersection over Union (IoU), which refers to the ratio between the intersection and the union of the detection frames. The movement of the target object in the continuous frame image is a continuous process. Therefore, in addition to establishing an association (adding a connection edge) between the target objects based on the feature similarity of the target objects, the embodiment of the present application can also establish an association between the target objects based on the overlap between the detection frames of the target objects in different frame images. The overlap condition can be pre-set to filter the nodes based on the overlap condition so as to add a connection edge between the nodes. For example, the overlap condition can be pre-set to IoU greater than 0.5. Therefore, the nodes can be matched and analyzed in pairs, and a second connection edge can be added between the node combinations with IoU greater than 0.5.

[0044] After determining the feature map of the target object, a classification result for each target object can be determined based on the feature map of each target object in step 206 to determine the target object's movement trajectory in the video data. After determining the feature map of the target object, feature extraction can be performed on the nodes and connecting edges in the feature map to obtain node features and edge features, thereby determining the classification result of the target object. Classifying the target object can be understood as classifying the detection frames belonging to the same target into the same category, and connecting the detection frames in each frame image to obtain the target object's movement trajectory in the video data.

[0045] The embodiment of the present application can use a neural network model (NN) to identify and process the feature graph to obtain the classification result of the target object. The neural network model is a complex network system formed by a large number of simple processing units (called neurons) widely connected to each other. It is a highly complex nonlinear dynamic learning system. Specifically, as an optional embodiment, the classification result of each target object is determined based on the feature graph of each target object, including: inputting the feature graph into a feature processing model to determine the classification result of the target object, and the feature processing model is used to determine the node features of the node and the edge features of the connecting edge, and determine the classification result of the target object. The feature processing model can be understood as one of the neural network models. For example, the feature processing model of the embodiment of the present application can adopt a graph convolutional network (GCN), a graph network (GNN), etc. In an optional embodiment, the feature processing model can adopt a graph convolutional network (or graph convolution model), which includes a convolution layer and a fully connected layer (FC). The convolution layer is used to extract the node features of the feature graph and the edge features of the connecting edges and perform optimization processing. For example, the extracted node features and edge features can be reduced in dimension to subsequently classify the target object. Each node of the fully connected layer is connected to all the nodes of the previous layer to integrate the features extracted by the convolution layer and then determine the classification result. In an example, the feature processing model can adopt the dynamic graph convolutional neural network (Evolve Graph Convolutional Network, E-GCN) in the GCN model. The embodiment of the present application can set four convolution layers and two fully connected layers to process the data and obtain the classification result of the target object. The graph convolution model can still have a high analysis accuracy when processing data before training. Therefore, in the embodiment of the present application, the graph convolution model can use a trained graph convolution model or an untrained graph convolution model to process data, which can be configured according to needs.

[0046] In addition, during the training of the feature processing model, the feature processing model can integrate the features extracted by the convolution layer through the fully connected layer, and then determine the classification result and the corresponding loss function, so that the feature processing model can be reversely adjusted according to the loss function to obtain a trained feature processing model. During the training of the feature processing model, some existing fully connected layers only consider whether the classification result is correct, but do not consider the probability of the target object corresponding to each category. Therefore, using this method to adjust the edge features of the feature graph can easily lead to classification errors. Specifically, as an optional embodiment, the method further includes: determining the probability of the target object corresponding to each category in each frame image through the fully connected layer of the feature processing model, and determining the corresponding cross entropy as a loss function. Cross entropy is mainly used to measure the difference information between two probability distributions. In the embodiment of the present application, during the training of the feature processing model, the corresponding cross entropy can be determined as a loss function based on the probability of the target object corresponding to each category, which can take into account the probability of the target object corresponding to different categories, so that the edge features can be adjusted according to the loss function to obtain a more accurate classification result.

[0047] After classifying the target objects, the present application can connect target objects belonging to the same category in different frame images to obtain corresponding movement trajectories. Specifically, as an optional embodiment, the determination of the movement trajectory of the target object in the video data includes: connecting the target objects in each frame image according to the classification results of the target objects in each frame image to obtain the movement trajectory of the target object in the video data. After determining the movement trajectory, other data processing can be performed based on the movement trajectory. For example, in the action prediction scenario, the next movement trajectory of the target object can be predicted based on the movement trajectory; for another example, in the autonomous driving scenario, the driving instructions that the vehicle needs to execute can be determined based on the movement trajectory; for another example, in the road security scenario, whether the vehicle or pedestrian violates the traffic rules can be determined based on the movement trajectory, and then the behavior analysis results can be output.

[0048] In addition, during the training process of the feature processing model, the corrected movement trajectory can be determined by manual correction, so as to determine the loss function of the feature processing model through the corrected movement trajectory. Specifically, as an optional embodiment, the method further includes: providing a correction page to display the movement trajectory of the target object through the correction page; obtaining correction information for the movement trajectory of the target object, determining the correction result, and determining the loss function of the feature processing model through the correction result. During the training process of the feature processing model, the corresponding movement trajectory can be determined based on the classification result output by the feature processing model, and the movement trajectory can be displayed on the correction page, and the correction page can include a calibration control for the movement trajectory. The analyst can correct the movement trajectory of the target object in the video by triggering the calibration control, thereby determining the corresponding correction result. After that, the loss function can be determined based on the difference between the correction result (corrected movement trajectory) and the movement trajectory of the target object, so as to reversely adjust the feature processing model through the loss function, thereby improving the accuracy of the feature processing model.

[0049] In an embodiment of the present application, a frame image of video data can be obtained, and the feature vector of the target object in the detection frame of the frame image and the frame image combination can be determined; then, based on the feature vector of the target object in the frame image combination, the feature similarity between each target object in the frame image combination can be determined; based on the detection frame, the spatial similarity between each target object can be determined; then, based on the feature similarity and spatial similarity between each target object, an association can be established for the same target object in different frame images to form a corresponding feature map. Then, based on the feature map, the classification result corresponding to the target object can be determined, and the movement trajectory of the target object in the video data can be obtained. In the process of data association, the embodiment of the present application can perform data association based on the feature similarity and spatial similarity between the target objects, so that the association of features and the association of spatial positions between the target objects can be considered, and a more accurate classification can be obtained, thereby improving the accuracy of the analysis of the movement trajectory of the target object in the video data.

[0050] On the basis of the above embodiments, the present application also provides a data processing method that can be applied at the processing end, specifically, Figure 3 As shown, the method includes:

[0051] Step 302: Acquire a frame image of the video data.

[0052] Step 304: Determine an image group queue and add a target number of consecutive frame images to the image group queue to form a frame image combination. As an optional embodiment, this embodiment can adopt a dynamic evolution method to continuously update the frame image combination. Specifically, the step of updating the frame images in the frame image combination includes: deleting at least one frame image from the image group queue and adding a corresponding number of frame images to the image group queue to form the frame image combination.

[0053] Step 306: Determine a detection frame within the frame image, and extract a detection image within the detection frame.

[0054] Step 308: Extract the features of the target object in the detection image to obtain a feature vector.

[0055] Step 310 : Filter out a first object related to the target object of the target frame image based on the feature vector of the target object in the frame image combination, and filter out a second object related to the first object.

[0056] Step 312: Determine the feature vectors of the first object and the second object to form a node.

[0057] Step 314: Determine the feature similarity between the nodes based on the feature vector of each node, and determine the nodes with adjacent feature vectors based on the feature similarity to add a first connecting edge.

[0058] Step 316: Determine the spatial similarity between the nodes based on the overlap between the detection frames of the nodes, and determine the nodes that meet the overlap condition to add a second connecting edge to form a feature map of the target object in the corresponding target frame image.

[0059] Step 318: Input the feature map into the feature processing model to determine the classification result of the target object.

[0060] Step 320 : Connect the target objects in each frame image based on the classification results of the target objects in each frame image to obtain the movement trajectory of the target objects in the video data.

[0061] In an embodiment of the present application, a frame image of video data can be obtained. Then, on the one hand, a group queue can be determined, and a target number of continuous frame images can be added to the group queue to form a frame image combination; on the other hand, a detection frame for locating the target object in the frame image can be determined, and features of the target object in the detection frame can be extracted to obtain a feature vector of the target object. Afterwards, the first object and the second object corresponding to the target object in the target frame image can be screened out based on the feature vector of the target object in the frame image combination, and the corresponding feature vectors can be extracted to form nodes; then, based on the feature similarity between the nodes and the overlap of the detection frame, connecting edges can be added between the nodes to establish data associations between the nodes and form a corresponding feature graph. Then, based on the feature graph, the classification result corresponding to the target object is determined, and the movement trajectory of the target object in the video data is obtained.

[0062] On the basis of the above embodiments, the embodiments of the present application also provide a data processing method, which can be applied to a processing end. The processing end can be understood as a device for receiving and analyzing video data. The method can analyze target objects in different frame images in road video data, determine the detection frame and feature vector of the target object, and then perform data association on the target objects in different frame images, and then determine the classification result of the target object to determine the moving trajectory of the target object in the road video data. After that, it can be determined whether the target object has violated the traffic rules based on the moving trajectory, and obtain the behavior analysis result. Specifically, if Figure 4 As shown, the method includes:

[0063] Step 402: Acquire frame images of road video data and determine a feature vector and a frame image combination of a target object within a detection frame of the frame image. In an optional embodiment, the road video data may be acquired by a road video acquisition component. Specifically, the method further includes: acquiring the road video data based on the road video acquisition component. The road video acquisition component may be a camera installed on the road. In an optional embodiment, determining the feature vector of the target object within the detection frame of the frame image includes: determining a detection frame within the frame image, extracting a detection image within the detection frame; and extracting features of the target object within the detection image to obtain a feature vector. The detection frame is used to locate the target object. The target object in the road video data may be a vehicle, pedestrian, obstacle, animal, etc. In an optional embodiment, determining the frame image combination includes: acquiring a target number of consecutive frame images as a frame image combination, and updating the frame images in the frame image combination after determining a feature map corresponding to the frame image combination. In an optional example, acquiring a target number of consecutive frame images as a frame image combination includes: acquiring a target number of consecutive frame images; determining a group queue, and adding the target number of consecutive frame images to the group queue to form the frame image combination. The updating of the frame images in the frame image combination includes: deleting at least one frame image from the image group queue and adding a corresponding number of frame images to the image group queue to form an updated frame image combination.

[0064] Step 404: Determine a feature graph based on the similarity between the target objects in the frame image combination, wherein the similarity includes feature similarity and / or spatial similarity. The feature graph is used to associate the same target object in different frame images. As an optional embodiment, step 404 specifically includes: based on the feature vector of the target object in the frame image combination, filter out related objects corresponding to the target object in the target frame image, and extract the corresponding feature vector to form a node; based on the similarity between the nodes, add connecting edges between the nodes to form a feature graph corresponding to the target object in the target frame image. As an optional embodiment, the filtering out related objects corresponding to the target object in the target frame image based on the feature vector of the target object in the frame image combination, and extracting the corresponding feature vector to form a node, includes: based on the feature vector of the target object in the frame image combination, filtering out a first object related to the target object of the target frame image, and filtering out a second object related to the first object; and determining the feature vectors of the first object and the second object to form a node. As an optional embodiment, the method of adding connecting edges between nodes based on the similarity between each node includes: determining the feature similarity between each node based on the feature vector of each node, and determining the nodes with adjacent feature vectors based on the feature similarity to add a first connecting edge; determining the spatial similarity between each node based on the overlap between the detection frames of each node, and determining the nodes that meet the overlap conditions to add a second connecting edge.

[0065] Step 406: Determine the classification result of each target object based on the feature graph of each target object to determine the target object's movement trajectory in the road video data. As an optional embodiment, determining the classification result of each target object based on the feature graph of each target object includes: inputting the feature graph into a feature processing model to determine the classification result of the target object, wherein the feature processing model is used to determine the node features of the nodes and the edge features of the connecting edges, and determine the classification result of the target object. As an optional embodiment, determining the target object's movement trajectory in the road video data includes: connecting the target objects in each frame image based on the classification result of the target object in each frame image to obtain the target object's movement trajectory in the road video data.

[0066] Step 408: Determine the behavior type of the target object based on the movement trajectory to determine the corresponding behavior analysis result. Specifically, in one example, the movement trajectory can be the movement trajectory of a vehicle. The embodiment of the present application can collect the movement trajectory of the vehicle within a period of time (such as three seconds) to determine the vehicle's driving speed, and then determine whether the vehicle has violated the law (such as speeding) based on the vehicle's driving speed, thereby outputting the behavior analysis result. In another example, the movement trajectory can be the movement trajectory of a pedestrian. The embodiment of the present application can determine the pedestrian's walking direction based on the pedestrian's movement direction, and then determine whether the pedestrian is walking against the flow, and output the corresponding behavior analysis result.

[0067] The implementation of this embodiment is similar to that of the above embodiment. The specific implementation can refer to the specific implementation of the above embodiment, which will not be repeated here.

[0068] In an embodiment of the present application, road video data can be acquired through a road camera, and then a frame image of the road video data can be acquired, and the feature vector of the target object in the detection frame of the frame image and the frame image combination can be determined; then, based on the feature vector of the target object in the frame image combination, the relevant objects corresponding to the target object in the target frame image can be screened out, and the corresponding feature vectors can be extracted to form nodes; then, based on the feature similarity and spatial similarity between the nodes, connecting edges can be added between the nodes to establish data associations between the nodes and form a corresponding feature graph. Then, based on the feature graph, the classification result corresponding to the target object is determined, and the movement trajectory of the target object in the video data is obtained. Then, based on the movement trajectory, it can be determined whether the target object has violated traffic regulations (such as driving against traffic lights, not following traffic lights, etc.) to obtain behavior analysis results.

[0069] On the basis of the above embodiments, the embodiments of the present application provide a data processing method that can be applied to a processing end. The processing end can be understood as a device for acquiring and analyzing video data. The method can analyze target objects in different frame images in live video data, determine the detection frame and feature vector of the target object, and then perform data association on the target objects in different frame images, thereby determining the classification result of the target object, and determining the movement trajectory of the target object in the video data. Afterwards, virtual information can be added to the target object in the video data based on the movement trajectory to achieve an augmented reality effect. Specifically, as Figure 5 As shown, the method includes:

[0070] Step 502: Acquire a frame image of live video data, and determine a feature vector of a target object within a detection frame of the frame image and a frame image combination.

[0071] Step 504: Determine a feature map based on the similarity between target objects in the frame image combination, wherein the similarity includes feature similarity and / or spatial similarity. The feature map is used to associate the same target object in different frame images. Specifically, as an optional embodiment, step 504 includes: screening related objects corresponding to the target object in the target frame image based on the feature vectors of the target objects in the frame image combination, and extracting the corresponding feature vectors to form nodes; and adding connecting edges between the nodes based on the similarity between the nodes to form a feature map corresponding to the target object in the target frame image.

[0072] Step 506: Determine the classification result of each target object based on the feature map of each target object to determine the movement trajectory of the target object in the live video data.

[0073] Step 508: Add virtual information to the target object in the live video data based on the movement trajectory.

[0074] The implementation of this embodiment is similar to that of the above embodiment. The specific implementation can refer to the specific implementation of the above embodiment, which will not be repeated here.

[0075] Among them, this embodiment can add virtual information to the target object in the live video based on augmented reality (AR) technology. AR technology is a technology that cleverly integrates virtual information with the real world. In this embodiment of the application, a frame image of the live video data can be obtained, and the feature vector of the target object within the detection frame of the frame image and the frame image combination can be determined; then, based on the feature vector of the target object in the frame image combination, the related objects corresponding to the target object in the target frame image can be filtered out, and the corresponding feature vectors can be extracted to form nodes; then, based on the feature similarity and spatial similarity between the nodes, connecting edges can be added between the nodes to establish data associations between the nodes and form a corresponding feature graph. Then, based on the feature graph, the classification result corresponding to the target object is determined, and the movement trajectory of the target object in the video data is obtained. Then, based on the movement trajectory, virtual information can be added to the target object to enhance the live broadcast effect. For example, based on the movement trajectory of the host's face in the live video, virtual information of facial decorations (such as sunglasses) can be added to the host's face, achieving an effect of combining virtual and real life.

[0076] On the basis of the above embodiments, the embodiments of the present application provide a data processing method that can be applied to a processing end. The processing end can be understood as a device for acquiring and analyzing video data. The method can be applied in an autonomous driving scenario. The target objects (such as vehicles, pedestrians, obstacles) in different frame images of the driving video data during vehicle driving can be analyzed to determine the detection frame and feature vector of the target object. Then, data association is performed on the target objects in different frame images to determine the classification result of the target object, so as to determine the moving trajectory of the target object in the video data. After that, the driving instruction can be determined based on the moving trajectory to control the vehicle driving. Specifically, if Figure 6A As shown, the method includes:

[0077] Step 602: Acquire a frame image of driving video data, and determine a feature vector of a target object within a detection frame of the frame image and a frame image combination.

[0078] Step 604: Determine a feature map based on the similarity between target objects in the frame image combination, where the similarity includes feature similarity and / or spatial similarity. The feature map is used to associate the same target object in different frame images. Specifically, as an optional embodiment, step 604 includes: screening related objects corresponding to the target object in the target frame image based on the feature vectors of the target objects in the frame image combination, and extracting the corresponding feature vectors to form nodes; and adding connecting edges between the nodes based on the similarity between the nodes to form a feature map corresponding to the target object in the target frame image.

[0079] Step 606: Determine the classification result of each target object based on the feature map of each target object to determine the movement trajectory of the target object in the driving video data.

[0080] Step 608: Determine driving instructions based on the movement trajectory to control the vehicle.

[0081] In an embodiment of the present application, the target object's spatial movement trajectory in three-dimensional space can be determined based on the target object's movement trajectory in driving video data, combined with the target object's depth information in the driving video data, and then the corresponding driving instructions can be determined. Specifically, as an optional embodiment, determining the driving instructions based on the movement trajectory includes: obtaining the depth information of the target object in each frame image; determining the target object's spatial movement trajectory based on the target object's depth information and movement trajectory; and determining the driving instructions based on the spatial movement trajectory to control the vehicle's travel. In an embodiment of the present application, a depth recognition model can be pre-set, and the depth recognition model can be used to recognize the depth information of the target object in each frame image. Then, the spatial movement trajectory of the target object in the corresponding three-dimensional space can be determined based on the depth information and the target object's movement trajectory in the video data. Thereafter, the driving instructions can be determined based on the target object's spatial movement trajectory in three-dimensional space and the vehicle's spatial movement trajectory in three-dimensional space. Specifically, as an optional embodiment, determining the driving instructions based on the spatial movement trajectory includes: determining the vehicle's movement path based on the target object's spatial movement trajectory; and determining the corresponding driving instructions based on the movement path to control the vehicle's travel. This embodiment can plan vehicle overtaking, following and other behaviors based on the spatial movement trajectory of each target object, thereby determining the corresponding movement path and thus the corresponding driving instructions. The driving instructions may include instructions such as turning left, turning right, accelerating, decelerating, and braking, thereby controlling the vehicle.

[0082] The implementation of this embodiment is similar to that of the above embodiment. The specific implementation can refer to the specific implementation of the above embodiment, which will not be repeated here.

[0083] In an embodiment of the present application, a frame image of driving video data can be obtained, and the feature vector of the target object within the detection frame of the frame image and the frame image combination can be determined; then, based on the feature vector of the target object in the frame image combination, the related objects corresponding to the target object in the target frame image can be screened out, and the corresponding feature vectors can be extracted to form nodes; then, based on the feature similarity and spatial similarity between the nodes, connecting edges can be added between the nodes to establish data associations between the nodes and form a corresponding feature graph. Then, based on the feature graph, the classification result corresponding to the target object can be determined, and the movement trajectory of the target object in the video data can be obtained. Then, based on the movement trajectory, the corresponding driving instructions can be determined, such as deceleration, acceleration, turning, braking, and other driving instructions to control the driving of the vehicle.

[0084] Figure 6B A comparative schematic diagram of determining the target movement trajectory by using a feature association method and a method according to an embodiment of the present application is provided, such as Figure 6B As shown, in the process of determining the movement trajectory by using feature association (such as Figure 6B In the process of the left side), in the frame images of the first and third frames on the left side, vehicle 2 is not blocked, while in the frame image of the second frame on the left side, vehicle 2 is partially blocked. Therefore, in the process of data association using feature association, the features of vehicle 2 in the frame image of the second frame on the left side have low similarity with the features in the frame images of the other two frames, and may not be associated. Therefore, in the process of data association using feature association, data association can be performed on vehicle 2 in the frame images of the first and third frames to determine the movement trajectory (such as trajectory Figure 1 ).

[0085] In the process of determining the movement trajectory using the method of this embodiment (feature similarity and spatial similarity) (e.g. Figure 6B In the process on the right), in the first and third frames on the right, vehicle 2 is not blocked, so it can be associated by feature association. In the second frame on the right, vehicle 2 is partially blocked. Therefore, the spatial association method is used to determine the overlap of the detection frames of the target object in the first and second frames on the right, thereby associating the vehicles in the first and second frames on the right, and then determining the vehicle's movement trajectory (such as trajectory Figure 2 ). It can be seen that in the scenario of autonomous driving of a vehicle, the embodiment of the present application can perform trajectory recognition of multiple targets based on features and spatial associations, which can reduce the probability of target positions jumping, thereby being able to more accurately determine the relative position between the obstacle and the vehicle, thereby improving the safety of autonomous driving.

[0086] Based on the above embodiments, the embodiments of the present application provide a data processing method that can be applied to a processing end. The processing end can be understood as a device that interacts with a terminal to receive video data and analyzes the video data. The method can provide an interactive page to the terminal to facilitate users to upload video data to be analyzed; after receiving the uploaded video data, the movement trajectory of each target object can be analyzed to form an analysis result, and the analysis result can be issued through the interactive page. Specifically, Figure 7A As shown, the method includes:

[0087] Step 702: Provide an interactive page to obtain video data to be processed based on the interactive page.

[0088] Step 704: Acquire a frame image of the video data, and determine a feature vector of the target object within the detection frame of the frame image and a frame image combination.

[0089] Step 706: Determine a feature map based on the similarity between target objects in the frame image combination. The similarity includes feature similarity and / or spatial similarity. The feature map is used to associate the same target object in different frame images.

[0090] Step 708: Determine the classification result of each target object based on the feature map of each target object to determine the movement trajectory of the target object in the video data as the analysis result, and issue the analysis result.

[0091] like Figure 7B As shown, in an embodiment of the present application, the processing end can provide an interactive page to the terminal, and the interactive page includes a data upload control. The user of the terminal can upload the video data to be processed to the processing end by triggering the data upload control. The processing end can obtain the frame image of the video data and determine the feature vector of the target object in the detection frame of the frame image and the frame image combination; then, based on the feature vector of the target object in the frame image combination, the feature similarity between each target object in the frame image combination can be determined; based on the detection frame, the spatial similarity between each target object can be determined; then, based on the feature similarity and spatial similarity between each target object, an association can be established for the same target object in different frame images to form a corresponding feature map. Then, based on the feature map, the classification result corresponding to the target object is determined, and the movement trajectory of the target object in the video data is obtained as the analysis result, and the analysis result is sent down to display the analysis result in the interactive page. The embodiment of the present application can use the processing end to provide a service for identifying the trajectory of the target to the terminal, which can facilitate users to identify the trajectory of the target in the video data.

[0092] It should be noted that for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the order of the actions described, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0093] Based on the above embodiment, this embodiment further provides a data processing device, referring to Figure 8 , specifically including the following modules:

[0094] The video data acquisition module 802 is used to acquire a frame image of the video data and determine a feature vector of a target object within a detection frame of the frame image and a frame image combination.

[0095] The feature map acquisition module 804 is used to determine a feature map based on the similarity between target objects in the frame image combination, wherein the similarity includes feature similarity and / or spatial similarity. The feature map is used to associate the same target object in different frame images. As an optional embodiment, the feature map acquisition module 804 includes: a related node acquisition module, which is used to filter out related objects corresponding to the target object in the target frame image based on the feature vector of the target object in the frame image combination, and extract the corresponding feature vector to form a node; and an association relationship acquisition module, which is used to add connecting edges between nodes based on the similarity between the nodes, to form a feature map corresponding to the target object in the target frame image.

[0096] The movement trajectory acquisition module 806 is used to determine the classification result of each target object based on the feature map of each target object, so as to determine the movement trajectory of the target object in the video data.

[0097] In summary, in the embodiment of the present application, the frame image of the video data can be obtained, and the feature vector of the target object in the detection frame of the frame image and the frame image combination can be determined; then, the feature similarity between each target object in the frame image combination can be determined based on the feature vector of the target object in the frame image combination; the spatial similarity between each target object can be determined based on the detection frame; then, based on the feature similarity and spatial similarity between each target object, an association can be established for the same target object in different frame images to form a corresponding feature map. Then, based on the feature map, the classification result corresponding to the target object is determined, and the movement trajectory of the target object in the video data is obtained. In the process of data association, the embodiment of the present application can perform data association based on the feature similarity and spatial similarity between the target objects, so that the association of features and the association of spatial positions between the target objects can be considered, and a more accurate classification can be obtained, thereby improving the accuracy of the analysis of the movement trajectory of the target object in the video data.

[0098] Based on the above embodiment, this embodiment further provides a data processing device, which may specifically include the following modules:

[0099] The frame image acquisition processing module is used to acquire frame images of video data.

[0100] The frame image combination acquisition processing module is configured to determine an image group queue and add a target number of consecutive frame images to the image group queue to form a frame image combination. As an optional embodiment, this embodiment can adopt a dynamic evolutionary approach to continuously update the frame image combination. Specifically, the frame image combination acquisition processing module is further configured to delete at least one frame image from the image group queue and add a corresponding number of frame images to the image group queue to form a frame image combination.

[0101] The detection image acquisition processing module is used to determine the detection frame within the frame image and extract the detection image within the detection frame.

[0102] The feature vector acquisition processing module is used to extract the features of the target object in the detection image and obtain the feature vector.

[0103] The related object screening processing module is used to screen out a first object related to the target object of the target frame image according to the feature vector of the target object in the frame image combination, and to screen out a second object related to the first object.

[0104] The related node acquisition processing module is used to determine the feature vectors of the first object and the second object to form nodes.

[0105] The first connection processing module is used to determine the feature similarity between the nodes based on the feature vector of each node, and determine the nodes with adjacent feature vectors based on the feature similarity to add a first connection edge.

[0106] The second connection processing module is used to determine the spatial similarity between the nodes based on the overlap between the detection frames of the nodes, and determine the nodes that meet the overlap conditions to add a second connection edge to form a feature map of the target object in the corresponding target frame image.

[0107] The feature extraction processing module is used to input the feature map into the feature processing model to determine the classification result of the target object.

[0108] The movement trajectory acquisition processing module is used to connect the target objects in each frame image according to the classification results of the target objects in each frame image to obtain the movement trajectory of the target objects in the video data.

[0109] In an embodiment of the present application, a frame image of video data can be obtained. Then, on the one hand, a group queue can be determined, and a target number of continuous frame images can be added to the group queue to form a frame image combination; on the other hand, a detection frame for locating the target object in the frame image can be determined, and features of the target object in the detection frame can be extracted to obtain a feature vector of the target object. Afterwards, the first object and the second object corresponding to the target object in the target frame image can be screened out based on the feature vector of the target object in the frame image combination, and the corresponding feature vectors can be extracted to form nodes; then, based on the feature similarity between the nodes and the overlap of the detection frame, connecting edges can be added between the nodes to establish data associations between the nodes and form a corresponding feature graph. Then, based on the feature graph, the classification result corresponding to the target object is determined, and the movement trajectory of the target object in the video data is obtained.

[0110] Based on the above embodiment, this embodiment further provides a data processing device, referring to Figure 9 , specifically including the following modules:

[0111] Video data determination module 902 is configured to acquire frame images of road video data and determine a feature vector and frame image combination of a target object within a detection frame of the frame image. As an optional embodiment, the road video data may be acquired via a road video acquisition component. Specifically, the apparatus further includes a video data acquisition and processing module configured to acquire road video data based on the road video acquisition component. The road video acquisition component may be a camera positioned on the road. As an optional embodiment, video data determination module 902 may specifically include: determining a detection frame within the frame image and extracting a detection image within the detection frame; and extracting features of the target object within the detection image to obtain a feature vector. The detection frame is used to locate the target object. Target objects in road video data may include vehicles, pedestrians, obstacles, animals, and the like. As an optional embodiment, video data determination module 902 may specifically include: acquiring a target number of consecutive frame images as a frame image combination, and updating the frame images within the frame image combination after determining a feature map corresponding to the frame image combination. As an optional example, the video data determination module 902 specifically includes: obtaining a target number of consecutive frame images; determining an image group queue, and adding the target number of consecutive frame images to the image group queue to form a frame image combination. Updating the frame images in the frame image combination includes: deleting at least one frame image from the image group queue and adding a corresponding number of frame images to the image group queue to form an updated frame image combination.

[0112] The feature map determination module 904 determines a feature map based on the similarity between target objects in the frame image combination, wherein the similarity includes feature similarity and / or spatial similarity. The feature map is used to associate the same target object in different frame images. As an optional embodiment, the feature map determination module 904 specifically includes: a related node determination module for filtering out related objects corresponding to the target object in the target frame image based on the feature vector of the target object in the frame image combination, and extracting the corresponding feature vector to form a node; an association relationship determination module for adding connecting edges between nodes based on the similarity between each node to form a feature map corresponding to the target object in the target frame image. As an optional embodiment, the related node determination module specifically includes: filtering out a first object related to the target object in the target frame image based on the feature vector of the target object in the frame image combination, and filtering out a second object related to the first object; and determining the feature vectors of the first object and the second object to form a node. As an optional embodiment, the association relationship determination module specifically includes: determining the feature similarity between each node based on the feature vector of each node, and determining the nodes with adjacent feature vectors based on the feature similarity to add a first connecting edge; determining the spatial similarity between each node based on the overlap between the detection frames of each node, and determining the nodes that meet the overlap conditions to add a second connecting edge.

[0113] The movement trajectory determination module 906 is used to determine the classification result of each target object based on the feature map of each target object, so as to determine the movement trajectory of the target object in the road video data. As an optional embodiment, the movement trajectory determination module 906 specifically includes: inputting the feature map into a feature processing model to determine the classification result of the target object, and the feature processing model is used to determine the node features of the nodes and the edge features of the connecting edges, and determine the classification result of the target object. As an optional embodiment, the movement trajectory determination module 906 specifically includes: connecting the target objects in each frame image based on the classification result of the target object in each frame image to obtain the movement trajectory of the target object in the road video data.

[0114] The behavior result determination module 908 is used to determine the behavior type of the target object according to the movement trajectory to determine the corresponding behavior analysis result.

[0115] In an embodiment of the present application, a frame image of road video data can be obtained, and the feature vector of the target object in the detection frame of the frame image and the frame image combination can be determined; then, based on the feature vector of the target object in the frame image combination, the relevant objects corresponding to the target object in the target frame image can be screened out, and the corresponding feature vectors can be extracted to form nodes; then, based on the feature similarity and spatial similarity between the nodes, connecting edges can be added between the nodes to establish data associations between the nodes and form a corresponding feature graph. Then, based on the feature graph, the classification result corresponding to the target object can be determined, and the movement trajectory of the target object in the video data can be obtained. Then, based on the movement trajectory, it can be determined whether the target object has violated traffic regulations (such as driving against traffic lights, not following traffic lights, etc.) to obtain behavior analysis results.

[0116] Based on the above embodiment, this embodiment further provides a data processing device, referring to Figure 10 , specifically including the following modules:

[0117] The video data acquisition module 1002 is used to obtain a frame image of live video data and determine a feature vector of a target object within a detection frame of the frame image and a frame image combination.

[0118] The feature map acquisition module 1004 is used to determine a feature map based on the similarity between target objects in the frame image combination, wherein the similarity includes feature similarity and / or spatial similarity, and the feature map is used to associate the same target object in different frame images. As an optional embodiment, the feature map acquisition module 1004 specifically includes: a related node acquisition module, which is used to filter out related objects corresponding to the target object in the target frame image based on the feature vector of the target object in the frame image combination, and extract the corresponding feature vector to form a node; and an association relationship acquisition module, which is used to add connecting edges between nodes based on the similarity between the nodes, to form a feature map corresponding to the target object in the target frame image.

[0119] The movement trajectory obtaining module 1006 is used to determine the classification result of each target object based on the feature map of each target object, so as to determine the movement trajectory of the target object in the live video data.

[0120] The virtual information adding module 1008 is used to add virtual information to the target object in the live video data according to the movement trajectory.

[0121] This embodiment can add virtual information to the target object in the live video. In the embodiment of the present application, the frame image of the live video data can be obtained, and the feature vector of the target object in the detection frame of the frame image and the frame image combination can be determined; then, based on the feature vector of the target object in the frame image combination, the related objects corresponding to the target object in the target frame image can be screened out, and the corresponding feature vectors can be extracted to form nodes; then, based on the feature similarity and spatial similarity between the nodes, connecting edges can be added between the nodes to establish data associations between the nodes and form a corresponding feature graph. Then, based on the feature graph, the classification result corresponding to the target object is determined, and the movement trajectory of the target object in the video data is obtained. Then, based on the movement trajectory, virtual information can be added to the target object to enhance the live broadcast effect. For example, based on the movement trajectory of the host's face in the live video, virtual information of facial decorations (such as sunglasses) can be added to the host's face to achieve the effect of combining virtual and reality.

[0122] Based on the above embodiment, this embodiment further provides a data processing device, referring to Figure 11 , specifically including the following modules:

[0123] The video data acquisition module 1102 is used to obtain frame images of driving video data and determine the feature vector and frame image combination of the target object in the detection frame of the frame image.

[0124] The feature map acquisition module 1104 is used to determine a feature map based on the similarity between target objects in the frame image combination, where the similarity includes feature similarity and / or spatial similarity. The feature map is used to associate the same target object in different frame images. Specifically, as an optional embodiment, the feature map acquisition module 1104 includes: a related node acquisition module, which is used to filter out related objects corresponding to the target object in the target frame image based on the feature vector of the target object in the frame image combination, and extract the corresponding feature vector to form a node; and an associated information acquisition module, which is used to add connecting edges between nodes based on the similarity between the nodes, to form a feature map corresponding to the target object in the target frame image.

[0125] The movement trajectory acquisition module 1106 is used to determine the classification result of each target object based on the feature map of each target object, so as to determine the movement trajectory of the target object in the driving video data.

[0126] The driving control module 1108 is used to determine driving instructions based on the movement trajectory to control the vehicle.

[0127] The embodiment of the present application can determine the spatial movement trajectory of the target object in three-dimensional space based on the movement trajectory of the target object in the driving video data, combined with the depth information of the target object in the driving video data, and then determine the corresponding driving instructions. Specifically, as an optional embodiment, the driving control module 1108 specifically includes: a depth information acquisition processing module for acquiring the depth information of the target object in each frame image; a spatial movement trajectory acquisition processing module for determining the spatial movement trajectory of the target object based on the depth information and movement trajectory of the target object; a driving instruction acquisition processing module for determining the driving instructions based on the spatial movement trajectory to control the vehicle driving. The embodiment of the present application can pre-set a depth recognition model, use the depth recognition model to identify the depth information of the target object in each frame image, and then determine the corresponding spatial movement trajectory in the three-dimensional space based on the depth information and the movement trajectory of the target object in the video data. Subsequently, driving instructions can be determined based on the spatial movement trajectory of the target object in three-dimensional space and the spatial movement trajectory of the vehicle in three-dimensional space. Specifically, as an optional embodiment, the driving instruction acquisition and processing module specifically includes: determining the vehicle's movement path based on the spatial movement trajectory of the target object; and determining corresponding driving instructions based on the movement path to control vehicle travel. This embodiment can plan vehicle behaviors such as overtaking and following based on the spatial movement trajectory of each target object, thereby determining the corresponding movement path and corresponding driving instructions. The driving instructions may include instructions such as turning left, turning right, accelerating, decelerating, and braking, thereby controlling the vehicle.

[0128] In an embodiment of the present application, a frame image of driving video data can be obtained, and the feature vector of the target object within the detection frame of the frame image and the frame image combination can be determined; then, based on the feature vector of the target object in the frame image combination, the related objects corresponding to the target object in the target frame image can be screened out, and the corresponding feature vectors can be extracted to form nodes; then, based on the feature similarity and spatial similarity between the nodes, connecting edges can be added between the nodes to establish data associations between the nodes and form a corresponding feature graph. Then, based on the feature graph, the classification result corresponding to the target object can be determined, and the movement trajectory of the target object in the video data can be obtained. Then, based on the movement trajectory, the corresponding driving instructions can be determined, such as deceleration, acceleration, turning, braking, and other driving instructions to control the driving of the vehicle.

[0129] Based on the above embodiment, this embodiment further provides a data processing device, referring to Figure 12 , specifically including the following modules:

[0130] The interactive page providing module 1202 is configured to provide an interactive page to obtain video data to be processed based on the interactive page.

[0131] The video data processing module 1204 is configured to obtain a frame image of the video data and determine a feature vector of a target object within a detection frame of the frame image and a frame image combination.

[0132] The feature vector processing module 1206 is used to determine a feature map based on the similarity between target objects in the frame image combination, where the similarity includes feature similarity and / or spatial similarity. The feature map is used to associate the same target object in different frame images.

[0133] The analysis result sending module 1208 is used to determine the classification result of each target object based on the feature map of each target object, so as to determine the movement trajectory of the target object in the video data as the analysis result, and send the analysis result.

[0134] In an embodiment of the present application, the processing end can provide an interactive page to the terminal, and the interactive page includes a data upload control. The user of the terminal can upload the video data to be processed to the processing end by triggering the data upload control. The processing end can obtain the frame image of the video data and determine the feature vector of the target object in the detection frame of the frame image and the frame image combination; then, based on the feature vector of the target object in the frame image combination, the feature similarity between each target object in the frame image combination can be determined; based on the detection frame, the spatial similarity between each target object can be determined; then, based on the feature similarity and spatial similarity between each target object, an association can be established for the same target object in different frame images to form a corresponding feature map. Then, based on the feature map, the classification result corresponding to the target object is determined, and the movement trajectory of the target object in the video data is obtained as the analysis result, and the analysis result is sent down to display the analysis result in the interactive page. The embodiment of the present application can use the processing end to provide a service for identifying the trajectory of the target to the terminal, which can facilitate users to identify the trajectory of the target in the video data.

[0135] An embodiment of the present application further provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0136] The present application provides one or more machine-readable media having instructions stored thereon, which, when executed by one or more processors, cause an electronic device to perform one or more of the methods described in the above embodiments. In the present application, the electronic device includes a server, a terminal device, and the like.

[0137] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as a server (cluster), a terminal, etc. Figure 13 An exemplary apparatus 1300 that can be used to implement various embodiments described in this application is schematically illustrated.

[0138] For one embodiment, Figure 13 An exemplary apparatus 1300 is shown having one or more processors 1302, a control module (chip set) 1304 coupled to at least one of the processor(s) 1302, a memory 1306 coupled to the control module 1304, a non-volatile memory (NVM) / storage device 1308 coupled to the control module 1304, one or more input / output devices 1310 coupled to the control module 1304, and a network interface 1312 coupled to the control module 1304.

[0139] The processor 1302 may include one or more single-core or multi-core processors, and the processor 1302 may include any combination of general-purpose processors or dedicated processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, the apparatus 1300 can serve as a server, terminal, or other device described in the embodiments of the present application.

[0140] In some embodiments, the apparatus 1300 may include one or more computer-readable media (e.g., memory 1306 or NVM / storage 1308) having instructions 1314 and one or more processors 1302 configured in conjunction with the one or more computer-readable media to execute the instructions 1314 to implement a module to perform the actions described in the present disclosure.

[0141] For one embodiment, the control module 1304 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1302 and / or any suitable device or component in communication with the control module 1304 .

[0142] The control module 1304 may include a memory controller module to provide an interface to the memory 1306. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0143] The memory 1306 can be used, for example, to load and store data and / or instructions 1314 for the device 1300. For one embodiment, the memory 1306 can include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 1306 can include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0144] For one embodiment, control module 1304 may include one or more input / output controllers to provide interfaces to NVM / storage device 1308 and input / output device(s) 1310 .

[0145] For example, NVM / storage 1308 may be used to store data and / or instructions 1314. NVM / storage 1308 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0146] NVM / storage device 1308 may include storage resources that are part of the device on which apparatus 1300 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 1308 may be accessible over a network via input / output device(s) 1310.

[0147] (One or more) input / output devices 1310 may provide an interface for apparatus 1300 to communicate with any other appropriate devices. Input / output devices 1310 may include communication components, audio components, sensor components, etc. Network interface 1312 may provide an interface for apparatus 1300 to communicate via one or more networks. Apparatus 1300 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0148] For one embodiment, at least one of the processor(s) 1302 may be packaged together with the logic of one or more controllers of the control module 1304 (e.g., a memory controller module). For one embodiment, at least one of the processor(s) 1302 may be packaged together with the logic of one or more controllers of the control module 1304 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with the logic of one or more controllers of the control module 1304. For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with the logic of one or more controllers of the control module 1304 to form a system-on-chip (SoC).

[0149] In various embodiments, the apparatus 1300 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the apparatus 1300 may have more or fewer components and / or a different architecture. For example, in some embodiments, the apparatus 1300 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0150] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in the memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0151] An embodiment of the present application further provides an electronic device, comprising: a processor; and a memory on which executable code is stored. When the executable code is executed, the processor executes one or more methods described in the embodiments of the present application.

[0152] The embodiments of the present application further provide one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes one or more methods described in the embodiments of the present application.

[0153] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0154] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0155] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0156] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0158] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0159] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0160] The above is a detailed introduction to a data processing method, a data processing device, an electronic device and a storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A data processing method, characterized in that: The method includes: Acquire a frame image of the video data and determine a feature vector of a target object within a detection frame of the frame image; Acquire a target number of continuous frame images as a frame image combination; According to the feature vector of the target object in the frame image combination, the related objects corresponding to the target object in the target frame image are screened out, and the corresponding feature vectors are extracted to form nodes; Determining the feature similarity between target objects in the frame image combination based on feature vectors of the target objects in the frame image combination; Determine the spatial similarity between target objects based on the overlap between the detection frames of the target objects; Determine the similarity between nodes based on feature similarity and spatial similarity, add connecting edges between nodes, and form a feature map of the target object in the corresponding target frame image; According to the feature map of each target object, the classification result of each target object is determined to determine the movement trajectory of the target object in the video data.

2. The method according to claim 1, characterized in that Also includes: After determining the feature map corresponding to the frame image combination, the frame images in the frame image combination are updated.

3. The method according to claim 2, characterized in that The step of acquiring a target number of continuous frame images as a frame image combination includes: Acquire a target number of continuous frame images; An image group queue is determined, and a target number of continuous frame images are added to the image group queue to form a frame image combination.

4. The method according to claim 3, characterized in that The updating of the frame images in the frame image combination includes: At least one frame image is deleted from the image group queue and a corresponding number of frame images are added to the image group queue to form an updated frame image combination.

5. The method according to claim 1, wherein The determining of a feature vector of a target object within a detection frame of a frame image includes: Determine the detection frame within the frame image and extract the detection image within the detection frame; Extract the features of the target object in the detection image and obtain the feature vector.

6. The method according to claim 1, wherein The method of screening out related objects corresponding to the target object in the target frame image based on the feature vector of the target object in the frame image combination, and extracting the corresponding feature vector to form a node includes: Screening out a first object related to the target object of the target frame image based on a feature vector of the target object in the frame image combination, and screening out a second object related to the first object; Determine the feature vectors of the first object and the second object to form a node.

7. The method according to claim 1, characterized in that Adding connecting edges between nodes based on the similarity between the nodes includes: Determining feature similarities between nodes based on feature vectors of each node, and determining nodes with adjacent feature vectors based on the feature similarities to add a first connecting edge; According to the overlap between the detection frames of the nodes, the spatial similarity between the nodes is determined, and the nodes that meet the overlap condition are determined to add the second connecting edge.

8. The method according to claim 1, characterized in that Determining the classification result of each target object based on the feature map of each target object includes: The feature graph is input into a feature processing model to determine a classification result of the target object. The feature processing model is used to determine node features of nodes and edge features of connecting edges, and determine the classification result of the target object.

9. The method according to claim 1, characterized in that Determining the movement trajectory of the target object in the video data includes: According to the classification results of the target objects in each frame image, the target objects in each frame image are connected to obtain the movement trajectory of the target objects in the video data.

10. A data processing method, characterized in that: The method includes: Acquire a frame image of road video data, and determine a feature vector of a target object within a detection frame of the frame image; Acquire a target number of continuous frame images as a frame image combination; According to the feature vector of the target object in the frame image combination, the related objects corresponding to the target object in the target frame image are screened out, and the corresponding feature vectors are extracted to form nodes; Determining the feature similarity between target objects in the frame image combination based on feature vectors of the target objects in the frame image combination; Determine the spatial similarity between target objects based on the overlap between the detection frames of the target objects; Determine the similarity between nodes based on feature similarity and spatial similarity, add connecting edges between nodes, and form a feature map of the target object in the corresponding target frame image. The feature map is used to associate the same target object in different frame images. Determine the classification result of each target object based on the feature map of each target object to determine the movement trajectory of the target object in the road video data; Based on the movement trajectory, the behavior type of the target object is determined to determine the corresponding behavior analysis result.

11. A data processing method, characterized in that: The method includes: Obtain a frame image of live video data and determine a feature vector of a target object within a detection frame of the frame image; Acquire a target number of continuous frame images as a frame image combination; According to the feature vector of the target object in the frame image combination, the related objects corresponding to the target object in the target frame image are screened out, and the corresponding feature vectors are extracted to form nodes; Determining the feature similarity between target objects in the frame image combination based on feature vectors of the target objects in the frame image combination; Determine the spatial similarity between target objects based on the overlap between the detection frames of the target objects; Determine the similarity between nodes based on feature similarity and spatial similarity, add connecting edges between nodes, and form a feature map of the target object in the corresponding target frame image. The feature map is used to associate the same target object in different frame images. Determine the classification result of each target object based on the feature map of each target object to determine the movement trajectory of the target object in the live video data; According to the movement trajectory, virtual information is added to the target object in the live video data.

12. A data processing method, characterized in that: The method includes: Acquire a frame image of driving video data, and determine a feature vector of a target object within a detection frame of the frame image; Acquire a target number of continuous frame images as a frame image combination; According to the feature vector of the target object in the frame image combination, the related objects corresponding to the target object in the target frame image are screened out, and the corresponding feature vectors are extracted to form nodes; Determining the feature similarity between target objects in the frame image combination based on feature vectors of the target objects in the frame image combination; Determine the spatial similarity between target objects based on the overlap between the detection frames of the target objects; Determine the similarity between nodes based on feature similarity and spatial similarity, add connecting edges between nodes, and form a feature map of the target object in the corresponding target frame image. The feature map is used to associate the same target object in different frame images. Determine the classification result of each target object based on the feature map of each target object to determine the movement trajectory of the target object in the driving video data; According to the movement trajectory, a driving instruction is determined to control the vehicle.

13. The method according to claim 12, characterized in that The determining of the driving instruction according to the movement trajectory includes: Obtain the depth information of the target object in each frame image; Determine the spatial movement trajectory of the target object based on the depth information and movement trajectory of the target object; According to the spatial movement trajectory, a driving instruction is determined to control the vehicle.

14. The method according to claim 13, characterized in that Determining the driving instruction based on the spatial movement trajectory includes: Determine the vehicle's moving path based on the target object's spatial movement trajectory; According to the moving path, corresponding driving instructions are determined to control the vehicle's travel.

15. A data processing method, characterized in that: The method includes: Providing an interactive page to obtain video data to be processed based on the interactive page; Acquire a frame image of the video data and determine a feature vector of a target object within a detection frame of the frame image; Acquire a target number of continuous frame images as a frame image combination; According to the feature vector of the target object in the frame image combination, the related objects corresponding to the target object in the target frame image are screened out, and the corresponding feature vectors are extracted to form nodes; Determining the feature similarity between target objects in the frame image combination based on feature vectors of the target objects in the frame image combination; Determine the spatial similarity between target objects based on the overlap between the detection frames of the target objects; Determine the similarity between nodes based on feature similarity and spatial similarity, add connecting edges between nodes, and form a feature map of the target object in the corresponding target frame image. The feature map is used to associate the same target object in different frame images. Based on the feature map of each target object, a classification result of each target object is determined to determine a movement trajectory of the target object in the video data as an analysis result, and the analysis result is issued.

16. A data processing device, characterized in that: The device comprises: The video data acquisition module is used to acquire frame images of video data and determine the feature vector of the target object within the detection frame of the frame image; and acquire a target number of consecutive frame images as a frame image combination; A feature map acquisition module is used to screen out related objects corresponding to the target object in the target frame image based on the feature vector of the target object in the frame image combination, and extract the corresponding feature vector to form a node; determine the feature similarity between the target objects in the frame image combination based on the feature vector of the target object in the frame image combination; determine the spatial similarity between the target objects based on the overlap between the detection frames of the target objects; determine the similarity between the nodes based on the feature similarity and the spatial similarity, add connecting edges between the nodes, and form a feature map corresponding to the target object in the target frame image, wherein the feature map is used to associate the same target object in different frame images; The movement trajectory acquisition module is used to determine the classification result of each target object based on the feature map of each target object, so as to determine the movement trajectory of the target object in the video data.

17. An electronic device, characterized in that: include: processor; and A memory having executable code stored thereon, which, when executed, causes the processor to perform the method according to one or more of claims 1-15.

18. One or more machine-readable media having executable code stored thereon, which, when executed, causes a processor to perform the method of one or more of claims 1-15.

Images