Method, device, equipment and storage medium for detecting and tracking video object association

Abstract

The present invention relates to artificial intelligence technology and discloses a method for detecting and tracking the association of video objects, comprising: obtaining an object video, extracting features from the object video frame by frame using a preset feature extraction network, and obtaining a video frame feature sequence corresponding to the object video; sequentially selecting two consecutive video frame features from the video frame feature sequence as target feature blocks, performing object detection on the target feature blocks using a preset attention model, and obtaining an object detection frame corresponding to the target feature block; using a preset multi-level node conversion method to convert the object detection frames of the target feature blocks one by one into a node network, and obtaining a node network graph corresponding to the target feature block; performing association analysis on the node network graph, and determining whether the object in the object video is being tracked based on the results of the association analysis. The present invention also provides a device, apparatus, and storage medium for detecting and tracking the association of video objects. The present invention can improve the accuracy of tracking detection.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and storage medium for detecting and tracking the association of video objects. Background Art

[0002] With the development of artificial intelligence technology, more and more fields have begun to use artificial intelligence technology to assist work. For example, in order to reduce the workload of staff in criminal investigations, the method of video object correlation detection can be used to identify personnel information in surveillance videos and assist staff in determining whether the target person is being tracked.

[0003] Existing methods for detecting the association of video objects mostly use image detection methods based on neural networks to track and detect video objects, and then judge the tracking status based on the target person. In practical applications, the image detection method has a large amount of computation and lacks an intelligent solution to the problem of person association. The results of manual judgment are less accurate, which leads to lower accuracy in tracking detection. Summary of the Invention

[0004] The present invention provides a method, device, equipment and storage medium for detecting and tracking the association of video objects, the main purpose of which is to solve the problem of low accuracy during tracking detection.

[0005] To achieve the above objectives, the present invention provides a method for detecting and tracking the association of video objects, comprising:

[0006] Obtain an object video, and extract features from the object video frame by frame using a preset feature extraction network to obtain a video frame feature sequence corresponding to the object video;

[0007] Selecting two consecutive video frame features from the video frame feature sequence as target feature blocks, performing object detection on the target feature blocks using a preset attention model, and obtaining an object detection frame corresponding to the target feature block;

[0008] The object detection frames of the target feature blocks are converted into node networks one by one using a preset multi-level node conversion method to obtain a node network graph corresponding to the target feature blocks;

[0009] A preset correlation analysis network is used to perform correlation analysis on the node network graph, and whether the object in the object video is tracked is determined based on the result of the correlation analysis.

[0010] Optionally, extracting features from the object video frame by frame using a preset feature extraction network to obtain a video frame feature sequence corresponding to the object video includes:

[0011] Using a preset video conversion tool to convert the object video into a video frame atlas;

[0012] Selecting video frame images from the video frame atlas one by one as target images, and extracting features from the target images using a bottom-up convolutional network structure to obtain a feature pyramid consisting of multiple features with gradually increasing feature levels;

[0013] Selecting features in the feature pyramid from top to bottom as target layer features, using an upsampling algorithm to enlarge the target layer features to the same size as the lower layer features, and using a lateral connection feature fusion method to fuse the lower layer features with the upsampled target layer features to obtain feature layer features corresponding to the target layer features;

[0014] All feature layer features of the target image are aggregated into video frame features, and all video frame features of the video frame atlas are aggregated into a video frame feature sequence.

[0015] Optionally, performing object detection on the target feature block using a preset attention model to obtain an object detection frame corresponding to the target feature block includes:

[0016] Encoding the target feature block into a feature code using an encoder of the attention model;

[0017] Decoding the entity to be queried from the feature code using a decoder of the attention model;

[0018] The entity to be queried is mapped through a multi-layer perceptron of the attention model to obtain the object detection frame.

[0019] Optionally, the converting the object detection frames of the target feature blocks into node networks one by one using a preset multi-level node conversion method to obtain a node network graph corresponding to the target feature blocks includes:

[0020] Taking the object bounding box in the object detection box as the vertex of the node network;

[0021] Multiplying the object category corresponding to the object bounding box by the object confidence corresponding to the object bounding box to obtain a vertex value corresponding to the vertex;

[0022] The vertices are connected in series using straight lines to obtain vertex edges of the node network, and the center vectors of the object bounding boxes corresponding to the two ends of the vertex edges are converted into weights of the vertex edges to complete the transformation of the node network graph.

[0023] Optionally, converting the center vectors of the object bounding boxes corresponding to both ends of the vertex edge into the weights of the vertex edge includes:

[0024] Selecting the object bounding box corresponding to one end of the vertex edge as the first bounding box, and the object bounding box corresponding to the other end of the vertex edge as the second bounding box;

[0025] Calculating a first center vector corresponding to the first bounding box and a second center vector corresponding to the second bounding box using a center vector algorithm;

[0026] The second center vector is subtracted from the first center vector to obtain a standard vertex edge vector, and a modulo operation is performed on the standard vertex edge vector to obtain a weight of the vertex edge.

[0027] Optionally, the performing correlation analysis on the node network graph one by one using a preset correlation analysis network includes:

[0028] Encoding each vertex in the node network graph according to the vertex value and the weight of the vertex edge of the node network using the encoder of the association analysis network to obtain a vertex vector corresponding to each vertex;

[0029] The vertex vector is decoded by a decoder of the association analysis network to obtain the association type of the vertex edge.

[0030] Optionally, the encoder using the associative network encodes each vertex in the node network graph according to the vertex value of the node network and the weight of the vertex edge to obtain a vertex vector corresponding to each vertex, including:

[0031] The vertex values ​​of the node network are sampled one by one by an encoder of the association analysis network;

[0032] The weight of the vertex edge of the vertex is obtained by sampling the adjacent vertices of the vertex, and the timing information of the vertex is obtained by encoding the current timestamp;

[0033] The vertex vector is obtained by performing aggregate coding on the timing information, the vertex value, and the weight.

[0034] In order to solve the above problems, the present invention further provides a device for detecting and tracking the association of video objects, the device comprising:

[0035] A feature extraction module is used to obtain an object video and extract features from the object video frame by frame using a preset feature extraction network to obtain a video frame feature sequence corresponding to the object video;

[0036] An object detection module is configured to sequentially select two consecutive video frame features from the video frame feature sequence as target feature blocks, perform object detection on the target feature blocks using a preset attention model, and obtain an object detection frame corresponding to the target feature block;

[0037] A node network module is used to convert the object detection frames of the target feature block into a node network one by one using a preset multi-level node conversion method to obtain a node network graph corresponding to the target feature block;

[0038] A correlation analysis module, configured to perform correlation analysis on the node network graph using a preset correlation analysis network;

[0039] A tracking detection module is used to determine whether the object in the object video is tracked according to the result of the correlation analysis.

[0040] In order to solve the above problem, the present invention further provides a device, comprising:

[0041] at least one processor; and,

[0042] a memory communicatively connected to the at least one processor; wherein,

[0043] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the above-mentioned method for detecting and tracking the association of video objects.

[0044] In order to solve the above problem, the present invention further provides a storage medium, wherein the storage medium stores at least one computer program, and the at least one computer program is executed by a processor in a device to implement the above-mentioned method for detecting and tracking the association of video objects.

[0045] The embodiment of the present invention uses a feature extraction network to extract features from an object video, which can effectively extract high-level features from low-resolution features and retain image details of high-resolution features, thereby improving the accuracy of feature extraction and providing high-quality detection samples for subsequent correlation detection. By performing object detection on the target feature block through the attention model, the feature dependency relationship between all target objects in the target feature block can be captured and the original features can be enhanced with global context information. At the same time, the trajectory of the target object can be further determined by the introduced position encoding vector, thereby ensuring the accuracy of subsequent correlation analysis. By performing correlation analysis on the node network graph through the correlation analysis network, the detailed features of the object in the object video can be effectively retained, the accuracy of object selection and category determination can be improved, and the accuracy of tracking detection can be improved. Therefore, the detection and tracking method, device, equipment and storage medium of video object correlation proposed by the present invention can solve the problem of low accuracy during tracking detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flowchart of a method for detecting and tracking the association of video objects provided by one embodiment of the present invention;

[0047] Figure 2 A schematic diagram of the object detection process according to an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of a node conversion process according to an embodiment of the present invention;

[0049] Figure 4 A functional module diagram of a device for detecting and tracking the association of video objects provided by one embodiment of the present invention;

[0050] Figure 5 A schematic structural diagram of a device for implementing the method for detecting and tracking video object association provided by an embodiment of the present invention.

[0051] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0053] The embodiment of the present application provides a method for detecting and tracking the association of video objects. The execution subject of the method for detecting and tracking the association of video objects includes but is not limited to at least one of the devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present application. In other words, the method for detecting and tracking the association of video objects can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0054] Reference Figure 1 FIG. 1 is a flow chart of a method for detecting and tracking the association of video objects provided by an embodiment of the present invention. In this embodiment, the method for detecting and tracking the association of video objects includes:

[0055] S1. Obtain an object video, and extract features from the object video frame by frame using a preset feature extraction network to obtain a video frame feature sequence corresponding to the object video;

[0056] In the embodiment of the present invention, the object video may be a video containing multiple moving objects, such as a criminal investigation case video or a surveillance video.

[0057] In an embodiment of the present invention, extracting features from the object video frame by frame using a preset feature extraction network to obtain a video frame feature sequence corresponding to the object video includes:

[0058] Using a preset video conversion tool to convert the object video into a video frame atlas;

[0059] Selecting video frame images from the video frame atlas one by one as target images, and extracting features from the target images using a bottom-up convolutional network structure to obtain a feature pyramid consisting of multiple features with gradually increasing feature levels;

[0060] Selecting features in the feature pyramid from top to bottom as target layer features, using an upsampling algorithm to enlarge the target layer features to the same size as the lower layer features, and using a lateral connection feature fusion method to fuse the lower layer features with the upsampled target layer features to obtain feature layer features corresponding to the target layer features;

[0061] All feature layer features of the target image are aggregated into video frame features, and all video frame features of the video frame atlas are aggregated into a video frame feature sequence.

[0062] In detail, the video conversion tool may be the cv2.VideoCapture tool in the OpenCV tool library.

[0063] In detail, upsampling may be achieved by using a nearest neighbor difference algorithm, where upsampling refers to magnifying the original image so that the original image can be displayed at a higher resolution.

[0064] Specifically, the feature fusion method using lateral connection is used to fuse the lower layer features with the upsampled target layer features to obtain feature layer features corresponding to the target layer features, including:

[0065] Perform a convolution operation on the lower layer features using a 1x1 convolution kernel to obtain standard lower layer features;

[0066] Adding the characteristic value of the standard lower layer feature to the characteristic value of the upsampled target layer feature, and updating the characteristic value of the target layer feature using the result of the addition;

[0067] A 3x3 convolution kernel is used to perform a convolution operation on the target layer features after the eigenvalues ​​are updated to obtain feature layer features corresponding to the target layer feature layer.

[0068] In detail, the convolution operation performed on the lower feature layer using a 1x1 convolution kernel can ensure that the number of channels of the standard lower feature layer is consistent with the number of channels of the target layer feature, which facilitates the subsequent feature value addition operation.

[0069] Specifically, the 3x3 convolution kernel is used to perform a convolution operation on the target layer features after the eigenvalues ​​are updated, which can adjust the channel size of the feature layer features, further extract object features, and facilitate docking with subsequent attention models.

[0070] In an embodiment of the present invention, by using a feature extraction network to extract features from object videos, high-level features can be effectively extracted from low-resolution features, and image details of high-resolution features can be retained, thereby improving the accuracy of feature extraction and providing high-quality detection samples for subsequent correlation detection.

[0071] S2. Select two consecutive video frame features from the video frame feature sequence as target feature blocks, perform object detection on the target feature blocks using a preset attention model, and obtain an object detection frame corresponding to the target feature block;

[0072] In an embodiment of the present invention, the object detection frame includes an object bounding box, an object category, and an object confidence level.

[0073] In detail, the attention model can be a TrackFormer model, which can realize target detection between consecutive frames and form a tracking track for each detected target.

[0074] In the embodiment of the present invention, referring to Figure 2 As shown, the object detection is performed on the target feature block by using a preset attention model to obtain an object detection frame corresponding to the target feature block, including:

[0075] S21, using the encoder of the attention model to encode the target feature block into a feature code;

[0076] S22, using the decoder of the attention model to decode the entity to be queried from the feature code;

[0077] S23. Map the entity to be queried through the multi-layer perceptron of the attention model to obtain the object detection frame.

[0078] Among them, the encoder of the attention model can be the encoder of the TrackFormer model. The encoder of the TrackFormer model includes a self-attention mechanism, residual connection and activation feedback, which can capture the feature dependencies between all elements in the sequence and enhance the original features with global context information, so that the TrackFormer model can learn recognizable features for locating the target object.

[0079] Specifically, the feature encoding refers to embedding the feature vector of the target feature block into a position encoding vector, which can introduce the position information of the target object while capturing the object features, thereby facilitating the detection of the target object trajectory.

[0080] In detail, the decoder of the attention model may refer to the Decoder of the TrackFormer model, and the Decoder of the TrackFormer model includes a cross attention mechanism, a residual connection, and activation feedback.

[0081] Furthermore, the entity to be queried refers to a target object to be detected in the video frame image corresponding to the target feature block.

[0082] In detail, the multilayer perceptron is a special neural network that has multiple hidden layers between the input layer and the output layer in addition to the input layer and the output layer.

[0083] In an embodiment of the present invention, object detection is performed on the target feature block through the attention model, which can capture the feature dependencies between all target objects in the target feature block and enhance the original features with global context information. At the same time, the trajectory of the target object can be further determined by the introduced position encoding vector, thereby ensuring the accuracy of subsequent correlation analysis.

[0084] S3, using a preset multi-level node conversion method to convert the object detection frames of the target feature block into a node network one by one, to obtain a node network graph corresponding to the target feature block;

[0085] In the embodiment of the present invention, referring to Figure 3 As shown, the object detection frames of the target feature blocks are converted into node networks one by one using a preset multi-level node conversion method to obtain a node network diagram corresponding to the target feature blocks, including:

[0086] S31, taking the object bounding box in the object detection box as the vertex of the node network;

[0087] S32. Multiply the object category corresponding to the object bounding box by the object confidence corresponding to the object bounding box to obtain a vertex value corresponding to the vertex;

[0088] S33. Connect the vertices in series using straight lines to obtain vertex edges of the node network, and convert the center vectors of the object bounding boxes corresponding to the two ends of the vertex edges into weights of the vertex edges to complete the transformation of the node network graph.

[0089] In detail, the object bounding box refers to a selection box used to frame the object feature portion in the target feature block.

[0090] Specifically, the object category refers to the category of the object in the target feature block, including but not limited to people, animals, machines, plants, etc.

[0091] Furthermore, the object confidence refers to the probability that the object is within the object bounding box.

[0092] In detail, converting the center vectors of the object bounding boxes corresponding to the two ends of the vertex edge into the weights of the vertex edge includes:

[0093] Selecting the object bounding box corresponding to one end of the vertex edge as the first bounding box, and the object bounding box corresponding to the other end of the vertex edge as the second bounding box;

[0094] Calculating a first center vector corresponding to the first bounding box and a second center vector corresponding to the second bounding box using a center vector algorithm;

[0095] The second center vector is subtracted from the first center vector to obtain a standard vertex edge vector, and a modulo operation is performed on the standard vertex edge vector to obtain a weight of the vertex edge.

[0096] In detail, the center vector of the object bounding box refers to the position vector of the midpoint of the object bounding box in the target feature block.

[0097] In an embodiment of the present invention, the object detection frames of the target feature blocks are converted into corresponding node network diagrams one by one through a preset multi-level node conversion method, which can network the important information between the target feature blocks and facilitate the input of parameters of the subsequent correlation analysis network.

[0098] S4. Performing a correlation analysis on the node network graph using a preset correlation analysis network, and determining whether the object in the object video is tracked based on a result of the correlation analysis.

[0099] In an embodiment of the present invention, the correlation analysis network may refer to a temporal graph network (TGN), which is a network for analyzing the correlation between dynamic graph elements.

[0100] In an embodiment of the present invention, the performing correlation analysis on the node network graph one by one using a preset correlation analysis network includes:

[0101] Encoding each vertex in the node network graph according to the vertex value and the weight of the vertex edge of the node network using the encoder of the association analysis network to obtain a vertex vector corresponding to each vertex;

[0102] The vertex vector is decoded by a decoder of the association analysis network to obtain the association type of the vertex edge.

[0103] Specifically, the encoder using the association analysis network encodes each vertex in the node network graph according to the vertex value of the node network and the weight of the vertex edge to obtain the vertex vector corresponding to each vertex, including:

[0104] The vertex values ​​of the node network are sampled one by one by an encoder of the association analysis network;

[0105] The weight of the vertex edge of the vertex is obtained by sampling the adjacent vertices of the vertex, and the timing information of the vertex is obtained by encoding the current timestamp;

[0106] The vertex vector is obtained by performing aggregate coding on the timing information, the vertex value, and the weight.

[0107] Specifically, the encoder of the correlation analysis network may refer to an encoder of a time sequence graph network.

[0108] In detail, the decoder of the relevance analysis network may be the multilayer perceptron.

[0109] In detail, judging whether the object in the object video is tracked according to the result of the correlation analysis includes:

[0110] When the association type of the vertex edge corresponding to the object in the object video is related, determining that the object is being tracked by another object corresponding to a vertex edge of the related type;

[0111] When the association type of the vertex edges corresponding to the object in the object video is irrelevant, it is determined that the object is not tracked.

[0112] In the embodiment of the present invention, by performing correlation analysis on the node network diagram through the correlation analysis network, the detailed features of the object in the object video can be effectively retained, the accuracy of object selection and category determination can be improved, and the accuracy of tracking detection can be improved.

[0113] The embodiment of the present invention uses a feature extraction network to extract features from an object video, which can effectively extract high-level features from low-resolution features and retain image details of high-resolution features, thereby improving the accuracy of feature extraction and providing high-quality detection samples for subsequent correlation detection. By performing object detection on the target feature block through the attention model, the feature dependency relationship between all target objects in the target feature block can be captured and the original features can be enhanced with global context information. At the same time, the trajectory of the target object can be further determined by the introduced position encoding vector, thereby ensuring the accuracy of subsequent correlation analysis. By performing correlation analysis on the node network graph through the correlation analysis network, the detailed features of the object in the object video can be effectively retained, the accuracy of object selection and category determination can be improved, and the accuracy of tracking detection can be improved. Therefore, the detection and tracking method of video object correlation proposed by the present invention can solve the problem of low accuracy during tracking detection.

[0114] like Figure 4 , which is a functional module diagram of a device for detecting and tracking the association of video objects provided by one embodiment of the present invention.

[0115] The video object correlation detection and tracking device 100 described in the present invention can be installed in a device. Depending on the functionality implemented, the video object correlation detection and tracking device 100 may include a feature extraction module 101, an object detection module 102, a node network module 103, an association analysis module 104, and a tracking detection module 105. A module, also referred to as a unit, is a series of computer program segments that can be executed by a device processor and perform a fixed function, and is stored in the device's memory.

[0116] In this embodiment, the functions of each module / unit are as follows:

[0117] The feature extraction module 101 is used to obtain an object video and extract features from the object video frame by frame using a preset feature extraction network to obtain a video frame feature sequence corresponding to the object video;

[0118] The object detection module 102 is configured to sequentially select two consecutive video frame features from the video frame feature sequence as target feature blocks, perform object detection on the target feature blocks using a preset attention model, and obtain an object detection frame corresponding to the target feature block;

[0119] The node network module 103 is configured to convert the object detection frames of the target feature block into a node network one by one using a preset multi-level node conversion method to obtain a node network graph corresponding to the target feature block;

[0120] The association analysis module 104 is used to perform association analysis on the node network diagram using a preset association analysis network;

[0121] The tracking detection module 105 is configured to determine whether the object in the object video is being tracked based on the result of the correlation analysis.

[0122] In detail, each module in the apparatus for detecting and tracking the association of video objects in the embodiment of the present invention adopts the same method as above. Figures 1 to 3 The method for detecting and tracking the association of video objects described in the foregoing is the same technical means and can produce the same technical effects, so it will not be repeated here.

[0123] like Figure 5 FIG. 1 is a schematic diagram of the structure of a device for implementing a method for detecting and tracking video object association provided by an embodiment of the present invention.

[0124] The device 1 may include a processor 10 , a memory 11 , a communication bus 12 , and a communication interface 13 , and may also include a computer program stored in the memory 11 and executable on the processor 10 , such as a video object correlation detection and tracking program.

[0125] In some embodiments, the processor 10 may be comprised of an integrated circuit, such as a single packaged integrated circuit or a plurality of packaged integrated circuits with the same or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control core (control unit) of the device, connecting various components of the device using various interfaces and circuits. It executes programs or modules stored in the memory 11 (e.g., a program for detecting and tracking video object associations) and accesses data stored in the memory 11 to perform various device functions and process data.

[0126] The memory 11 includes at least one type of readable storage medium, including flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the device, such as a mobile hard disk of the device. In other embodiments, the memory 11 may also be an external storage device of the device, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. Furthermore, the memory 11 may include both an internal storage unit of the device and an external storage device. The memory 11 can be used not only to store application software installed on the device and various types of data, such as the code of a video object correlation detection and tracking program, but also to temporarily store data that has been output or is about to be output.

[0127] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0128] The communication interface 13 is used for communication between the above-mentioned device and other devices, and includes a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the device and other devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the device and to display a visual user interface.

[0129] The figure only shows a device with components. Those skilled in the art will understand that the structure shown in the figure does not constitute a limitation on the device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0130] For example, although not shown, the device may further include a power source (e.g., a battery) to power various components. Preferably, the power source may be logically connected to the at least one processor 10 via a power management device, thereby enabling functions such as charge management, discharge management, and power consumption management via the power management device. The power source may further include any of one or more DC or AC power sources, a recharging device, a power failure detection circuit, a power converter or inverter, a power status indicator, and the like. The device may also include various sensors, Bluetooth modules, Wi-Fi modules, and the like, which are not further detailed here.

[0131] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0132] The video object association detection and tracking program stored in the memory 11 of the device 1 is a combination of multiple instructions. When executed in the processor 10, it can achieve the following:

[0133] Obtain an object video, and extract features from the object video frame by frame using a preset feature extraction network to obtain a video frame feature sequence corresponding to the object video;

[0134] Selecting two consecutive video frame features from the video frame feature sequence as target feature blocks, performing object detection on the target feature blocks using a preset attention model, and obtaining an object detection frame corresponding to the target feature block;

[0135] The object detection frames of the target feature blocks are converted into node networks one by one using a preset multi-level node conversion method to obtain a node network graph corresponding to the target feature blocks;

[0136] A preset correlation analysis network is used to perform correlation analysis on the node network graph, and whether the object in the object video is tracked is determined based on the result of the correlation analysis.

[0137] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, which will not be repeated here.

[0138] Furthermore, if the modules / units integrated into the device 1 are implemented as software functional units and sold or used as independent products, they may be stored in a storage medium. The storage medium may be volatile or non-volatile. For example, the storage medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0139] The present invention further provides a storage medium, wherein the readable storage medium stores a computer program, and when the computer program is executed by a processor of a device, the computer program can implement:

[0140] Obtain an object video, and extract features from the object video frame by frame using a preset feature extraction network to obtain a video frame feature sequence corresponding to the object video;

[0141] Selecting two consecutive video frame features from the video frame feature sequence as target feature blocks, performing object detection on the target feature blocks using a preset attention model, and obtaining an object detection frame corresponding to the target feature block;

[0142] The object detection frames of the target feature blocks are converted into node networks one by one using a preset multi-level node conversion method to obtain a node network graph corresponding to the target feature blocks;

[0143] A preset correlation analysis network is used to perform correlation analysis on the node network graph, and whether the object in the object video is tracked is determined based on the result of the correlation analysis.

[0144] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical function division, and other division methods may be used in actual implementation.

[0145] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0146] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0147] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0148] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. Any reference to a figure in a claim should not be construed as limiting the claim to which it relates.

[0149] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0150] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. Terms such as "first" and "second" are used to indicate names and do not imply any particular order.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for detecting and tracking the association of video objects, characterized in that: The method comprises: Obtain an object video, and extract features from the object video frame by frame using a preset feature extraction network to obtain a video frame feature sequence corresponding to the object video; Selecting two consecutive video frame features from the video frame feature sequence as target feature blocks, performing object detection on the target feature blocks using a preset attention model, and obtaining an object detection frame corresponding to the target feature block; The object detection frames of the target feature blocks are converted into node networks one by one using a preset multi-level node conversion method to obtain a node network graph corresponding to the target feature blocks; Performing a correlation analysis on the node network graph using a preset correlation analysis network, and determining whether the object in the object video is tracked based on a result of the correlation analysis; The method of converting the object detection frames of the target feature blocks into node networks one by one using a preset multi-level node conversion method to obtain a node network graph corresponding to the target feature blocks includes: Taking the object bounding box in the object detection box as the vertex of the node network; Multiplying the object category corresponding to the object bounding box by the object confidence corresponding to the object bounding box to obtain a vertex value corresponding to the vertex; The vertices are connected in series using straight lines to obtain vertex edges of the node network, and the center vectors of the object bounding boxes corresponding to the two ends of the vertex edges are converted into weights of the vertex edges to complete the transformation of the node network graph.

2. The method for detecting and tracking the association of video objects according to claim 1, wherein: The step of extracting features from the object video frame by frame using a preset feature extraction network to obtain a video frame feature sequence corresponding to the object video includes: Using a preset video conversion tool to convert the object video into a video frame atlas; Selecting video frame images from the video frame atlas one by one as target images, and extracting features from the target images using a bottom-up convolutional network structure to obtain a feature pyramid consisting of multiple features with gradually increasing feature levels; Selecting features in the feature pyramid from top to bottom as target layer features, using an upsampling algorithm to enlarge the target layer features to the same size as the lower layer features, and using a lateral connection feature fusion method to fuse the lower layer features with the upsampled target layer features to obtain feature layer features corresponding to the target layer features; All feature layer features of the target image are aggregated into video frame features, and all video frame features of the video frame atlas are aggregated into a video frame feature sequence.

3. The method for detecting and tracking the association of video objects according to claim 1, wherein: The performing object detection on the target feature block by using a preset attention model to obtain an object detection frame corresponding to the target feature block includes: Encoding the target feature block into a feature code using an encoder of the attention model; Decoding the entity to be queried from the feature code using a decoder of the attention model; The entity to be queried is mapped through a multi-layer perceptron of the attention model to obtain the object detection frame.

4. The method for detecting and tracking the association of video objects according to claim 1, wherein: The converting the center vectors of the object bounding boxes corresponding to the two ends of the vertex edge into the weights of the vertex edge includes: Selecting the object bounding box corresponding to one end of the vertex edge as the first bounding box, and the object bounding box corresponding to the other end of the vertex edge as the second bounding box; Calculating a first center vector corresponding to the first bounding box and a second center vector corresponding to the second bounding box using a center vector algorithm; The second center vector is subtracted from the first center vector to obtain a standard vertex edge vector, and a modulo operation is performed on the standard vertex edge vector to obtain a weight of the vertex edge.

5. The method for detecting and tracking the association of video objects according to claim 1, wherein: The method of performing correlation analysis on the node network diagram one by one using a preset correlation analysis network includes: Encoding each vertex in the node network graph according to the vertex value and the weight of the vertex edge of the node network using the encoder of the association analysis network to obtain a vertex vector corresponding to each vertex; The vertex vector is decoded by a decoder of the association analysis network to obtain the association type of the vertex edge.

6. The method for detecting and tracking the association of video objects according to claim 5, wherein: The encoder using the association analysis network encodes each vertex in the node network graph according to the vertex value and the weight of the vertex edge of the node network to obtain a vertex vector corresponding to each vertex, including: The vertex values ​​of the node network are sampled one by one by an encoder of the association analysis network; The weight of the vertex edge of the vertex is obtained by sampling the adjacent vertices of the vertex, and the timing information of the vertex is obtained by encoding the current timestamp; The vertex vector is obtained by performing aggregate coding on the timing information, the vertex value, and the weight.

7. A device for detecting and tracking the association of video objects, for implementing the method for detecting and tracking the association of video objects according to any one of claims 1 to 6, characterized in that: The device comprises: A feature extraction module is used to obtain an object video and extract features from the object video frame by frame using a preset feature extraction network to obtain a video frame feature sequence corresponding to the object video; An object detection module is configured to sequentially select two consecutive video frame features from the video frame feature sequence as target feature blocks, perform object detection on the target feature blocks using a preset attention model, and obtain an object detection frame corresponding to the target feature block; A node network module is used to convert the object detection frames of the target feature block into a node network one by one using a preset multi-level node conversion method to obtain a node network graph corresponding to the target feature block; A correlation analysis module, configured to perform correlation analysis on the node network graph using a preset correlation analysis network; A tracking detection module is used to determine whether the object in the object video is tracked according to the result of the correlation analysis.

8. A device, characterized in that The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the method for detecting and tracking the association of video objects according to any one of claims 1 to 6.

9. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for detecting and tracking the association of video objects according to any one of claims 1 to 6 is implemented.

Images