An efficient video scene graph generation method and device based on a tracking algorithm

Abstract

The application discloses a high-efficiency video scene graph generation method and device based on a tracking algorithm, adopts an object element tracking method to acquire element information in each frame of picture, contacts time context information again to fit a scene relationship between object elements in each frame of picture, and generates a corresponding scene graph according to the scene relationship. The application avoids the method of identifying object elements for each frame of picture in a traditional element information acquisition scheme, reduces the detection times of object elements, and greatly improves the scene graph generation efficiency. In addition, the application only tracks object elements with the highest element prediction confidence, and discards the interference of the prediction results of low-confidence objects on the element information acquisition accuracy.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to an efficient method and apparatus for generating video scene graphs based on a tracking algorithm. Background Technology

[0002] A scene graph is a structured data structure used to describe a visual scene, reflecting information such as spatial structure, object positions, and object interactions. The purpose of a video scene graph is to extract and understand the content of each frame in a video, enabling deeper analysis and processing of the video data. Although each frame of a video is a single image, generating a scene graph allows for the extraction of key information from each frame, forming an abstract representation of the entire video content.

[0003] Current video scene graph generation strategies involve performing object detection on each frame of the video and then using the generated object element information to create the scene graph. However, in the most advanced video scene graph generation workflows, the object detection process accounts for 70.3% of the time, significantly slowing down the generation efficiency of the video scene graph. Summary of the Invention

[0004] The purpose of this application is to provide an efficient video scene graph generation method and apparatus based on a tracking algorithm, which can improve the above-mentioned problems.

[0005] The embodiments of this application are implemented as follows: In a first aspect, this application provides an efficient video scene graph generation method based on a tracking algorithm, which includes the following steps: S1: Track the element information of each object element in each frame of the video file; S2: Based on the element information, and in conjunction with the temporal context information, fit the scene relationship between the object elements in each frame; S3: Based on the scene relationships, generate a scene diagram corresponding to each frame in the video file. S1, S2, etc. are merely step identifiers. The execution order of the method does not necessarily follow the numerical order from smallest to largest. For example, step S2 can be executed first and then step S1. This application does not impose any restrictions.

[0006] This application discloses an efficient video scene graph generation method based on a tracking algorithm. It employs object element tracking to acquire element information in each frame, then combines this with temporal context information to fit the scene relationships between object elements in each frame, and generates the corresponding scene graph based on these relationships. This application avoids the traditional method of identifying object elements in each frame when acquiring element information, reducing the number of object element detections and significantly improving the efficiency of scene graph generation. Furthermore, this application only tracks object elements with the highest prediction confidence, discarding the interference of low-confidence object predictions on the accuracy of the generated video scene graph.

[0007] In an optional embodiment of this application, step S1 includes the following steps: S11: After the previous round of tracking is completed, the object elements of the current first frame in the video file are identified by the object detection network to obtain the initial element information of each object element in this round of tracking.

[0008] The aforementioned object detection network is the Faster Region Convolutional Neural Network (Faster R-CNN), an end-to-end deep learning detection algorithm. Its innovation lies in generating several candidate boxes through a region proposal network and an anchor mechanism, and integrating four modules—feature extraction, candidate box selection, bounding box regression, and classification—into a single network, thereby effectively improving detection accuracy and efficiency.

[0009] The initial element information mentioned above includes: initial element bounding box information, initial element predicted class label, and element prediction confidence.

[0010] S12: Track the object elements in subsequent frames, record the changes in element information between adjacent frames, until the current round of tracking stops.

[0011] Optionally, S12 includes: S121: The first Frames The first feature map is obtained by inputting it into the first convolutional network.

[0012] S122: The first Frames The second feature map is obtained by inputting it into a second convolutional network with the same structure and weights as the first convolutional network.

[0013] S123: The first Frames of Heatmap formed by the center points of each object element The third feature map is obtained by inputting it into a third convolutional network with the same structure and weights as the second convolutional network.

[0014] S124: The first feature map, the second feature map, and the third feature map are added bit-by-bit and then input into the Pose ResNet network to obtain the third feature map. Frames and the first Frames The element information change values ​​include the displacement information of each object element, the bounding box size of each element, and the tracking confidence.

[0015] S125: Determine whether to continue this round of tracking based on the tracking confidence level in the element information change value.

[0016] The loss function for the displacement information in the Pose ResNet network is as follows: ; in, The loss function representing the displacement information. Indicates the first Frames The number of object elements in the text. Indicates the first The object element from the first Frames up to the first Frames Actual displacement information Indicates the first The object element from the first Frames up to the first Frames Predicted displacement information; The loss function for the element bounding box size in the Pose ResNet network is as follows: ; in, The loss function represents the size of the bounding box of the element. Indicates the first The first frame in the picture The actual element bounding box size of each object element. Indicates the first The object element from the first Frames up to the first Frames The predicted element bounding box size; The total loss function of the Pose ResNet network As shown in the following formula: .

[0017] S13: Summarize the initial element information and the element information change values ​​obtained in each round of tracking to obtain the element information of each object element in each frame.

[0018] Optionally, S13 includes: S131: Obtain the center position of the initial element bounding box based on the initial element bounding box information; S132: Move the center position of the initial element bounding box according to the corresponding displacement information to obtain the center position of the element bounding box of each object element in each frame. S133: Determine the element bounding box information of the corresponding object element in each frame based on the element bounding box size and the element bounding box center position; S134: Continue the initial element prediction class label as the element prediction class label of the object element in each frame; S135: Use the element bounding box information and the element prediction class label in each frame as the element information of the corresponding object element.

[0019] In an optional embodiment of this application, step S2 includes the following steps: S21: Encode and fuse single-frame images using a spatial encoder to obtain the element relationship between any two object elements in each frame, as a relationship set; S22: Based on the refined relationship set, the scene relationship between any two object elements in each frame is fitted by using a time decoder in conjunction with the time context information.

[0020] In an optional embodiment of this application, step S21 includes the following steps: S211: The single frame is encoded and fused using a spatial encoder according to the following formula to obtain the first frame. The element relationship between any two of the aforementioned object elements in a frame: ; in, Indicates the first The first frame in the picture The visual features of an object element obtained after passing through the first convolutional layer during the tracking of element information. Indicates the first The first frame in the picture The visual features of an object element obtained after passing through the first convolutional layer during the tracking of element information. Indicates the first The first frame in the picture The bounding box information of each object element. Indicates the first The first frame in the picture The bounding box information of each object element. Indicates the first The first frame in the picture The first object element and the first The feature map corresponding to the union region of the bounding boxes of each object element. Indicates the first The first frame in the picture Predicted class label of each object element. Indicates the first The first frame in the picture Predicted class label of each object element. Indicates a connection operation. This represents the tensor flattening operation. , , Represents a linear matrix that compresses dimensions. This indicates a bitwise addition operation. express and Convert to Feature maps of the same dimension; S212: Summarize the above... The relationships between each element in the frame are used to obtain a set of relationships. : ;in, Indicates the first of the above. The number of element relationships described in the frame; S213: Refine the element relations in the relation set using a multi-head attention mechanism according to the following formula, and update the relation set: ,in, , , These represent the query vector, key vector, and value vector output by the refinement layer of the multi-head attention mechanism, respectively. This indicates the output result of the multi-head attention layer in the multi-head attention mechanism corresponding to the spatial encoder.

[0021] It can be understood that the attention mechanism is essentially an allocation mechanism, its core idea being to highlight certain important features of an object. Based on the importance of the objects in the attention mechanism, resources, i.e., weights, are reallocated. The core idea is to find the correlations between existing data and then highlight certain important features. Attention can be described as mapping a query and a set of key-value pairs to an output, where the query, key, value, and output are all vectors. The output is calculated using a compatibility function in the form of a weighted sum, which calculates the weight assigned to each value.

[0022] In an optional embodiment of this application, step S22 includes the following steps: S221: Constructing a sliding window : ,in, Indicates the size of the sliding window; S222: The sliding window is used as the input to the time decoder, and the result is obtained through a multi-head attention mechanism. Scene relationships in frames : , It is the function representation of the multi-head attention layer in the multi-head attention mechanism corresponding to the time decoder. It is understandable that the scene graph of a video file needs to consider not only spatial context information, but also the temporal continuity of the relationships between objects. Therefore, the first... Scene graph generation of frames requires not only the bounding boxes and object element information of the object instances in the current frame, but also the information from previous frames (i.e., frames 1, 2, ...). The bounding box and object element information of the object instance are obtained from each frame. By constructing a sliding window and using it as input to the time decoder, the scene relationship between any two object elements in each frame can be fitted by combining the time context information.

[0023] In a second aspect, this application discloses an efficient video scene graph generation apparatus based on a tracking algorithm, comprising a processor, an input device, an output device, and a memory, wherein the processor, input device, output device, and memory are interconnected, wherein the memory is used to store a computer program, the computer program comprising program instructions, and the processor is configured to invoke the program instructions to execute the method as described in any of the first aspects.

[0024] Thirdly, this application discloses a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as described in any of the first aspects.

[0025] Beneficial effects: This application discloses an efficient video scene graph generation method based on a tracking algorithm. It uses an object element tracking method to obtain element information in each frame, then combines temporal context information to fit the scene relationship between object elements in each frame, and generates the corresponding scene graph based on the scene relationship.

[0026] This application avoids the traditional method of identifying object elements in each frame of the image, thus reducing the number of object element detections and significantly improving the efficiency of scene graph generation. Furthermore, this application only tracks object elements with the highest prediction confidence, discarding the interference of low-confidence object predictions on the accuracy of element information acquisition.

[0027] This application constructs a sliding window and uses it as the input to a time decoder. It needs to consider not only spatial context information but also the temporal continuity of the relationship between objects. By combining the temporal context information, it fits the scene relationship between any two object elements in each frame.

[0028] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, optional embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of the workflow of an object detection network provided in this application; Figure 2 This is a schematic diagram of initial element information provided in this application; Figure 3 This is a schematic diagram of the working process of a tracking algorithm provided in this application; Figure 4 This is a schematic diagram of the operation of a spatial encoder provided in this application; Figure 5 This is a schematic diagram of the operation of a time decoder provided in this application; Figure 6 This is a schematic diagram of the structure of an efficient video scene graph generation device based on a tracking algorithm provided in this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0032] In a first aspect, this application provides an efficient video scene graph generation method based on a tracking algorithm, which includes the following steps: S1: Tracks the element information of each object element in each frame of the video file.

[0033] In the video scene graph generation process, object detection accounts for 70.3% of the time. Therefore, the key to improving scene generation efficiency lies in reducing the number of object detections for object elements. Based on the first... Instead of performing object detection on every single frame, this algorithm uses object tracking to determine the object elements in subsequent frames based on the detected objects. This strategy of using a more efficient object tracking method instead of inefficient object detection can significantly improve the speed of scene graph generation.

[0034] The aforementioned element information includes element bounding box information and element prediction class labels in each frame, wherein the element bounding box information includes element bounding box size and element bounding box position.

[0035] S2: Based on the element information and the temporal context information, fit the scene relationship between the object elements in each frame.

[0036] This application constructs a sliding window and uses it as the input to a time decoder. It needs to consider not only spatial context information but also the temporal continuity of the relationship between objects. By combining the temporal context information, it fits the scene relationship between any two object elements in each frame.

[0037] S3: Generate a scene diagram corresponding to each frame in the video file based on scene relationships. S1, S2, etc. are merely step identifiers. The execution order of the method does not necessarily follow the numerical order from smallest to largest. For example, step S2 can be executed first and then step S1. This application does not impose any restrictions.

[0038] As understood, this application discloses an efficient video scene graph generation method based on a tracking algorithm. It employs an object element tracking method to acquire element information in each frame, then combines this with temporal context information to fit the scene relationships between object elements in each frame, and generates the corresponding scene graph based on these relationships. This application avoids the traditional method of identifying object elements in each frame when acquiring element information, reducing the number of object element detections and significantly improving the efficiency of scene graph generation. Furthermore, this application only tracks object elements with the highest prediction confidence, discarding the interference of low-confidence object prediction results on the accuracy of element information acquisition.

[0039] In an optional embodiment of this application, S1 includes the following steps: S11: After the previous round of tracking is completed, the object elements of the current first frame in the video file are identified by the object detection network to obtain the initial element information of each object element in this round of tracking.

[0040] The aforementioned object detection network is the Faster Region Convolutional Neural Network (Faster R-CNN), an end-to-end deep learning detection algorithm. Its innovation lies in generating several candidate boxes through a region proposal network and an anchor mechanism, and integrating four modules—feature extraction, candidate box selection, bounding box regression, and classification—into a single network, thereby effectively improving detection accuracy and efficiency.

[0041] like Figure 1 The object detection network shown first extracts the feature map of an input image using 13 convolutional layers and 4 pooling layers. This feature map is then fed into a Region Proposal Network (RPN) and a Region of Interest (ROI) pooling layer. The RPN generates candidate bounding boxes for object elements, outputting region proposals and region scores. Non-maximum suppression (NMS) is applied to the region scores. The output region proposals are then fed into the ROI pooling layer. The ROI pooling layer receives the image's feature map and the RPN's output region proposals for object elements, and extracts proposed feature maps for the object elements after combining the information. The proposed feature maps of different sizes are normalized before being fed into subsequent recognition. Finally, regression prediction of the object element bounding boxes is performed, and a softmax layer is used for object element classification.

[0042] The initial element information mentioned above includes: initial element bounding box information, initial element predicted class labels, and element prediction confidence scores. The initial element bounding box information includes the initial element bounding box size and the initial element bounding box position. For example... Figure 2 As shown, the input image includes object elements. , , , ;object element The corresponding element information includes the element prediction class label. and element prediction confidence ; object element The corresponding element information includes the element prediction class label. and element prediction confidence ; object element The corresponding element information includes the element prediction class label. and element prediction confidence ; object element The corresponding element information includes the element prediction class label. and element prediction confidence .

[0043] S12: Track object elements in subsequent frames, record the changes in element information between adjacent frames, until the current tracking cycle stops.

[0044] Optionally, such as Figure 3 As shown, S12 includes: S121: The first Frames The first feature map is obtained by inputting it into the first convolutional network.

[0045] S122: The first Frames The second feature map is obtained by inputting it into a second convolutional network with the same structure and weights as the first convolutional network.

[0046] S123: The first Frames of Heatmap formed by the center points of each object element The third feature map is obtained by inputting it into a third convolutional network with the same structure and weights as the second convolutional network.

[0047] Optionally, heatmap The rendering process can be represented by the following Gaussian function: ; in, Indicates the first Frames Lost The coordinates of the center point of each object element This indicates the size of the heatmap to be rendered. This represents the variance of the Gaussian kernel function.

[0048] S124: Add the first, second, and third feature maps position by position and then input them into the pose residual network (Pose ResNet) to obtain the... Frames and the Frames The element information change values ​​include the displacement information of each object element, the bounding box size of each element, and the tracking confidence. .

[0049] S125: Determine whether to continue this round of tracking based on the tracking confidence level in the element information change value.

[0050] The loss function for displacement information in the Pose ResNet network is as follows: ; in, The loss function representing displacement information, Indicates the first Frames The number of object elements in the text. Indicates the first The nth object element starts from the nth Frames To the Frames Actual displacement information Indicates the first The nth object element starts from the nth Frames To the Frames Predicted displacement information; The loss function for element bounding box size in the Pose ResNet network is as follows: ; in, The loss function represents the size of the element's bounding box. Indicates the first The first frame in the picture The actual element bounding box size of each object element. Indicates the first The nth object element starts from the nth Frames To the Frames The predicted element bounding box size; The total loss function of the Pose ResNet network As shown in the following formula: .

[0051] S13: Summarize the initial element information and element information change values ​​obtained from each round of tracking to obtain the element information of each object element in each frame.

[0052] Optionally, S13 includes: S131: Obtain the center position of the initial element bounding box based on the initial element bounding box information; S132: Move the center position of the initial element bounding box according to the corresponding displacement information to obtain the center position of the element bounding box of each object element in each frame. S133: Determine the element bounding box information of the corresponding object element in each frame based on the element bounding box size and the element bounding box center position; S134: Continue using the initial element prediction class label as the element prediction class label of the object element in each frame; S135: Use the bounding box information and predicted class label of the elements in each frame as the element information of the corresponding object element.

[0053] In an optional embodiment of this application, S2 includes the following steps: S21: Encode and fuse single-frame images using a spatial encoder to obtain the element relationships between any two object elements in each frame, which are then used as a set of relationships.

[0054] S22: Based on the refined relationship set, the scene relationship between any two object elements in each frame is fitted by connecting the time context information through the time decoder.

[0055] In optional embodiments of this application, such as Figure 4 As shown, S21 includes the following steps: S211: The single frame is encoded and fused using a spatial encoder according to the following formula to obtain the first frame. The element relationship between any two object elements in a frame: ; in, Indicates the first The first frame in the picture The visual features of an object element obtained after passing through the first convolutional layer during the tracking of element information. Indicates the first The first frame in the picture The visual features of an object element obtained after passing through the first convolutional layer during the tracking of element information. Indicates the first The first frame in the picture The bounding box information of each object element. Indicates the first The first frame in the picture The bounding box information of each object element. Indicates the first The first frame in the picture The first object element and the first The feature map corresponding to the union region of the bounding boxes of each object element. Indicates the first The first frame in the picture Predicted class label of each object element. Indicates the first The first frame in the picture Predicted class label of each object element. Indicates a connection operation. This represents the tensor flattening operation. , , Represents a linear matrix that compresses dimensions. This indicates a bitwise addition operation. express and Convert to Feature maps of the same dimension; S212: Organizing the... The relationships between each element in the frame are used to obtain a set of relationships. : ;in, Indicates the first The number of element relationships in a frame; S213: Refine the element relationships in the relationship set using a multi-head attention mechanism according to the following formula, and update the relationship set: ,in, , , These represent the query vector, key vector, and value vector output by the refinement layer of the multi-head attention mechanism, respectively. This represents the output of the multi-head attention layer in the multi-head attention mechanism corresponding to the spatial encoder.

[0056] It can be understood that the attention mechanism is essentially an allocation mechanism, its core idea being to highlight certain important features of an object. Based on the importance of the objects in the attention mechanism, resources, i.e., weights, are reallocated. The core idea is to find the correlations between existing data and then highlight certain important features. Attention can be described as mapping a query and a set of key-value pairs to an output, where the query, key, value, and output are all vectors. The output is calculated using a compatibility function in the form of a weighted sum, which calculates the weight assigned to each value.

[0057] In optional embodiments of this application, such as Figure 5 As shown, S22 includes the following steps: S221: Constructing a sliding window : ,in, Indicates the size of the sliding window; S222: Using the sliding window as input to the temporal decoder, the result is obtained through a multi-head attention mechanism. Scene relationships in frames : , This is the function representation of the multi-head attention layer in the multi-head attention mechanism corresponding to the temporal decoder. It is understandable that the scene graph of a video file needs to consider not only spatial context information, but also the temporal continuity of the relationships between objects. Therefore, the first... Scene graph generation of a frame requires not only the bounding boxes and object element information of the current frame, but also the information of the frames before it (i.e., frames 1, 2, ...). The bounding boxes and object element information of the objects in each frame are obtained. By constructing a sliding window and using it as input to the time decoder, the scene relationship between any two object elements in each frame can be fitted by combining the temporal context information.

[0058] Secondly, this application provides an efficient video scene graph generation device based on a tracking algorithm. For example... Figure 6 As shown, the efficient video scene graph generation apparatus based on a tracking algorithm includes one or more processors 601; one or more input devices 602; one or more output devices 603; and a memory 604. The processors 601, input devices 602, output devices 603, and memory 604 are connected via a bus 605. The memory 604 stores a computer program, which includes program instructions, and the processors 601 execute the program instructions stored in the memory 604. The processors 601 are configured to invoke the program instructions to perform the operation of any of the methods in the first aspect. It should be understood that, in this embodiment of the invention, the processor 601 may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0059] Input device 602 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 603 may include a display (LCD, etc.), a speaker, etc.

[0060] The memory 604 may include read-only memory and random access memory, and provides instructions and data to the processor 601. A portion of the memory 604 may also include non-volatile random access memory. For example, the memory 604 may also store device type information.

[0061] In specific implementations, the processor 601, input device 602, and output device 603 described in the embodiments of the present invention can execute the implementation methods described in any of the methods in the first aspect, or they can execute the implementation methods of the terminal devices described in the embodiments of the present invention, which will not be repeated here.

[0062] Thirdly, the present invention provides a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, implement the steps of any of the methods of the first aspect.

[0063] The aforementioned computer-readable storage medium can be an internal storage unit of the terminal device in any of the foregoing embodiments, such as a hard disk or memory of the terminal device. The aforementioned computer-readable storage medium can also be an external storage device of the terminal device, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the terminal device. Furthermore, the aforementioned computer-readable storage medium may include both internal storage units and external storage devices of the terminal device. The aforementioned computer-readable storage medium is used to store the aforementioned computer program and other programs and data required by the terminal device. The aforementioned computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0064] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0065] In the several embodiments provided in this application, it should be understood that the disclosed terminal devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices or units, or it may be an electrical, mechanical or other form of connection.

[0066] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0067] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0068] If the aforementioned integrated units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0069] The terms "first," "second," "first," or "second" as used in the various embodiments of this disclosure may modify various components regardless of their order and / or importance, but these terms do not limit the corresponding components. The above terms are configured only for the purpose of distinguishing an element from other elements. For example, "first user equipment" and "second user equipment" refer to different user equipments, although both are user equipment. For example, without departing from the scope of this disclosure, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.

[0070] When a component (e.g., a first component) is referred to as being "(operably or communicatively) coupled" or "(operably or communicatively) coupled to" or "connected to" another component (e.g., a second component), it should be understood that the first component is directly connected to the second component or that the first component is indirectly connected to the second component via yet another component (e.g., a third component). Conversely, it can be understood that when a component (e.g., a first component) is referred to as being "directly connected" or "directly coupled" to another component (the second component), no component (e.g., a third component) is inserted between the two.

[0071] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0072] The above description is merely an optional embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in this application.

[0073] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrases “if determination” or “if detection (of the stated condition or event)” can be interpreted as “when determination” or “in response to determination” or “when detection (of the stated condition or event)” or “in response to detection (of the stated condition or event).”

[0074] The above description is merely an optional embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in this application.

[0075] The above description is merely an optional embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for generating efficient video scene graphs based on a tracking algorithm, characterized in that, Includes the following steps: S1: Track the element information of each object element in each frame of the video file; S2: Based on the element information, and in conjunction with the temporal context information, fit the scene relationship between the object elements in each frame; S3: Generate a scene diagram corresponding to each frame in the video file based on the scene relationships; S1 includes the following steps: S11: After the previous round of tracking is completed, the object elements of the current first frame in the video file are identified by the object detection network to obtain the initial element information of each object element in this round of tracking; S12: Track the object elements in subsequent frames and record the changes in element information between adjacent frames until the current round of tracking stops; S13: Summarize the initial element information and the element information change values ​​obtained in each round of tracking to obtain the element information of each object element in each frame; S12 includes: S121: The first Frames The first feature map is obtained by inputting it into the first convolutional network; S122: The first Frames The second feature map is obtained by inputting it into a second convolutional network with the same structure and weights as the first convolutional network. S123: The first Frames of Heatmap formed by the center points of each object element The third feature map is obtained by inputting it into a third convolutional network with the same structure and weights as the second convolutional network. S124: The first feature map, the second feature map, and the third feature map are added bit-by-bit and then input into the PoseResNet network to obtain the first feature map. Frames and the first Frames The element information change value includes the displacement information of each object element, the bounding box size of each element, and the tracking confidence. S125: Based on the tracking confidence level in the change value of the element information, determine whether to continue this round of tracking; S13 includes: S131: Obtain the center position of the initial element bounding box based on the initial element bounding box information; S132: Move the center position of the initial element bounding box according to the corresponding displacement information to obtain the center position of the element bounding box of each object element in each frame. S133: Determine the element bounding box information of the corresponding object element in each frame based on the element bounding box size and the element bounding box center position; S134: Continue the initial element prediction class label as the element prediction class label of the object element in each frame; S135: Use the element bounding box information and the element predicted class label in each frame as the element information of the corresponding object element; The loss function for the displacement information in the Pose ResNet network is as follows: ； in, The loss function representing the displacement information. Indicates the first Frames The number of object elements in the text. Indicates the first The object element from the first Frames up to the first Frames Actual displacement information Indicates the first The object element from the first Frames up to the first Frames Predicted displacement information; The loss function for the element bounding box size in the Pose ResNet network is as follows: ； in, The loss function represents the size of the bounding box of the element. Indicates the first The first frame in the picture The actual element bounding box size of each object element. Indicates the first The object element from the first Frames up to the first Frames The predicted element bounding box size; The total loss function of the Pose ResNet network As shown in the following formula: .

2. The efficient video scene graph generation method based on tracking algorithm according to claim 1, characterized in that, The target detection network is the Faster R-CNN network; The initial element information includes: initial element bounding box information, initial element predicted class label, and element prediction confidence.

3. The efficient video scene graph generation method based on tracking algorithm according to any one of claims 1 to 2, characterized in that, S2 includes the following steps: S21: Encode and fuse single-frame images using a spatial encoder to obtain the element relationship between any two object elements in each frame, as a relationship set; S22: Based on the refined relationship set, the scene relationship between any two object elements in each frame is fitted by using a time decoder to connect with the time context information.

4. The efficient video scene graph generation method based on tracking algorithm according to claim 3, characterized in that, S21 includes the following steps: S211: The single frame is encoded and fused using a spatial encoder according to the following formula to obtain the first frame. The element relationship between any two of the aforementioned object elements in a frame: ； in, Indicates the first The first frame in the picture The visual features of an object element obtained after passing through the first convolutional layer during the tracking of element information. Indicates the first The first frame in the picture The visual features of an object element obtained after passing through the first convolutional layer during the tracking of element information. Indicates the first The first frame in the picture The bounding box information of each object element. Indicates the first The first frame in the picture The bounding box information of each object element. Indicates the first The first frame in the picture The first object element and the first The feature map corresponding to the union region of the bounding boxes of each object element. Indicates the first The first frame in the picture Predicted class label of each object element. Indicates the first The first frame in the picture Predicted class label of each object element. Indicates a connection operation. This represents the tensor flattening operation. , , Represents a linear matrix that compresses dimensions. This indicates a bitwise addition operation. express and Convert to Feature maps of the same dimension; S212: Summarize the above... The relationships between each element in the frame are used to obtain a set of relationships. : ;in, Indicates the first of the above. The number of element relationships described in the frame; S213: Update the relation set by refining the element relations in the relation set according to the following formula using a multi-head attention mechanism: ,in, , , These represent the query vector, key vector, and value vector output by the refinement layer of the multi-head attention mechanism, respectively. This represents the output result of the multi-head attention layer in the multi-head attention mechanism corresponding to the spatial encoder.

5. The efficient video scene graph generation method based on tracking algorithm according to claim 3, characterized in that, S22 includes the following steps: S221: Constructing a sliding window : ,in, Indicates the size of the sliding window; S222: The sliding window is used as the input to the time decoder, and the result is obtained through a multi-head attention mechanism. Scene relationships in frames : , This represents the output result of the multi-head attention layer in the multi-head attention mechanism corresponding to the time decoder.

6. A high-efficiency video scene graph generation device based on a tracking algorithm, characterized in that, The system includes a processor, an input device, an output device, and a memory, which are interconnected. The memory is used to store a computer program, which includes program instructions. The processor is configured to invoke the program instructions to perform the method as described in any one of claims 1 to 5.

Images