Video relationship prediction method and device based on dual attention, equipment and medium

Abstract

The present application belongs to the technical field of video processing, and particularly relates to a video relationship prediction method and device based on dual attention, equipment and a medium, wherein the method comprises: obtaining a video to be predicted; preprocessing the video to obtain a preprocessed video; predicting a target object in the preprocessed video based on a dual attention mechanism, and outputting an action predicate prediction result and a position predicate prediction result of the target object; combining the action predicate prediction result and the position predicate prediction result, and optimizing the combined result according to a preset weight adjustment parameter to obtain an optimized prediction result. The present application can accurately predict the action predicate and the position predicate of the target object in the video by analyzing and predicting the video, and can obtain a prediction result with high confidence and in accordance with a grammatical expression form by combining and optimizing the action predicate and the position predicate, thereby optimizing the action and position relationship prediction technology of the target object in the video.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a video relationship prediction method, apparatus, device, and medium based on dual attention. Background Technology

[0002] Visual relationship detection in static images has been extensively studied, but it is far more complex and varied in video. The initial challenge of visual relationship detection was modeling the vast label space of relationships. Some studies have attempted to improve efficiency by pruning unlikely relationship candidates or refining the training data. Current techniques for visual relationship detection in video follow a three-stage framework: detecting object trajectories, predicting object relationships, and greedily associating relationships. However, current research focuses solely on extracting features from target objects or identifying their trajectories, neglecting the impact of predictions of relationship components (such as predicates) on the final relationship prediction results.

[0003] Therefore, how to accurately predict the action predicates and position predicates of target objects in videos, and predict the relationship between the actions and positions of target objects based on the action predicates and position predicates, so as to obtain relationship prediction results with high confidence and conforming to grammatical expression forms, is a problem that needs to be solved. Summary of the Invention

[0004] This invention provides a video relationship prediction method, apparatus, device, and medium based on dual attention, which aims to accurately predict the action predicates and position predicates of target objects in a video, and predict the relationship between the actions and positions of the target objects based on the action predicates and position predicates, so as to obtain relationship prediction results with high confidence and conforming to grammatical expression, thereby optimizing the relationship prediction technology.

[0005] To achieve the aforementioned objective, a first aspect of this invention proposes a video relationship prediction method based on dual attention, the method comprising:

[0006] Obtain the video to be predicted;

[0007] The video is preprocessed to obtain a preprocessed video;

[0008] The target object in the preprocessed video is predicted based on the dual attention mechanism, and the action predicate prediction result and the position predicate prediction result of the target object are output.

[0009] The action predicate prediction result and the position predicate prediction result are combined to obtain the initial prediction result;

[0010] The initial prediction result is optimized based on preset weight adjustment parameters to obtain an optimized prediction result.

[0011] Further, the preprocessing of the video to obtain a preprocessed video includes:

[0012] The video is deinterlaced to obtain the first video;

[0013] The first video is resampled to obtain the second video;

[0014] The second video is subjected to noise reduction processing to obtain a preprocessed video.

[0015] Furthermore, the dual attention mechanism includes an encoding component and a decoding component. The prediction of the target object in the preprocessed video based on the dual attention mechanism, and the output of the action predicate prediction result and the position predicate prediction result of the target object, includes:

[0016] The preprocessed video is decomposed into a set of overlapping segments, and trajectory suggestions for the target object are generated on each segment;

[0017] Feature extraction is performed on the trajectory suggestion to obtain feature data of the target object, wherein the feature data includes trajectory features, motion features, and visual features;

[0018] Relationship features are obtained by modeling the relationship based on the trajectory features, the motion features, and the visual features.

[0019] The trajectory features, motion features, and visual features are merged using an association rule algorithm combined with the relation features, and the action predicate prediction result and position predicate prediction result of the target object are output.

[0020] Further, the step of combining the action predicate prediction result and the position predicate prediction result to obtain the initial prediction result includes:

[0021] The action predicate prediction result and the position predicate prediction result are combined based on string concatenation to obtain the initial prediction result.

[0022] Further, optimizing the initial prediction result according to preset weight adjustment parameters to obtain an optimized prediction result includes:

[0023] Read the preset weight adjustment parameters, wherein the weight adjustment parameters include action predicate adjustment parameters and position predicate adjustment parameters;

[0024] The initial prediction results are weighted according to the action predicate adjustment parameters and the position predicate adjustment parameters to obtain the calculation result with the highest weighted score;

[0025] Based on the calculation results, an optimized prediction result is output.

[0026] This application also provides a video relationship prediction based on dual attention, characterized in that the apparatus comprises:

[0027] The acquisition module is used to acquire the video to be predicted;

[0028] The processing module is used to preprocess the video to obtain a preprocessed video;

[0029] The prediction module is used to predict target objects in the preprocessed video based on the dual attention mechanism, and output the action predicate prediction results and position predicate prediction results of the target objects.

[0030] The combination module is used to combine the action predicate prediction result and the position predicate prediction result to obtain an initial prediction result;

[0031] The optimization module is used to optimize the initial prediction result according to preset weight adjustment parameters to obtain an optimized prediction result.

[0032] Furthermore, the prediction module includes:

[0033] A decomposition unit is used to decompose the preprocessed video into a set of overlapping segments and generate trajectory suggestions for target objects on each segment;

[0034] An extraction unit is used to extract features from the trajectory suggestion to obtain feature data of the target object, wherein the feature data includes trajectory features, motion features, and visual features;

[0035] A modeling unit is used to perform relationship modeling based on the trajectory features, the motion features, and the visual features to obtain relationship features;

[0036] The association unit is used to combine the trajectory features, motion features, and visual features using an association rule algorithm and the relationship features, and output the action predicate prediction result and the position predicate prediction result of the target object.

[0037] Furthermore, the optimization module includes:

[0038] A reading unit is used to read preset weight adjustment parameters, wherein the weight adjustment parameters include action predicate adjustment parameters and position predicate adjustment parameters;

[0039] The calculation unit is used to perform weighted calculation on the initial prediction result based on the action predicate adjustment parameter and the position predicate adjustment parameter, and obtain the calculation result with the highest weighted score;

[0040] The output unit is used to output optimized prediction results based on the calculation results.

[0041] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, characterized in that the processor executes the computer program to implement the steps of the video relationship prediction method based on dual attention as described above.

[0042] This application also provides a computer-readable storage medium storing a computer program thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the video relationship prediction method based on dual attention described above.

[0043] Beneficial effects: The system acquires a video to be predicted, preprocesses the video to enhance its image and remove interference, resulting in a preprocessed video. Based on a dual attention mechanism, it predicts the target object in the preprocessed video to accurately predict the action predicate and position predicate of the target object, outputting the predicted action predicate and position predicate results. The predicted action predicate and position predicate results are combined to obtain a semantically coherent initial prediction result. The initial prediction result is optimized according to preset weight adjustment parameters, ensuring that the combination of action predicate and position predicate in the initial prediction result conforms to grammatical expression, thereby obtaining a high-confidence relation prediction result that conforms to grammatical expression, thus optimizing the relation prediction technology. Attached Figure Description

[0044] Figure 1 This is a schematic flowchart of an embodiment of the video relationship prediction method based on dual attention in this application;

[0045] Figure 2 This is a schematic flowchart of another embodiment of the video relationship prediction method based on dual attention in this application;

[0046] Figure 3 This is a schematic flowchart of another embodiment of the video relationship prediction method based on dual attention in this application;

[0047] Figure 4 This is a schematic flowchart of another embodiment of the video relationship prediction method based on dual attention in this application;

[0048] Figure 5 This is a schematic diagram of an embodiment of the video relationship prediction device based on dual attention according to this application;

[0049] Figure 6 This is a schematic block diagram of an embodiment of the computer device of this application.

[0050] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any modules and all combinations of one or more associated listed items.

[0053] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0054] Reference Figure 1 This invention provides a video relationship prediction method based on dual attention, comprising the following steps S1-S5:

[0055] S1: Obtain the video to be predicted.

[0056] The video to be predicted can be captured via a camera or obtained from a public website.

[0057] S2: Preprocess the video to obtain a preprocessed video.

[0058] After acquiring the video to be predicted, the video is preprocessed to obtain a preprocessed video. Specifically, the video is deinterlaced to eliminate or greatly reduce the drawbacks of interlaced scanning, resulting in a first video; the first video is resampled to recover the image in the video, resulting in a second video; the second video is then denoised to reduce noise in the video image, resulting in the preprocessed video. By preprocessing the video, the image is enhanced and interference information is removed, providing an effective guarantee for subsequent video prediction and obtaining high-confidence relationship prediction results.

[0059] S3: Based on the dual attention mechanism, predict the target object in the preprocessed video and output the action predicate prediction result and the position predicate prediction result of the target object.

[0060] The video is preprocessed to obtain a preprocessed video. Then, a dual attention mechanism is used to predict the target object in the preprocessed video, outputting the action predicate prediction result and the location predicate prediction result of the target object. Specifically, a target object is selected, the preprocessed video is decomposed into a set of overlapping segments, and a trajectory suggestion for the target object is generated on each segment. Feature extraction is performed on the trajectory suggestions to obtain feature data of the target object, wherein the feature data includes trajectory features, motion features, and visual features. Relationship modeling is performed based on the trajectory features, motion features, and visual features to obtain relationship features. An association rule algorithm is used to combine the relationship features with the trajectory features, motion features, and visual features to obtain the action predicate prediction result and the location predicate prediction result of the target object. Obtaining the action predicate prediction result and the location predicate prediction result provides a valid basis for subsequently obtaining the relationship prediction result.

[0061] S4: Combine the action predicate prediction result and the position predicate prediction result to obtain the initial prediction result.

[0062] Based on the dual attention mechanism, the target object in the preprocessed video is predicted. After the action predicate prediction result and the position predicate prediction result of the target object are output, the action predicate prediction result and the position predicate prediction result are combined based on string concatenation to obtain a character-coherent initial prediction result. The initial prediction result expresses the relationship between the action predicate and the position predicate, thus completing the initial prediction of the relationship between the action and the position in the video.

[0063] S5: Optimize the initial prediction result according to the preset weight adjustment parameters to obtain the optimized prediction result.

[0064] After combining the action predicate prediction results and the location predicate prediction results to obtain an initial prediction result, the initial prediction result is optimized according to preset weight adjustment parameters to obtain an optimized prediction result. Specifically, preset weight adjustment parameters are read, including action predicate adjustment parameters and location predicate adjustment parameters; the initial prediction result is weighted according to the action predicate adjustment parameters and the location predicate adjustment parameters, and the calculation result with the highest weighted score is obtained, which is the prediction result of the most likely correct predicate; the optimized prediction result is output according to the calculation result. In the prediction result, the expression of its action predicate and location predicate is close to the conventional expression form (correct grammatical expression) and matches the action and position of the target object in the video, which is more conducive to user interpretation and thus improves user experience.

[0065] This embodiment provides a video relation prediction method based on dual attention. The method involves acquiring a video to be predicted, preprocessing the video to enhance its image and remove interference, resulting in a preprocessed video; predicting target objects in the preprocessed video using a dual attention mechanism to accurately predict their action and position predicates, and outputting the predicted action and position predicates; combining these results to obtain a semantically coherent initial prediction; and optimizing the initial prediction based on preset weight adjustment parameters to ensure that the combination of action and position predicates in the initial prediction conforms to grammatical expression, thereby obtaining a high-confidence relation prediction result that conforms to grammatical expression, thus optimizing the relation prediction technique.

[0066] Reference Figure 2 In one embodiment, the above-described preprocessing of the video to obtain a preprocessed video includes:

[0067] S21: Deinterlacing is performed on the video to obtain the first video.

[0068] Deinterlacing converts interlaced images into progressive scan images, eliminating or significantly reducing the drawbacks of interlacing. The deinterlacing process includes: detecting moving objects by comparing the captured image with a static background and obtaining preliminary motion vectors; selecting a block containing the moving object in a field as the target block through block-based image matching, then determining the matching block based on candidate blocks to obtain accurate motion vectors; and finally, using the motion vectors to obtain a complete frame image from the two fields.

[0069] S21: Perform image resampling processing on the first video to obtain the second video.

[0070] Image resampling involves resampling a digital image composed of discrete data after sampling, according to the desired pixel positions or pixel spacing, to construct a new image after geometric transformation. The image resampling process includes: reconstructing a two-dimensional continuous function representing the original image from the input discrete digital image; and then sampling according to the new pixel spacing and pixel positions to recover the image.

[0071] S21: Perform noise reduction processing on the second video to obtain a preprocessed video.

[0072] Noise reduction refers to the process of reducing noise in digital images.

[0073] As described above, video preprocessing includes important operations such as deinterlacing, image resampling, and noise reduction. Specifically, moving objects are detected by comparing the captured image with a static background, and preliminary motion vectors are obtained. Through block-based image matching, a block containing the moving object in a field is selected as the target block. Then, matching blocks are determined based on candidate blocks, thereby obtaining accurate motion vectors. The motion vectors are used to obtain a complete first video from the two fields. A two-dimensional continuous function representing the original image is reconstructed using the input discrete digital image, and then sampled according to the new pixel spacing and pixel position to recover the image, obtaining a second video. Noise reduction processing is performed on the second video to obtain a preprocessed video. By preprocessing the video to enhance the image and remove interference information, a high-confidence prediction result is effectively guaranteed for subsequent video prediction.

[0074] Reference Figure 3 In one embodiment, the dual attention mechanism includes an encoding component and a decoding component. The step of predicting the target object in the preprocessed video based on the dual attention mechanism and outputting the action predicate prediction result and the position predicate prediction result of the target object includes:

[0075] S31: Decompose the preprocessed video into a set of overlapping segments, and generate trajectory suggestions for the target object on each segment.

[0076] The preprocessed video is decomposed into a set of overlapping segments, for example, into segments of length L with L / 2 overlapping frames (e.g., L = 30), and trajectory suggestions for the target object are generated on each segment. This approach can reduce drift problems in target tracking algorithms caused by changes in lighting and occlusion. Furthermore, the trajectory suggestions for individual target objects in each segment can generate a more diverse set of candidates.

[0077] S32: Perform feature extraction on the trajectory suggestion to obtain feature data of the target object, wherein the feature data includes trajectory features, motion features, and visual features.

[0078] Improved dense trajectory features of target objects are extracted segmentally using HoG, HoF, and MBH to capture motion features and low-level visual features. HoG (Histogram of Oriented Gradient) is a feature descriptor used in computer vision and image processing for object detection; it constructs features by calculating and statistically analyzing the gradient orientation histograms of local image regions. HoF (Histograms of Oriented Optical Flow) is a weighted statistical method for optical flow directions, obtaining an optical flow direction information histogram; it is commonly used in action recognition. MBH (Motion Boundary Histograms) treats the optical flow images in the x and y directions as two grayscale images and then extracts the gradient histograms of these grayscale images.

[0079] S33: Based on the trajectory features, motion features, and visual features, perform relationship modeling to obtain relationship features.

[0080] A relational model is constructed based on the scores of the predictors of the subject, predicate, and object. The relational triples (subject, predicate, and object) are predicted based on the relational model, and then the relational features are obtained.

[0081] S34: The trajectory features, motion features, and visual features are merged using an association rule algorithm combined with the relation features, and the action predicate prediction result and position predicate prediction result of the target object are output.

[0082] Association rule algorithms are rule-based machine learning algorithms that can discover relationships of interest in large databases; their purpose is to use certain metrics to identify strong rules in the database.

[0083] As described above, the dual attention mechanism includes an encoding component and a decoding component. The encoding component is a component that encodes and converts signals (such as bitstreams) or data into a signal form that can be used for communication, transmission, and storage. The decoding component refers to a component that converts the input analog video signal into a digital signal format for further compression and transmission. The preprocessed video is decomposed into a set of overlapping segments, for example, the video is decomposed into segments of length L with L / 2 overlapping frames (e.g., L=30), and trajectory suggestions for the target object are generated on each segment. Improved dense trajectory features of the target object are extracted using HoG, HoF, and MBH segmentation to capture motion features and low-level visual features. A relation model is constructed based on the scores of the comprehensive subject, predicate, and object prediction factors. Relation triples (subject, predicate, and object) are predicted based on the relation model to obtain relation features. The trajectory features, motion features, and visual features are merged using an association rule algorithm combined with the relation features to output the action predicate prediction results and position predicate prediction results of the target object. Obtaining the predicate prediction results and the position predicate prediction results provides a valid basis for obtaining subsequent prediction results.

[0084] In one embodiment, the combination of the action predicate prediction result and the position predicate prediction result to obtain an initial prediction result includes:

[0085] The action predicate prediction result and the position predicate prediction result are combined based on string concatenation to obtain the initial prediction result.

[0086] As mentioned above, string concatenation refers to combining discrete characters to regenerate a new string. The string concatenation techniques include using +, concat, StringBuffer, StringBuilder, and StringUtils.join. In this embodiment, any of these string concatenation methods can be used to combine the action predicate prediction result and the position predicate prediction result to obtain a coherent initial prediction result.

[0087] Reference Figure 4 In one embodiment, optimizing the initial prediction result according to a preset weight adjustment parameter to obtain an optimized prediction result includes:

[0088] S51: Read the preset weight adjustment parameters, wherein the weight adjustment parameters include action predicate adjustment parameters and position predicate adjustment parameters.

[0089] The weight adjustment parameters are empirical parameters obtained through pre-training. These empirical parameters include action predicate adjustment parameters and position predicate adjustment parameters. The action predicate adjustment parameters are used to adjust the action predicates in the prediction results so that the action predicates in the prediction results conform to conventional expressions (syntactic expressions) while also matching the actions of the target object in the video. The position predicate adjustment parameters are used to adjust the position predicates in the prediction results so that the position predicates in the prediction results conform to conventional expressions (syntactic expressions) while also matching the position of the target object in the video.

[0090] S52: The initial prediction result is weighted according to the action predicate adjustment parameter and the position predicate adjustment parameter to obtain the calculation result with the highest weighted score.

[0091] S53: Output optimized prediction results based on the calculation results.

[0092] As described above, prediction training is performed in advance based on video relationship prediction requirements to obtain empirical parameters for adjusting the prediction results, namely weight adjustment parameters. These weight adjustment parameters are not infinitely large. Although a larger weight adjustment parameter results in a higher confidence score for the prediction results, the dependency between the action predicate prediction results and the position predicate prediction results must also be considered. Therefore, when performing prediction training to obtain weight adjustment parameters based on video relationship prediction requirements, if the weight adjustment parameter is greater than a preset threshold (training empirical value), a weight adjustment parameter greater than the preset threshold can be used as a fixed adjustment parameter. This satisfies the training requirements and reduces the resource consumption of overtraining. The weight adjustment parameters are then used to adjust the action predicate prediction results and position predicate prediction results in the initial prediction results. The location predicate prediction results are weighted to increase the weight of the action predicate prediction results and the location predicate prediction results in the overall prediction result. This ensures that the action predicate in the prediction result conforms to conventional expression (grammatical expression) while also matching the action of the target object in the video; and that the location predicate in the prediction result conforms to conventional expression (grammatical expression) while also matching the location of the target object in the video. The result with the highest weighted score is then obtained, which is the prediction result of the most likely correct predicate. Based on this result, an optimized prediction result is output. Compared to the initial prediction result, this optimized prediction result has higher confidence and its predicate expression is closer to grammatical form, making it easier for users to understand and thus improving the user experience.

[0093] Reference Figure 5 This invention also provides a video relationship prediction device based on dual attention, comprising:

[0094] Module 10 is used to acquire the video to be predicted;

[0095] Processing module 20 is used to preprocess the video to obtain a preprocessed video;

[0096] Prediction module 30 is used to predict target objects in the preprocessed video based on dual attention mechanism, and output the action predicate prediction result and the position predicate prediction result of the target object;

[0097] The combination module 40 is used to combine the action predicate prediction result and the position predicate prediction result to obtain an initial prediction result;

[0098] The optimization module 50 is used to optimize the initial prediction result according to the preset weight adjustment parameters to obtain the optimized prediction result.

[0099] As described above, the video relationship prediction device based on dual attention can realize the video relationship prediction method based on dual attention.

[0100] In one embodiment, the acquisition module 10 further includes:

[0101] A deinterlacing unit is used to deinterlac the video to obtain a first video;

[0102] A resampling unit is used to perform image resampling processing on the first video to obtain a second video;

[0103] The noise reduction unit is used to perform noise reduction processing on the second video to obtain a pre-processed video.

[0104] In one embodiment, the processing module 20 further includes:

[0105] A decomposition unit is used to decompose the preprocessed video into a set of overlapping segments and generate trajectory suggestions for target objects on each segment;

[0106] An extraction unit is used to extract features from the trajectory suggestion to obtain feature data of the target object, wherein the feature data includes trajectory features, motion features, and visual features;

[0107] A modeling unit is used to perform relationship modeling based on the trajectory features, the motion features, and the visual features to obtain relationship features;

[0108] The association unit is used to combine the trajectory features, motion features, and visual features using an association rule algorithm and the relationship features, and output the action predicate prediction result and the position predicate prediction result of the target object.

[0109] In one embodiment, the positioning module 30 further includes:

[0110] The concatenation unit is used to combine the action predicate prediction result and the position predicate prediction result based on string concatenation to obtain the initial prediction result.

[0111] In one embodiment, the trimming module 40 further includes:

[0112] A reading unit is used to read preset weight adjustment parameters, wherein the weight adjustment parameters include action predicate adjustment parameters and position predicate adjustment parameters;

[0113] The calculation unit is used to perform weighted calculation on the initial prediction result based on the action predicate adjustment parameter and the position predicate adjustment parameter, and obtain the calculation result with the highest weighted score;

[0114] The output unit is used to output optimized prediction results based on the calculation results.

[0115] Reference Figure 6 The present invention also provides a computer device, the internal structure of which can be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor is designed to provide computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores operating devices, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores relevant data such as those related to video relationship prediction methods based on dual attention. The network interface is used for communication with external terminals via a network connection. Furthermore, the computer device may also include input devices and a display screen. When the aforementioned computer program is executed by a processor, it implements a video relationship prediction method based on dual attention, comprising the following steps: acquiring the video to be predicted; preprocessing the video to obtain a preprocessed video; predicting the target object in the preprocessed video based on a dual attention mechanism, and outputting the action predicate prediction result and the position predicate prediction result of the target object; combining the action predicate prediction result and the position predicate prediction result to obtain an initial prediction result; and optimizing the initial prediction result according to preset weight adjustment parameters to obtain an optimized prediction result. Those skilled in the art will understand that... Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment on which the present application is applied.

[0116] One embodiment of this application also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a video relationship prediction method based on dual attention, including the following steps: acquiring a video to be predicted; preprocessing the video to obtain a preprocessed video; predicting a target object in the preprocessed video based on a dual attention mechanism, and outputting action predicate prediction results and position predicate prediction results for the target object; combining the action predicate prediction results and the position predicate prediction results to obtain an initial prediction result; and optimizing the initial prediction result according to preset weight adjustment parameters to obtain an optimized prediction result. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0117] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media provided in this application and in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0118] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0119] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A video relationship prediction method based on dual attention, characterized in that, The method includes: Obtain the video to be predicted; The video is preprocessed to obtain a preprocessed video; The target object in the preprocessed video is predicted based on the dual attention mechanism, and the action predicate prediction result and the position predicate prediction result of the target object are output. The action predicate prediction result and the position predicate prediction result are combined to obtain the initial prediction result; The initial prediction result is optimized according to the preset weight adjustment parameters to obtain the optimized prediction result; The step of optimizing the initial prediction result according to preset weight adjustment parameters to obtain an optimized prediction result includes: Pre-training is performed based on video relationship prediction requirements to obtain empirical parameters for adjusting the prediction results, namely weight adjustment parameters. Preset weight adjustment parameters are read. If the weight adjustment parameters are greater than a preset threshold, the weight adjustment parameters greater than the preset threshold are used as fixed adjustment parameters. The weight adjustment parameters include action predicate adjustment parameters and position predicate adjustment parameters. The initial prediction result is weighted according to the action predicate adjustment parameters and the position predicate adjustment parameters to increase the weight of the action predicate prediction result and the position predicate prediction result in the prediction result. This makes the action predicate of the prediction result conform to the conventional expression and fit the action of the target object in the video; and makes the position predicate of the prediction result conform to the conventional expression and match the position of the target object in the video, thereby obtaining the calculation result with the highest weighted score. Based on the calculation results, an optimized prediction result is output.

2. The video relationship prediction method based on dual attention according to claim 1, characterized in that, The preprocessing of the video to obtain a preprocessed video includes: The video is deinterlaced to obtain the first video; The first video is resampled to obtain the second video; The second video is subjected to noise reduction processing to obtain a preprocessed video.

3. The video relationship prediction method based on dual attention according to claim 1, characterized in that, The dual attention mechanism includes an encoding component and a decoding component. The prediction of target objects in the preprocessed video based on the dual attention mechanism, and the output of action predicate prediction results and position predicate prediction results for the target objects, includes: The preprocessed video is decomposed into a set of overlapping segments, and trajectory suggestions for the target object are generated on each segment; Feature extraction is performed on the trajectory suggestion to obtain feature data of the target object, wherein the feature data includes trajectory features, motion features, and visual features; Relationship features are obtained by modeling the relationship based on the trajectory features, the motion features, and the visual features. The trajectory features, motion features, and visual features are merged using an association rule algorithm combined with the relation features, and the action predicate prediction result and position predicate prediction result of the target object are output.

4. The video relationship prediction method based on dual attention according to claim 1, characterized in that, The step of combining the action predicate prediction result and the position predicate prediction result to obtain the initial prediction result includes: The action predicate prediction result and the position predicate prediction result are combined based on string concatenation to obtain the initial prediction result.

5. A video relationship prediction device based on dual attention, used to implement the method according to any one of claims 1-4, characterized in that, The device includes: The acquisition module is used to acquire the video to be predicted; The processing module is used to preprocess the video to obtain a preprocessed video; The prediction module is used to predict target objects in the preprocessed video based on the dual attention mechanism, and output the action predicate prediction results and position predicate prediction results of the target objects. The combination module is used to combine the action predicate prediction result and the position predicate prediction result to obtain an initial prediction result; The optimization module is used to optimize the initial prediction result according to preset weight adjustment parameters to obtain an optimized prediction result.

6. The video relationship prediction device based on dual attention according to claim 5, characterized in that, The prediction module includes: A decomposition unit is used to decompose the preprocessed video into a set of overlapping segments and generate trajectory suggestions for target objects on each segment; An extraction unit is used to extract features from the trajectory suggestion to obtain feature data of the target object, wherein the feature data includes trajectory features, motion features, and visual features; A modeling unit is used to perform relationship modeling based on the trajectory features, the motion features, and the visual features to obtain relationship features; The association unit is used to combine the trajectory features, motion features, and visual features using an association rule algorithm and the relationship features, and output the action predicate prediction result and the position predicate prediction result of the target object.

7. The video relationship prediction device based on dual attention according to claim 5, characterized in that, The optimization module includes: A reading unit is used to read preset weight adjustment parameters, wherein the weight adjustment parameters include action predicate adjustment parameters and position predicate adjustment parameters; The calculation unit is used to perform weighted calculation on the initial prediction result based on the action predicate adjustment parameter and the position predicate adjustment parameter, and obtain the calculation result with the highest weighted score; The output unit is used to output optimized prediction results based on the calculation results.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the video relationship prediction method based on dual attention as described in any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the video relationship prediction method based on dual attention as described in any one of claims 1 to 4.

Images