Object interaction analysis method and device based on visual features, equipment and medium

Abstract

The application relates to the technical field of artificial intelligence, can be applied to business scenes such as financial technology and medical health, and discloses an object interaction analysis method and device based on visual features, equipment and a medium, which comprises the following steps: obtaining multi-scale visual features of a prediction frame, detecting a significant object, generating an action description pair in combination with historical video frames, constructing an action context, extracting a language feature sequence and a visual feature sequence, splicing and mapping into a unified dimension, realizing interaction fusion by using a self-attention mechanism, reorganizing into a multi-scale fusion feature map, and outputting the spatial position distribution, the action category and the time contact feature of the interactive object. The application generates a textual action context by combining the action description information of the historical video frames with the significant object list, combines the multi-scale visual features and the language feature sequence, realizes unified mapping, cross-modal interaction and feature fusion, and improves the accuracy and stability of object interaction analysis in a variable environment.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for object interaction analysis based on visual features. Background Technology

[0002] In the field of object interaction analysis in self-centered videos, existing technologies generally rely on visual information extraction methods based on neural networks. Common approaches include 3D convolutional neural networks and feature modeling based on spatiotemporal attention mechanisms. These methods typically process fixed-length video frame sequences directly, extracting visual features for short-term object interaction prediction. However, in practical applications, existing technologies generally lack the ability to effectively express "action context," failing to accurately capture the impact of continuous actions and object interactions in historical video frames on the current prediction task. This makes it difficult for the system to fully understand the semantic logic of user operations. For example, when performing actions such as "cutting vegetables" in a kitchen environment, the user's historical actions and their relationship with key objects such as knives and cutting boards jointly determine the operational intent. Relying solely on a single visual frame sequence is insufficient to reconstruct the complete semantic information chain.

[0003] Meanwhile, visual information in complex environments is easily affected by interference and limitations. Existing methods rely heavily on visual input, leading to the loss of important information when the background is cluttered, objects are occluded, or lighting changes drastically. Especially in scenarios involving interactions with rare or long-tailed objects, traditional methods have weak generalization ability and are prone to missing key interaction processes, severely impacting the accuracy and stability of interaction prediction.

[0004] In the fintech sector, interactive behavior analysis based on video data is widely used in scenarios such as counter service processing, smart terminal operation monitoring, and remote identity verification. Existing systems mostly focus on extracting visual features from local video clips, lacking a systematic understanding of the combined information of historical operation steps and key objects in the physical environment. This results in insufficient ability to identify the user's true intentions when business processes are complex, item combinations are diverse, or abnormal operational behaviors occur, affecting intelligent risk control and business compliance levels.

[0005] In the healthcare field, video interaction analytics has been used for surgical monitoring, recording of medical procedures, and assessment of patient rehabilitation training. However, existing technologies mainly focus on extracting static visual information from video frames, failing to effectively integrate historical action trajectories with the interaction relationships between objects in the medical environment. This makes it difficult for the system to capture timely and accurate interactive behaviors closely related to medical safety when dealing with complex surgical procedures, medical device occlusion, or environmental changes, thus limiting the improvement of decision support and risk warning capabilities. Summary of the Invention

[0006] The main objective of this invention is to provide a method, apparatus, device, and storage medium for object interaction analysis based on visual features, aiming to solve the technical problems of existing technologies failing to effectively integrate visual features and action context information, lacking a unified expression and correlation analysis for object interaction prediction tasks, resulting in insufficient accuracy in interaction understanding and future prediction in complex scenarios.

[0007] To achieve the above objectives, the present invention provides a method for object interaction analysis based on visual features, comprising:

[0008] Acquire predicted frames from video data, extract multi-scale visual features from the predicted frames, and detect salient objects in the predicted frames to form a list of salient objects;

[0009] Initial motion description pairs are generated based on historical video frames, and the same initial motion description pairs in consecutive frames in the historical video frames are aggregated to generate an aggregated motion description pair sequence.

[0010] By combining the aggregated action description pair sequence and the list of salient objects, a textualized action context is generated;

[0011] The visual features at each scale in the multi-scale visual features are divided into image blocks and positional codes are added to generate a multi-scale visual feature sequence.

[0012] The textualized action context is encoded using a pre-trained semantic encoding model to generate a sequence of linguistic features;

[0013] For each scale, the visual feature sequence and the language feature sequence of that scale are spliced ​​together, and the splicing result is mapped to a unified dimension to form a hybrid input sequence for the corresponding scale;

[0014] The mixed input sequences at each scale are subjected to cross-modal interaction through a multi-layer self-attention mechanism to obtain a multi-scale fused feature sequence;

[0015] The multi-scale fusion feature sequence is recombined into a multi-scale fusion feature map;

[0016] Based on the multi-scale fused feature map, the analysis results of the spatial location distribution, action semantic category and temporal contact features of future interactive objects are generated.

[0017] Furthermore, to achieve the above objectives, the present invention provides an object interaction analysis device based on visual features, comprising:

[0018] The visual perception module is used to acquire predicted frames of video data, extract multi-scale visual features of the predicted frames, and detect salient objects in the predicted frames to form a list of salient objects.

[0019] The action semantic extraction module is used to generate initial action description pairs based on historical video frames, and to aggregate the same initial action description pairs in consecutive frames in the historical video frames to generate an aggregated action description pair sequence.

[0020] A context building module is used to generate a textualized action context by combining the aggregated action description pair sequence and the list of salient objects;

[0021] The visual encoding module is used to divide the visual features at each scale of the multi-scale visual features into image blocks and add positional encoding to generate a multi-scale visual feature sequence.

[0022] The language encoding module is used to encode the textualized action context using a pre-trained semantic encoding model to generate a sequence of language features;

[0023] The fusion and stitching module is used to stitch together the visual feature sequence and the language feature sequence of each scale, and map the stitching result to a unified dimension to form a hybrid input sequence of the corresponding scale.

[0024] The cross-modal interaction module is used to perform cross-modal interaction on the mixed input sequences at each scale through a multi-layer self-attention mechanism to obtain a multi-scale fused feature sequence;

[0025] The feature recombination module is used to recombine the multi-scale fused feature sequence into a multi-scale fused feature map.

[0026] The interaction analysis module is used to generate analysis results of the spatial location distribution, action semantic category, and temporal contact features of future interactive objects based on the multi-scale fused feature map.

[0027] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a visual feature-based object interaction analysis program stored in the memory and executable on the processor, wherein when the visual feature-based object interaction analysis program is executed by the processor, it implements the steps of the visual feature-based object interaction analysis method as described above.

[0028] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a visual feature-based object interaction analysis program, wherein the visual feature-based object interaction analysis program, when executed by a processor, implements the steps of the visual feature-based object interaction analysis method as described above.

[0029] Beneficial Effects: This invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as fintech and healthcare. It discloses a method, apparatus, device, and medium for object interaction analysis based on visual features, comprising: acquiring predicted frames of video data; extracting multi-scale visual features from the predicted frames; detecting salient objects in the predicted frames to form a list of salient objects; generating initial action description pairs based on historical video frames; aggregating consecutive frames of the same action description pair to form an aggregated action description pair sequence; combining the aggregated action description pair sequence with the list of salient objects to generate a textualized action context; dividing the multi-scale visual features into image blocks and adding positional encoding to generate a multi-scale visual feature sequence; encoding the textualized action context using a pre-trained semantic encoding model to generate a linguistic feature sequence; for each scale, concatenating the visual feature sequence and the linguistic feature sequence and mapping them to a unified dimension to form a hybrid input sequence; performing cross-modal interaction through a multi-layer self-attention mechanism to obtain a multi-scale fusion feature sequence; recombining the multi-scale fusion feature sequence to form a multi-scale fusion feature map; and generating analysis results of the spatial location distribution, action semantic category, and temporal contact features of future interactive objects based on the multi-scale fusion feature map. This invention generates textualized action context by combining action description information from historical video frames with a list of salient objects. By combining multi-scale visual features and language feature sequences, and through unified mapping, cross-modal interaction, and feature fusion, it effectively enhances the semantic understanding and key interaction information extraction capabilities in complex scenes, thereby improving the accuracy and stability of object interaction analysis in changing environments. Attached Figure Description

[0030] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0031] Figure 1 This is a schematic diagram of an application environment for an object interaction analysis method based on visual features according to an embodiment of the present invention;

[0032] Figure 2 This is a flowchart illustrating an embodiment of the object interaction analysis method based on visual features of the present invention.

[0033] Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the object interaction analysis device based on visual features of the present invention.

[0034] Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0035] Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0036] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0037] The object interaction analysis method based on visual features provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can obtain predicted frames of video data from the client, extract multi-scale visual features from the predicted frames, detect salient objects in the predicted frames to form a salient object list; generate initial action description pairs based on historical video frames, and aggregate consecutive frames of the same action description pair to form an aggregated action description pair sequence; combine the aggregated action description pair sequence with the salient object list to generate a textualized action context; divide the multi-scale visual features into image patches and add positional encoding to generate a multi-scale visual feature sequence; use a pre-trained semantic encoding model to encode the textualized action context to generate a linguistic feature sequence; for each scale, concatenate the visual feature sequence and the linguistic feature sequence, and map them to a unified dimension to form a hybrid input sequence; perform cross-modal interaction through a multi-layer self-attention mechanism to obtain a multi-scale fusion feature sequence; reorganize the multi-scale fusion feature sequence to form a multi-scale fusion feature map; based on the multi-scale fusion feature map, generate analysis results on the spatial location distribution, action semantic category, and temporal contact features of future interactive objects. This invention generates textualized action context by jointly combining action description information from historical video frames with a list of salient objects. By integrating multi-scale visual features and linguistic feature sequences, and through unified mapping, cross-modal interaction, and feature fusion, it effectively enhances semantic understanding and key interaction information extraction capabilities in complex scenes, improving the accuracy and stability of object interaction analysis in changing environments. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention is described in detail below through specific embodiments.

[0038] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the object interaction analysis method based on visual features provided by the present invention. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0039] like Figure 2 As shown, the object interaction analysis method based on visual features proposed in this invention includes the following steps:

[0040] S10, acquire the predicted frames of the video data, extract the multi-scale visual features of the predicted frames, and detect salient objects in the predicted frames to form a list of salient objects;

[0041] In this embodiment, a predicted frame of video data is acquired. The video data can originate from any device with video capture capabilities. Video data refers to a continuous sequence of visual information, typically composed of multiple single-frame images arranged chronologically. Specifically, it can be acquired through sensors, cameras, mobile terminals, or monitoring systems. The predicted frame refers to the last frame in the video data sequence, serving as the image corresponding to the target time point for inference or analysis. It can be acquired by directly extracting the last frame from the video sequence using a time index, or by setting logical trigger conditions, such as reaching a specific time threshold or receiving an external analysis request. The existence of the predicted frame provides the visual information foundation for the current environment, supporting subsequent data processing.

[0042] Extracting multi-scale visual features from predicted frames involves identifying numerical representations of image content, encompassing multiple dimensions such as spatial structure, texture boundaries, and color information. These features are typically extracted using image coding networks. The term "multi-scale" refers to constructing feature representations at different spatial resolution levels for the same predicted frame. This can be achieved by building image pyramids or employing downsampling structures of different scales within the neural network architecture. For example, convolutional neural networks can extract high-resolution detail information and low-resolution global information by concatenating convolutional and pooling layers of varying depths, forming a multi-scale visual feature combination. Multi-scale visual features help capture both local fine information and overall structural information in an image simultaneously, improving the accuracy of subsequent object detection and environmental understanding.

[0043] The detection of salient objects in the prediction frame is crucial. Salient objects are defined as object regions in an image based on visual salience or semantic importance, typically including targets with distinctive appearance features, relevance to user interaction, or inference value. The detection process can be based on visual features combined with a predefined set of object categories, implemented through classification networks, object detection models, or semantic segmentation methods. The predefined set of object categories can be flexibly configured according to specific application scenarios, including common tools in a kitchen environment, office supplies in a financial setting, and medical instruments in a medical facility. By defining the category set, the specificity and application adaptability of the detection range are ensured. The detection operation typically includes region proposal, category determination, and bounding box localization, generating results including parameter information such as object category, spatial location, and size range.

[0044] A salient object list is created, which is a data set that organizes all detected salient object information into a fixed structure for subsequent retrieval. This list is typically stored as a list, array, or structure. Each object's information includes parameters such as category identifier, spatial location, size range, and confidence score. The list structure facilitates parallel analysis and multi-task collaborative processing of multiple objects in subsequent workflows. Furthermore, it allows for flexible expansion of object information fields according to task requirements, supporting data adaptation needs in different scenarios.

[0045] In the specific implementation process, the predicted frames of video data can be acquired by configuring high frame rate video acquisition equipment to transmit continuous image data to the analysis end in real time. The latest frame is then extracted as the predicted frame based on the system timestamp or business logic conditions. When extracting multi-scale visual features, deep learning-based image coding networks, such as residual structures or pyramid networks, can be used to generate multi-level feature maps by setting different downsampling ratios, representing the local details and global structural information of the predicted frame respectively. The detection of salient objects can be based on a general object detection model, such as a single-stage detection structure or a two-stage detection structure. By loading a set of object categories associated with the application scenario, and combining visual features, the corresponding object category and location information are output. During the formation of the salient object list, the detection results can be stored in a structured data format, such as a two-dimensional array or dictionary structure, to ensure the integrity and structured representation of object information, facilitating subsequent information retrieval and correlation analysis.

[0046] The acquisition method of prediction frames can be adjusted according to different application scenarios. For real-time inference requirements in dynamic environments, a higher frequency video data acquisition mechanism can be adopted to improve the timeliness of prediction frames. The structural depth or width parameters of the image coding network can also be adjusted to balance the detail preservation capability and computational efficiency of multi-scale visual features. For environmental backgrounds of varying complexity, the size and specific content of the object category set can be flexibly configured to enhance the generalization capability of the detection system in open scenarios and improve the accuracy of identifying uncertain object categories. In the data structure design of the salient object list, additional fields can be added according to business needs, such as adding object tracking numbers, interaction history information, or object status indicators, to adapt to more complex multi-task processing scenarios.

[0047] This embodiment effectively enhances the ability to simultaneously perceive local details and overall structure in complex environments by constructing multi-scale visual information representations. Combined with a salient object detection method based on category sets, it improves the accuracy of target extraction in different application environments. The structured organization of the salient object list ensures efficient management and task invocation of multi-object information, improving the stability and scalability of the overall data processing flow.

[0048] S20, generate initial motion description pairs based on historical video frames, and aggregate the same initial motion description pairs of consecutive frames in the historical video frames to generate an aggregated motion description pair sequence;

[0049] In this embodiment, initial action description pairs are generated based on historical video frames. Historical video frames refer to continuous image frame data in a video sequence earlier than the current time point, typically including dynamic information related to the user or environment. The source can be on-site video surveillance, mobile device photography, or data acquired by wearable cameras. The process of generating initial action description pairs first requires semantic understanding of the content in each frame. Specifically, frame-level description information can be extracted using a pre-trained image captioning model. Frame-level descriptions are linguistic expressions reflecting key actions and object information in the image frame, usually presented as natural language phrases or structured text. To ensure standardized description structure, part-of-speech tagging can be used to perform syntactic analysis on the frame-level descriptions, identifying verb and noun combinations. Verbs reflect changes in behavior or state, while nouns refer to objects or scenes involved in the behavior or related to the state. The combination of these two forms a complete semantic expression, constructing the initial action description pair. The initial action description pair reflects the direct semantic connection between actions and objects in the user or environment, possessing a clear semantic structure and strong interpretability.

[0050] This method aggregates identical initial action description pairs from consecutive frames in historical video. Consecutive frames refer to a sequence of video frames that are adjacent and sequentially connected on the timeline. Determining whether initial action description pairs are identical within consecutive frames can be achieved through text matching, semantic similarity analysis, or rule retrieval methods. If the initial action description pairs extracted from consecutive frames are completely identical in semantics or textual expression, it indicates that these frames reflect the same continuous action or interaction state, possessing temporal consistency and behavioral coherence. These consecutive frames with identical initial action description pairs are grouped into the same action segment. The existence of action segments helps integrate dynamic information in the temporal dimension, forming a higher-level semantic expression.

[0051] An aggregated action description pair sequence is generated, using action segments as the basic unit. Representative initial action description pairs are extracted from each action segment and combined in chronological order to form the aggregated action description pair sequence. This sequence not only preserves the structural information of temporal continuity but also expresses the semantic relationship between actions and objects through description pairs, facilitating subsequent semantic reasoning and behavior prediction.

[0052] In practical applications, historical video frames can be dynamically acquired from the video data stream by setting time windows or buffers. The length of the time window can be adjusted according to task requirements, making it suitable for analysis needs of varying complexity. When generating initial action description pairs, a deep learning-based image captioning generation model can be used. Through visual encoding and language decoding structures, it outputs frame-level descriptions that conform to natural language expressions. Combined with part-of-speech tagging techniques, such as conditional random fields and dependency parsing, it accurately extracts verb and noun combinations from the descriptions, improving the accuracy of semantic expression of actions and objects. When determining whether action description pairs in consecutive frames are identical, text fingerprinting technology, semantic embedding space matching, or semantic similarity determination using large-scale pre-trained language models can be combined to enhance the system's robustness to differences in expression, synonyms, or structural deformations.

[0053] This embodiment extracts action and object joint information from historical video frames to form a well-structured initial action description pair. Combined with semantic consistency analysis of consecutive frames, it constructs a temporally coherent and semantically clear sequence of aggregated action description pairs. This can effectively capture historical behavior trajectories and environmental interaction states, improve the system's ability to perceive and reason about user intentions, environmental dynamics, and potential behavior patterns, reduce semantic understanding bias in complex environments, and enhance the accuracy of behavior prediction and scene analysis.

[0054] S30, combine the aggregated action description pair sequence and the list of salient objects to generate a textualized action context;

[0055] In this embodiment, a textualized action context is generated by combining an aggregated action description pair sequence and a salient object list. The aggregated action description pair sequence refers to a set of linguistic information arranged chronologically, reflecting the association between actions and objects in historical video frames. It originates from the structured integration of initial action description pairs in historical video frames, embodying temporal coherence and semantic logic. The salient object list is a set of objects with potential interactive value or task relevance detected based on the predicted frame. It typically includes information such as object category, spatial location, and appearance features, forming a standardized structure to ensure consistency and accuracy in subsequent processing.

[0056] Generating textualized action contexts involves jointly representing the aggregated action description pairs and information from a list of salient objects into text content that conforms to the structure of natural language. This textualized representation enhances the clarity and logic of semantic expression, facilitating efficient understanding and processing by downstream models or inference modules. By organically integrating actions in the temporal dimension with object information in the spatial dimension, this textualized action context constructs a complete semantic description of the environment, helping to overcome the limitations of single information sources and improving the system's semantic understanding of complex scenes.

[0057] In the implementation process, each action description pair in the aggregated action description pair sequence can first be transformed into a phrase or sentence in language, maintaining the integrity and accuracy of actions, objects, and their semantic relationships. Simultaneously, the category information of each object in the salient object list is also transformed into a unified language expression to avoid semantic ambiguity or inconsistent expression. These two types of information are then connected using natural language generation methods or template combination techniques to form a coherent text that conforms to linguistic logic, ensuring the integrity, coherence, and richness of the expressed content.

[0058] In practical implementation, the sequence of aggregated action description pairs can be transformed one by one into a linguistic expression combining actions and objects through sequence traversal and text template matching. For example, the description pair "pick up - cup" can be transformed into the linguistic phrase "the user picks up the cup". The object information in the salient object list can be expressed as context-appropriate text content such as "the phone on the desktop" or "the fruit on the table" by combining object categories with contextual structure.

[0059] The fusion of these two types of information can be achieved using a static template structure, which integrates action and object information by setting fixed language templates to ensure standardized and consistent expression. Alternatively, a generation method based on a large-scale language model can be used, taking as input a sequence of aggregated action description pairs and a list of salient objects to generate overall text content with natural language expression capabilities, thereby improving expressive flexibility and semantic richness.

[0060] The level of detail and information density of textual expressions can be adjusted to suit different application environments. For example, in scenarios with high information security requirements, the textual expression can emphasize object category, location, or status information to facilitate system monitoring and early warning; in environments with high user experience requirements, the textual expression can highlight the interaction logic and environmental relationship to enhance the system's ability to express semantic chains and the accuracy of reasoning.

[0061] This embodiment expresses the aggregated action description pair sequence and the list of salient objects as a textual action context. The system can effectively integrate historical behavior information in the time dimension and environmental object information in the spatial dimension to construct a clear and semantically complete textual expression, improve the accuracy of scene semantic understanding, enhance the system's ability to perceive user intent, environmental state and potential interaction relationships, reduce the risk of semantic ambiguity and information omission in complex environments, and improve the overall analysis and reasoning level.

[0062] S40, the visual features at each scale in the multi-scale visual features are divided into image blocks and position codes are added to generate a multi-scale visual feature sequence;

[0063] In this embodiment, the visual features at each scale in the multi-scale visual feature set are divided into image patches and positional encodings are added to generate a multi-scale visual feature sequence. The multi-scale visual features refer to a set of image features obtained from different spatial resolution levels through a feature extraction network. These features typically cover multiple levels of information, from low-resolution, high-semantic-abstraction layers to high-resolution, detail-rich layers, ensuring that the system can simultaneously capture both macroscopic scene structure and local detail information. Specifically, each scale's visual feature refers to a feature map obtained for a single spatial resolution level, structurally represented as a two-dimensional or three-dimensional tensor, containing spatial and channel dimensions.

[0064] Dividing visual features at each scale into image patches involves segmenting continuous spatial feature regions into independent, structured sub-units according to a preset size. This image patch division helps unify the data structure, facilitating subsequent serialization processing and cross-modal information fusion. Positional encoding refers to generating additional encoded information for each image patch based on its spatial location within the overall visual features. This supplements the expression of spatial location information and prevents the loss of spatial structure during serialization. Generating a multi-scale visual feature sequence involves organizing the positionally encoded image patches into a linear sequence in a predetermined order. This serialized representation facilitates unified input to downstream sequence processing modules or cross-modal fusion structures, improving the standardization and structural clarity of the overall information representation.

[0065] In the implementation process, the image patch segmentation size and sliding step size can be set to ensure a balance between spatial information representation and computational efficiency in the visual feature segmentation results at different scales. Generally, smaller image patch sizes can be used for high-resolution scales, while larger image patch sizes can be used for low-resolution scales, to adapt to different levels of spatial information density. The generation of positional codes can be based on absolute positional coding schemes, which calculate the row and column indices of image patches in the overall feature map to generate corresponding numerical codes; or it can be based on relative positional coding, which enhances the system's overall perception of spatial structure by expressing the spatial relative positional relationships between image patches.

[0066] In the specific implementation, for each scale of visual features, the image can be divided into fixed-size rectangular blocks using a row and column slicing method, ensuring that each image block contains continuous and complete spatial information fragments. After segmentation, a positional code is generated based on the top-left corner coordinates or center coordinates of each image block. The encoding form can be a numerical vector, a positional embedding matrix, or high-dimensional positional mapping information. The positional code is then weighted and superimposed with the visual features within the image block, directly concatenated, or fused through a mapping function to ensure that spatial positional information and visual feature information are jointly retained in the data structure of each image block.

[0067] All image blocks are organized into a linear sequence in row-first or column-first order to form a single-scale visual feature sequence. Visual feature sequences at different scales can be organized independently to preserve the information expression structure at different resolution levels, or they can be cascaded or fused in multiple channels to form an overall multi-scale visual feature sequence, which is convenient for subsequent unified processing and information integration.

[0068] In different production environments, the size, location encoding format, and sequence organization structure can be flexibly adjusted according to system computing resources, task requirements, and target scenarios. For example, in environments sensitive to spatial structure, high-precision location encoding and fine-grained image block partitioning can be used to enhance spatial representation capabilities; in resource-constrained scenarios or scenarios with high requirements for information compression, low-dimensional location encoding and large-size image blocks can be used to reduce computational complexity and data scale.

[0069] This embodiment divides multi-scale visual features into image blocks and adds positional encoding. The system can completely preserve spatial structure information while serializing the representation, avoiding spatial information loss and structural disorder caused by the sequence structure. This improves the system's ability to express spatial relationships and multi-level visual information in complex scenes, enhances the overall accuracy and stability of downstream tasks in spatial localization, structural understanding and information fusion, and optimizes the system's expression and processing effect on complex visual inputs.

[0070] S50, Encode the textualized action context using a pre-trained semantic encoding model to generate a sequence of language features;

[0071] In this embodiment, a pre-trained semantic encoding model is used to encode textualized action context to generate a sequence of linguistic features. The textualized action context refers to a structured natural language information expression constructed based on aggregated action descriptions of sequences and a list of salient objects. It typically includes semantic text content related to historical actions, object information, and interaction relationships, possessing clear semantic logic and information summarization structure. The semantic encoding model refers to an encoding network that has been trained on large-scale corpora or multi-domain data and possesses stable semantic understanding capabilities. Common structures include, but are not limited to, encoders based on self-attention mechanisms, bidirectional context-aware models, or multi-level semantic representation networks. This model can efficiently extract semantic information from text and convert it into a high-dimensional feature representation.

[0072] The encoding process refers to inputting the textual action context into the semantic encoding model, generating a structured semantic feature expression based on the feature extraction and expression mechanism within the model. The language feature sequence is the serialized expression of the encoding result, usually represented as a fixed-dimensional vector sequence. It has the ability to retain the original semantic information, structural logic and content association of the text, which is convenient for subsequent fusion with other modal information or further information reasoning.

[0073] The input format for textual action context can be natural language text, structured sentence sequences, or standardized text data. The input end of the semantic coding model adapts to the input format according to the model design requirements. It often converts text data into numerical expression form through word vectors, character encoding, or sub-word unit expression, which facilitates feature extraction and expression operations within the model.

[0074] In practical implementation, a pre-trained large-scale semantic coding model can be selected, such as a language coding network built on a transformer architecture. The textual action context is input to the model's input end, and after multi-layer semantic feature extraction and context information fusion operations, the corresponding language feature sequence is output. Before text input, preprocessing operations such as word segmentation, removal of redundant information, and standardization of expression can be performed to improve coding efficiency and expression accuracy.

[0075] The generated language feature sequence can retain action information, object information and interaction logic in the textual action context. Each element in the sequence corresponds to a specific word, phrase or semantic unit in the input text. The overall sequence structure retains the original order and logical relationship of the text, which is convenient for alignment, fusion and joint analysis with visual information or other information sequences.

[0076] In different application environments, the type, parameter size, and training data source of the semantic coding model can be flexibly adjusted according to system requirements and application scenarios. For example, in complex tasks with high requirements for semantic understanding, a coding model with a large parameter size and abundant training data can be selected to improve semantic expression capabilities; in scenarios with limited computing resources or high real-time requirements, a lightweight coding network can be selected to balance expressive power and computational efficiency.

[0077] This embodiment uses a pre-trained semantic coding model to encode the textual action context and generate a sequence of language features. The system can efficiently acquire high-level semantic expressions related to historical actions, object information, and interaction logic, improve the clarity and semantic integrity of the information structure, enhance the information association and semantic understanding capabilities in downstream information fusion, reasoning, and analysis processes, and optimize the overall expression effect and analysis accuracy of the system in complex task environments.

[0078] S60, for each scale, the visual feature sequence and the language feature sequence of the scale are spliced ​​together, and the splicing result is mapped to a unified dimension to form a hybrid input sequence of the corresponding scale;

[0079] In this embodiment, for each scale, the visual feature sequence and the linguistic feature sequence of the scale are spliced ​​together and mapped to a unified dimension to form a hybrid input sequence of the corresponding scale. The visual feature sequence of the scale refers to the structured sequence expression formed by image block segmentation and position encoding addition steps for a specific scale in multi-scale visual features. It has visual information and spatial position information at that scale. The linguistic feature sequence refers to the semantic information expression sequence generated after encoding the textual action context through a semantic encoding model. It includes textual information features related to historical actions, object interactions, and semantic logic.

[0080] Concatenation refers to the sequence-level combination of visual and linguistic feature sequences at the feature level. This is typically manifested as a connection along the sequence or feature dimensions, ensuring a structurally unified representation of visual and linguistic information, facilitating subsequent joint analysis and interaction information fusion. The concatenated result can be represented as a joint feature sequence, with a structural length equal to the sum of the lengths of the visual and linguistic feature sequences. The sequence contains interleaved feature units from both image and linguistic modalities, forming a multimodal structured input foundation. In this sequence, the visual component retains spatial perception and multi-scale localization, while the linguistic component embeds behavioral context and interaction logic. The concatenated result constitutes a unified representation with significant cross-modal correlation characteristics at the sequence level, serving the subsequent fusion operations in attention mechanisms.

[0081] Mapping to a unified dimension refers to using linear projection, feature transformation, or other representation methods to uniformly adjust information from different sources to the same feature dimension after splicing the mixed information sequence. This ensures the consistency and structural integrity of information expression, and facilitates subsequent joint modeling, information fusion, and cross-modal reasoning of the system.

[0082] A mixed input sequence corresponding to a scale refers to a structured expression sequence formed independently for each scale. The sequence simultaneously contains visual feature information, spatial location information, and textual semantic information for that scale. The overall structure has information integrity, expression clarity, and multi-dimensional information synergy characteristics, which facilitates subsequent multi-layer self-attention mechanism processing and information interaction analysis.

[0083] In its implementation, the system can independently extract the corresponding visual feature sequences and linguistic feature sequences for each scale in the multi-scale visual features, and combine the two types of information into a hybrid sequence using a sequence-level concatenation method. The concatenation operation can be implemented through sequence concatenation, feature dimension expansion, or other sequence processing mechanisms to ensure the consistency of the information structure and the logical clarity of the combined expression.

[0084] After the splicing is completed, the system adjusts the mixed information sequence to a preset unified dimension by setting a unified feature dimension standard and using linear transformation, feature mapping network or other parameter learnable mapping structure. This ensures that the mixed input sequences at all scales in the system have a unified structure and consistent feature expression, thereby optimizing the information fusion efficiency and the overall expression effect of the system.

[0085] In practical applications, mapping operations can be adapted to different system requirements, computational resources, and information expression requirements through different parameter settings, network structures, or transformation methods, thereby improving the system's expressive flexibility and applicability.

[0086] This embodiment, by splicing visual feature sequences and linguistic feature sequences for each scale and mapping them to a unified dimension to form a hybrid input sequence for the corresponding scale, can efficiently achieve structural unification and feature fusion of multi-source information, improve the system's ability to collaboratively express visual, spatial, and semantic information, optimize the overall effect of subsequent cross-information interaction, information reasoning, and joint analysis, and enhance the system's expression integrity, information fusion efficiency, and reasoning accuracy in complex information environments.

[0087] S70, the mixed input sequences at each scale are subjected to cross-modal interaction through a multi-layer self-attention mechanism to obtain a multi-scale fused feature sequence;

[0088] In this embodiment, the mixed input sequence at each scale is subjected to cross-modal interaction through a multi-layer self-attention mechanism to obtain a multi-scale fusion feature sequence. The mixed input sequence refers to the unified structure expression sequence formed by splicing and mapping visual feature sequences and language feature sequences for each scale in the preceding steps. The sequence contains visual information, spatial location information and text semantic information at the same time, and has the characteristics of multi-source information collaborative expression.

[0089] Multi-layer self-attention mechanism refers to a structure based on self-attention computation structure that achieves global dependency modeling, dynamic information weighting, and semantic feature enhancement among information units within a sequence through multi-level, progressive information interaction and feature update operations. It typically improves information expression capabilities, enhances information fusion depth, and optimizes the overall multi-information interaction effect of the system by stacking multiple layers of self-attention computation units.

[0090] Cross-modal interaction refers to the use of the structural advantages of self-attention mechanisms within a mixed input sequence to enhance the association between visual, spatial, and semantic information by dynamically calculating the correlation and influence weights between different information units. This improves the overall information fusion capability and joint semantic expression level of the sequence, enabling deep fusion and collaborative reasoning of multi-source information.

[0091] Multi-scale fusion feature sequences refer to sequence structures that have undergone multi-layer self-attention mechanism processing for each scale. The sequences have stronger information expression capabilities, higher information fusion degree, and more complete multi-source collaborative expression effect. The overall structure can be applied to subsequent high-order reasoning, spatial structure reorganization, and task output processing of the system.

[0092] In the specific implementation process, the system can independently set up a multi-layer self-attention mechanism processing structure for each scale, adopt an attention computing network with adjustable parameters and scalable structure, and progressively process mixed input sequences to achieve dynamic weighting of information within the sequence, global dependency modeling across information units, and deep fusion of multi-source information.

[0093] Multi-layer self-attention mechanisms can employ standard multi-head attention structures, improved cross-modal attention modules, or other information interaction frameworks based on sequence structures. In the specific implementation process, the system can flexibly adjust the number of attention modules, structural parameters, and information update strategies according to information complexity, sequence length, and task requirements to ensure the optimization of information interaction effects at different scales and the overall expressive integrity of the system.

[0094] During information interaction, the system can improve the robustness and integrity of sequences in complex information environments by setting separate attention update paths for visual, spatial and semantic information, or by adopting a unified information fusion framework, to achieve collaborative updating, joint expression and overall structural optimization among information categories.

[0095] This embodiment effectively enhances the system's ability to jointly model multi-source information, information fusion depth, and expression completeness by using a multi-layer self-attention mechanism to conduct cross-modal interaction of the mixed input sequences at each scale. It also optimizes the synergistic expression effect between visual, spatial, and semantic information, enhances the system's overall information reasoning ability and multi-dimensional information fusion level, and improves the system's expression accuracy, information fusion efficiency, and analysis reliability in complex information environments.

[0096] S80, the multi-scale fusion feature sequence is reorganized into a multi-scale fusion feature map;

[0097] In this embodiment, the multi-scale fusion feature sequence is recombined into a multi-scale fusion feature map. The multi-scale fusion feature sequence refers to the sequence structure containing information at different scales, formed through multi-layer self-attention mechanisms in the preceding steps. This sequence integrates visual, spatial location, and semantic information, possessing multi-dimensional joint expression characteristics. Recombination refers to restoring, arranging, and constructing a spatial structure expression based on the internal information of the sequence, forming a multi-scale fusion feature map with two-dimensional spatial distribution characteristics, facilitating subsequent information parsing and feature extraction based on the spatial structure.

[0098] Multi-scale refers to the system's independent construction of multi-level information representation structures for different spatial resolutions, information fineness, and semantic levels during the feature representation process. Each scale corresponds to a set of feature representation results with different resolutions, expression densities, and information depths. Fusion feature maps refer to graph-like structures generated based on multi-scale fused feature sequences through spatial information recovery and structural rearrangement operations. These structures possess spatial layout information, complete expression, and information fusion effects, exhibiting two-dimensional spatial distribution characteristics and information visualization advantages, facilitating spatial reasoning, semantic recognition, and high-order information analysis in subsequent tasks.

[0099] In practical implementation, the system can first analyze the spatial location information, feature category information and information content expression within the fused feature sequence for each scale, and then restore the two-dimensional spatial arrangement relationship of the feature elements based on the spatial location information to reconstruct the spatial structure of the corresponding scale.

[0100] The system can determine the specific position of each feature element in a two-dimensional spatial structure by calculating the spatial index, row and column coordinates of the feature elements or by parsing the positional encoding. The feature elements in the sequence are then arranged sequentially according to the spatial distribution pattern to form a fused feature map with spatial continuity and complete information expression.

[0101] For different scales, the system can independently perform the above spatial structure reorganization operation to construct fusion feature maps with different resolutions, information granularities and semantic levels, forming multi-scale information expression results. This allows the system to flexibly call information expression structures at different scales according to task requirements, thereby improving the overall information expression depth and multi-level information collaborative reasoning effect.

[0102] This embodiment effectively restores the spatial structure in information representation by recombining multi-scale fused feature sequences into multi-scale fused feature maps, improves the spatial visualization effect and spatial reasoning ability of information, optimizes the collaborative layout expression between visual information, spatial information and semantic information, enhances the system's information understanding ability and multi-dimensional information fusion depth in complex scenarios, and improves the spatial location reasoning, semantic logic recognition and information association analysis effects in subsequent tasks.

[0103] S90, Based on the multi-scale fusion feature map, generate analysis results of the spatial location distribution, action semantic category and temporal contact features of future interactive objects.

[0104] In this embodiment, based on a multi-scale fused feature map, analysis results are generated regarding the spatial distribution, action semantic categories, and temporal contact features of future interactive objects. This involves using the multi-scale fused feature map as input information and, through structured information interpretation and feature reasoning, obtaining three key types of information about future interactive objects: spatial distribution, action semantic categories, and temporal contact features. The multi-scale fused feature map originates from previous steps and possesses different spatial scales, information levels, and fusion expression effects, comprehensively reflecting the joint expression information of objects, actions, and spatial relationships in the scene.

[0105] Spatial location distribution refers to the prediction of the spatial location of future interactive objects in a scene by reasoning the spatial information and object expression content in the multi-scale fusion feature map. Specifically, it can be the bounding box coordinates, spatial region distribution, or specific location point information in a two-dimensional plane, reflecting the system's ability to reason about the spatial positioning and layout of future interactive objects.

[0106] Action semantic category refers to the semantic label expression of future interactive actions that is analyzed and inferred based on the information content of the fused feature map. Specifically, it is semantic category information that reflects the action type, action content or action purpose, which facilitates the system's understanding and expression of action type, operation logic and behavioral intention in future interactive scenarios.

[0107] Temporal contact features refer to the prediction of future interactive object contact information in the temporal dimension based on the temporal information, spatial layout and semantic association in the feature map during the reasoning and analysis process. Specifically, it is expressed as time offset, contact time or duration, reflecting the behavioral characteristics of future interactive objects in the temporal dimension.

[0108] In actual implementation, the system can use a multi-task information interpretation mechanism to input multi-scale fused feature maps into the spatial location reasoning branch, the action semantic analysis branch, and the temporal feature prediction branch, and obtain the corresponding prediction results respectively.

[0109] The spatial location reasoning branch can predict the spatial location distribution of future interactive objects based on the spatial location information in the feature map and the fusion representation result, through convolution operations, spatial decoding structures or location regression algorithms, and output specific bounding box coordinates, location points or spatial region information.

[0110] The action semantic analysis branch can generate semantic category labels for actions by combining spatial information and semantic expression in multi-scale feature maps through feature extraction structures, semantic classification networks, or contextual semantic reasoning modules, accurately reflecting the semantic logic and behavioral content of future interactive actions.

[0111] The temporal feature prediction branch can be based on temporal information encoding structures, sequence analysis modules, or temporal regression networks. It can utilize dynamic information, spatial layout, and semantic content in multi-scale fusion feature maps to infer and predict the contact time characteristics of future interactive objects, and output time offsets or temporal behavior indicators.

[0112] The system can structurally combine spatial location distribution, action semantic category, and temporal contact features to form a unified expression of analysis results. It provides joint information output on the spatial location, semantic category, and temporal behavior of future interactive objects, supporting information understanding and decision support in subsequent stages such as interaction prediction, task planning, or operation assistance.

[0113] This embodiment generates analysis results of the spatial location distribution, action semantic categories, and temporal contact features of future interactive objects based on multi-scale fusion feature maps. It can comprehensively integrate scene information, action logic, and temporal behavior expression, improve the system's ability to predict the spatial layout of future interactive scenes, recognize semantic actions, and reason about temporal behavior, enhance its adaptability to complex interactive scenes, increase the depth of information understanding and the accuracy of operation prediction, and optimize the system's overall expression effect in interaction monitoring, behavior analysis, and task assistance.

[0114] In one embodiment, step S10 includes:

[0115] S101, determine the last frame in the video data as the prediction frame;

[0116] S102, The predicted frame is processed by a convolutional neural network to generate a multi-scale feature map;

[0117] S103, Extract multi-scale visual features containing different spatial resolutions from the multi-scale feature map;

[0118] S104, Get the predefined set of object categories;

[0119] S105, Use a visual language model to determine the similarity between the predicted frame and each object category in the predefined object category set;

[0120] S106, Select a preset number of object categories based on the similarity to form a list of prominent objects containing the selected object categories.

[0121] In this embodiment, acquiring the prediction frame of video data refers to locating the time node frame used for interactive analysis within a continuous video data stream. The video data is typically a continuous image sequence, and its source may include mobile devices, fixed monitoring systems, smart terminals, or other video acquisition devices. The prediction frame is determined by selecting the last frame in the video data. The last frame refers to the image frame with the latest time dimension in the currently acquired video sequence, reflecting the need for predictive analysis based on the latest environmental state, avoiding the introduction of future information, and maintaining the rationality of the prediction scenario.

[0122] Extracting multi-scale visual features involves using a convolutional neural network (CNN) structure to extract features layer by layer from the predicted frame. A CNN is a neural network architecture capable of hierarchical feature representation and can include multiple layers of convolutional, pooling, and normalization structural units. Through the feature processing of the network's multi-layer structure, multi-scale feature maps are generated. Feature maps are intermediate data structures that express image information at different spatial resolutions. Multi-scale feature maps reflect the visual expression of the same image at different perceptual ranges and information levels, helping the system to simultaneously capture fine-grained local information and global spatial relationships.

[0123] Extracting multi-scale visual features from multi-scale feature maps, which contain different spatial resolutions, refers to acquiring visual representations that reflect image details, structural layout, and global information at different levels or output resolutions. This can be achieved through direct downsampling or upsampling of feature maps, or by specific scale aggregation strategies. Multi-scale visual features provide a rich foundation of visual information, supporting subsequent salient object detection and interaction analysis.

[0124] Obtain a predefined set of object categories, which refers to a set of object category information stored or dynamically generated in the system. This set is used for comparison and analysis with the predicted frame. The set of object categories can come from a domain knowledge base, a standard object dictionary, or user-defined configurations. The category content includes object name, semantic description, visual reference information, etc.

[0125] Visual language models are used to determine the similarity between a predicted frame and each object category in a predefined set of object categories. A visual language model is a joint representation structure that simultaneously processes image information and textual semantic information. Common implementations include cross-modal contrastive learning models and graph-text coding networks. Similarity calculation is based on the multi-scale visual features of the predicted frame and the semantic information in the object category set. Through semantic alignment, feature mapping, or similarity measurement methods, numerical information reflecting the degree of semantic association between image content and object categories is output.

[0126] Based on similarity, a preset number of object categories are selected to form a salient object list containing these categories. The preset number is an integer parameter configured by the system according to actual application needs. The salient object list is a set of object categories highly relevant to the predicted frame content, reflecting key object information that may exist or will be interacted with in the current scene. This process ensures high relevance and discriminative power in the output results through similarity ranking, threshold filtering, or probabilistic inference, providing a precise object-level information foundation for subsequent interaction analysis.

[0127] This embodiment selects the last frame of the video data as the prediction frame, which can accurately reflect the latest visual state of the current environment and effectively avoid analysis bias caused by information lag. By extracting multi-scale feature maps based on convolutional neural networks and obtaining multi-scale visual features at different spatial resolutions, it can balance detailed representation and global structural information, improving the accuracy of object detection and scene understanding. Combining a predefined set of object categories with a visual language model for similarity calculation enables effective alignment of image content and semantic information. The selected list of salient objects focuses on key objects closely related to the current scene, reducing information redundancy and improving the relevance and discriminativeness of object information during interactive analysis, thus enhancing the system's comprehensive ability to perceive objects and predict interactions in complex environments.

[0128] In one embodiment, step S20 above includes:

[0129] S201, Use a pre-trained image captioning model to process each frame in the historical video frames and generate the corresponding frame-level description;

[0130] S202, the frame-level description is analyzed by part-of-speech tagging, and verb-noun combinations are extracted and the intermediate words are simplified to form initial action description pairs;

[0131] S203, Calculate the frequency of occurrence of the initial action description pair in historical video frames;

[0132] S204, Select the most frequent initial action description pair in each historical video frame as the representative action description pair of the historical video frame;

[0133] S205, Identify consecutive video frame sequences in historical video frame sequences that represent action description pairs;

[0134] S206, aggregate each representative action description pair into an action segment by combining the same consecutive video frame sequence;

[0135] S207, combine all action segments in chronological order to form a sequence of aggregated action description pairs.

[0136] In this embodiment, initial action description pairs are generated based on historical video frames. Historical video frames refer to a collection of video data that is earlier than the current analysis frame in time, and their sources include continuous video clips, buffered frame sequences, or past portions of real-time acquired data streams. A pre-trained image captioning model is used to process each frame in the historical video frames. This model combines visual information with language generation capabilities, taking an image frame as input and outputting textual descriptions related to the image content. This model can employ a joint architecture of visual encoding and language generation, possessing the ability to extract semantic information from image data and generate spoken text. Frame-level descriptions refer to the independent textual expressions generated for each image frame, reflecting information about actions, scenes, objects, or events in the image.

[0137] Frame-level descriptions are analyzed using part-of-speech tagging (POT). POT is a fundamental analytical operation in natural language processing, identifying the grammatical categories of words in the text, including verbs, nouns, adjectives, and adverbs. Extracting verb-noun combinations specifically involves extracting verbs describing actions and nouns related to entities from the text, reflecting the association between actions and objects. Simplifying intermediate words involves removing irrelevant conjunctions, modifiers, or structural words, retaining the core expressions of actions and objects, and forming initial action description pairs. These initial action description pairs are combinations of actions and objects, reflecting the behavioral semantic information in the video frames.

[0138] The frequency of initial action description pairs in historical video frames is statistically analyzed. Frequency refers to the number of times or the percentage of a particular description pair appearing in all historical video frames, and is used to measure the representativeness and importance of the description pair. The initial action description pair with the highest frequency in each historical video frame is selected as the representative action description pair for that historical video frame. The representative action description pair reflects the most semantically informative or most frequent behavioral expression in the current frame, which helps to simplify information expression and highlight core content.

[0139] In historical video frame sequences, consecutive video frame sequences with identical representative action description pairs are identified. A consecutive video frame sequence is a set of temporally adjacent and uninterrupted video frames. Identical representative action description pairs indicate that these frames express unified behavioral semantic information. Each consecutive video frame sequence with identical representative action description pairs is aggregated into an action segment. An action segment is a time segment composed of several consecutive video frames, with unified action expression and semantic continuity within it. All action segments are combined chronologically to form an aggregated action description pair sequence. The chronological order is based on the timestamps or frame indices of the video frames. The aggregated action description pair sequence is a time-ordered sequence of action expressions, reflecting the continuous behavioral process and action change information in the video.

[0140] This embodiment effectively extracts core action information from video sequences by performing frame-level semantic extraction and linguistic representation on historical video frames, combined with part-of-speech tagging and simplification. Frequency statistics and representativeness filtering improve the refinement and reliability of information representation, avoiding redundant information from interfering with the analysis. Aggregation of continuous frame sequences preserves the continuity of behavior over time, enhancing the system's ability to capture the evolution of actions. The resulting aggregated action description pairs accurately represent the key behavioral chains within a historical time period, providing a high-quality, structured linguistic information foundation for subsequent scene understanding, semantic reasoning, and interaction prediction, thus improving the overall accuracy of the system's behavior analysis and its semantic expression capabilities.

[0141] In one embodiment, step S40 above includes:

[0142] S401, for each scale of visual features, the visual features of the scale are divided into image blocks of a preset size;

[0143] S402, determine the spatial coordinates of each image block;

[0144] S403, Generate a position code based on the spatial coordinates;

[0145] S404, Add the location code to the corresponding image block;

[0146] S405, Arrange the image blocks with added position coding in spatial order to form a visual feature sequence of the corresponding scale;

[0147] S406 combines visual feature sequences at all scales to form a multi-scale visual feature sequence.

[0148] In this embodiment, multi-scale visual features refer to a set of image features with different spatial resolutions or receptive field ranges obtained by extracting features from the predicted frame. These features typically include high-resolution detail features and low-resolution semantic features, and can originate from convolutional neural networks, feature pyramid networks, or other multi-scale structures. The visual features at each scale are two-dimensional matrices or tensors at the corresponding resolution, representing the spatial structure and content information of the image at that scale.

[0149] The visual features at each scale are divided into image blocks of a preset size. An image block is a small region of a fixed size that is divided in the visual feature matrix. The preset size is determined according to application requirements, network structure or hardware constraints. The division operation can be carried out by sliding window, uniform segmentation or other methods. The purpose is to divide the overall feature information into local units, which facilitates the subsequent serialization processing and the introduction of position encoding.

[0150] Determine the spatial coordinates of each image block. The spatial coordinates are the two-dimensional position identifiers of the image block in the visual feature matrix. They may include horizontal position, vertical position, hierarchical index or other spatial information, which are derived from the matrix index or physical position mapping of each image block during the segmentation process.

[0151] Position codes are generated based on spatial coordinates. Position coding refers to converting spatial location information into a vector representation that can be recognized and utilized by the model. Common methods include sinusoidal function coding, learnable position embedding, or other forms of numerical representation. Position coding is used to supplement the spatial information of image patches and avoid the loss of spatial structure during serialization.

[0152] The location encoding is added to the corresponding image patch. The addition operation can be vector concatenation, element-wise addition, or other fusion methods. The image patch with added location encoding has both local content information and spatial location expression, which makes it easier for downstream models to capture spatial relationships.

[0153] The image blocks with added positional encoding are arranged in spatial order to form a visual feature sequence of the corresponding scale. The spatial order is determined by the spatial coordinates of the image blocks, usually using row priority, column priority or other serialization rules. The resulting one-dimensional sequence is easy to combine with the sequence modeling structure for subsequent information processing.

[0154] Combining visual feature sequences at all scales forms a multi-scale visual feature sequence. The combination operation involves splicing visual feature sequences at different scales according to scale index, resolution information, or other rules to form a sequence expression with a unified overall structure and clear hierarchy, reflecting multi-scale information fusion and spatial hierarchical structure.

[0155] This embodiment fully preserves the spatial structure and multi-level information representation of the image by performing image patching and positional encoding on multi-scale visual features, thus solving the problem of spatial information loss during serialization. Image patching improves the granularity of local information processing, helping to capture detailed features and local changes. The introduction of positional encoding supplements spatial location information, enhancing the model's ability to understand structural relationships. The construction of multi-scale visual feature sequences achieves unified expression and efficient combination of information at different scales, improving the system's performance in subsequent multi-level information processing and complex structural reasoning.

[0156] In one embodiment, step S70 above includes:

[0157] S701, For each scale, the mixed input sequence of the scale is input into the multilayer self-attention module;

[0158] S702, in the multi-layer self-attention module, determine the attention weight between visual feature sequence elements and language feature sequence elements;

[0159] S703, The visual feature sequence elements are updated with weights according to the attention weights to generate an updated visual feature sequence;

[0160] S704, The language feature sequence elements are updated with weights according to the attention weights to generate the updated language feature sequence;

[0161] S705, merge the updated visual feature sequence and the updated language feature sequence to generate an updated feature sequence;

[0162] S706, The updated feature sequence is iteratively processed through a multi-layer self-attention module;

[0163] S707 uses the iteratively processed feature sequence as the fused feature sequence for the corresponding scale.

[0164] S708 combines fusion feature sequences from all scales to form a multi-scale fusion feature sequence.

[0165] In this embodiment, the hybrid input sequence is a structure formed by concatenating visual feature sequences and linguistic feature sequences under a unified dimension. This sequence combines image information and semantic information, possessing the ability to express multi-source data fusion. Each scale of the hybrid input sequence corresponds to a different level in the multi-scale visual feature structure, expressing joint image and linguistic information at different spatial resolutions.

[0166] A multi-layer self-attention module is a layered stacked structure that achieves global information interaction and feature updates between elements within a sequence through multiple self-attention calculations. It originates from transformer structures or improved attention mechanism designs and has the ability to fuse cross-modal information and capture dependencies.

[0167] In a multi-layer self-attention module, visual feature sequence elements and linguistic feature sequence elements are different components in the input sequence. The correlation between them is dynamically calculated through an attention mechanism. The attention weight reflects the dependence strength and information importance between different elements, and is usually implemented through similarity functions or parameter matrix transformations.

[0168] Weighted updates are performed on visual feature sequence elements and linguistic feature sequence elements according to attention weights. The update operation refers to the integration of information contributed by other sequence elements while retaining the original information, thereby improving the global consistency and cross-modal collaborative effect of the overall expression. The updated visual and linguistic feature sequences contain the joint enhanced expression of image structure information and semantic content information, respectively.

[0169] The updated visual feature sequence is merged with the language feature sequence to generate an updated feature sequence. The merging operation can be splicing, overlaying or other fusion methods to form a unified and information-rich comprehensive sequence expression, which facilitates subsequent unified modeling and further information interaction.

[0170] Through iterative processing of multi-layer self-attention modules, the updated feature sequence is input into the self-attention structure multiple times, and the higher-order correlation between different elements within the sequence and cross-modal semantic integration are enhanced layer by layer. The iterative process realizes multi-round aggregation and deep fusion of information flow.

[0171] The feature sequence after iterative processing is used as the fused feature sequence at the corresponding scale. The fused feature sequence retains the original visual and linguistic information and has a stronger joint expressive ability after multi-layer interaction. It is suitable for multimodal feature modeling and task reasoning at different scale levels.

[0172] By combining the fusion feature sequences of all scales to form a multi-scale fusion feature sequence, the combination operation integrates multimodal representations of information with different spatial resolutions and levels, ensuring the overall consistency and expressive richness of the system when processing multi-scale structural information and complex semantic relationships.

[0173] This embodiment achieves deep fusion of visual and linguistic information within the sequence through a multi-layer self-attention mechanism, significantly enhancing the consistency and semantic synergy of multi-scale information expression, and improving the system's ability to integrate and analyze multi-source information in complex scenarios. Through iterative processing and high-order interactions, it solves the problem of isolated cross-modal information in traditional sequence modeling, improves the effectiveness of semantic association capture and global dependency modeling, and ensures that the multi-scale fused feature sequence possesses rich structural information and semantic expression capabilities, meeting the needs of subsequent high-precision multimodal analysis and task prediction.

[0174] In one embodiment, step S80 above includes:

[0175] S801, For each scale of the multi-scale fusion feature sequence, parse the spatial location information in the fusion feature sequence;

[0176] S802, Reconstruct the two-dimensional spatial structure based on the spatial location information;

[0177] S803, according to the reconstructed two-dimensional spatial structure, the feature elements in the fused feature sequence are precisely positioned and arranged for the low-resolution features at the bottom layer, and semantic reasoning is performed to arrange the low-resolution features at the top layer, forming a fused feature map of the corresponding scale.

[0178] S804 combines fused feature maps from all scales to form a multi-scale fused feature map.

[0179] In this embodiment, the multi-scale fusion feature sequence is composed of fusion feature sequences at different scale levels. The fusion feature sequence combines visual and semantic information, possesses cross-modal expression capabilities, and originates from the joint information expression of different spatial resolutions or different structural levels. Each scale's fusion feature sequence contains serialized image patch features and positional encoding information, providing a basis for reconstructing the spatial structure.

[0180] Parsing and fusing spatial location information in a feature sequence refers to extracting the corresponding position of each feature element in the original space based on the spatial location encoding or index information carried in the sequence. Spatial location information is usually represented by position encoding, grid indexing, coordinate mapping, etc., to ensure that the serialized information can be accurately mapped back to the corresponding spatial structure layout.

[0181] Reconstructing a two-dimensional spatial structure based on spatial location information involves mapping feature elements in the sequence to a two-dimensional matrix or feature map structure according to their spatial location, thus restoring the spatial layout relationships that were disrupted during the serialization process. The two-dimensional spatial structure can be a regular grid, a matrix arrangement, or other spatial topological structures, ensuring the complete expression of spatial information.

[0182] In the reconstructed two-dimensional spatial structure, a differentiated arrangement strategy is adopted for the low-level high-resolution features and the high-level low-resolution features. High-resolution features usually come from the low-level structure of the image and have rich details and spatial positioning capabilities. Precise positioning arrangement refers to accurately mapping the feature elements to the corresponding positions in the two-dimensional space under high-precision spatial coordinates to ensure the detailed restoration of the spatial structure.

[0183] High-level low-resolution features originate from high-level abstract expressions of images or sequences. They possess strong semantic information but have low spatial resolution. Semantic reasoning arrangement refers to combining spatial location and semantic association to reasonably map high-level features into spatial structures, while preserving the abstract information and semantic generalization ability of high-level expressions, thereby improving the semantic integrity and information richness of the overall feature map.

[0184] A fusion feature map of corresponding scale is formed. The fusion feature map is a two-dimensional structure that combines the spatial structure after serialization information restoration with multimodal fusion information. It has the ability to jointly express multi-source information and is suitable for spatial correlation analysis and subsequent high-order information processing.

[0185] By combining fusion feature maps of all scales, a multi-scale fusion feature map is formed. The combination operation integrates fusion feature maps with different spatial resolutions and different expression levels, preserving the structural characteristics and semantic hierarchy of multi-scale information, and forming a joint expression structure with complete multi-scale information and clear structural layout, providing complete information support for subsequent spatial analysis and task reasoning.

[0186] In practice, the fused feature sequence contains multiple layers of output. Typically, the portion with fewer than or equal to a preset threshold L1 is classified as the bottom layer, and the portion with more than L1 is classified as the top layer. The specific threshold is set according to actual needs. Taking L1=2 as an example, the first two feature sequences are the bottom layer, and the remaining portion is the top layer.

[0187] Simultaneously, considering the resolution settings, fused feature maps with spatial dimensions greater than R1×R1 are considered high-resolution, while those smaller than R1×R1 are considered low-resolution. With an original input image size of 256×256 and R1 set to 64, the generated fused feature maps with dimensions of 128×128 and 64×64 are considered high-resolution, while those with dimensions of 32×32 and 16×16 are considered low-resolution.

[0188] In the process of reconstructing multi-scale fused feature maps, precise positioning and arrangement operations are performed on the low-resolution, bottom-level parts to ensure the preservation of spatial details and boundary information. For the high-resolution, top-level parts, semantic reasoning and arrangement are performed to enhance the semantic understanding of object relationships, scene background, and interaction intent, thereby improving the overall spatial representation and analysis capabilities. This approach effectively balances the preservation of spatial details with semantic generalization, making it suitable for object interaction analysis tasks that require the integration of spatial and semantic information.

[0189] Example Description: In the healthcare field, for the analysis of medical device operations in intraoperative videos, the multi-layer output of the fused feature sequence contains spatial and semantic information at different levels. In practice, considering the input image size of 256×256, a layer threshold of 2 and a resolution threshold of 64 pixels are set. In the specific implementation, the first two layers of the fused feature sequence with a spatial size greater than or equal to 64×64 are defined as low-level high-resolution features. Precise positioning and arrangement operations are applied to ensure that the edge contours and spatial relationships of surgical instruments are preserved. Subsequent layers, especially those with a spatial size less than 64×64, are considered high-level low-resolution features. Semantic reasoning and arrangement are performed to highlight the lesion area, medical operation behavior, and semantic association with the surrounding environment. Finally, a multi-scale fused feature map is formed, ensuring that the system has both fine-grained spatial reconstruction capabilities and comprehensive analytical effects for medical semantic understanding.

[0190] In the field of fintech business, during remote counter interaction monitoring, in the output of different scales of the fused feature sequence, the low-level features with a spatial resolution greater than 128×128 are used to accurately restore the operation interface and the location of key items (such as bank cards and signature boards). The accuracy of item recognition and position calibration is improved by precise positioning and arrangement. The high-level output with a spatial resolution less than 128×128 is combined with the context to perform semantic reasoning and arrangement, which helps to infer the customer's operation intention, interaction process and risk behavior, forming a unified multi-scale fused feature map, providing multi-level and rich visual and semantic information support for subsequent risk analysis and business compliance detection.

[0191] This embodiment effectively solves the problem of lost positional information when serialized information expresses spatial structure by recombining the fused feature sequence into a multi-scale fused feature map, realizing the complete reconstruction and joint expression of multi-source information in two-dimensional space. The combination of precise positioning arrangement and semantic reasoning arrangement ensures the reasonable fusion of high-resolution detailed information and high-level semantic information in the spatial structure, improving the system's ability to jointly express complex spatial structures and multi-level semantic information. The multi-scale fused feature map has higher information density and expression completeness, improving the accuracy and robustness of the system in subsequent spatial location analysis, semantic recognition, and interaction prediction tasks, and enhancing the overall utilization efficiency and expression effect of multimodal information in complex environments.

[0192] In one embodiment, step S90 above includes:

[0193] S901, the multi-scale fused feature map is input into the spatial location prediction branch to generate the bounding box coordinates of the future interactive object, and the bounding box coordinates are used as the spatial location distribution;

[0194] S902, the multi-scale fused feature map is input into the action classification branch to generate verb-noun combinations, and the verb-noun combinations are used as action semantic categories;

[0195] S903, input the multi-scale fused feature map into the time regression branch to generate the contact time offset, and use the contact time offset as the time contact feature;

[0196] S904, combine the spatial location distribution, action semantic category and temporal contact features to generate analysis results.

[0197] In this embodiment, the multi-scale fused feature map contains spatial structure information and semantic expression information at different scales and resolutions, and has the ability to express multi-dimensional features across scales. This feature map retains the spatial location distribution of the predicted frame, the local features of salient objects, the action description information and language coding information of historical video frames through the preceding operation steps, and forms a high-dimensional expression that combines global relationships and local details through a multi-layer self-attention mechanism.

[0198] The spatial location prediction branch, targeting the spatial structure information of the fused feature map, utilizes a network module containing spatial convolutional structures and scale aggregation operations to extract features related to the position of the target interactive object, generating bounding box coordinates. Bounding box coordinates include horizontal position, vertical position, width, and height information, typically expressed as quadruples or center point offsets and size changes. This can be achieved through regression modules, anchor box mechanisms, or center point localization algorithms.

[0199] The action classification branch takes the fused feature map as input, focusing on processing high-level and low-resolution features. Combined with embedded textual action context information, it utilizes a semantic classifier or sequence modeling network to output verb-noun combinations. These verb-noun combinations represent the predicted interaction behavior category and the category of the key object involved, forming an action semantic category that conforms to the semantic structure of natural language. This category output can be in the form of a probability distribution, using maximum value selection, or a multi-category joint output format to express complex interaction types.

[0200] The temporal regression branch takes the fused feature map as input, combines the timestamp information of the predicted frame with the temporal dynamic features in the fused features, and outputs a contact time offset based on the regression network structure. The contact time offset is a numerical scalar, reflecting the time interval from the current time point to the actual contact between the interacting objects in the future, in units of frames, milliseconds, or seconds. During the temporal regression process, a multilayer perceptron structure, residual connections, and regularization mechanisms can be incorporated to ensure the stability and numerical accuracy of the time offset prediction.

[0201] Spatial location distribution, action semantic category, and temporal contact features are generated independently through the aforementioned branch structure. The output results are combined through structured operations to form a unified set of analysis results. The overall analysis results have three dimensions: spatial positioning information, interaction semantic category, and temporal prediction information, which can comprehensively express the expected state and dynamic characteristics of future interactive objects.

[0202] During the overall operation, spatial location prediction, action semantic classification, and temporal regression can be performed in parallel or sequentially according to task priority. The specific execution order and task dependencies can be adjusted and optimized according to the actual scenario requirements. Through the joint design of the multi-task structure, the analysis results form synergistic constraints at the spatial, semantic, and temporal levels, improving the overall accuracy and robustness of the prediction and reducing the impact of single-task errors on the system output.

[0203] This embodiment improves the spatial positioning accuracy of future interactive objects, the semantic rationality of action category recognition, and the stability of time estimation by fusing spatial and semantic information from feature maps and combining spatial location prediction, action semantic classification, and temporal contact feature analysis. This enhances the system's ability to analyze complex and ever-changing interactive scenarios, especially in real-world environments with high uncertainty and diverse interactive targets, where it exhibits superior analysis and prediction capabilities.

[0204] In one embodiment, a visual feature-based object interaction analysis device is provided, which corresponds one-to-one with the visual feature-based object interaction analysis method described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the object interaction analysis device based on visual features of the present invention. The modules include: visual perception module 10, action semantic extraction module 20, context construction module 30, visual encoding module 40, language encoding module 50, fusion and splicing module 60, cross-modal interaction module 70, feature reorganization module 80, and interaction analysis module 90. Detailed descriptions of each functional module are as follows:

[0205] The visual perception module 10 is used to acquire the predicted frames of video data, extract the multi-scale visual features of the predicted frames, and detect salient objects in the predicted frames to form a list of salient objects.

[0206] The action semantic extraction module 20 is used to generate initial action description pairs based on historical video frames, and aggregate the same initial action description pairs of consecutive frames in the historical video frames to generate an aggregated action description pair sequence.

[0207] Context building module 30 is used to generate a textualized action context by combining the aggregated action description pair sequence and the list of salient objects;

[0208] The visual encoding module 40 is used to divide the visual features at each scale in the multi-scale visual features into image blocks and add position encoding to generate a multi-scale visual feature sequence.

[0209] The language encoding module 50 is used to encode the textualized action context using a pre-trained semantic encoding model to generate a language feature sequence;

[0210] The fusion and stitching module 60 is used to stitch together the visual feature sequence and the language feature sequence of each scale, and map the stitching result to a unified dimension to form a hybrid input sequence of the corresponding scale.

[0211] The cross-modal interaction module 70 is used to perform cross-modal interaction on the mixed input sequence at each scale through a multi-layer self-attention mechanism to obtain a multi-scale fused feature sequence;

[0212] Feature recombination module 80 is used to recombine the multi-scale fused feature sequence into a multi-scale fused feature map;

[0213] The interaction analysis module 90 is used to generate analysis results of the spatial location distribution, action semantic category and temporal contact features of future interactive objects based on the multi-scale fusion feature map.

[0214] In one embodiment, the visual perception module 10 is specifically used for:

[0215] The last frame in the video data is determined as the prediction frame;

[0216] The predicted frames are processed by a convolutional neural network to generate multi-scale feature maps.

[0217] Extract multi-scale visual features containing different spatial resolutions from the multi-scale feature map;

[0218] Get a predefined set of object categories;

[0219] The similarity between the predicted frame and each object category in the predefined set of object categories is determined using a visual language model.

[0220] Based on the similarity, a preset number of object categories are selected to form a list of prominent objects containing the selected object categories.

[0221] In one embodiment, the action semantic extraction module 20 is specifically used for:

[0222] Use a pre-trained image captioning model to process each frame in the historical video frames and generate a corresponding frame-level description;

[0223] By analyzing the frame-level description through part-of-speech tagging, verb-noun combinations are extracted and intermediate words are simplified to form initial action description pairs;

[0224] Statistically analyze the frequency of occurrence of the initial action description pair in historical video frames;

[0225] Select the most frequent initial motion description pair in each historical video frame as the representative motion description pair for that historical video frame;

[0226] Identify consecutive video frame sequences in historical video frame sequences that represent the same action description pair;

[0227] Each representative action description is aggregated into an action segment for the same consecutive video frame sequence;

[0228] All action segments are combined in chronological order to form a sequence of aggregated action description pairs.

[0229] In one embodiment, the visual encoding module 40 is specifically used for:

[0230] For each scale of visual features, the visual features at that scale are divided into image blocks of a preset size;

[0231] Determine the spatial coordinates of each image patch;

[0232] A location code is generated based on the spatial coordinates;

[0233] Add the location code to the corresponding image block;

[0234] The image patches with added location encoding are arranged in spatial order to form a visual feature sequence of the corresponding scale;

[0235] Combining visual feature sequences at all scales forms a multi-scale visual feature sequence.

[0236] In one embodiment, the cross-modal interaction module 70 is specifically used for:

[0237] For each scale, the mixed input sequence of that scale is input into a multi-layer self-attention module;

[0238] In the multi-layer self-attention module, the attention weights between visual feature sequence elements and language feature sequence elements are determined;

[0239] The visual feature sequence elements are updated by weighting according to the attention weights to generate the updated visual feature sequence.

[0240] The language feature sequence elements are updated by weighting according to the attention weights to generate the updated language feature sequence.

[0241] The updated visual feature sequence and the updated language feature sequence are merged to generate an updated feature sequence;

[0242] The updated feature sequence is iteratively processed using a multi-layer self-attention module;

[0243] The feature sequence after iterative processing is used as the fused feature sequence at the corresponding scale;

[0244] The fusion feature sequences of all scales are combined to form a multi-scale fusion feature sequence.

[0245] In one embodiment, the feature recombination module 80 is specifically used for:

[0246] For each scale of the multi-scale fusion feature sequence, the spatial location information in the fusion feature sequence is parsed.

[0247] Reconstruct the two-dimensional spatial structure based on the spatial location information;

[0248] The feature elements in the fused feature sequence are arranged according to the reconstructed two-dimensional spatial structure. The low-level high-resolution features are precisely located and arranged, and the high-level low-resolution features are arranged by semantic reasoning to form a fused feature map of the corresponding scale.

[0249] Combine the fused feature maps of all scales to form a multi-scale fused feature map.

[0250] In one embodiment, the interactive analysis module 90 is specifically used for:

[0251] The multi-scale fused feature map is input into the spatial location prediction branch to generate the bounding box coordinates of future interactive objects, and the bounding box coordinates are used as the spatial location distribution.

[0252] The multi-scale fused feature map is input into the action classification branch to generate verb-noun combinations, and the verb-noun combinations are used as action semantic categories;

[0253] The multi-scale fused feature map is input into the time regression branch to generate the contact time offset, and the contact time offset is used as the time contact feature.

[0254] The analysis results are generated by combining the spatial location distribution, action semantic category, and temporal contact features.

[0255] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a visual feature-based object interaction analysis method on the server side.

[0256] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a visual feature-based object interaction analysis method.

[0257] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0258] Acquire predicted frames from video data, extract multi-scale visual features from the predicted frames, and detect salient objects in the predicted frames to form a list of salient objects;

[0259] Initial motion description pairs are generated based on historical video frames, and the same initial motion description pairs in consecutive frames in the historical video frames are aggregated to generate an aggregated motion description pair sequence.

[0260] By combining the aggregated action description pair sequence and the list of salient objects, a textualized action context is generated;

[0261] The visual features at each scale in the multi-scale visual features are divided into image blocks and positional codes are added to generate a multi-scale visual feature sequence.

[0262] The textualized action context is encoded using a pre-trained semantic encoding model to generate a sequence of linguistic features;

[0263] For each scale, the visual feature sequence and the language feature sequence of that scale are spliced ​​together, and the splicing result is mapped to a unified dimension to form a hybrid input sequence for the corresponding scale;

[0264] The mixed input sequences at each scale are subjected to cross-modal interaction through a multi-layer self-attention mechanism to obtain a multi-scale fused feature sequence;

[0265] The multi-scale fusion feature sequence is recombined into a multi-scale fusion feature map;

[0266] Based on the multi-scale fused feature map, the analysis results of the spatial location distribution, action semantic category and temporal contact features of future interactive objects are generated.

[0267] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0268] Acquire predicted frames from video data, extract multi-scale visual features from the predicted frames, and detect salient objects in the predicted frames to form a list of salient objects;

[0269] Initial motion description pairs are generated based on historical video frames, and the same initial motion description pairs in consecutive frames in the historical video frames are aggregated to generate an aggregated motion description pair sequence.

[0270] By combining the aggregated action description pair sequence and the list of salient objects, a textualized action context is generated;

[0271] The visual features at each scale in the multi-scale visual features are divided into image blocks and positional codes are added to generate a multi-scale visual feature sequence.

[0272] The textualized action context is encoded using a pre-trained semantic encoding model to generate a sequence of linguistic features;

[0273] For each scale, the visual feature sequence and the language feature sequence of that scale are spliced ​​together, and the splicing result is mapped to a unified dimension to form a hybrid input sequence for the corresponding scale;

[0274] The mixed input sequences at each scale are subjected to cross-modal interaction through a multi-layer self-attention mechanism to obtain a multi-scale fused feature sequence;

[0275] The multi-scale fusion feature sequence is recombined into a multi-scale fusion feature map;

[0276] Based on the multi-scale fused feature map, the analysis results of the spatial location distribution, action semantic category and temporal contact features of future interactive objects are generated.

[0277] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0278] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0279] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0280] It should be noted that if any software tools or components not belonging to our company appear in the embodiments of this application, they are merely for illustrative purposes and do not represent actual use.

Claims

1. A method for analyzing object interactions based on visual features, characterized in that, Includes the following steps: Acquire predicted frames from video data, extract multi-scale visual features from the predicted frames, and detect salient objects in the predicted frames to form a list of salient objects; Initial motion description pairs are generated based on historical video frames, and the same initial motion description pairs in consecutive frames in the historical video frames are aggregated to generate an aggregated motion description pair sequence. By combining the aggregated action description pair sequence and the list of salient objects, a textualized action context is generated; The visual features at each scale in the multi-scale visual features are divided into image blocks and positional codes are added to generate a multi-scale visual feature sequence. The textualized action context is encoded using a pre-trained semantic encoding model to generate a sequence of linguistic features; For each scale, the visual feature sequence and the language feature sequence of that scale are spliced ​​together, and the splicing result is mapped to a unified dimension to form a hybrid input sequence for the corresponding scale; The mixed input sequences at each scale are subjected to cross-modal interaction through a multi-layer self-attention mechanism to obtain a multi-scale fused feature sequence; The multi-scale fusion feature sequence is recombined into a multi-scale fusion feature map; Based on the multi-scale fused feature map, the analysis results of the spatial location distribution, action semantic category and temporal contact features of future interactive objects are generated.

2. The object interaction analysis method based on visual features as described in claim 1, characterized in that, Acquire predicted frames from video data, extract multi-scale visual features from the predicted frames, and detect salient objects in the predicted frames to form a salient object list, including: The last frame in the video data is determined as the prediction frame; The predicted frames are processed by a convolutional neural network to generate multi-scale feature maps. Extract multi-scale visual features containing different spatial resolutions from the multi-scale feature map; Get a predefined set of object categories; The similarity between the predicted frame and each object category in the predefined set of object categories is determined using a visual language model. Based on the similarity, a preset number of object categories are selected to form a list of prominent objects containing the selected object categories.

3. The object interaction analysis method based on visual features as described in claim 1, characterized in that, Initial motion description pairs are generated based on historical video frames, and identical initial motion description pairs in consecutive frames of the historical video frames are aggregated to generate an aggregated motion description pair sequence, including: Use a pre-trained image captioning model to process each frame in the historical video frames and generate a corresponding frame-level description; By analyzing the frame-level description through part-of-speech tagging, verb-noun combinations are extracted and intermediate words are simplified to form initial action description pairs; Statistically analyze the frequency of occurrence of the initial action description pair in historical video frames; Select the most frequent initial motion description pair in each historical video frame as the representative motion description pair for that historical video frame; Identify consecutive video frame sequences in historical video frame sequences that represent action description pairs; Each representative action description is aggregated into an action segment for the same consecutive video frame sequence; All action segments are combined in chronological order to form a sequence of aggregated action description pairs.

4. The object interaction analysis method based on visual features as described in claim 1, characterized in that, The visual features at each scale in the multi-scale visual features are divided into image patches and positional codes are added to generate a multi-scale visual feature sequence, including: For each scale of visual features, the visual features at that scale are divided into image blocks of a preset size; Determine the spatial coordinates of each image patch; A location code is generated based on the spatial coordinates; Add the location code to the corresponding image block; The image patches with added location encoding are arranged in spatial order to form a visual feature sequence of the corresponding scale; Combining visual feature sequences at all scales forms a multi-scale visual feature sequence.

5. The object interaction analysis method based on visual features as described in claim 1, characterized in that, The mixed input sequences at each scale are subjected to cross-modal interaction through a multi-layer self-attention mechanism to obtain multi-scale fused feature sequences, including: For each scale, the mixed input sequence of that scale is input into a multi-layer self-attention module; In the multi-layer self-attention module, the attention weights between visual feature sequence elements and language feature sequence elements are determined; The visual feature sequence elements are updated by weighting according to the attention weights to generate the updated visual feature sequence. The language feature sequence elements are updated by weighting according to the attention weights to generate the updated language feature sequence. The updated visual feature sequence and the updated language feature sequence are merged to generate an updated feature sequence; The updated feature sequence is iteratively processed using a multi-layer self-attention module; The feature sequence after iterative processing is used as the fused feature sequence at the corresponding scale; The fusion feature sequences of all scales are combined to form a multi-scale fusion feature sequence.

6. The object interaction analysis method based on visual features as described in claim 1, characterized in that, Recombining the multi-scale fused feature sequence into a multi-scale fused feature map includes: For each scale of the multi-scale fusion feature sequence, the spatial location information in the fusion feature sequence is parsed. Reconstruct the two-dimensional spatial structure based on the spatial location information; The feature elements in the fused feature sequence are arranged according to the reconstructed two-dimensional spatial structure. The low-level high-resolution features are precisely located and arranged, and the high-level low-resolution features are arranged by semantic reasoning to form a fused feature map of the corresponding scale. Combine the fused feature maps of all scales to form a multi-scale fused feature map.

7. The object interaction analysis method based on visual features as described in claim 1, characterized in that, Based on the multi-scale fused feature map, the analysis results of the spatial location distribution, action semantic category, and temporal contact features of future interactive objects are generated, including: The multi-scale fused feature map is input into the spatial location prediction branch to generate the bounding box coordinates of future interactive objects, and the bounding box coordinates are used as the spatial location distribution. The multi-scale fused feature map is input into the action classification branch to generate verb-noun combinations, and the verb-noun combinations are used as action semantic categories; The multi-scale fused feature map is input into the time regression branch to generate the contact time offset, and the contact time offset is used as the time contact feature. The analysis results are generated by combining the spatial location distribution, action semantic category, and temporal contact features.

8. A device for analyzing object interaction based on visual features, characterized in that, The object interaction analysis device based on visual features includes: The visual perception module is used to acquire predicted frames of video data, extract multi-scale visual features of the predicted frames, and detect salient objects in the predicted frames to form a list of salient objects. The action semantic extraction module is used to generate initial action description pairs based on historical video frames, and to aggregate the same initial action description pairs in consecutive frames in the historical video frames to generate an aggregated action description pair sequence. A context building module is used to generate a textualized action context by combining the aggregated action description pair sequence and the list of salient objects; The visual encoding module is used to divide the visual features at each scale of the multi-scale visual features into image blocks and add positional encoding to generate a multi-scale visual feature sequence. The language encoding module is used to encode the textualized action context using a pre-trained semantic encoding model to generate a sequence of language features; The fusion and stitching module is used to stitch together the visual feature sequence and the language feature sequence of each scale, and map the stitching result to a unified dimension to form a hybrid input sequence of the corresponding scale. The cross-modal interaction module is used to perform cross-modal interaction on the mixed input sequences at each scale through a multi-layer self-attention mechanism to obtain a multi-scale fused feature sequence; The feature recombination module is used to recombine the multi-scale fused feature sequence into a multi-scale fused feature map. The interaction analysis module is used to generate analysis results of the spatial location distribution, action semantic category, and temporal contact features of future interactive objects based on the multi-scale fused feature map.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a visual feature-based object interaction analysis program stored in the memory and executable on the processor. When executed by the processor, the visual feature-based object interaction analysis program implements the steps of the visual feature-based object interaction analysis method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a visual feature-based object interaction analysis program, which, when executed by a processor, implements the steps of the visual feature-based object interaction analysis method as described in any one of claims 1-7.

Images