Object interaction analysis method and device based on visual features, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to business scenes such as financial science and technology and medical health, and discloses an object interaction analysis method and device based on visual features, equipment and a medium. The method comprises the following steps: generating an action description pair in combination with historical video frames, constructing an action context, extracting a language feature sequence and a visual feature sequence, splicing and mapping into a unified dimension, realizing interactive fusion by utilizing a self-attention mechanism, recombining into a multi-scale fusion feature map, and outputting spatial position distribution, action category and time contact features of an interactive object. According to the method, the textual action context is jointly generated through the action description information of the historical video frame and the significant object list, and the accuracy and stability of object interaction analysis in a variable environment are improved through unified mapping, cross-modal interaction and feature fusion in combination with the multi-scale visual feature and the language feature sequence.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an object interaction analysis method and device based on visual features, equipment and a medium. BACKGROUND

[0002] In the field of object interaction analysis of egocentric videos, existing technologies generally rely on neural network-based visual information extraction methods. Common solutions include three-dimensional convolutional neural networks and feature modeling based on spatiotemporal attention mechanisms. These methods usually directly process fixed-length video frame sequences to extract visual features for short-time object interaction prediction. However, in actual applications, existing technologies generally lack effective expression of "action context", and cannot accurately capture the influence of continuous actions in historical video frames and object interaction processes on current prediction tasks, making it difficult for the system to fully understand the semantic logic of user operations. For example, in a kitchen environment, performing actions such as "cutting vegetables", the user's historical actions and the relationship with key objects such as knives and cutting boards jointly determine the operation intent, and it is difficult to restore the complete semantic information chain by relying on a single visual frame sequence.

[0003] At the same time, visual information in complex environments is easily limited by interference, and the high dependence of existing methods on visual input makes the system have the problem of missing important information in the case of cluttered background, object occlusion or severe light changes. Especially in scenarios involving rare or long-tail object interactions, traditional methods have weak generalization ability and are prone to miss key interaction processes, seriously affecting the accuracy and stability of interaction prediction.

[0004] In the field of financial technology business, interaction behavior analysis based on video data is widely used in scenarios such as counter service handling, intelligent terminal operation monitoring, and remote identity verification. Existing systems focus more on extracting visual features from local video segments, lacking a systematic understanding of the joint information of historical operation steps and key objects in the physical environment, resulting in insufficient recognition of user's true intentions when the business process is complex, the combination of items is diverse, or abnormal operation behavior occurs, affecting the level of intelligent risk control and business compliance.

[0005] In the field of medical and health business, video interaction analysis has been used for surgical assistance monitoring, diagnosis and treatment process recording, and patient rehabilitation training evaluation. However, existing technologies mainly focus on static visual information extraction of video frames, and fail to effectively integrate historical action trajectories and item interaction relationships in the medical environment, making it difficult for the system to timely and accurately capture interaction behaviors closely related to medical safety when dealing with complex surgical operations, medical instrument occlusion, or environmental changes, limiting the improvement of assisted decision-making and risk warning capabilities. SUMMARY

[0006] The main purpose of the present application is to provide a visual feature-based object interaction analysis method, device, equipment and storage medium, aiming to solve the technical problems that the prior art fails to effectively fuse visual features and action context information, lacks unified expression and correlation analysis for object interaction prediction tasks, and leads to insufficient interaction understanding and future prediction accuracy in complex scenes.

[0007] To achieve the above-mentioned purpose, the present application provides a visual feature-based object interaction analysis method, comprising: obtaining a prediction frame of video data, extracting multi-scale visual features of the prediction frame, and detecting significant objects in the prediction frame to form a significant object list; generating an initial action description pair based on historical video frames, and aggregating the same initial action description pairs of consecutive frames in the historical video frames to generate an aggregated action description pair sequence; combining the aggregated action description pair sequence and the significant object list to generate a textual action context; dividing the visual features of each scale in the multi-scale visual features into image blocks and adding position encoding to generate a multi-scale visual feature sequence; encoding the textual action context using a pre-trained semantic encoding model to generate a language feature sequence; for each scale, splicing the visual feature sequence and the language feature sequence of the scale, and mapping the splicing result to a unified dimension to form a mixed input sequence of the corresponding scale; passing each scale's mixed input sequence through a multi-layer self-attention mechanism for cross-modal interaction to obtain a multi-scale fusion feature sequence; reorganizing the multi-scale fusion feature sequence into a multi-scale fusion feature map; based on the multi-scale fusion feature map, generating analysis results of the spatial position distribution, action semantic category and time contact feature of the future interaction object.

[0008] Further, to achieve the above-mentioned purpose, the present application provides a visual feature-based object interaction analysis device, comprising: a visual perception module for obtaining a prediction frame of video data, extracting multi-scale visual features of the prediction frame, and detecting significant objects in the prediction frame to form a significant object list; an action semantic extraction module for generating an initial action description pair based on historical video frames, and aggregating the same initial action description pairs of consecutive frames in the historical video frames to generate an aggregated action description pair sequence; a context construction module for combining the aggregated action description pair sequence and the significant object list to generate a textual action context; a visual coding module, configured to respectively divide the visual features of each scale in the multi-scale visual features into image blocks and add position encoding, to generate a multi-scale visual feature sequence; a language coding module, configured to encode the textual action context using a pre-trained semantic coding model, to generate a language feature sequence; a fusion splicing module, configured to, for each scale, splice the visual feature sequence and the language feature sequence of the scale, and map the splicing result to a uniform dimension to form a mixed input sequence of the corresponding scale; a cross-modal interaction module, configured to respectively perform cross-modal interaction on the mixed input sequence of each scale through a multi-layer self-attention mechanism, to obtain a multi-scale fusion feature sequence; a feature recombination module, configured to recombine the multi-scale fusion feature sequence into a multi-scale fusion feature map; an interaction analysis module, configured to generate an analysis result of the spatial position distribution, the action semantic category and the time contact feature of the future interactive object based on the multi-scale fusion feature map.

[0009] Further, to achieve the above object, the present application further provides a computer device, which comprises a memory, a processor and a visual feature-based object interaction analysis program stored in the memory and executable on the processor, and the visual feature-based object interaction analysis program, when executed by the processor, implements the steps of the visual feature-based object interaction analysis method.

[0010] Further, to achieve the above object, the present application further provides a computer readable storage medium, which stores a visual feature-based object interaction analysis program, and the visual feature-based object interaction analysis program, when executed by a processor, implements the steps of the visual feature-based object interaction analysis method.

[0011] Beneficial Effects: This invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as fintech and healthcare. It discloses a method, apparatus, device, and medium for object interaction analysis based on visual features, comprising: acquiring predicted frames of video data; extracting multi-scale visual features from the predicted frames; detecting salient objects in the predicted frames to form a list of salient objects; generating initial action description pairs based on historical video frames; aggregating consecutive frames of the same action description pair to form an aggregated action description pair sequence; combining the aggregated action description pair sequence with the list of salient objects to generate a textualized action context; dividing the multi-scale visual features into image blocks and adding positional encoding to generate a multi-scale visual feature sequence; encoding the textualized action context using a pre-trained semantic encoding model to generate a linguistic feature sequence; for each scale, concatenating the visual feature sequence and the linguistic feature sequence and mapping them to a unified dimension to form a hybrid input sequence; performing cross-modal interaction through a multi-layer self-attention mechanism to obtain a multi-scale fusion feature sequence; recombining the multi-scale fusion feature sequence to form a multi-scale fusion feature map; and generating analysis results of the spatial location distribution, action semantic category, and temporal contact features of future interactive objects based on the multi-scale fusion feature map. This invention generates textualized action context by combining action description information from historical video frames with a list of salient objects. By combining multi-scale visual features and language feature sequences, and through unified mapping, cross-modal interaction, and feature fusion, it effectively enhances the semantic understanding and key interaction information extraction capabilities in complex scenes, thereby improving the accuracy and stability of object interaction analysis in changing environments. Attached Figure Description

[0012] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for an object interaction analysis method based on visual features according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the object interaction analysis method based on visual features of the present invention. Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the object interaction analysis device based on visual features of the present invention. Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0013] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0014] The object interaction analysis method based on visual features provided in this invention can be applied to, for example...Figure 1 In an application environment of the client and the server, the client communicates with the server through a network. The server can obtain a predicted frame of video data through the client, extract a multi-scale visual feature of the predicted frame, detect a significant object in the predicted frame to form a significant object list, generate an initial action description pair based on historical video frames, aggregate continuous frames of the same action description pair to form an aggregated action description pair sequence, generate a textual action context by combining the aggregated action description pair sequence and the significant object list, divide the multi-scale visual feature into image blocks and add position encoding to generate a multi-scale visual feature sequence, encode the textual action context using a pre-trained semantic encoding model to generate a language feature sequence, splice the visual feature sequence and the language feature sequence for each scale, map to a unified dimension to form a hybrid input sequence, perform cross-modal interaction through a multi-layer self-attention mechanism to obtain a multi-scale fusion feature sequence, reorganize the multi-scale fusion feature sequence to form a multi-scale fusion feature map, and generate an analysis result of a spatial position distribution, an action semantic category, and a time contact feature of a future interactive object based on the multi-scale fusion feature map. The application generates a textual action context by combining the action description information of the historical video frames and the significant object list, combines the multi-scale visual feature and the language feature sequence, and performs unified mapping, cross-modal interaction, and feature fusion, which effectively enhances the semantic understanding ability and key interaction information extraction ability in a complex scene, and improves the accuracy and stability of object interaction analysis in a variable environment. The client can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers. The application will be described in detail through specific embodiments.

[0015] Please refer to Figure 2 , Figure 2 The flowchart of an embodiment of the object interaction analysis method based on visual features provided by the application is shown. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in an order different from that shown.

[0016] As Figure 2 shown, the object interaction analysis method based on visual features provided by the application includes the following steps: S10, obtaining a predicted frame of video data, extracting a multi-scale visual feature of the predicted frame, and detecting a significant object in the predicted frame to form a significant object list; In this embodiment, a prediction frame of video data is obtained, which can be sourced from any device with video capture function. Video data refers to a sequence of continuous visual information, usually composed of multiple single-frame images in chronological order, which can be obtained through sensors, cameras, mobile terminals or monitoring systems. The prediction frame refers to the last frame of image in the video data sequence, which is the image picture corresponding to the target time point for inference or analysis. Its acquisition method can be directly intercepting the tail frame image in the video sequence through time index, or extracting the corresponding frame image after setting logical trigger conditions, such as reaching a certain time threshold or receiving external analysis request. The existence of the prediction frame provides the basis for visual information in the current environment, supporting subsequent data processing.

[0017] Multi-scale visual features of the prediction frame are extracted. Visual features refer to numerical representations of image content, including spatial structure, texture boundary, color information and other dimensions, which are usually extracted through image encoding networks. The meaning of multi-scale is to construct feature representations of different spatial resolution levels for the same prediction frame. This can be achieved by constructing image pyramids or using different scale downsampling structures in neural network architecture. Taking convolutional neural network as an example, different depths of convolution and pooling layers can be concatenated to extract high-resolution detailed information and low-resolution global information, forming a multi-scale visual feature combination. Multi-scale visual features help to capture both local fine information and overall structure information in the image, improving the accuracy of subsequent object detection and environment understanding.

[0018] Significant objects in the prediction frame are detected. Significant objects refer to object regions in the image picture determined based on visual saliency or semantic importance, usually including objects with obvious appearance characteristics, related to user interaction or with inference reference value. The detection process can be based on visual features combined with a pre-defined object class set, implemented through classification networks, object detection models or semantic segmentation methods. The pre-defined object class set can be flexibly configured according to specific application scenarios, including common tools in kitchen environment, office supplies in financial scenarios, medical equipment in medical settings, etc. By setting the form of the class set, the relevance and application adaptability of the detection range are ensured. The detection operation usually includes region proposal, class determination and bounding box positioning, and the generated results include object class, spatial position, size range and other parameter information.

[0019] A list of salient objects is formed, which refers to organizing all the detected salient object information into a data set according to a fixed structure for subsequent calling, usually in the form of a list, array or structure. Each object information includes parameters such as class identifier, spatial position, size range, confidence score, etc. The list structure facilitates parallel analysis and multi-task cooperative processing of multiple objects in subsequent processing procedures, and can be flexibly expanded according to task requirements to support data adaptation requirements in different scenarios.

[0020] In the specific implementation process, the predicted frame of video data can be obtained by configuring a high frame rate video acquisition device, transmitting the continuous image data to the analysis end in real time, and extracting the latest frame as the predicted frame based on the system timestamp or business logic condition. When extracting multi-scale visual features, a deep learning-based image coding network can be used, such as a residual structure or a pyramid network, to set different down-sampling rates to generate multi-level feature maps, respectively expressing the local details and global structure information of the predicted frame. The implementation of detecting salient objects can be based on a general object detection model, such as a single-stage detection structure or a two-stage detection structure, by loading and applying an object class set associated with the application scenario, and combining the visual feature output to obtain the corresponding object class and position information. In the process of forming the salient object list, the detection results can be stored in a structured data format, such as a two-dimensional array or a dictionary structure, to ensure the integrity and structured expression of the object information, facilitating subsequent information calling and correlation analysis.

[0021] The acquisition method of the predicted frame can be adjusted according to different application scenarios. For real-time inference requirements in dynamic environments, a higher frequency video data acquisition mechanism can be used to improve the timeliness of the predicted frame. The structure depth or width parameters of the image coding network can also be adjusted to balance the detail retention capability and computational efficiency of multi-scale visual features. For different complexity of the environment background, the size and specific content of the object class set can be flexibly configured to enhance the generalization ability of the detection system in an open scene and improve the recognition accuracy of uncertain object classes. In the data structure design of the salient object list, additional fields can be extended according to business requirements, such as adding tracking numbers, interaction history information or object state indicators, to adapt to more complex multi-task processing scenarios.

[0022] The present embodiment effectively improves the simultaneous perception ability of local details and overall structure in complex environments by constructing multi-scale visual information representation, and enhances the target extraction accuracy of the system in different application environments by combining the salient object detection method based on the class set. The structured organization form of the salient object list ensures efficient management and task calling of multi-object information, and improves the stability and expansion ability of the overall data processing flow.

[0023] S20, generating initial action description pairs based on historical video frames, and aggregating the same initial action description pairs of consecutive frames in the historical video frames to generate an aggregated action description pair sequence; In this embodiment, initial action description pairs are generated based on historical video frames, which refer to consecutive image frame data in a video sequence earlier than the current time point, usually including dynamic information related to the user or the environment, and the source can be data obtained from live video monitoring, mobile device shooting or wearable camera equipment. The process of generating initial action description pairs first requires semantic understanding of the content in each frame of image. Specifically, frame-level description information can be extracted by a pre-trained image captioning model. Frame-level description is a language expression reflecting the key action and object information in the image frame, usually in the form of natural language phrases or structured text. To ensure the structural standardization of the description, part-of-speech tagging method can be used for syntactic analysis of the frame-level description to identify the combination of verbs and nouns. Verbs reflect behavior or state changes, and nouns refer to objects or scenes involved in the behavior or related to the state. The combination of the two forms a complete semantic expression, which is constructed as an initial action description pair. The initial action description pair embodies the direct semantic relationship between actions and objects in the user or the environment, with clear semantic structure and strong interpretability.

[0024] The same initial action description pairs of consecutive frames in the historical video frames are aggregated. Consecutive frames refer to a sequence of video frames that are adjacent and sequentially continuous in the time axis. Whether the initial action description pairs in consecutive frames are the same can be determined by text matching, semantic similarity analysis or rule retrieval method. If the initial action description pairs extracted from consecutive frames are completely consistent in semantic or textual expression, it means that these frames reflect the same continuous action or interactive state, with time consistency and behavior coherence. These consecutive frames with the same initial action description pairs are divided into the same action segment. The existence of the action segment helps to integrate dynamic information in the time dimension and form high-level semantic expression.

[0025] The aggregated action description pair sequence is generated. The representative initial action description pairs of each action segment are extracted, and these description pairs are combined in chronological order to form the aggregated action description pair sequence. This sequence not only retains the structural information of time continuity, but also expresses the semantic association between actions and objects through the description pairs, which is convenient for subsequent semantic reasoning and behavior prediction.

[0026] In practical applications, historical video frames can be dynamically obtained from a video data stream by setting a time window or a buffer. The length of the time window can be adjusted according to task requirements, and is suitable for environmental analysis requirements of different complexities. When generating an initial action description pair, a deep learning-based image caption generation model can be used to output a frame-level description conforming to natural language expression through a visual encoding and language decoding structure. In combination with part-of-speech tagging techniques such as conditional random fields and dependency syntax analysis, verbs and noun combinations in the description can be accurately extracted to improve the semantic expression accuracy of actions and objects. When judging whether the action description pairs of consecutive frames are the same, text fingerprint technology, semantic embedding space matching, or semantic similarity judgment of a large-scale pre-trained language model can be combined to enhance the robustness of the system to expression differences, synonymous expressions, or structural deformations.

[0027] By extracting joint information of actions and objects in historical video frames, the embodiment forms an initial action description pair with clear structure, and constructs a sequence of aggregated action description pairs with time continuity and clear semantics by combining semantic consistency analysis of consecutive frames. This can effectively capture historical behavior trajectories and environmental interaction states, improve the perception and reasoning ability of the system for user intent, environmental dynamics, and potential behavior patterns, reduce semantic understanding bias in complex environments, and enhance the accuracy of behavior prediction and scene analysis.

[0028] S30, generating a textual action context in combination with the sequence of aggregated action description pairs and the list of significant objects; In this embodiment, a textual action context is generated in combination with the sequence of aggregated action description pairs and the list of significant objects. The sequence of aggregated action description pairs is a set of language information reflecting the association between actions and objects in historical video frames arranged in chronological order, derived from the structured integration of initial action description pairs in historical video frames, and embodies time continuity and semantic logic. The list of significant objects is a set of objects detected in the predicted frame that have potential interaction value or task relevance, usually containing object categories, spatial positions, appearance features, etc., forming a standardized structure to ensure consistency and accuracy in subsequent processing.

[0029] Generating a textual action context specifically means expressing the information in the sequence of aggregated action description pairs and the list of significant objects as text content conforming to natural language structure. Textual representation can enhance the clarity and logic of semantic expression, making it easier for downstream models or reasoning modules to efficiently understand and process. The textual action context integrates action information in the time dimension and object information in the space dimension, constructing a complete environmental semantic description, which helps to overcome the limitations of single information source and improves the semantic understanding ability of the system in complex scenarios.

[0030] In the implementation process, each action description pair in the aggregated action description pair sequence can be first converted into a phrase or sentence in a language expression, while maintaining the integrity and accuracy of the actions, objects and their semantic relationships. At the same time, each object category information in the salient object list is also converted into a unified language expression to avoid semantic ambiguity or inconsistent expression. The two types of information are connected through natural language generation methods or template combination techniques to form a complete text that conforms to the language logic, ensuring the completeness, coherence and information richness of the expression content.

[0031] In actual implementation, the aggregated action description pair sequence can be converted into a language expression combined with actions and objects one by one through sequence traversal and text template matching, such as converting the "pick up-cup" description pair into the language phrase "user picks up the cup". The object information in the salient object list can be expressed as "phone on the table" and "fruit on the dining table" through the combination of object categories and context structures, which conforms to the context of the text content.

[0032] The fusion of the two types of information can use a static template structure to integrate the action and object information through a fixed language template to ensure the expression is standardized and unified. It can also use a large-scale language model-based generation method to input the aggregated action description pair sequence and the salient object list to generate a complete text content with natural language expression ability, improving the flexibility and semantic richness of the expression.

[0033] For different application environments, the detail level and information density of the textual expression can be adjusted. For example, in a high information security requirement scenario, the object category, location or state information can be strengthened in the text expression to facilitate system monitoring and early warning; in a high user experience requirement environment, the interaction logic and environment association can be highlighted in the text expression to enhance the system's expression ability and reasoning accuracy of the semantic chain.

[0034] By expressing the aggregated action description pair sequence and the salient object list as a textual action context, the system can effectively integrate the historical behavior information in the time dimension and the environmental object information in the space dimension, construct a text expression with clear structure and complete semantics, improve the accuracy of scene semantic understanding, enhance the system's perception ability of user intent, environmental state and potential interaction relationship, reduce the risk of semantic ambiguity and information omission in complex environments, and improve the overall analysis and reasoning level.

[0035] S40, dividing each scale of the multi-scale visual feature into an image block and adding a position code to generate a multi-scale visual feature sequence; In this embodiment, the visual features of each scale in the multi-scale visual features are respectively divided into image blocks and added with position encodings to generate a multi-scale visual feature sequence, wherein the multi-scale visual features refer to a set of image features obtained from different spatial resolution levels by a feature extraction network, which usually covers multi-level information from a low resolution, high semantic abstraction layer to a high resolution, detail-rich layer, ensuring that the system can capture both macro scene structure and local detail information. The visual features of each scale specifically refer to the feature map obtained for a single spatial resolution level, which is structurally represented as a two-dimensional or three-dimensional tensor, containing spatial dimensions and channel dimensions.

[0036] Dividing the visual features of each scale into image blocks means that the continuous spatial feature area is divided into independent, structured subunits according to a predetermined size. The division of image blocks helps to unify the data structure, facilitating subsequent serialization processing and cross-modal information fusion. Position encoding refers to generating additional encoding information based on the spatial position information of each image block in the overall visual features, which is used to supplement the expression of spatial position information and avoid the loss of spatial structure in the serialization process. Generating a multi-scale visual feature sequence means organizing the image blocks with position encodings into a linear sequence in a predetermined order. The serialized expression form facilitates the unified input of downstream sequence processing modules or cross-modal fusion structures, improving the standardization and structural clarity of overall information expression.

[0037] In the implementation process, the division size and sliding step of the image blocks can be set to ensure a balance between spatial information expression and computational efficiency for the division results of visual features of different scales. Generally, smaller image block sizes can be used for high resolution scales, and larger image block sizes can be used for low resolution scales to adapt to different levels of spatial information density. The generation of position encoding can be based on an absolute position encoding scheme, which generates corresponding numerical encoding by calculating the row and column indices of the image block in the overall feature map. Alternatively, it can be based on relative position encoding, which expresses the spatial relative position relationship between image blocks to enhance the overall perception ability of the system to spatial structure.

[0038] In the specific implementation process, for the visual features of each scale, they can be divided into fixed-size rectangular image blocks based on the row and column slicing method, ensuring that each image block contains continuous and complete spatial information fragments. After division, position encodings are generated based on the top-left corner coordinates or center coordinates of each image block. The encoding form can be a numerical vector, a position embedding matrix, or high-dimensional position mapping information. The position encoding and the visual features inside the image block are weighted and superimposed, directly spliced, or fused through a mapping function to ensure that the spatial position information and the visual feature information are jointly retained in the data structure of each image block.

[0039] All image blocks are organized into a linear sequence in row-major or column-major order, forming a single-scale visual feature sequence. Visual feature sequences at different scales can be independently organized to preserve the information expression structure of different resolution levels, or can be formed into an overall multi-scale visual feature sequence through cascading or multi-channel fusion, facilitating subsequent unified processing and information integration.

[0040] In different production environments, the size, position encoding form, and sequence organization structure can be flexibly adjusted according to system computing resources, task requirements, and target scenarios. For example, in an environment sensitive to spatial structure, high-precision position encoding and fine-grained image block division can be used to enhance spatial expression capability; in a resource-limited or high-demand information compression scenario, low-dimensional position encoding and large-size image blocks can be used to reduce computational complexity and data size.

[0041] By dividing multi-scale visual features into image blocks and adding position encoding, the system can preserve spatial structure information while expressing in sequence, avoiding spatial information loss and structure disorder caused by sequence structure, improving the system's expression capability for complex scene spatial relationships and multi-level visual information, enhancing the overall accuracy and stability of downstream tasks in the process of spatial positioning, structure understanding, and information fusion, and optimizing the system's expression and processing effect for complex visual input.

[0042] S50, encoding the textual action context using a pre-trained semantic encoding model to generate a language feature sequence; In this embodiment, a pre-trained semantic encoding model is used to encode the textual action context to generate a language feature sequence. The textual action context refers to structured natural language information expression based on the sequence and significant object list constructed from aggregated action descriptions, which typically includes semantic text content related to historical actions, object information, and interaction relationships, with clear semantic logic and information induction structure. The semantic encoding model refers to an encoding network that has been trained based on large-scale corpus or multi-domain data and has stable semantic understanding capability. Common structures include but are not limited to self-attention mechanism-based encoders, bidirectional context-aware models, or multi-level semantic expression networks. This model can efficiently extract semantic information from text and convert it into high-dimensional feature expression.

[0043] The encoding process refers to inputting the textual action context into the semantic encoding model, generating structured semantic feature expression based on the model's internal feature extraction and expression mechanism. The language feature sequence is a sequential expression form of the encoding result, usually represented as a fixed-dimensional vector sequence, with the ability to preserve the original semantic information, structure logic, and content association of the text, facilitating subsequent fusion with other modal information or further information reasoning.

[0044] The input format of the textual action context can be natural language text, structured sentence sequence or standardized text data. The input end of the semantic encoding model is adapted to the input format according to the design requirements of the model, and the text data is usually converted into a numerical expression form through word vectors, character encoding or sub-word units, so as to facilitate feature extraction and expression operations inside the model.

[0045] In the specific implementation process, a pre-trained large semantic encoding model can be selected, such as a language encoding network based on a transformer architecture. The textual action context is input into the input end of the model, and after multiple layers of semantic feature extraction and context information fusion operations, the corresponding language feature sequence is output. Before text input, preprocessing operations such as word segmentation, removing redundant information and standardized expression can be performed to improve encoding efficiency and expression accuracy.

[0046] The generated language feature sequence can retain the action information, object information and interaction logic in the textual action context. Each element in the sequence corresponds to a specific word, phrase or semantic unit in the input text. The overall sequence structure retains the original order and logical relationship of the text, facilitating alignment, fusion and joint analysis with visual information or other information sequences.

[0047] In different application environments, the type, parameter size and training data source of the semantic encoding model can be flexibly adjusted according to system requirements and application scenarios. For example, in complex tasks with high semantic understanding requirements, a large parameter size and rich training data encoding model can be selected to improve semantic expression capability; in scenarios with limited computing resources or high real-time requirements, a lightweight encoding network can be selected to balance expression capability and computing efficiency.

[0048] This embodiment encodes the textual action context using a pre-trained semantic encoding model to generate a language feature sequence. The system can efficiently obtain high-level semantic expressions related to historical actions, object information and interaction logic, improve the clarity of information structure expression and semantic integrity, enhance the information association capability and semantic understanding capability in the downstream information fusion, reasoning and analysis process, and optimize the overall expression effect and analysis accuracy of the system in complex task environments.

[0049] S60, for each scale, splicing the visual feature sequence and the language feature sequence of the scale, and mapping the splicing result to a unified dimension to form a mixed input sequence of the corresponding scale; In this embodiment, for each scale, the visual feature sequence and the language feature sequence of the scale are spliced and mapped to a unified dimension to form a mixed input sequence of the corresponding scale, wherein the visual feature sequence of the scale refers to a structured sequence representation formed by image block division and position coding addition for a specific scale in the multi-scale visual feature, which has visual information and spatial position information of the scale, and the language feature sequence refers to a semantic information representation sequence generated by encoding the textual action context by a semantic coding model, which contains textual information features related to historical actions, object interactions and semantic logic.

[0050] Splicing refers to a sequence-level combination operation of visual feature sequences and language feature sequences of the same scale at the feature level, which is usually represented as connection of sequence dimensions or feature dimensions, to ensure unified expression of visual information and language information in structure, facilitating subsequent joint analysis and interaction information fusion. The result of splicing can be represented as a joint feature sequence, the structure length of which is equal to the sum of the lengths of the visual feature sequence and the language feature sequence, and the sequence contains interleaved feature units from image modalities and language modalities, forming a multi-modal structured input basis. In this sequence, the visual part retains spatial perception ability and multi-scale positioning expression, while the language part embeds behavior context and interaction logic, and the splicing result of the two at the sequence level constitutes a unified expression carrier with significant cross-modal correlation characteristics, serving the fusion operation in the subsequent attention mechanism.

[0051] Mapping to a unified dimension means that for the spliced mixed information sequence, linear projection, feature transformation or other expression methods are used to uniformly adjust different source information to the same feature dimension, to ensure consistency and structural integrity of information expression, facilitating subsequent joint modeling, information fusion and cross-modal reasoning of the system.

[0052] The mixed input sequence of the corresponding scale refers to a structured expression sequence independently formed for each scale, which contains visual feature information, spatial position information and textual semantic information of the scale, and the overall structure has information integrity, clear expression and multi-dimensional information collaboration characteristics, facilitating subsequent multi-layer self-attention mechanism processing and information interaction analysis.

[0053] In the specific implementation process, the system can independently extract the corresponding visual feature sequence and language feature sequence for each scale in the multi-scale visual feature, and combine the two types of information to form a mixed sequence by sequence-level splicing. The splicing operation can be realized by sequence connection, feature dimension expansion or other sequence processing mechanisms, to ensure the consistency of information structure and the logical clarity of combined expression.

[0054] After splicing, the system sets a unified feature dimension standard, uses linear transformation, feature mapping network or other parameter-learnable mapping structure to adjust the mixed information sequence to the preset unified dimension, ensures that all scale mixed input sequences in the system have the standard of structural unity and consistent feature expression, optimizes information fusion efficiency and overall expression effect of the system.

[0055] In specific applications, the mapping operation can be adapted to different system requirements, computing resources and information expression requirements through different parameter settings, network structures or transformation methods, improving the expression flexibility and application scope of the system.

[0056] This embodiment can efficiently realize structural unity and feature fusion of multi-source information by splicing visual feature sequences and language feature sequences for each scale and mapping to a unified dimension to form a mixed input sequence of the corresponding scale, improving the collaborative expression capability of the system for visual information, spatial information and semantic information, optimizing the overall effect of subsequent cross-information interaction, information reasoning and joint analysis, and enhancing the expression integrity, information fusion efficiency and reasoning accuracy of the system in complex information environment.

[0057] S70, the mixed input sequence of each scale is respectively interacted through a multi-layer self-attention mechanism to obtain a multi-scale fusion feature sequence; In this embodiment, the mixed input sequence of each scale is respectively interacted through a multi-layer self-attention mechanism to obtain a multi-scale fusion feature sequence, wherein the mixed input sequence refers to the unified structural expression sequence formed by splicing and mapping of visual feature sequences and language feature sequences for each scale in the previous step, which contains visual information, spatial position information and text semantic information, and has the characteristics of multi-source information collaborative expression.

[0058] The multi-layer self-attention mechanism refers to a structure based on self-attention calculation, which realizes global dependency modeling, information dynamic weighting and semantic feature strengthening between information units in the sequence through multi-level, progressive information interaction and feature updating operations. It usually stacks multiple layers of self-attention calculation units to improve information expression capability, enhance information fusion depth and optimize the overall multi-information interaction effect of the system.

[0059] Cross-modal interaction refers to the structure advantage of self-attention mechanism in the mixed input sequence, which strengthens the associated expression between visual information, spatial information and semantic information by dynamically calculating the correlation and influence weight between different information units, improves the information fusion capability and joint semantic expression level of the sequence as a whole, and realizes deep fusion and collaborative reasoning of multi-source information.

[0060] The multi-scale fusion feature sequence refers to a sequence structure after multi-layer self-attention mechanism processing for each scale. The sequence has stronger information expression capability, higher information fusion degree, and more perfect multi-source collaborative expression effect. The overall structure can be applied to subsequent high-order reasoning, spatial structure reorganization, and task output processing of the system.

[0061] In the specific implementation process, the system can independently set the multi-layer self-attention mechanism processing structure for each scale, use a parameter-adjustable and structure-extensible attention calculation network, and progressively process the mixed input sequence to realize dynamic weighting of internal sequence information, global dependency modeling of cross-information units, and deep fusion of multi-source information.

[0062] The multi-layer self-attention mechanism can use a standard multi-head attention structure, an improved cross-modal attention module, or other information interaction frameworks based on sequence structure. In the specific implementation process, the system can flexibly adjust the number of layers, structure parameters, and information update strategies of the attention module according to information complexity, sequence length, and task requirements to ensure optimization of information interaction effects at different scales and expression integrity of the overall system.

[0063] In the information interaction process, the system can set separate attention update paths for visual information, spatial information, and semantic information, or use a unified information fusion framework to realize collaborative update, joint expression, and fusion optimization of the overall structure among information categories, thereby improving the expression robustness and information integrity of the sequence in a complex information environment.

[0064] In this embodiment, the mixed input sequence of each scale is interacted through the multi-layer self-attention mechanism to effectively improve the joint modeling capability, information fusion depth, and expression integrity of the system for multi-source information, optimize the collaborative expression effect among visual information, spatial information, and semantic information, enhance the overall information reasoning capability and multi-dimensional information fusion level of the system, and improve the expression accuracy, information fusion efficiency, and analysis reliability of the system in a complex information environment.

[0065] S80, reorganizing the multi-scale fusion feature sequence into a multi-scale fusion feature map; In this embodiment, the multi-scale fusion feature sequence is reorganized into a multi-scale fusion feature map. The multi-scale fusion feature sequence refers to a sequence structure containing information expression of different scales formed by multi-layer self-attention mechanism interaction processing in the previous step. The sequence internally fuses visual information, spatial position information, and semantic information, and has multi-dimensional joint expression characteristics. Reorganization refers to restoring, arranging, and constructing spatial structure expression based on internal sequence information to form a multi-scale fusion feature map with two-dimensional spatial distribution characteristics, which facilitates subsequent information analysis and feature extraction based on spatial structure of the system.

[0066] Multi-scale refers to the feature expression process, the system independently constructs a multi-level information expression structure for different spatial resolutions, information granularity and semantic levels. Each scale corresponds to a set of feature expression results with different resolutions, different expression densities and different information depths. The fused feature map refers to the graph structure generated by spatial information recovery and structure rearrangement operation based on the multi-scale fused feature sequence, which has spatial layout information, expression integrity and information fusion effect. It has two-dimensional spatial distribution characteristics and information visualization advantages, which is convenient for spatial reasoning, semantic recognition and high-order information analysis in subsequent tasks.

[0067] In actual implementation, the system can first analyze the spatial position information, feature category information and information content expression in the sequence, based on the spatial position information, recover the two-dimensional spatial arrangement relationship of the feature elements, and reconstruct the spatial structure of the corresponding scale.

[0068] The system can determine the specific position of each feature element in the two-dimensional spatial structure by calculating the spatial index, row and column coordinates or based on the position coding analysis result, arrange the feature elements in the sequence according to the spatial distribution rule, and form a fused feature map with spatial continuity and information expression integrity.

[0069] For different scales, the system can independently perform the above spatial structure reconstruction operation to construct fused feature maps with different resolutions, information granularities and semantic levels, forming multi-scale information expression results, which facilitates the system to flexibly call information expression structures at different scales according to task requirements, and improves the overall information expression depth and multi-level information collaborative reasoning effect.

[0070] This embodiment can effectively recover the spatial structure in information expression by reorganizing the multi-scale fused feature sequence into a multi-scale fused feature map, improve the spatial visualization effect and spatial reasoning ability of information, optimize the collaborative layout expression among visual information, spatial information and semantic information, enhance the information understanding ability and multi-dimensional information fusion depth of the system in complex scenes, and improve the spatial position reasoning, semantic logic recognition and information correlation analysis effect in subsequent tasks.

[0071] S90, based on the multi-scale fused feature map, generates analysis results of spatial position distribution, action semantic category and time contact characteristics of future interactive objects.

[0072] In the embodiment, based on the multi-scale fusion feature map, the analysis results of the spatial position distribution, the action semantic category and the time contact feature of the future interactive object are generated, which involves inputting the multi-scale fusion feature map as information, obtaining three types of key information about the future interactive object through structured information interpretation and feature reasoning, including the spatial position distribution, the action semantic category and the time contact feature. The multi-scale fusion feature map is derived from the previous step and has different spatial scales, information levels and fusion expression effects, which can comprehensively reflect the joint expression information of objects, actions and spatial relationships in the scene.

[0073] The spatial position distribution refers to predicting the spatial position expression result of the future interactive object in the scene by reasoning the spatial information and object expression content in the multi-scale fusion feature map, which can be specifically the boundary box coordinates, spatial region distribution or specific position point information in the two-dimensional plane, and reflects the spatial positioning and layout reasoning ability of the system for the future interactive object.

[0074] The action semantic category refers to analyzing and reasoning the semantic label expression of the future interactive action based on the information content in the fusion feature map, which is specifically the semantic category information reflecting the action type, action content or action purpose, facilitating the understanding and expression of the system for the action type, operation logic and behavior intention in the future interactive scene.

[0075] The time contact feature refers to predicting the contact information of the future interactive object in the time dimension based on the time information, spatial layout and semantic association in the feature map during the reasoning analysis process, which is specifically the time offset, contact time or duration expression, reflecting the behavior characteristics of the future interactive object in the time sequence dimension.

[0076] In the actual implementation process, the system can input the multi-scale fusion feature map into the spatial position reasoning branch, the action semantic analysis branch and the time feature prediction branch through the multi-task information interpretation mechanism, and obtain the corresponding prediction results respectively.

[0077] The spatial position reasoning branch can predict the spatial position distribution of the future interactive object based on the spatial position information and fusion expression result in the feature map through convolution operation, spatial decoding structure or position regression algorithm, and output specific boundary box coordinates, position points or spatial region information.

[0078] The action semantic analysis branch can generate the semantic category label of the action by combining the spatial information and semantic expression in the multi-scale feature map through the feature extraction structure, semantic classification network or context semantic reasoning module, and accurately reflect the semantic logic and behavior content of the future interactive action.

[0079] The time feature prediction branch can be based on a time information coding structure, a sequence analysis module, or a time regression network, use dynamic information, spatial layout, and semantic content in the multi-scale fusion feature map to infer and predict the contact time feature of the future interactive object, and output a time offset or a time sequence behavior indicator.

[0080] The system can structurally combine the spatial position distribution, action semantic category, and time contact feature to form a unified analysis result expression, provide joint information output about the spatial position, semantic category, and time behavior of the future interactive object, and support information understanding and decision support of subsequent systems in interactive prediction, task planning, or operation assistance.

[0081] The embodiment can comprehensively integrate scene information, action logic, and time behavior expression by generating the analysis results of the spatial position distribution, action semantic category, and time contact feature of the future interactive object based on the multi-scale fusion feature map, improve the spatial layout prediction, semantic action recognition, and time behavior inference capability of the system for the future interactive scene, enhance the adaptability, information understanding depth, and operation prediction accuracy of the system for complex interactive scenes, and optimize the comprehensive expression effect of the system in interactive monitoring, behavior analysis, and task assistance.

[0082] In one embodiment, the above step S10 includes: S101, determining the last frame in the video data as a prediction frame; S102, processing the prediction frame through a convolutional neural network to generate a multi-scale feature map; S103, extracting multi-scale visual features containing different spatial resolutions from the multi-scale feature map; S104, obtaining a set of predefined object categories; S105, determining the similarity between the prediction frame and each object category in the set of predefined object categories using a visual language model; S106, selecting a preset number of object categories according to the similarity to form a salient object list containing the selected object categories.

[0083] In the embodiment, the prediction frame of the video data is obtained, which refers to locating a time node frame for interactive analysis in a continuous video data stream. The video data is usually a continuous image sequence, and the source can include a mobile device, a fixed monitoring system, a smart terminal, or other video acquisition devices. The prediction frame is determined by selecting the last frame in the video data. The last frame refers to the image frame at the end of the time dimension in the currently obtained video sequence, which reflects the prediction analysis requirement based on the latest environment state, avoids introducing future information, and maintains the rationality of the prediction scene.

[0084] Extracting multi-scale visual features involves using a convolutional neural network structure to perform layer-by-layer feature extraction on the predicted frame. The convolutional neural network is a neural network architecture with hierarchical feature expression capability, which can include multiple layers of convolution, pooling, normalization, and other structural units. Through the feature processing process of the multi-layer network structure, a multi-scale feature map is generated. The feature map is an intermediate data structure that expresses image information at different spatial resolutions. The multi-scale feature map reflects the visual expression of the same image at different perception ranges and different information levels, which helps the system to capture both fine-grained local information and global spatial relationships.

[0085] Extracting multi-scale visual features from the multi-scale feature map involves obtaining visual expressions that reflect image details, structural layouts, and global information at different levels or different output resolutions. This can be achieved through direct downsampling, upsampling, or specific scale aggregation strategies of the feature map. Multi-scale visual features provide a rich foundation of visual information to support subsequent significant object detection and interaction analysis.

[0086] Obtaining a predefined set of object categories involves storing or dynamically generating a set of object category information in the system for comparative analysis with the predicted frame. The object category set can be derived from domain knowledge bases, standard object dictionaries, or user-defined configurations, and the category content includes object names, semantic descriptions, visual reference information, etc.

[0087] Determining the similarity between the predicted frame and each object category in the predefined set of object categories using a visual language model. The visual language model is a joint expression structure that simultaneously processes image information and text semantic information. Common implementation forms include cross-modal contrast learning models, image-text encoding networks, etc. The similarity calculation is based on the multi-scale visual features of the predicted frame and the semantic information in the object category set. Through semantic alignment, feature mapping, or similarity measurement methods, numerical information reflecting the degree of semantic association between image content and object categories is output.

[0088] Selecting a preset number of object categories based on similarity to form a significant object list containing selected object categories. The preset number is an integer parameter configured by the system based on actual application requirements. The significant object list is a set of object categories that are highly relevant to the content of the predicted frame, reflecting key object information that may exist or will interact in the current scene. This process ensures that the output results have high relevance and discriminability through similarity sorting, threshold filtering, or probability reasoning methods, providing accurate object-level information for subsequent interaction analysis.

[0089] By selecting the last frame of the video data as the prediction frame, the embodiment can accurately reflect the latest visual state of the current environment, effectively avoiding analysis deviation caused by information lag. Based on the convolutional neural network, multi-scale feature maps are extracted and multi-scale visual features with different spatial resolutions are obtained, which can balance the expression of details and global structure information, and improve the accuracy of object detection and scene understanding. Combined with the pre-defined object category set and the visual language model, similarity calculation can be performed to effectively align the image content and semantic information. The selected significant object list focuses on the key objects closely related to the current scene, reduces information redundancy, improves the pertinence and discriminability of object information in the interactive analysis process, and enhances the comprehensive ability of the system to perceive and interact with objects in complex environments.

[0090] In one embodiment, the above step S20 comprises: S201, processing each frame in the historical video frame using a pre-trained image captioning model to generate a corresponding frame-level description; S202, analyzing the frame-level description by part-of-speech tagging, extracting verb-noun combinations and simplifying intermediate words to form initial action description pairs; S203, counting the frequency of occurrence of the initial action description pairs in the historical video frames; S204, selecting the initial action description pair with the highest frequency in each historical video frame as the representative action description pair of the historical video frame; S205, identifying continuous video frame sequences with the same representative action description pairs in the historical video frame sequence; S206, aggregating each continuous video frame sequence with the same representative action description pairs into an action segment; S207, combining all action segments in chronological order to form an aggregated action description pair sequence.

[0091] In the embodiment, the initial action description pairs are generated based on historical video frames, which refer to a set of video data earlier in time than the current analysis frame, and the sources include continuous video segments, buffer storage frame sequences, or past parts of real-time data streams. Each frame in the historical video frame is processed using a pre-trained image captioning model. The image captioning model is a structure that combines visual information and language generation capabilities. By inputting an image frame, it outputs text description information related to the image content. The model can use a joint architecture of visual encoding and language generation, and has the ability to extract semantic information from image data and generate language text. Frame-level description refers to independent text expression generated for each frame of image, reflecting action, scene, object or event information in the image.

[0092] The frame-level description is analyzed by part-of-speech tagging. Part-of-speech tagging is a basic analysis operation in natural language processing, which identifies the grammatical categories of words in the text, including verbs, nouns, adjectives, adverbs, etc. The verb-noun combination is extracted, which specifically refers to extracting verbs that describe actions and nouns that involve entities from the text, reflecting the association information between actions and objects, simplifying the intermediate words, removing irrelevant conjunctions, modifiers or structural words, and retaining the core expression of actions and objects to form initial action description pairs. The initial action description pair is a combination structure of action and object, which embodies the behavior semantic information in the video frame.

[0093] The frequency of the initial action description pair in the historical video frame is counted. The frequency is the number of times or the proportion of a certain description pair appearing in all historical video frames, which is used to measure the representativeness and importance of the description pair. The initial action description pair with the highest frequency in each historical video frame is selected as the representative action description pair of the historical video frame. The representative action description pair reflects the most semantic information or the most frequent behavior expression in the current frame, which helps to simplify information expression and highlight the core content.

[0094] The same continuous video frame sequence of the representative action description pair is identified in the historical video frame sequence. The continuous video frame sequence is a set of adjacent and uninterrupted video frames in time, and the same representative action description pair indicates that these frames express unified behavior semantic information. Each continuous video frame sequence with the same representative action description pair is aggregated into an action segment. The action segment is a time segment composed of several continuous video frames, with unified internal action expression and continuous semantics. All action segments are combined in time sequence to form an aggregated action description pair sequence. The time sequence is based on the timestamp or frame index of the video frame. The aggregated action description pair sequence is a time-ordered action expression sequence, reflecting the continuous behavior process and action change information in the video.

[0095] The embodiment can effectively extract the core action information in the video sequence by performing frame-level semantic extraction and language expression on the historical video frames, combining part-of-speech tagging and simplification operations. Through frequency statistics and representative selection, the refinement and reliability of information expression are improved, and redundant information interference is avoided. The aggregation operation of the continuous frame sequence preserves the behavior continuity in the time dimension, enhancing the system's ability to capture the action evolution process. The finally generated aggregated action description pair sequence accurately expresses the key behavior chain in the historical time period, providing a high-quality and structured language information basis for subsequent scene understanding, semantic reasoning and interaction prediction, and improving the overall behavior analysis accuracy and semantic expression ability of the system.

[0096] In one embodiment, the above step S40 includes: S401, for the visual features of each scale, dividing the visual features of the scale into image blocks of a preset size; S402, determine the spatial position coordinates of each image block; S403, generate position encoding according to the spatial position coordinates; S404, add the position encoding to the corresponding image block; S405, arrange the image blocks with added position encoding in spatial order to form a visual feature sequence of the corresponding scale; S406, combine the visual feature sequences of all scales to form a multi-scale visual feature sequence.

[0097] In this embodiment, multi-scale visual features refer to a collection of image features with different spatial resolutions or receptive field ranges obtained after feature extraction on the predicted frame. Typically, it includes high-resolution detail features and low-resolution semantic features. The source can be a convolutional neural network, a feature pyramid network, or other multi-scale structures. The visual features of each scale are two-dimensional matrices or tensors at the corresponding resolution, representing the spatial structure and content information of the image at that scale.

[0098] Divide the visual features of each scale into image blocks of a preset size. An image block refers to a small area of fixed size divided in the visual feature matrix. The preset size is determined according to application requirements, network structure, or hardware constraints. The division operation can use sliding windows, uniform segmentation, etc. The purpose is to cut the overall feature information into local units for subsequent serialization processing and the introduction of position encoding.

[0099] Determine the spatial position coordinates of each image block. The spatial position coordinates are two-dimensional position identifiers of the image block in the visual feature matrix, which can include horizontal position, vertical position, hierarchical index, or other spatial information. They are derived from the matrix index or physical position mapping of each image block in the division process.

[0100] Generate position encoding according to the spatial position coordinates. Position encoding refers to the conversion of spatial position information into a vector representation that can be recognized and utilized by the model. Common methods include sine function encoding, learnable position embedding, or other numerical representations. Position encoding is used to supplement the spatial information of the image block to avoid losing spatial structure during serialization.

[0101] Add the position encoding to the corresponding image block. The addition operation can be vector concatenation, element addition, or other fusion methods. The image block with added position encoding has both local content information and spatial position representation, making it easier for downstream models to capture spatial relationships.

[0102] The image blocks with position coding are arranged in spatial order to form a visual feature sequence of a corresponding scale, and the spatial order is determined according to the spatial position coordinates of the image blocks, and a one-dimensional sequence is formed by using row priority, column priority or other sequencing rules, which facilitates subsequent information processing by combining a sequence modeling structure.

[0103] The visual feature sequences of all scales are combined to form a multi-scale visual feature sequence, and the combination operation is to splice the visual feature sequences of different scales according to scale indexes, resolution information or other rules to form a sequence expression with unified overall structure and clear hierarchy, which reflects multi-scale information fusion and spatial hierarchical structure.

[0104] The embodiment fully retains the spatial structure and multi-level information expression of the image by dividing the multi-scale visual features into image blocks and coding the positions, and solves the problem of spatial information loss in the sequencing process. The image block division operation improves the processing granularity of local information, which helps to capture detailed features and local changes. The introduction of position coding supplements the spatial position information and enhances the understanding ability of the model to structural relationships. The construction of the multi-scale visual feature sequence realizes the unified expression and efficient combination of information of different scales, and improves the performance of the system in subsequent multi-level information processing and complex structure reasoning.

[0105] In one embodiment, the above step S70 comprises: S701, for each scale, input the mixed input sequence of the scale into a multi-layer self-attention module; S702, in the multi-layer self-attention module, determine the attention weight between the visual feature sequence element and the language feature sequence element; S703, update the visual feature sequence element according to the attention weight to generate an updated visual feature sequence; S704, update the language feature sequence element according to the attention weight to generate an updated language feature sequence; S705, combine the updated visual feature sequence and the updated language feature sequence to generate an updated feature sequence; S706, iteratively process the updated feature sequence by the multi-layer self-attention module; S707, take the iteratively processed feature sequence as a fusion feature sequence of a corresponding scale; S708, combine the fusion feature sequences of all scales to form a multi-scale fusion feature sequence.

[0106] In this embodiment, the mixed input sequence is a structure formed by splicing the visual feature sequence and the language feature sequence in a unified dimension, which combines image information and semantic information and has multi-source data fusion expression capability. The mixed input sequence of each scale corresponds to different levels in the multi-scale visual feature structure, expressing image and language joint information at different spatial resolutions.

[0107] The multi-layer self-attention module is a hierarchical stacked structure that realizes global information interaction and feature update between sequence elements through multiple self-attention calculations. It is derived from the transformer structure or improved attention mechanism design and has cross-modal information fusion and dependency capture capability.

[0108] In the multi-layer self-attention module, the visual feature sequence elements and the language feature sequence elements are different constituent units in the input sequence, and the correlation between them is dynamically calculated through the attention mechanism. The attention weight reflects the dependency strength and information importance between different elements, which is usually realized through a similarity function or a parameter matrix transformation.

[0109] According to the attention weight, the visual feature sequence elements and the language feature sequence elements are respectively updated, and the update operation means that the information contributed by other sequence elements is fused on the basis of preserving the original information, so as to improve the global consistency and cross-modal collaboration effect of the overall expression. The updated visual and language feature sequences respectively contain joint enhanced expression of image structure information and semantic content information.

[0110] The updated visual feature sequence and the language feature sequence are combined to generate an updated feature sequence. The combination operation can be splicing, superposition or other fusion methods to form a comprehensive sequence expression with unified structure and rich information, which is convenient for subsequent unified modeling and further information interaction.

[0111] Through the iterative processing of the multi-layer self-attention module, the updated feature sequence is input into the self-attention structure multiple times, and the high-order correlation between different elements in the sequence and the cross-modal semantic integration are enhanced layer by layer. The iterative process realizes multi-round aggregation and deep fusion of information flow.

[0112] The feature sequence after iterative processing is used as the fusion feature sequence of the corresponding scale. The fusion feature sequence retains the original visual and language information and has stronger joint expression capability after multi-layer interaction, which is suitable for multi-modal feature modeling and task reasoning at different scale levels.

[0113] The fusion feature sequences of all scales are combined to form a multi-scale fusion feature sequence. The combination operation integrates multi-modal expression of different spatial resolutions and level information, ensuring the overall consistency and expression richness of the system when processing multi-scale structure information and complex semantic relationships.

[0114] The embodiment realizes the deep fusion of visual and language information within the sequence through the multi-layer self-attention mechanism, significantly enhances the consistency and semantic collaboration ability of multi-scale information expression, and improves the integration and analysis level of the system in complex scenes. Through iterative processing and high-order interaction, the problem of isolated cross-modal information in traditional sequence modeling is solved, the effect of semantic association capture and global dependency modeling is improved, and the multi-scale fusion feature sequence is ensured to have rich structure information and semantic expression ability, meeting the subsequent high-precision multi-modal analysis and task prediction needs.

[0115] In one embodiment, the above step S80 comprises: S801, for each scale of the multi-scale fusion feature sequence, analyzing the spatial position information in the fusion feature sequence; S802, reconstructing a two-dimensional spatial structure according to the spatial position information; S803, arranging the feature elements in the fusion feature sequence according to the reconstructed two-dimensional spatial structure, performing accurate positioning and arrangement of the bottom layer high-resolution features, and performing semantic reasoning arrangement of the high layer low-resolution features, to form a fusion feature map corresponding to the scale; S804, combining all scale fusion feature maps to form a multi-scale fusion feature map.

[0116] In the embodiment, the multi-scale fusion feature sequence is composed of fusion feature sequences of different scale levels, and the fusion feature sequence combines visual information and semantic information, has cross-modal expression ability, and is derived from joint information expression of different spatial resolutions or different structure levels. Each scale of the fusion feature sequence contains the image block features and position encoding information after the sequence, which has the basis for restoring the spatial structure.

[0117] Analyzing the spatial position information in the fusion feature sequence refers to extracting the corresponding position of each feature element in the original space based on the spatial position encoding or index information carried in the sequence. The spatial position information is usually represented by position encoding, grid index, coordinate mapping, etc., to ensure that the sequenced information can be accurately mapped back to the corresponding spatial structure layout.

[0118] According to the spatial position information, the two-dimensional spatial structure is reconstructed. The reconstruction process maps the feature elements in the sequence to a two-dimensional matrix or feature map structure according to the spatial position, restores the spatial layout relationship scattered in the sequence process, and the two-dimensional spatial structure can be a regular grid, matrix arrangement or other spatial topology, to ensure complete expression of spatial information.

[0119] In the reconstructed two-dimensional spatial structure, a differentiated arrangement strategy is adopted for the bottom layer high-resolution features and the high layer low-resolution features. The high-resolution features usually come from the low-level structure of the image and have rich detail information and spatial positioning ability. Precise positioning arrangement means accurately mapping feature elements to the corresponding position in the two-dimensional space under high-precision spatial coordinates, ensuring the detail restoration of the spatial structure.

[0120] The high layer low-resolution features come from the high-level abstract expression of the image or sequence and have strong semantic information but low spatial resolution. Semantic reasoning arrangement means reasonably mapping high-level features to the spatial structure combined with spatial position and semantic association, while preserving the abstract information and semantic generalization ability of high-level expression, improving the semantic integrity and information richness of the overall feature map.

[0121] The corresponding scale fusion feature map is formed, which is in the form of a two-dimensional structure and combines the spatial structure after the restoration of the sequence information and the multi-modal fusion information, has the ability of joint expression of multi-source information, and is suitable for spatial correlation analysis and subsequent high-order information processing.

[0122] Combining all scale fusion feature maps forms a multi-scale fusion feature map. The combination operation integrates fusion feature maps with different spatial resolutions and different expression levels, preserves the structural characteristics and semantic levels of multi-scale information, and forms a joint expression structure with complete multi-scale information and clear structural layout, providing complete information support for subsequent spatial analysis and task reasoning.

[0123] In actual operation, the fusion feature sequence contains multiple layers of output. The part with a layer number less than or equal to a preset threshold L1 is divided into the bottom layer, and the part with a layer number greater than L1 is divided into the high layer. The specific threshold is set according to actual needs. Taking L1 = 2 as an example, the first two layers of feature sequence are the bottom layer, and the remaining part is the high layer.

[0124] At the same time, combined with the resolution setting, the fusion feature map with a spatial size greater than or equal to R1xR1 is considered as high-resolution, and the one smaller than R1xR1 is considered as low-resolution. Taking the original input image size as 256x256 and R1 as 64, in the generated fusion feature map, the sizes of 128x128 and 64x64 are high-resolution, and the sizes of 32x32 and 16x16 are low-resolution.

[0125] In the process of recombining the multi-scale fusion feature map, for the bottom layer high-resolution part, precise positioning arrangement operation is performed to ensure the preservation of spatial details and boundary information; for the high layer low-resolution part, semantic reasoning arrangement is performed to strengthen the semantic understanding of object relationship, scene background and interactive intention, and improve the overall spatial expression and analysis ability. This way effectively balances the preservation of spatial details and the expression of semantic generalization, and is suitable for object interaction analysis tasks that need to integrate spatial and semantic information.

[0126] In the medical health business field, for intraoperative video medical instrument operation analysis, the multi-layer output of the fused feature sequence respectively contains spatial information and semantic information of different levels. In practice, combined with the input image size of 256x256, the layer threshold is set to 2, and the resolution threshold is set to 64 pixels. In the specific implementation process, the output of the first two layers in the fused feature sequence and with a spatial size greater than or equal to 64x64 is defined as the bottom layer high-resolution feature, and the precise positioning arrangement operation is applied to ensure the retention of the edge profile and spatial position relationship of the surgical instrument. The output of the subsequent layers, especially the output with a spatial size less than 64x64, is regarded as a high layer low-resolution feature, and the semantic reasoning arrangement is performed to highlight the semantic association of the lesion area, medical operation behavior and the surrounding environment, and finally the multi-scale fused feature map is reorganized to ensure that the system has both fine-grained spatial restoration capability and comprehensive analysis effect for medical semantic understanding.

[0127] In the financial technology business field, in the remote counter interaction monitoring process, the bottom layer feature with a spatial resolution greater than 128x128 in the different scale outputs of the fused feature sequence is used for accurate restoration of the operation interface and the position of key items (such as bank cards and signature plates), and the precise positioning arrangement is used to improve the accuracy of item recognition and position calibration; the high layer output with a spatial resolution less than 128x128 is combined with the context to perform semantic reasoning arrangement, which assists in inferring the customer operation intention, interaction process and risk behavior, and forms a unified multi-scale fused feature map, providing multi-level and rich visual and semantic information support for subsequent risk analysis and business compliance detection.

[0128] The embodiment effectively solves the problem of loss of position information when the serialized information expresses the spatial structure by reorganizing the fused feature sequence into a multi-scale fused feature map, and realizes the complete reconstruction and joint expression of multi-source information in a two-dimensional space. The combination of precise positioning arrangement and semantic reasoning arrangement ensures the reasonable fusion of high-resolution detailed information and high-level semantic information in the spatial structure, and improves the joint expression capability of the system for complex spatial structure and multi-level semantic information. The multi-scale fused feature map has higher information density and expression integrity, improves the accuracy and robustness of the system in subsequent spatial position analysis, semantic recognition and interaction prediction tasks, and improves the overall utilization efficiency and expression effect of multi-modal information in complex environments.

[0129] In one embodiment, the above step S90 includes: S901, inputting the multi-scale fused feature map into a spatial position prediction branch to generate a bounding box coordinate of a future interaction object, and taking the bounding box coordinate as a spatial position distribution; S902, inputting the multi-scale fused feature map into an action classification branch to generate a verb-noun combination, and taking the verb-noun combination as an action semantic category; S903, inputting the multi-scale fusion feature map into a time regression branch to generate a contact time offset, and taking the contact time offset as a time contact feature; S904, combining the spatial position distribution, the action semantic category, and the time contact feature to generate an analysis result.

[0130] In the embodiment, the multi-scale fusion feature map contains spatial structure information and semantic expression information at different scales and different resolutions, and has multi-dimensional feature expression capability of cross-scale fusion. The feature map retains spatial position distribution of the predicted frame, local features of the salient object, action description information and language coding information of the historical video frame through the preceding operation steps, and forms a high-dimensional expression with global relationship and local details through the multi-layer self-attention mechanism.

[0131] The spatial position prediction branch uses network modules containing spatial convolution structure and scale aggregation operation to extract features related to the position of the target interactive object based on the spatial structure information of the fusion feature map, and generates the bounding box coordinates. The bounding box coordinates include horizontal position, vertical position, width and height information, and are usually expressed in the form of four-tuple or center point offset and size change amount, which can be realized by regression module, anchor mechanism or center point positioning algorithm.

[0132] The action classification branch inputs the fusion feature map, focuses on processing the high-level low-resolution features therein, combines the embedded textual action context information, uses a semantic classifier or a sequence modeling network to output a verb-noun combination. The verb-noun combination represents the predicted interactive behavior category and the key object category involved, forming an action semantic category conforming to the natural language semantic structure. The category output can be in the form of probability distribution, selected by maximum value, or expressed in the form of multi-class joint output to represent complex interaction types.

[0133] The time regression branch takes the fusion feature map as input, combines the timestamp information of the predicted frame and the time dynamic feature in the fusion feature map, and outputs the contact time offset based on the regression network structure. The contact time offset is a numerical scalar, which reflects the time interval from the current time point to the actual contact of the future interactive object, and the unit is frame number, millisecond or second. During the time regression process, multi-layer perceptron structure, residual connection and regularization mechanism can be combined to ensure the stability and numerical precision of the time offset prediction.

[0134] The spatial position distribution, the action semantic category, and the time contact feature are independently generated through the above branch structure, and the output results are combined through a structured combination operation to form a unified analysis result set. The overall analysis result has three-dimensional content of spatial positioning information, interactive semantic category and time prediction information, and can fully express the expected state and dynamic characteristics of the future interactive object.

[0135] During the whole operation process, the spatial position prediction, the action semantic classification and the time regression can be executed in parallel, or can be completed in turn according to the task priority, and the specific execution sequence and the task dependency relationship can be adjusted and optimized according to the actual scene demand. Through the joint design of the multi-task structure, the analysis results form a synergistic constraint in the space, the semantics and the time three levels, improve the overall accuracy and robustness of the prediction, and reduce the influence of single task error on the system output.

[0136] The embodiment fuses the spatial and semantic information of the feature map, jointly analyzes the spatial position prediction, the action semantic classification and the time contact feature, improves the spatial positioning accuracy of the future interactive object, the semantic rationality of the action category recognition and the stability of the time prediction, enhances the analysis ability of the system to the complex and variable interactive scene, and has better analysis effect and prediction ability in the actual environment with strong uncertainty and multiple interactive targets.

[0137] In an embodiment, a visual feature-based object interaction analysis device is provided, which corresponds to the visual feature-based object interaction analysis method in the above embodiment. Referring to Figure 3 , Figure 3 A functional module schematic diagram of a preferred embodiment of the visual feature-based object interaction analysis device of the present application is shown. The visual perception module 10, the action semantic extraction module 20, the context construction module 30, the visual coding module 40, the language coding module 50, the fusion splicing module 60, the cross-modal interaction module 70, the feature recombination module 80 and the interaction analysis module 90. The detailed description of each functional module is as follows: The visual perception module 10 is used for acquiring a predicted frame of video data, extracting multi-scale visual features of the predicted frame, and detecting significant objects in the predicted frame to form a significant object list; The action semantic extraction module 20 is used for generating an initial action description pair based on historical video frames, and aggregating the same initial action description pairs of consecutive frames in the historical video frames to generate an aggregated action description pair sequence; The context construction module 30 is used for generating a textual action context in combination with the aggregated action description pair sequence and the significant object list; The visual coding module 40 is used for dividing the visual features of each scale in the multi-scale visual features into image blocks and adding position coding respectively, to generate a multi-scale visual feature sequence; The language coding module 50 is used for encoding the textual action context using a pre-trained semantic coding model to generate a language feature sequence; The fusion splicing module 60 is used for splicing the visual feature sequence and the language feature sequence of each scale for each scale, and mapping the splicing result to a unified dimension to form a mixed input sequence of the corresponding scale; The cross-modal interaction module 70 is configured to perform cross-modal interaction on each scale of the mixed input sequence through a multi-layer self-attention mechanism to obtain a multi-scale fusion feature sequence. The feature reorganization module 80 is configured to reorganize the multi-scale fusion feature sequence into a multi-scale fusion feature map. The interaction analysis module 90 is configured to generate an analysis result of a spatial position distribution, an action semantic category, and a time contact feature of a future interaction object based on the multi-scale fusion feature map.

[0138] In an embodiment, the visual perception module 10 is specifically configured to: determine a last frame in the video data as a prediction frame; process the prediction frame through a convolutional neural network to generate a multi-scale feature map; extract multi-scale visual features with different spatial resolutions from the multi-scale feature map; obtain a set of predefined object categories; determine a similarity between the prediction frame and each object category in the set of predefined object categories using a visual language model; select a preset number of object categories according to the similarity to form a list of salient objects containing the selected object categories.

[0139] In an embodiment, the action semantic extraction module 20 is specifically configured to: process each frame in the historical video frames using a pre-trained image captioning model to generate a corresponding frame-level description; analyze the frame-level description through part-of-speech tagging to extract verb-noun combinations and simplify intermediate words to form initial action description pairs; count a frequency of occurrence of the initial action description pairs in the historical video frames; select the initial action description pair with the highest frequency in each historical video frame as a representative action description pair of the historical video frame; identify continuous video frame sequences with the same representative action description pairs in the sequence of historical video frames; aggregate each continuous video frame sequence with the same representative action description pairs into an action segment; combine all the action segments in chronological order to form an aggregated action description pair sequence.

[0140] In an embodiment, the visual encoding module 40 is specifically configured to: divide the visual features of each scale into image blocks of a preset size; determine spatial position coordinates of each image block; generate position encodings according to the spatial position coordinates; and adding the position code to a corresponding image block; arranging the image blocks with added position codes in spatial order to form a visual feature sequence of a corresponding scale; combining visual feature sequences of all scales to form a multi-scale visual feature sequence.

[0141] In an embodiment, the cross-modal interaction module 70 is specifically configured to: for each scale, inputting the mixed input sequence of the scale into a multi-layer self-attention module; in the multi-layer self-attention module, determining attention weights between visual feature sequence elements and language feature sequence elements; weighting and updating the visual feature sequence elements according to the attention weights to generate an updated visual feature sequence; weighting and updating the language feature sequence elements according to the attention weights to generate an updated language feature sequence; merging the updated visual feature sequence and the updated language feature sequence to generate an updated feature sequence; iteratively processing the updated feature sequence through the multi-layer self-attention module; taking the iteratively processed feature sequence as a fusion feature sequence of a corresponding scale; combining fusion feature sequences of all scales to form a multi-scale fusion feature sequence.

[0142] In an embodiment, the feature reorganization module 80 is specifically configured to: for each fusion feature sequence of a scale in the multi-scale fusion feature sequence, analyzing spatial position information in the fusion feature sequence; reconstructing a two-dimensional spatial structure according to the spatial position information; arranging feature elements in the fusion feature sequence according to the reconstructed two-dimensional spatial structure to perform accurate positioning and arrangement of bottom-layer high-resolution features and perform semantic reasoning arrangement of high-layer low-resolution features, forming a fusion feature map of a corresponding scale; combining fusion feature maps of all scales to form a multi-scale fusion feature map.

[0143] In an embodiment, the interaction analysis module 90 is specifically configured to: inputting the multi-scale fusion feature map into a spatial position prediction branch to generate a bounding box coordinate of a future interactive object, and taking the bounding box coordinate as a spatial position distribution; inputting the multi-scale fusion feature map into an action classification branch to generate a verb-noun combination, and taking the verb-noun combination as an action semantic category; input the multi-scale fused feature map into a time regression branch to generate a contact time offset, and take the contact time offset as a time contact feature; combine the spatial position distribution, the action semantic category, and the time contact feature to generate an analysis result.

[0144] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram thereof can be as shown in Figure 4 The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. The processor of the computer device is configured to provide determination and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with an external client through a network connection. The computer program, when executed by the processor, implements the functions or steps of a visual feature-based object interaction analysis method server side.

[0145] In one embodiment, a computer device is provided, which can be a client, and an internal structure diagram thereof can be as shown in Figure 5 The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. The processor of the computer device is configured to provide determination and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with an external server through a network connection. The computer program, when executed by the processor, implements the functions or steps of a visual feature-based object interaction analysis method client side. In one embodiment, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the following steps when executing the computer program: obtain a predicted frame of video data, extract multi-scale visual features of the predicted frame, and detect significant objects in the predicted frame to form a significant object list; generate an initial action description pair based on historical video frames, and aggregate the same initial action description pairs of consecutive frames in the historical video frames to generate an aggregated action description pair sequence; combine the aggregated action description pair sequence and the significant object list to generate a textual action context; dividing each scale of visual features in the multi-scale visual features into image blocks and adding position encoding to generate a multi-scale visual feature sequence; encoding the textual action context using a pre-trained semantic encoding model to generate a language feature sequence; for each scale, splicing the visual feature sequence and the language feature sequence of the scale, and mapping the splicing result to a unified dimension to form a mixed input sequence of the corresponding scale; performing cross-modal interaction on each scale of the mixed input sequence through a multi-layer self-attention mechanism to obtain a multi-scale fusion feature sequence; reorganizing the multi-scale fusion feature sequence into a multi-scale fusion feature map; based on the multi-scale fusion feature map, generating an analysis result of the spatial position distribution, action semantic category and time contact feature of the future interactive object.

[0146] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the following steps: obtaining a predicted frame of video data, extracting multi-scale visual features of the predicted frame, and detecting significant objects in the predicted frame to form a significant object list; generating an initial action description pair based on historical video frames, and aggregating the same initial action description pairs of consecutive frames in the historical video frames to generate an aggregated action description pair sequence; combining the aggregated action description pair sequence and the significant object list to generate a textual action context; dividing each scale of visual features in the multi-scale visual features into image blocks and adding position encoding to generate a multi-scale visual feature sequence; encoding the textual action context using a pre-trained semantic encoding model to generate a language feature sequence; for each scale, splicing the visual feature sequence and the language feature sequence of the scale, and mapping the splicing result to a unified dimension to form a mixed input sequence of the corresponding scale; performing cross-modal interaction on each scale of the mixed input sequence through a multi-layer self-attention mechanism to obtain a multi-scale fusion feature sequence; reorganizing the multi-scale fusion feature sequence into a multi-scale fusion feature map; based on the multi-scale fusion feature map, generating an analysis result of the spatial position distribution, action semantic category and time contact feature of the future interactive object.

[0147] It should be noted that the above functions or steps that can be implemented by the computer readable storage medium or the computer device can correspond to the related descriptions of the server side and the client side in the foregoing method embodiments. To avoid repetition, they will not be described here.

[0148] Those skilled in the art can understand that all or part of the processes in the foregoing method embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the foregoing method embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. The non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. The volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0149] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified. In actual applications, the above functions can be completed by different functional units or modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0150] It should be noted that if non-company software tools or components appear in the embodiments of the present application, they are only used for example introduction and do not represent actual use.

Claims

1. A method for object interaction analysis based on visual features, characterized in that, The method comprises the following steps: obtaining a predicted frame of video data, extracting multi-scale visual features of the predicted frame, and detecting significant objects in the predicted frame to form a significant object list; generating initial action description pairs based on historical video frames, and aggregating the same initial action description pairs of consecutive frames in the historical video frames to generate an aggregated action description pair sequence; combining the aggregated action description pair sequence and the significant object list to generate a textual action context; dividing the visual features of each scale in the multi-scale visual features into image blocks respectively and adding position encoding to generate a multi-scale visual feature sequence; encoding the textual action context using a pre-trained semantic encoding model to generate a language feature sequence; for each scale, splicing the visual feature sequence and the language feature sequence of the scale, and mapping the splicing result to a unified dimension to form a mixed input sequence of the corresponding scale; performing cross-modal interaction on the mixed input sequence of each scale through a multi-layer self-attention mechanism to obtain a multi-scale fusion feature sequence; reorganizing the multi-scale fusion feature sequence into a multi-scale fusion feature map; based on the multi-scale fusion feature map, generating analysis results of spatial position distribution, action semantic category and time contact feature of future interactive objects.

2. The visual feature based object interaction analysis method of claim 1, wherein, obtaining a predicted frame of video data, extracting multi-scale visual features of the predicted frame, and detecting significant objects in the predicted frame to form a significant object list, comprising: determining the last frame in the video data as the predicted frame; processing the predicted frame through a convolutional neural network to generate a multi-scale feature map; extracting multi-scale visual features containing different spatial resolutions from the multi-scale feature map; obtaining a set of predefined object categories; determining the similarity between the predicted frame and each object category in the set of predefined object categories using a visual language model; selecting a preset number of object categories according to the similarity to form a significant object list containing the selected object categories.

3. The visual feature based object interaction analysis method of claim 1, wherein, generating initial action description pairs based on historical video frames, and aggregating the same initial action description pairs of consecutive frames in the historical video frames to generate an aggregated action description pair sequence, comprising: processing each frame in the historical video frames using a pre-trained image captioning model to generate a corresponding frame-level description; analyzing the frame-level description through part-of-speech tagging, extracting verb-noun combinations and simplifying intermediate words to form initial action description pairs; counting the frequency of occurrence of the initial action description pairs in the historical video frames; selecting the initial action description pair with the highest frequency in each historical video frame as the representative action description pair of the historical video frame; identifying consecutive video frame sequences with the same representative action description pairs in the historical video frame sequence; aggregating each consecutive video frame sequence with the same representative action description pair into an action segment; combining all action segments in chronological order to form an aggregated action description pair sequence.

4. The visual feature based object interaction analysis method of claim 1, wherein, dividing the visual features of each scale in the multi-scale visual features into image blocks respectively and adding position encoding to generate a multi-scale visual feature sequence, comprising: for the visual features of each scale, dividing the visual features of the scale into image blocks of a preset size; determining the spatial position coordinates of each image block; generating a position code according to the spatial position coordinates; adding the position code into a corresponding image block; arranging the image block with the added position code in a spatial order to form a visual feature sequence of a corresponding scale; combining visual feature sequences of all scales to form a multi-scale visual feature sequence.

5. The visual feature based object interaction analysis method of claim 1, wherein, performing cross-modal interaction on each scale of the mixed input sequence through a multi-layer self-attention mechanism to obtain a multi-scale fusion feature sequence, including: inputting the mixed input sequence of each scale into a multi-layer self-attention module; determining attention weights between visual feature sequence elements and language feature sequence elements in the multi-layer self-attention module; weighting and updating the visual feature sequence elements according to the attention weights to generate updated visual feature sequence elements; weighting and updating the language feature sequence elements according to the attention weights to generate updated language feature sequence elements; combining the updated visual feature sequence elements and the updated language feature sequence elements to generate an updated feature sequence; iteratively processing the updated feature sequence through the multi-layer self-attention module; taking the iteratively processed feature sequence as a fusion feature sequence of a corresponding scale; combining fusion feature sequences of all scales to form a multi-scale fusion feature sequence.

6. The visual feature based object interaction analysis method of claim 1, wherein, reorganizing the multi-scale fusion feature sequence into a multi-scale fusion feature map, including: analyzing spatial position information in the fusion feature sequence for each scale of the multi-scale fusion feature sequence; reconstructing a two-dimensional spatial structure according to the spatial position information; performing precise positioning arrangement on bottom layer high-resolution features and semantic reasoning arrangement on high layer low-resolution features according to the reconstructed two-dimensional spatial structure to form a fusion feature map of a corresponding scale; combining fusion feature maps of all scales to form a multi-scale fusion feature map.

7. The visual feature based object interaction analysis method of claim 1, wherein, generating analysis results of spatial position distribution, action semantic category and time contact feature of a future interactive object based on the multi-scale fusion feature map, including: inputting the multi-scale fusion feature map into a spatial position prediction branch to generate a bounding box coordinate of the future interactive object, and taking the bounding box coordinate as a spatial position distribution; inputting the multi-scale fusion feature map into an action classification branch to generate a verb-noun combination, and taking the verb-noun combination as an action semantic category; inputting the multi-scale fusion feature map into a time regression branch to generate a contact time offset, and taking the contact time offset as a time contact feature; combining the spatial position distribution, the action semantic category and the time contact feature to generate analysis results.

8. An object interaction analysis apparatus based on visual features, characterized by, The object interaction analysis device based on visual features includes: a visual perception module configured to obtain a predicted frame of video data, extract multi-scale visual features of the predicted frame, and detect significant objects in the predicted frame to form a significant object list; an action semantic extraction module configured to generate an initial action description pair based on historical video frames, and aggregate the same initial action description pairs of consecutive frames in the historical video frames to generate an aggregated action description pair sequence; a context construction module configured to generate a textual action context by combining the sequence of actions and the list of salient objects with the aggregated action description; a visual encoding module configured to divide the visual features of each scale in the multi-scale visual features into image blocks respectively and add position encodings to generate a sequence of multi-scale visual features; a language encoding module configured to encode the textual action context using a pre-trained semantic encoding model to generate a sequence of language features; a fusion and concatenation module configured to, for each scale, concatenate the sequence of visual features and the sequence of language features of the scale and map the concatenation result to a uniform dimension to form a mixed input sequence of the corresponding scale; a cross-modal interaction module configured to perform cross-modal interaction on the mixed input sequence of each scale through a multi-layer self-attention mechanism to obtain a sequence of multi-scale fusion features; a feature reorganization module configured to reorganize the sequence of multi-scale fusion features into a multi-scale fusion feature map; an interaction analysis module configured to generate an analysis result of spatial position distribution, action semantic category and time contact feature of a future interaction object based on the multi-scale fusion feature map.

9. A computer device, comprising: The computer device comprises a memory, a processor, and a visual feature-based object interaction analysis program stored in the memory and executable on the processor, and the visual feature-based object interaction analysis program, when executed by the processor, implements the steps of the visual feature-based object interaction analysis method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a visual feature-based object interaction analysis program, and the visual feature-based object interaction analysis program, when executed by the processor, implements the steps of the visual feature-based object interaction analysis method according to any one of claims 1-7.

Images