A visual co-processing method, system, device and medium

Abstract

The application discloses a visual cooperative processing method, system, device and medium, and relates to the technical field of big data processing.The method comprises the following steps: obtaining a target detection result by detecting a target object and / or an attribute of the target object in target video data; aggregating the target detection result to obtain multi-dimensional object attribute information; judging whether the multi-dimensional object attribute information has a change in a preset range for a current frame and a previous frame in the target video data to obtain a judgment result; calling a corresponding processing model according to the judgment result and outputting a model cooperative processing result.The complex visual information is decoupled into multiple dimensions to generate rich object attribute information.A lightweight model is used in a simple task, and a large model is called in a complex task, so that the balance between high performance and low power consumption is achieved.The system can flexibly adapt to different task complexities and ensure efficient operation of the system in a resource-limited environment.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, specifically to a visual collaborative processing method, system, device, and medium. Background Technology

[0002] In recent years, multimodal intelligent systems have made significant progress, encompassing key modules such as open-vocabulary object detection, multimodal language models, and intelligent agent frameworks. In open-vocabulary object detection, the technology has continuously evolved, from traditional methods like Faster R-CNN and the YOLO series, to DETR incorporating the Transformer architecture, and then to object detection methods based on natural language prompts using image-text pre-trained models (such as CLIP). Multimodal models have also progressed from early simple concatenation and fusion to the contrastive learning-based unified modeling pioneered by CLIP, and further to advanced task processing models integrating language models and visual encoders. Furthermore, agent architectures with large language models as the core scheduling center are gradually taking shape, improving execution flexibility and system scalability in complex task scenarios.

[0003] Despite progress, existing technologies still face numerous challenges. First, task understanding and scheduling capabilities are limited; systems largely rely on static processes and lack dynamic analysis and scheduling capabilities, leading to resource waste or response delays. Second, the loose coupling between perception and language models results in insufficient collaboration, making it difficult to achieve fine-grained, real-time multi-model interaction. Furthermore, large multimodal models have numerous parameters and complex inference, and direct invocation incurs significant resource overhead, making it difficult to meet real-time and high-concurrency requirements. Finally, the granularity of processing visual content such as images and videos is unreasonable, making it impossible to complete basic visual tasks using lightweight models first, resulting in overall low operating efficiency. Summary of the Invention

[0004] The main objective of this invention is to provide a visual collaborative processing method, system, device, and medium. It rapidly identifies target objects and their attributes using a lightweight visual perception model, reducing computational complexity. Complex visual information is decoupled into multiple dimensions, generating rich object attribute information to support deep semantic understanding. The system dynamically determines whether to invoke a large multimodal understanding model based on changes in the object and its attributes, avoiding unnecessary resource waste and improving system efficiency. In complex tasks, the system invokes the large multimodal understanding model to generate detailed semantic descriptions or answer semantic questions, supporting the execution of complex tasks. Through a dynamic scheduling mechanism, it flexibly adapts to different task complexities, ensuring efficient system operation in resource-constrained environments.

[0005] To achieve the above objectives, the embodiments of this application provide the following technical solutions:

[0006] According to a first aspect of the embodiments of this application, a visual collaborative processing method is provided, the method comprising:

[0007] Detect the target object and / or the attributes of the target object in the target video data to obtain the target detection result;

[0008] By aggregating the target detection results, multi-dimensional object attribute information is obtained;

[0009] For the current frame and the previous frame in the target video data, determine whether there is a change in the multi-dimensional object attribute information within a preset range, and obtain the determination result;

[0010] Based on the judgment result, the corresponding processing model is invoked, and the collaborative processing result of the model is output.

[0011] Optionally, the judgment result includes whether there is a change or not; the step of calling the corresponding processing model based on the judgment result and outputting the model collaborative processing result includes:

[0012] If there are changes, the multi-dimensional object attribute information is input into the multimodal understanding big model for deep semantic parsing, and the deep semantic understanding result is output.

[0013] If there are no changes, output a list of lightweight object properties, which includes the object that has not changed and its corresponding properties.

[0014] Optionally, the multi-dimensional object attribute information is input into a multimodal understanding model for deep semantic parsing, and the deep semantic understanding results are output, including:

[0015] Based on the object identifier and static attribute features of the multi-dimensional object attribute information, the scene category is determined;

[0016] The determined results are combined with spatiotemporal correlation data to analyze the object's behavioral intent, resulting in an analysis result; the spatiotemporal correlation data includes time-series event chains and spatial topological relationship graphs.

[0017] Based on the parsing results, a multi-turn conversational semantic result output is generated.

[0018] Optionally, determining whether the multi-dimensional object attribute information has a preset range of changes, and obtaining a determination result, includes:

[0019] Track the trajectory of objects in consecutive frames, and mark them when the confidence level is lower than a threshold. The mark includes a new mark or a disappearance mark; and / or, calculate the Mahalanobis distance between the current frame attribute and the historical average. If the distance exceeds a set threshold, it is determined that a change has occurred.

[0020] Optionally, the aggregation of the target detection results to obtain multi-dimensional object attribute information includes:

[0021] Based on the target detection results, the following actions are performed: assigning a unique identifier to each detected object and establishing an object identity unique identifier mapping table; and / or constructing an inherent attribute feature vector library, the inherent attributes including the object's color, shape, and size; and / or generating a state change event log, the state change event log being used to record dynamic states; and / or statistically analyzing the object quantity distribution matrix; and / or marking the positional relationships between objects to construct a spatial topology graph; and / or recording temporal association event chains; and / or generating basic semantic tags.

[0022] Optionally, after aggregating the target detection results to obtain multi-dimensional object attribute information, the method further includes:

[0023] Static attribute features are selected according to preset filtering rules, while dynamic attribute features are filtered out. The preset filtering rules include: extracting object identifiers and at least one inherent attribute from color, shape, and age; and / or filtering features describing changes in object state, including falling, damage, and abnormal movement trajectories; and / or deleting object quantity distribution data; and / or removing spatial topological relationships and temporal event chains between objects; and / or compressing the complete semantic description into keyword phrases.

[0024] Optionally, detecting the target object and / or the attributes of the target object in the target video data to obtain the target detection result includes:

[0025] A lightweight neural network is used to perform target detection on the target video data to identify target objects, including people, vehicles, and objects; and / or, an inherent attribute of the detected target object is extracted by a classifier, including color attribute, shape attribute, and size attribute; and / or, the target detection results of the current frame are compared with those of historical frames by an inter-frame difference analysis algorithm to generate object identifiers and / or attribute change threshold quantization values.

[0026] According to a second aspect of the embodiments of this application, a visual collaborative processing system is provided, the system comprising:

[0027] The target detection module is used to detect target objects and / or attributes of the target objects in the target video data, and obtain target detection results;

[0028] The aggregation module is used to aggregate the target detection results to obtain multi-dimensional object attribute information;

[0029] The judgment module is used to determine whether there is a change within a preset range in the multi-dimensional object attribute information for the current frame and the previous frame in the target video data, and to obtain the judgment result;

[0030] The result output module is used to call the corresponding processing model based on the judgment result and output the model collaborative processing result.

[0031] According to a third aspect of the present application, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0032] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided having computer-readable instructions stored thereon, the computer-readable instructions being executable by a processor to implement the method described in the first aspect above.

[0033] In summary, this application provides a visual collaborative processing method, system, device, and medium. By detecting target objects and / or their attributes in target video data, target detection results are obtained; the target detection results are aggregated to obtain multi-dimensional object attribute information; for the current frame and previous frames in the target video data, it is determined whether the multi-dimensional object attribute information exhibits a change within a preset range, resulting in a judgment result; based on the judgment result, a corresponding processing model is invoked, and the collaborative processing result is output. A lightweight visual perception model is used to quickly identify target objects and their attributes, reducing computational complexity. Complex visual information is decoupled into multiple dimensions, generating rich object attribute information to support deep semantic understanding. The invocation of a large multimodal understanding model is dynamically determined based on changes in the object and its attributes, avoiding unnecessary resource waste and improving system efficiency. In complex tasks, the large multimodal understanding model is invoked to generate detailed semantic descriptions or answer semantic questions, supporting the execution of complex tasks. A dynamic scheduling mechanism flexibly adapts to different task complexities, ensuring efficient system operation in resource-constrained environments. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0035] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0036] Figure 1 This is a flowchart of the visual collaborative processing method provided in the embodiments of this application;

[0037] Figure 2 A schematic diagram of an intelligent agent architecture for large and small vision model collaboration provided in this application embodiment;

[0038] Figure 3 This is a schematic diagram of a visual collaborative processing system provided in an embodiment of this application;

[0039] Figure 4 This paper shows a structural diagram of an electronic device provided in an embodiment of this application;

[0040] Figure 5 A diagram of a computer-readable storage medium provided in an embodiment of this application is shown.

[0041] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0043] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0044] Furthermore, in this invention, descriptions involving "first," "second," etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0045] In this invention, unless otherwise explicitly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0046] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are feasible for those skilled in the art. If the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0047] Figure 1 This application illustrates a visual collaborative processing method provided by an embodiment of the present application, the method comprising:

[0048] Step 101: Detect the target object and / or the attributes of the target object in the target video data to obtain the target detection result;

[0049] Step 102: Aggregate the target detection results to obtain multi-dimensional object attribute information;

[0050] Step 103: For the current frame and the previous frame in the target video data, determine whether there is a change in the multi-dimensional object attribute information within a preset range, and obtain the determination result;

[0051] Step 104: Based on the judgment result, call the corresponding processing model and output the model collaborative processing result.

[0052] In one possible implementation, step 101 involves detecting a target object and / or attributes of the target object in the target video data to obtain a target detection result, including:

[0053] A lightweight neural network is used to perform target detection on the target video data to identify target objects, including people, vehicles, and objects; and / or, an inherent attribute of the detected target object is extracted by a classifier, including color attribute, shape attribute, and size attribute; and / or, the target detection results of the current frame are compared with those of historical frames by an inter-frame difference analysis algorithm to generate object identifiers and / or attribute change threshold quantization values.

[0054] This system efficiently extracts key information from target video data using lightweight neural networks and classifiers, while dynamically monitoring changes in target objects and their attributes using inter-frame difference analysis algorithms. Specifically, the lightweight neural network performs target detection on the video data, quickly identifying target objects such as people, vehicles, and objects. This lightweight design aims to reduce computational resource consumption, ensuring efficient operation even in resource-constrained environments. For example, in a surveillance scenario, the system can quickly identify pedestrians or vehicles entering the monitored area.

[0055] The system extracts inherent attributes of detected target objects using a classifier, including color, shape, and size. These attributes provide richer details for subsequent analysis and decision-making. For example, the system can not only identify a car in a video, but also further determine that the car is red, sedan-shaped, and medium-sized.

[0056] By comparing the target detection results of the current frame with those of historical frames using an inter-frame difference analysis algorithm, a threshold quantification value for object identification and / or attribute changes is generated. This mechanism enables the system to dynamically monitor changes in target objects and their attributes, thereby promptly detecting newly appearing objects or significant changes in object attributes. For example, if a car's color changes from white to black, or a person suddenly appears or disappears from the surveillance footage, the system can quickly capture these changes.

[0057] These technologies together enable efficient perception and dynamic monitoring of video content, providing a foundation for subsequent intelligent decision-making and resource scheduling.

[0058] In one possible implementation, in step 102, aggregating the target detection results to obtain multi-dimensional object attribute information includes: for the target detection results, performing the following: assigning a unique identifier to each detected object and establishing an object identity unique identifier mapping table; and / or constructing an inherent attribute feature vector library, the inherent attributes including the object's color, shape, and size; and / or generating a state change event log, the state change event log being used to record dynamic states; and / or statistically analyzing the object quantity distribution matrix; and / or marking the positional relationships between objects to construct a spatial topology graph; and / or recording time-related event chains; and / or generating basic semantic tags.

[0059] The target detection results are aggregated from multiple dimensions to generate structured and rich object attribute information, thereby supporting subsequent intelligent analysis and decision-making. Specifically:

[0060] 1. Assign a unique identifier to each detected object and establish an object identity unique identifier mapping table. This process is similar to assigning a unique identifier to each device or entity in the Internet of Things (IoT) to facilitate tracking and management. For example, in an intelligent transportation system, assigning a unique identifier to each vehicle makes it easy to track the vehicle's driving trajectory and state changes.

[0061] 2. Construct an intrinsic attribute feature vector library covering the object's intrinsic attributes such as color, shape, and size. This feature extraction and aggregation method is similar to extracting geometric and color features in image analysis, enhancing the model's ability to extract information unique to the target itself. For example, by extracting the color attributes (e.g., red), shape attributes (e.g., sedan), and size attributes (e.g., medium-sized) of a vehicle, the target object can be described and identified more accurately.

[0062] 3. Generate a state change event log to record dynamic states. This is similar to recording changes in the motion state of an object in object detection, to facilitate subsequent behavior analysis. For example, recording the process of a car accelerating from a standstill, or a person moving from standing to walking.

[0063] 4. Statistical object quantity distribution matrix: This helps to understand the distribution of different objects in a scene. For example, in a monitoring scenario, counting the number of pedestrians in different areas can provide data support for crowd management.

[0064] 5. Mark the positional relationships between objects to construct a spatial topology map. This is similar to using unique identifiers in a Geographic Information System (GIS) to manage spatial data and construct spatial relationships between entities. For example, in a smart factory, production processes and logistics management can be optimized by marking the positional relationships between different devices.

[0065] 6. Recording time-related event chains helps in analyzing an object's behavioral patterns over time. For example, by recording a person's location changes at different points in time, their movement trajectory can be inferred.

[0066] 7. Generate basic semantic labels, transforming the detected object and attribute information into simple semantic descriptions. This is similar to converting object detection results into natural language descriptions in image description generation, making them easier for humans to understand and process. For example, transforming the detected "red car" and "acceleration" behavior into the semantic label "a red car is accelerating."

[0067] These technologies together enable multi-dimensional aggregation and structured processing of target detection results, providing a foundation for subsequent intelligent decision-making and resource scheduling.

[0068] In one possible implementation, after aggregating the target detection results to obtain multi-dimensional object attribute information, the method further includes: filtering out static attribute features and removing dynamic attribute features according to preset filtering rules; the preset filtering rules include: extracting object identifiers and at least one inherent attribute among color, shape, and age; and / or filtering features describing changes in object state, the features describing changes in object state including falling, damage, and abnormal movement trajectories; and / or deleting object quantity distribution data; and / or removing spatial topological relationships and temporal event chains between objects; and / or compressing the complete semantic description into keyword phrases.

[0069] After aggregating target detection results and generating multi-dimensional object attribute information, this information is further optimized and simplified using preset filtering rules. This process aims to extract key static attribute features and filter out dynamic and redundant attribute features, thereby providing more efficient and concise data support for subsequent intelligent decision-making and resource scheduling. Specifically:

[0070] 1. Based on preset filtering rules, extract at least one of the following: object identifier and inherent attributes such as color, shape, and age. For example, in an intelligent monitoring system, the system may retain the vehicle's unique identifier and color attribute (such as "a red sedan with license plate number 1234") while ignoring other dynamic attributes in order to quickly identify and track target objects.

[0071] 2. Filter out features describing changes in the state of an object, such as falling, damage, and abnormal movement trajectories. For example, in an intelligent transportation system, the system may ignore temporary stops or slight trajectory deviations of vehicles, focusing instead on the stable driving state of the vehicle, thereby reducing unnecessary alarms and actions.

[0072] 3. Remove data on object quantity distribution, spatial topological relationships between objects, and temporal event chains. This data may be unnecessary in some scenarios, and filtering it out can reduce computational burden. For example, in a simple object recognition task, the system may only need to know the existence and basic attributes of the target object, without requiring detailed spatial and temporal information.

[0073] 4. Compress complete semantic descriptions into keyword phrases. For example, compress "a person is running fast" into "a person running fast," thereby simplifying information and improving the system's processing efficiency.

[0074] Through these techniques, this method can effectively extract key static attribute features from a large amount of complex visual information, while removing unnecessary dynamic and redundant information. This not only improves the system's operating efficiency but also enhances its adaptability and flexibility, enabling it to perform tasks efficiently in different application scenarios.

[0075] In one possible implementation, in step 103, determining whether the multi-dimensional object attribute information has changed within a preset range and obtaining a determination result includes: tracking the trajectory of objects in consecutive frames and marking them when the confidence level is lower than a threshold, the marking including new markings or disappearance markings; and / or calculating the Mahalanobis distance between the current frame attribute and the historical average, and determining that a change has occurred if the distance exceeds a set threshold.

[0076] By dynamically monitoring changes in multi-dimensional object attribute information, it efficiently determines whether objects in target video data have undergone significant changes. This process aims to provide accurate data for subsequent intelligent decision-making and resource scheduling. Specifically:

[0077] 1. By tracking the trajectory of objects in consecutive frames, the system monitors the motion status of objects in real time. When the confidence level of an object (i.e., the system's confidence level in the accuracy of object recognition) falls below a preset threshold, it is marked. Marking types include "new" (indicating a new object entering the field of view) and "disappearance" (indicating an object leaving the field of view). For example, in an intelligent monitoring system, if a pedestrian enters the monitored area, the system will mark it as "new"; if the pedestrian leaves the monitored area, the system will mark it as "disappearance." This mechanism can promptly detect the appearance and disappearance of objects, ensuring the system's sensitivity to scene changes.

[0078] 2. Calculate the Mahalanobis distance between the attributes (such as color, shape, size, etc.) of objects in the current frame and their historical mean. Mahalanobis distance is a statistical measure used to assess the degree of difference between an observed value and the population mean. If the calculated Mahalanobis distance exceeds a set threshold, it is determined that the object's attribute has changed significantly. For example, if a car's color changes from blue to red, or a person's walking speed suddenly increases, these changes will be detected through Mahalanobis distance calculation. This quantitative evaluation method can accurately identify subtle changes in object attributes, thus providing the system with a more accurate basis for decision-making.

[0079] Using these two techniques, this method can dynamically monitor changes in the trajectory and attributes of objects, and promptly detect significant changes in target video data. This not only improves the system's response speed and accuracy but also optimizes resource allocation, enabling the system to invoke more advanced processing models when necessary, thereby achieving efficient and intelligent visual processing.

[0080] In one possible implementation, in step 104, the judgment result includes whether there is a change or not; the step of calling the corresponding processing model according to the judgment result and outputting the model collaborative processing result includes: if there is a change, inputting the multi-dimensional object attribute information into the multimodal understanding big model for deep semantic parsing and outputting the deep semantic understanding result; if there is no change, outputting a lightweight object attribute list, the lightweight object attribute list including objects that have not changed and their corresponding attributes.

[0081] In one possible implementation, in step 104, the multi-dimensional object attribute information is input into a multimodal understanding big model for deep semantic parsing, and the deep semantic understanding result is output, including: determining the scene category based on the object identifier and static attribute features of the multi-dimensional object attribute information; parsing the object's behavioral intent by combining the judgment result with spatiotemporal correlation data to obtain the parsing result; the spatiotemporal correlation data includes time series event chains and spatial topology relationship graphs; and generating multi-turn conversational semantic result output based on the parsing result.

[0082] Based on changes in object attribute information, processing models of varying complexity can be flexibly invoked to achieve efficient resource utilization and optimized processing results. Specifically:

[0083] 1. Judgment Result Classification: In step 104, the judgment result includes two cases: "change exists" and "no change exists." If the judgment result is "change exists," the multi-dimensional object attribute information is input into the multimodal understanding model for deep semantic parsing, and the deep semantic understanding result is output. This mechanism ensures that when the object or its attributes undergo significant changes, the system can call a more complex model for deep processing to provide richer semantic information. If the judgment result is "no change exists," a lightweight object attribute list is output, which includes objects that have not changed and their corresponding attributes. This mechanism avoids calling complex models when the object state is stable, thereby saving computational resources.

[0084] 2. In step 104, multi-dimensional object attribute information is input into the multimodal understanding big data model for deep semantic parsing. This includes the following steps: Based on the object identifier and static attribute features in the multi-dimensional object attribute information, scene category determination is performed. For example, by analyzing the object's color, shape, size, and other attributes, it is determined whether the current scene is a traffic scene, an indoor scene, or another specific scene. Further, the determined results are combined with spatiotemporal correlation data (including time-series event chains and spatial topology graphs) to analyze the object's behavioral intent. For example, by analyzing the object's movement trajectory and spatial location relationships in the time series, it is determined whether the object is performing a specific behavior (such as whether a vehicle is overtaking or a pedestrian is running). Further, multi-turn conversational semantic results are generated based on the parsing results. For example, the system can generate semantic descriptions such as "a red car is overtaking" or "a person is running quickly," and supports multi-turn dialogue for further interaction and confirmation.

[0085] To address the problem that existing systems lack dynamic analysis and scheduling capabilities during task execution and cannot intelligently select processing paths based on task complexity, this application uses a visual perception model to process video sequences in real time and dynamically monitor changes in objects and their attributes. Only when significant changes are detected is a multimodal understanding model invoked for deep understanding. This dynamic scheduling mechanism can flexibly select processing paths based on task complexity, avoiding wasted resources on simple tasks while ensuring that complex tasks are adequately processed.

[0086] To address the problem of insufficient collaboration between visual and language models in existing systems, hindering fine-grained, real-time multi-model interaction, this application proposes an object attribute aggregation module. This module decomposes visual understanding into seven dimensions: object, attribute, state, quantity, space, time, and semantics. The visual perception model focuses on changes in objects and attributes, while the larger multi-modal understanding model handles deep semantic understanding. This hierarchical approach enables closer collaboration between the visual and language models. The visual model provides basic perceptual information, while the language model performs deep semantic parsing, thus achieving fine-grained multi-model interaction.

[0087] Existing technologies often employ large-scale multimodal models with massive parameters and complex inference processes, making them difficult to deploy widely in computationally constrained environments. This application addresses this by employing a lightweight visual perception model to handle basic visual tasks, only invoking the large multimodal understanding model when necessary. This "on-demand" mechanism significantly reduces the system's average complexity and power consumption, enabling efficient operation even in resource-constrained environments.

[0088] To address the issue of low overall efficiency in some existing systems due to the inability to complete basic visual tasks using lightweight models first, this application employs a visual perception model to handle basic visual tasks (such as object detection and attribute perception), while a large multimodal understanding model handles deep semantic understanding. This hierarchical approach allows the system to complete coarse-grained basic visual tasks like object detection and segmentation using lightweight models first, avoiding unnecessary calls to heavyweight models and thus improving overall operational efficiency.

[0089] Figure 2 This diagram illustrates an intelligent agent architecture for collaborative large and small visual models provided in this application embodiment. A camera captures continuous images in real time, transmitting multiple frames per second to the system to provide raw video data for subsequent processing. The acquired video data is organized into video frame sequences, which serve as input to the visual perception model. This model, primarily focused on object detection and attribute perception, has a small number of parameters to ensure fast inference speed and extremely low power consumption. The visual perception model is responsible for extracting key visual information from the video frames, such as object identifiers and their inherent attribute information (color, shape, size, etc.). The output of the visual perception model is fed into an object attribute aggregation module. This module aggregates and analyzes the object detection results, generating multi-dimensional object attribute information, including object identity, inherent attributes, state changes, quantity distribution, spatial relationships, time series, and semantic descriptions. The output of the object attribute aggregation module is further structured and described, providing input to a multimodal understanding model. When deep understanding is required, the structured description is input into the multimodal understanding model. This is a large-scale visual-semantic bimodal model, typically with over 1 billion parameters, capable of providing detailed descriptions of video content or answering semantic questions. The model performs hierarchical reasoning, including scene classification, spatiotemporal correlation analysis, and multi-turn conversational semantic output. Ultimately, the output of the multimodal understanding model is used for final output, providing the system with detailed semantic information to support the execution of complex tasks. The entire process from video acquisition to final output is demonstrated, emphasizing the synergistic effect of the visual perception model and the multimodal understanding model in video content processing, and how a dynamic scheduling mechanism can achieve efficient resource utilization and optimized processing results.

[0090] The method provided in the embodiments of this application will be described in detail below. It includes the following stages:

[0091] Phase 1: Video acquisition and preliminary processing, and visual perception model processing.

[0092] Step 1: Acquire video sequence; the camera captures continuous images in real time, transmitting multiple frames per second to the system.

[0093] Step 2: Input the video frames one by one into the visual perception model to obtain video frame data containing image features.

[0094] Step 3: Input the video frame data into the visual perception model for object detection and attribute perception, and output the object detection results in the video frame, including object identifiers and their inherent attribute information. Object detection is used to identify objects such as people, vehicles, and objects in the video frame. Attribute perception is used to extract attribute information such as the object's color, shape, and size.

[0095] Video frame data serves as input to the visual perception model, used for basic visual processing. The visual perception model performs basic visual processing on video sequences, extracting key information and providing raw visual data for subsequent processing. It quickly extracts key visual information from video frames to determine if there are changes in objects or attributes, providing foundational data for subsequent decision-making. For example, detecting a red car.

[0096] Visual perception models extract object detection results in the following ways:

[0097] (i) A lightweight neural network architecture is used to perform target detection on video frames and identify objects such as people, vehicles, and objects;

[0098] (ii) Use a classifier to extract the inherent attributes of the object, including color (e.g., RGB values), shape (e.g., edge contour encoding), and size (e.g., area / length ratio).

[0099] (iii) By comparing the current frame with historical frames using a difference analysis algorithm, object addition / disappearance markers and attribute change thresholds are generated.

[0100] Phase Two: Analysis of the Object Attribute Aggregation Module.

[0101] Step 1: The object attribute aggregation module aggregates and analyzes the above target detection results, outputting aggregated object attribute information. Object attribute information includes seven dimensions: object identity, inherent attributes, state changes, quantity distribution, spatial relationships, time series, and semantic description.

[0102] The object attribute aggregation module decouples information in the following ways: (i) Object identity: Establishes a unique identifier mapping table for each detected object and assigns a unique identifier; (ii) Inherent attributes: Constructs an inherent attribute feature vector library, including color histograms, geometric shape descriptors, etc., to extract inherent attributes such as color, shape, and size of objects; (iii) State changes: Generates a state change event log to record object state changes, such as collisions and abnormal movement trajectories; (iv) Quantity distribution: Statistically calculates the object quantity distribution matrix to record the quantity of each type of object in video frames; (vi) Spatial relationships: Constructs a spatial topology graph to mark the spatial positional relationships between objects, such as front-back, top-bottom, etc.; (vii) Time series: Records time series event chains, such as the time association of "removing a hat → accelerating running"; (viii) Semantic description: Generates basic semantic tags, such as associating "pedestrian" with "running" to form a simple semantic description.

[0103] The analysis is broken down into seven dimensions: object, attribute, state, quantity, space, time, and semantics. Objects include identifying the main objects in video frames (such as people, vehicles, and objects). Attributes include extracting inherent attributes of objects such as color, shape, and size. State includes determining the state of objects or between objects (such as fallen, damaged, or fighting). Quantity includes counting the number of objects (such as two pedestrians). Space includes analyzing the spatial relationships between objects (such as front and back, up and down). Time includes analyzing changes in objects over time (such as a person running after removing their hat). Semantics includes generating simple semantic descriptions (such as "a person is running").

[0104] Step 2: Filter static attribute features and dynamic attribute information according to preset rules.

[0105] The preset rules include the following filtering conditions: (i) Retain object identifiers and static attributes such as color / shape / age: retain the object's identifiers and inherent attributes such as color, shape, and age; (ii) Exclude dynamic state information: filter out dynamic state information, such as falling or being damaged; (iii) Ignore quantity statistics: do not retain object quantity statistics; (iv) Remove spatial relationships and time series data: ignore spatial relationships and time series information between objects; (v) Truncate the complete semantic description to keyword phrases: simplify the complete semantic description to keyword phrases.

[0106] By filtering, key static attribute features are extracted, and dynamic and complex redundant information is filtered out, providing concise and efficient data support for subsequent dynamic scheduling decisions.

[0107] Phase 3: Dynamic scheduling decision-making.

[0108] Step 1: The visual perception model generates aggregated object attribute information based on the filtered static attribute features; the filtered static attribute features (such as object identifier, color, shape, age, etc.) are integrated into a structured dataset to provide input for subsequent change judgment.

[0109] Step 2: Based on the aggregated object attribute information, determine whether there are significant changes in the objects and their attributes between the current frame and the previous frame.

[0110] The object list and attribute list of the current frame are compared item by item with the data of the previous frame. If a new object is found (e.g., a dog appears in the current frame but not the previous one), or an attribute change occurs (e.g., the car color changes from white to black), it is marked as "requiring in-depth processing". Object changes include determining whether an object has been added or disappeared; significant changes in the object's inherent attributes (e.g., color change, shape change); and detection of interaction between objects (e.g., collision, chase). Other dimensions are ignored: the increase or decrease in the number of objects, movement of positions, or event semantics (e.g., "fighting," "falling down") are not checked.

[0111] Step 3: Output a scheduling decision signal based on the judgment result, indicating whether the multimodal understanding model needs to be invoked. If there is a significant change, the multimodal understanding model will be invoked; otherwise, only the visual perception model will continue to run.

[0112] Monitor changes in objects and their attributes. If significant changes are detected, invoke the multimodal understanding model. Dynamically select the processing path based on task complexity, and dynamically decide whether to invoke the multimodal understanding model for deep understanding to avoid unnecessary resource waste.

[0113] Phase 4: Multimodal understanding and deep processing of large models (conditional execution). If no changes are detected, skip this step.

[0114] Step 1: If the scheduling decision signal indicates that the multimodal understanding big model needs to be invoked, then input the current frame or video segment into the multimodal understanding big model.

[0115] Step 2: The multimodal understanding big data model performs deep semantic understanding on the video content and outputs deep semantic understanding results. This generates detailed semantic descriptions or answers semantic questions.

[0116] The multimodal understanding big model processes input in the following ways: (i) receiving structured meta data packets (containing object ID, filtered attribute vectors, and spatiotemporal coordinate markers); (ii) performing hierarchical reasoning: the primary layer completes scene classification based on visual features, the intermediate layer analyzes intent by combining spatiotemporal correlation, and the advanced layer generates multi-turn conversational semantic output; (iii) the output results include detailed semantic descriptions (such as "red car speeding and changing lanes") and action suggestions.

[0117] Phase 6: Output Results and Feedback

[0118] The multimodal understanding big model generates a final semantic description or answer based on the deep semantic understanding results, outputting the deep understanding results. This provides the system with detailed semantic information, supporting the execution of complex tasks. If there are changes, a deep description of the big model is output. If there are no changes, a list of objects and attributes of the lightweight model is output. The output results are applied in the following ways: (i) inputting the deep semantic understanding results into the embodied system's motion control module; (ii) synchronously updating the environmental knowledge graph database; and (iii) generating a task execution report for manual review.

[0119] Through the above steps, the goal of visual understanding is achieved while maintaining extremely low average complexity and power consumption.

[0120] This application proposes a visual collaborative processing method, aiming to achieve efficient understanding and processing of video content through the collaborative work of a visual perception model and a multimodal understanding model. The visual perception model, as a lightweight model, focuses on extracting objects and their inherent attributes (such as color, shape, size, etc.) from the input video sequence, and performing preliminary object detection and attribute perception. Due to its small parameter count, this model can perform fast inference while maintaining low power consumption. The object attribute aggregation module further processes the output of the visual perception model, aggregating object information and attribute information, but does not involve dynamic information such as state, quantity, space, time, and semantics. The purpose of this module is to monitor changes in objects and their attributes between the current frame and the previous frame to determine whether significant changes have occurred in the visual content.

[0121] When the visual perception model detects significant changes in an object or its attributes—such as the appearance of a new object or a sudden change in the attributes of an existing object—the system invokes the multimodal understanding big model for deep semantic parsing. The multimodal understanding big model is a parameter-intensive model capable of providing deep understanding, including scene classification, intent analysis, and multi-turn conversational semantic output. This on-demand invocation mechanism ensures that the complex multimodal understanding big model is used only when necessary, thus significantly reducing average complexity and power consumption while maintaining high performance. Conversely, if the object and its attributes do not change significantly, the system does not invoke the multimodal understanding big model and only utilizes the output of the visual perception model. This approach allows the system to operate with low power while maintaining high performance, making it suitable for resource-constrained environments such as embodied agents like robots. Through the collaborative work of the visual perception model and the multimodal understanding big model, efficient understanding and processing of video content is achieved while maintaining the system's flexibility and scalability. This design not only optimizes resource utilization but also improves the system's execution efficiency in complex task scenarios.

[0122] In summary, this application provides a visual collaborative processing method. It obtains target detection results by detecting target objects and / or their attributes in target video data; aggregates these results to obtain multi-dimensional object attribute information; determines whether the multi-dimensional object attribute information exhibits a change within a preset range between the current frame and previous frames in the target video data, obtaining a judgment result; and calls the corresponding processing model based on the judgment result, outputting the model collaborative processing result. This method rapidly identifies target objects and their attributes using a lightweight visual perception model, reducing computational complexity. It decouples complex visual information into multiple dimensions, generating rich object attribute information to support deep semantic understanding. It dynamically determines whether to call a large multimodal understanding model based on changes in the object and its attributes, avoiding unnecessary resource waste and improving system efficiency. In complex tasks, it calls the large multimodal understanding model to generate detailed semantic descriptions or answer semantic questions, supporting the execution of complex tasks. Through a dynamic scheduling mechanism, it flexibly adapts to different task complexities, ensuring efficient system operation in resource-constrained environments.

[0123] Based on the same technical concept, embodiments of this application also provide a visual collaborative processing system, such as... Figure 3 As shown, the system includes:

[0124] The target detection module 301 is used to detect target objects and / or attributes of the target objects in the target video data, and obtain target detection results;

[0125] Aggregation module 302 is used to aggregate the target detection results to obtain multi-dimensional object attribute information;

[0126] The judgment module 303 is used to determine whether there is a change within a preset range in the multi-dimensional object attribute information between the current frame and the previous frame in the target video data, and to obtain a judgment result.

[0127] The result output module 304 is used to call the corresponding processing model according to the judgment result and output the model collaborative processing result.

[0128] This application also provides an electronic device corresponding to the method provided in the foregoing embodiments. Please refer to... Figure 4 The diagram illustrates an electronic device provided by some embodiments of this application. The electronic device 20 may include: a processor 200, a memory 201, a bus 202, and a communication interface 203, wherein the processor 200, the communication interface 203, and the memory 201 are connected via the bus 202; the memory 201 stores a computer program that can run on the processor 200, and when the processor 200 runs the computer program, it executes the method provided by any of the foregoing embodiments of this application.

[0129] The memory 201 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one physical port (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0130] Bus 202 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used to store programs. After receiving an execution instruction, the processor 200 executes the program. The method disclosed in any of the foregoing embodiments of this application can be applied to the processor 200, or implemented by the processor 200.

[0131] The processor 200 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 200 or by instructions in software form. The processor 200 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 201. The processor 200 reads the information in memory 201 and, in conjunction with its hardware, completes the steps of the above method.

[0132] The electronic devices and methods provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.

[0133] This application also provides a computer-readable storage medium corresponding to the method provided in the foregoing embodiments. Please refer to... Figure 5The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored, which, when run by a processor, executes the methods provided in any of the foregoing embodiments.

[0134] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0135] The computer-readable storage medium provided in the above embodiments of this application and the method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0136] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0137] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0138] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made under the concept of the present invention using the contents of the present invention specification and drawings, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A method of visual co-processing, characterized by, The method comprises: detecting a target object and / or attributes of the target object in target video data to obtain a target detection result; aggregating the target detection result to obtain multi-dimensional object attribute information; judging whether the multi-dimensional object attribute information has a preset range of changes for a current frame and a previous frame in the target video data to obtain a judgment result; calling a corresponding processing model according to the judgment result and outputting a model collaborative processing result; The judgment result includes changes and no changes. The judgment result includes changes and no changes. According to the judgment result, the corresponding processing model is called, and the model collaborative processing result is output. If there is a change, the multi-dimensional object attribute information is input into a multi-modal understanding large model for deep semantic analysis, and a deep semantic understanding result is output. If there is no change, a lightweight object attribute list is output. The lightweight object attribute list includes objects and corresponding attributes that have no changes.

2. The method of claim 1, wherein, The multi-dimensional object attribute information is input into a multi-modal understanding large model for deep semantic analysis, and a deep semantic understanding result is output, which comprises: Based on the object identifier and the static attribute feature of the multi-dimensional object attribute information, scene category determination is performed; The result after determination is combined with the spatio-temporal correlation data to analyze the object behavior intention, and an analysis result is obtained. The spatio-temporal correlation data includes a time sequence event chain and a spatial topology relationship graph. Based on the analysis result, a multi-round dialogue semantic result is output.

3. The method of claim 1, wherein, Judging whether the multi-dimensional object attribute information has a preset range of changes to obtain a judgment result comprises: tracking the object trajectory of consecutive frames, and when the confidence is lower than a threshold, performing identification, the identification including adding identification or disappearing identification; and / or, calculating the Mahalanobis distance of the current frame attribute and the historical mean, and if the distance exceeds a set threshold, determining that a change has occurred.

4. The method of claim 1, wherein, The aggregation of the target detection result to obtain multi-dimensional object attribute information comprises: For the target detection result, perform: assign a unique identifier to each detected object, establish an object identity unique identifier mapping table; and / or, construct an inherent attribute feature vector library, the inherent attribute including the color, shape and size of the object; and / or, generate a state change event log, the state change event log being used to record dynamic state; and / or, count the object number distribution matrix; and / or, mark the positional relationship between objects to construct a spatial topology relationship graph; and / or, record the time correlation event chain; and / or, generate a basic semantic label.

5. The method of claim 4, wherein, After aggregating the target detection result to obtain multi-dimensional object attribute information, the method further comprises: According to a preset screening rule, filter out the dynamic attribute features. The preset screening rule includes: extracting the object identity identifier and at least one inherent attribute of color, shape and age; and / or, filtering features describing object state changes, the features describing object state changes including falling, damage and abnormal moving track; and / or, deleting object number distribution data; and / or, removing the spatial topology relationship between objects and the time correlation event chain; and / or, compressing complete semantic description into keyword phrases.

6. The method of claim 1, wherein, Detecting a target object and / or an attribute of the target object in target video data to obtain a target detection result, comprising: Performing target detection on the target video data using a lightweight neural network to identify a target object, the target object including a person, a vehicle, and an object; and / or extracting an inherent attribute of the target object detected by a classifier, the inherent attribute including a color attribute, a shape attribute, and a size attribute; and / or comparing target detection results of a current frame and a historical frame using an inter-frame difference analysis algorithm to generate an object identifier and / or an attribute change threshold quantization value.

7. A vision co-processing system, characterized by The system comprises: A target detection module configured to detect a target object and / or an attribute of the target object in target video data to obtain a target detection result; An aggregation module configured to aggregate the target detection result to obtain multi-dimensional object attribute information; A judgment module configured to judge whether the multi-dimensional object attribute information has a change within a preset range for a current frame and a previous frame in the target video data to obtain a judgment result; A result output module configured to call a corresponding processing model according to the judgment result and output a model collaborative processing result; the judgment result includes a change and no change; the calling of the corresponding processing model according to the judgment result and the output of the model collaborative processing result include: if there is a change, inputting the multi-dimensional object attribute information into a multi-modal understanding large model for deep semantic analysis to output a deep semantic understanding result; if there is no change, outputting a lightweight object attribute list, the lightweight object attribute list including an object and a corresponding attribute that have no change.

8. An electronic device comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that the processor executes the computer program to implement the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, A computer readable medium having stored thereon computer readable instructions executable by a processor to implement the method of any one of claims 1-6.

Images