Universal visual perception method for open world based on language guidance

Abstract

The invention provides a universal visual perception method for an open world based on language guidance, and belongs to the crossing field of computer vision and natural language processing, and the method comprises the steps: generating an initial multi-modal fusion representation, and obtaining the initial multi-modal fusion representation through the collection of a video frame sequence and a language instruction through processing and fusion; then determining a target candidate region, and carrying out matching, screening and optimization based on semantic keywords and visual region features; then generating a target identifier, and allocating a unique identifier to the high-confidence region; a continuous tracking trajectory sequence is formed, and a bounding box is updated and the trajectory is smoothed in combination with an algorithm; when the target disappears, the state vector of the target is temporarily stored, and the tracking identifier is recovered when a similar region appears; optimizing the tracking sequence, and adjusting a bounding box to generate an optimized sequence; and finally, outputting target motion trail, position and state information. Through multi-modal fusion, an optimization algorithm and a recovery mechanism, the open world target identification and tracking effect is effectively improved, and the practical value is high.

Description

Technical Field

[0001] This invention relates to the intersection of computer vision and natural language processing, and in particular to a language-guided, open-world universal visual perception method. Background Technology

[0002] In open-world environments, visual perception technology is crucial for intelligent systems such as autonomous driving and robot navigation, enabling them to understand complex scenes and interact with humans. However, existing methods still face significant limitations in dynamic and complex scenarios. Many solutions rely on predefined target categories or fixed visual features, making it difficult to adapt to the changing targets and environments in open worlds and unable to flexibly respond to natural language commands. For example, when untrained targets appear in the scene or when ambient lighting changes drastically, the system often fails to accurately identify or continuously track the targets.

[0003] The core challenge lies in how to achieve dynamic locking and continuous tracking of visual targets through language commands. First, establishing a real-time mapping between language descriptions and visual features is difficult. In open worlds, language commands such as "lock onto the pedestrian wearing blue clothes" contain rich semantic information, but the vision system needs to accurately associate these semantics with dynamically changing visual features. The lack of a dynamic mapping mechanism can cause the system to lose lock-on when the target's appearance changes or when there is background interference. Second, continuous management of the target's state becomes another critical issue. In complex scenes, targets may be occluded or temporarily disappear. The system needs to maintain a stable state management mechanism to ensure accurate tracking when the target reappears. For example, on a crowded street, the system needs to be able to relock onto the target based on previous language commands after a pedestrian is briefly obscured by a vehicle, rather than misjudging it as a new target. Summary of the Invention

[0004] The present invention aims to provide a language-guided open-world universal visual perception method to solve the above-mentioned problems in the prior art and improve the accuracy, continuity and universality of visual perception in open-world environments.

[0005] To address the above problems, this invention provides a language-guided, open-world universal visual perception method, comprising the following steps:

[0006] The text features and visual pixel data containing semantic information are collected through the input device to generate an initial multimodal fusion representation;

[0007] Based on the initial multimodal fusion representation, a contrastive language image pre-trained model is used to match semantic keywords with visual regions to determine the target candidate region;

[0008] From the identified target candidate regions, regions with bounding box confidence exceeding a threshold are selected as the primary targets for locking, and target identifiers are generated.

[0009] For the locked primary target and its identifier, the bounding box position is updated based on an online real-time tracking algorithm to form a continuous tracking trajectory sequence;

[0010] In the above continuous tracking trajectory sequence, if the target disappears for three consecutive frames, its state vector is predicted by the Kalman filter and temporarily stored.

[0011] When a similar appearance region appears in a new video frame, feature matching is performed based on the temporarily stored target state vector. If the match is successful, the tracking identifier is restored.

[0012] Based on the recovered tracking identifier, the tracking algorithm parameters are updated and the predicted state is incorporated into the bounding box estimation to obtain an optimized continuous tracking sequence;

[0013] The optimized continuous tracking sequence is analyzed to output the target's motion trajectory, position, and state information, thus completing a language-guided open-world universal visual perception process.

[0014] Preferably, the generation of the initial multimodal fusion representation includes:

[0015] First, video frame sequences and language commands are acquired through an input device, and the first video frame sequence and the first language command text are obtained through sampling.

[0016] For the first video frame sequence, if its resolution is lower than a preset threshold, image enhancement processing is performed to generate the second video frame sequence. At the same time, the first language instruction text is segmented and vectorized by natural language processing tools to obtain the first text feature vector.

[0017] Based on the second video frame sequence and the first text feature vector, a first alignment feature set is generated by aligning cross-modal data through timestamp matching;

[0018] For the first alignment feature set, a fusion feature encoding method is used to extract visual pixel features and concatenate them with the first text feature vector in a weighted manner to obtain an initial multimodal fusion representation.

[0019] Preferably, determining the target candidate region includes:

[0020] Extract a set of semantic keywords from language instructions, and then use a contrastive language image pre-training tool to extract features from the video frame sequence to obtain a set of visual region features based on the set of semantic keywords.

[0021] If the semantic relevance score between the semantic keyword set and the visual region feature set exceeds a preset threshold, the two are matched using the cosine similarity calculation method to determine the initial target candidate region set.

[0022] For the initial set of target candidate regions, spatial location analysis tools are used to delineate the region boundaries and dynamic target tracking methods are used to optimize their continuity, thereby obtaining target candidate regions.

[0023] Preferably, the generation of the target identifier includes:

[0024] The target candidate region and its bounding box coordinates are obtained from the image data through region detection, and the appearance feature vector is extracted.

[0025] If the confidence level of the bounding box coordinates is higher than a preset threshold, the region is selected as the primary target for locking by using coordinate filtering.

[0026] For the filtered target regions, a unique identifier is assigned to them using a hash mapping method to obtain the target identifier.

[0027] Preferably, forming a continuous tracking trajectory sequence includes:

[0028] Obtain the initial bounding box of the target identifier from the image frame, and calculate the position offset and feature similarity of the target between adjacent frames;

[0029] If the feature similarity is greater than the preset similarity threshold, the position offset and feature similarity are fused to determine the updated position of the bounding box;

[0030] The trajectory sequence is generated based on the updated position of the bounding box, and then smoothed by Kalman filtering to obtain a continuous tracking trajectory sequence.

[0031] Preferably, the temporary target state vector includes:

[0032] The detection results of the target bounding box in each frame are obtained from the sequence tracking. If it is determined that the target bounding box has not been detected in three consecutive frames, the target is confirmed to have disappeared.

[0033] Based on the disappearance detection results, Kalman filtering is used to iteratively calculate the historical motion state of the target to obtain the target's potential state.

[0034] By combining the potential states with a pre-established database of appearance descriptions, a temporary state vector containing position, velocity, and appearance descriptions is generated, and a predicted trajectory is generated using a linear interpolation method.

[0035] Preferably, the recovery tracking identifier includes:

[0036] New video frames are acquired from the video sequence, and motion regions are detected within them to obtain target candidate regions containing regions with similar appearances;

[0037] Extract the appearance feature vector for the target candidate region. If the matching score between the vector and the temporarily stored target state vector exceeds a preset threshold, it is determined to be the same target object.

[0038] Based on the judgment result, the position and velocity information is obtained from the target state vector and associated with the tracking identifier to obtain the recovered target tracking identifier.

[0039] Preferably, the optimized tracking sequence includes:

[0040] Based on the recovered tracking identifier, the state vector of the target object is obtained from the historical frame data and updated using the particle filter algorithm to obtain the predicted position of the target in the current frame;

[0041] By combining the predicted location with the image data of the current frame, the first bounding box is calculated using an object detection algorithm;

[0042] If the deviation between the first bounding box and the predicted position exceeds a preset threshold, adjust the coordinates of the first bounding box to obtain the second bounding box estimate;

[0043] Based on the second bounding box estimation, the inter-frame correlation data is updated, and an optimized continuous tracking sequence is generated through a sequence matching algorithm.

[0044] The present invention has the following beneficial effects:

[0045] 1. This invention effectively combines visual and linguistic information through multimodal fusion technology, improving the ability to identify targets in the open world and meeting the perception needs of various unknown targets.

[0046] 2. This invention employs a series of optimized algorithms and processing steps, which improves the accuracy and continuity of target tracking and reduces tracking interruptions.

[0047] 3. This invention implements a recovery tracking mechanism after the target disappears, enhancing the system's adaptability and robustness in complex environments. Attached Figure Description

[0048] Figure 1 The flowchart of a language-guided open-world universal visual perception method provided by this invention. Detailed Implementation

[0049] The present invention will be further described below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0050] Combination Figure 1This invention provides a language-guided, open-world universal visual perception method, specifically including:

[0051] Step 1: Collect text features and visual pixel data containing semantic information through the input device to generate an initial multimodal fusion representation;

[0052] Step 2: Based on the initial multimodal fusion representation, a contrastive language image pre-trained model is used to match semantic keywords with visual regions to determine the target candidate region;

[0053] Step 3: From the determined candidate target regions, select regions with bounding box confidence exceeding the threshold as the primary targets for locking, and generate target identifiers;

[0054] Step 4: For the locked primary target and its identifier, update the bounding box position based on the online real-time tracking algorithm to form a continuous tracking trajectory sequence;

[0055] Step 5: In the above continuous tracking trajectory sequence, if the target disappears for three consecutive frames, predict its state vector using a Kalman filter and temporarily store it;

[0056] Step 6: When a similar appearance region appears in a new video frame, feature matching is performed based on the temporarily stored target state vector. If the match is successful, the tracking identifier is restored.

[0057] Step 7: Based on the recovered tracking identifier, update the tracking algorithm parameters and incorporate the predicted state into the bounding box estimation to obtain the optimized continuous tracking sequence;

[0058] Step 8: Analyze the optimized continuous tracking sequence, output the target's motion trajectory, position and status information, and complete the language-guided open-world universal visual perception process.

[0059] The specific implementation process of this embodiment is as follows:

[0060] Step 1: Generate initial multimodal fusion representation

[0061] First, video frame sequences and language commands are acquired through an input device, resulting in a first video frame sequence and a first language command text. For the first video frame sequence, if its resolution is below a preset threshold, image enhancement processing is performed to generate a second video frame sequence. Image enhancement algorithms, such as contrast enhancement and noise reduction, are used to effectively improve image quality and provide clearer visual information for subsequent feature extraction. Simultaneously, the first language command text is segmented and vectorized using natural language processing tools to obtain a first text feature vector. The natural language processing tools accurately capture the semantic information in the language commands, making the text feature vector more representative.

[0062] Furthermore, based on the second video frame sequence and the first text feature vector, a first aligned feature set is generated by aligning cross-modal data through timestamp matching. In this embodiment, timestamp matching ensures the temporal consistency of visual and textual information, laying a good foundation for multimodal fusion. Then, for the first aligned feature set, a fusion feature encoding method is used to extract visual pixel features and weighted concatenate them with the first text feature vector to obtain an initial multimodal fusion representation. Moreover, the fusion feature encoding method can fully combine the advantages of visual and textual features, making the initial multimodal fusion representation more comprehensively reflect scene information. Compared with single-modal features, the recognition accuracy of multimodal fusion features is higher.

[0063] Step 2: Determine the target candidate region

[0064] This embodiment extracts a set of semantic keywords from language instructions, and then uses a contrastive language image pre-training tool to extract features from the video frame sequence based on this set of semantic keywords to obtain a set of visual region features. In this step, the contrastive language image pre-training tool is trained using large-scale image-text data and has powerful cross-modal feature extraction capabilities, enabling it to accurately extract visual region features related to semantic keywords.

[0065] If the semantic relevance score between the semantic keyword set and the visual region feature set exceeds a preset threshold, the two are matched using a cosine similarity calculation method to determine a preliminary target candidate region set. The cosine similarity formula is:

[0066] Where A is the semantic keyword feature vector and B is the visual region feature vector. Using cosine similarity calculation can accurately measure the similarity between semantic keywords and visual region features, improving the accuracy of target candidate region determination.

[0067] Furthermore, for the initial set of target candidate regions, spatial location analysis tools are used to delineate the region boundaries, and dynamic target tracking methods are employed to optimize their continuity, thus obtaining target candidate regions. This step, utilizing the combination of spatial location analysis and dynamic target tracking methods, effectively optimizes the continuity of the target candidate regions and reduces jitter and jumps within them.

[0068] Step 3: Generate target identifier

[0069] This embodiment obtains target candidate regions and their bounding box coordinates from image data through region detection, and extracts appearance feature vectors. The region detection algorithm accurately identifies target regions in the image, providing accurate location information for subsequent processing. If the confidence level of the bounding box coordinates is higher than a preset threshold, the region is selected as the primary target for tracking through coordinate filtering. The confidence level of the bounding box coordinates reflects the reliability of the region detection; selecting high-confidence regions as primary targets improves the accuracy of subsequent tracking.

[0070] Furthermore, for the filtered target regions, a unique identifier is assigned to them using a hash mapping method, resulting in a target identifier. This hash mapping method can quickly assign unique identifiers to targets, facilitating target tracking and management and improving system efficiency.

[0071] Step 4: Form a continuous tracking trajectory sequence

[0072] In this embodiment, the initial bounding box of the target identifier is obtained from the image frame, and the position offset and feature similarity of the target between adjacent frames are calculated. The position offset is calculated using an optical flow algorithm, which can accurately capture the motion trend of the target and provide a reliable basis for updating the bounding box. The feature similarity is calculated by extracting the feature points of the target using the SIFT algorithm. The SIFT algorithm has scale invariance and rotation invariance, and can accurately extract target features under different conditions, thus improving the accuracy of feature similarity calculation.

[0073] If the feature similarity is greater than a preset similarity threshold, the positional offset and feature similarity are fused to determine the updated position of the bounding box. The fusion formula is:

[0074] P update =α·P offset +(1-α)·P similarity

[0075] Among them, P update For the updated bounding box position, P offset P is the bounding box position obtained based on the position offset. similarity The bounding box position is obtained based on feature similarity, and α is the fusion coefficient, with a value range of [0,1]. By reasonably setting the fusion coefficient, the influence of position offset and feature similarity on the bounding box update can be comprehensively considered, making the bounding box update more accurate.

[0076] Next, a trajectory sequence is generated based on the updated positions of the bounding boxes, and then smoothed using a Kalman filter to obtain a continuous tracking trajectory sequence. The Kalman filter formula is as follows:

[0077] Prediction steps:

[0078] Update steps:

[0079]

[0080] P k|k =(IK k H k )P k|k-1

[0081] in, Let F be the predicted state at time k. k Here is the state transition matrix. Estimated state at time B k To control the input matrix, u k To control the input, P k|k-1 Let Q be the covariance matrix of the predicted state. k Let K be the process noise covariance matrix. k For Kalman gain, H k Let z be the observation matrix. k Let R be the observation value at time k. k To observe the noise covariance matrix, For the estimated state at time k, P k|k Let I be the covariance matrix for estimating the state, where I is the identity matrix.

[0082] It should be noted that using Kalman filtering can effectively smooth the trajectory sequence, reduce the impact of noise on the trajectory, and make the tracking trajectory more stable and continuous.

[0083] Step 5: Temporarily store the target state vector

[0084] This embodiment obtains the target bounding box detection result for each frame from sequence tracking. If it is determined that the target bounding box has not been detected within three consecutive frames, the target is confirmed to have disappeared. This embodiment determines whether the target has disappeared by using the detection results of multiple consecutive frames, which can reduce the probability of false positives. Based on the disappearance detection results, Kalman filtering is used to iteratively calculate the target's historical motion state to obtain the target's potential state. The iterative calculation of Kalman filtering can make full use of the target's historical motion information and accurately predict the target's potential motion state.

[0085] Simultaneously, by combining the potential states with a pre-established appearance description database, a temporary state vector containing position, velocity, and appearance descriptions is generated, and a predicted trajectory is generated using a linear interpolation method. The appearance description database stores the target's appearance feature information, facilitating its identification and tracking after it disappears; the linear interpolation method generates a smooth predicted trajectory, providing a reference for subsequent target recovery and tracking.

[0086] Step 6: Restore the tracking identifier

[0087] To achieve rapid recovery tracking after a target disappears and improve tracking continuity, this embodiment acquires new video frames from the video sequence and detects motion regions within them to obtain target candidate regions containing similar appearance areas. Appearance feature vectors are extracted from these candidate regions. If the matching score between these candidate regions and the temporarily stored target state vector exceeds a preset threshold, they are determined to be the same target object. The matching score is calculated using Euclidean distance, with the formula:

[0088] Among them, F i S is the i-th component of the eigenvector. i Let i be the i-th component of the target state vector.

[0089] Based on the above judgment results, the position and velocity information is then obtained from the target state vector and associated with the tracking identifier to obtain the recovered target tracking identifier.

[0090] Step 7: Optimize the tracking sequence

[0091] In this embodiment, the state vector of the target object is obtained from the historical frame data based on the recovered tracking identifier and updated by the particle filter algorithm to obtain the predicted position of the target in the current frame. Combining the predicted position with the image data of the current frame, the first bounding box is calculated by the target detection algorithm. The target detection algorithm can accurately obtain the actual position information of the target in the current frame.

[0092] If the deviation between the first bounding box and the predicted position exceeds a preset threshold, the coordinates of the first bounding box are adjusted to obtain the second bounding box estimate. Deviation adjustment enables the bounding box to more accurately surround the target and improves tracking accuracy.

[0093] Finally, based on the second bounding box estimation, the inter-frame correlation data is updated, and an optimized continuous tracking sequence is generated through a sequence matching algorithm. The sequence matching algorithm can ensure the coherence of inter-frame data, making the tracking sequence more optimized.

[0094] Step 8: Output the perception results

[0095] The optimized continuous tracking sequence is analyzed to output the target's trajectory, position, and state information, completing a language-guided open-world universal visual perception process. This process provides strong support for subsequent decision-making and applications by outputting comprehensive target information.

[0096] Contents not described in detail in this specification are prior art known to those skilled in the art. It is hereby indicated that the above description is intended to help those skilled in the art understand this invention, but does not limit the scope of protection of this invention. Any equivalent substitutions, modifications, improvements, or simplifications of the above descriptions that do not depart from the essential content of this invention fall within the scope of protection of this invention.

Claims

1. A language-guided, open-world universal visual perception method, characterized in that, The method includes: The text features and visual pixel data containing semantic information are collected through the input device to generate an initial multimodal fusion representation; Based on the initial multimodal fusion representation, a contrastive language image pre-trained model is used to match semantic keywords with visual regions to determine the target candidate region; From the identified target candidate regions, regions with bounding box confidence exceeding a threshold are selected as the primary targets for locking, and target identifiers are generated. For the locked primary target and its identifier, the bounding box position is updated based on an online real-time tracking algorithm to form a continuous tracking trajectory sequence; In the above continuous tracking trajectory sequence, if the target disappears for three consecutive frames, its state vector is predicted by the Kalman filter and temporarily stored. When a similar appearance region appears in a new video frame, feature matching is performed based on the temporarily stored target state vector. If the match is successful, the tracking identifier is restored. Based on the recovered tracking identifier, the tracking algorithm parameters are updated and the predicted state is incorporated into the bounding box estimation to obtain an optimized continuous tracking sequence; The optimized continuous tracking sequence is analyzed to output the target's motion trajectory, position, and state information, thus completing a language-guided open-world universal visual perception process.

2. The method according to claim 1, characterized in that, The generation of the initial multimodal fusion representation includes: First, video frame sequences and language commands are acquired through an input device, and the first video frame sequence and the first language command text are obtained through sampling. For the first video frame sequence, if its resolution is lower than a preset threshold, image enhancement processing is performed to generate the second video frame sequence. At the same time, the first language instruction text is segmented and vectorized by natural language processing tools to obtain the first text feature vector. Based on the second video frame sequence and the first text feature vector, a first alignment feature set is generated by aligning cross-modal data through timestamp matching; For the first alignment feature set, a fusion feature encoding method is used to extract visual pixel features and concatenate them with the first text feature vector in a weighted manner to obtain an initial multimodal fusion representation.

3. The method according to claim 1, characterized in that, The determination of the target candidate region includes: Extract a set of semantic keywords from language instructions, and then use a contrastive language image pre-training tool to extract features from the video frame sequence to obtain a set of visual region features based on the set of semantic keywords. If the semantic relevance score between the semantic keyword set and the visual region feature set exceeds a preset threshold, the two are matched using the cosine similarity calculation method to determine the initial target candidate region set. For the initial set of target candidate regions, spatial location analysis tools are used to delineate the region boundaries and dynamic target tracking methods are used to optimize their continuity, thereby obtaining target candidate regions.

4. The method according to claim 1, characterized in that, The generated target identifier includes: The target candidate region and its bounding box coordinates are obtained from the image data through region detection, and the appearance feature vector is extracted. If the confidence level of the bounding box coordinates is higher than a preset threshold, the region is selected as the primary target for locking by using coordinate filtering. For the filtered target regions, a unique identifier is assigned to them using a hash mapping method to obtain the target identifier.

5. The method according to claim 1, characterized in that, The formation of a continuous tracking trajectory sequence includes: Obtain the initial bounding box of the target identifier from the image frame, and calculate the position offset and feature similarity of the target between adjacent frames; If the feature similarity is greater than the preset similarity threshold, the position offset and feature similarity are fused to determine the updated position of the bounding box; The trajectory sequence is generated based on the updated position of the bounding box, and then smoothed by Kalman filtering to obtain a continuous tracking trajectory sequence.

6. The method according to claim 1, characterized in that, The temporarily stored target state vector includes: The detection results of the target bounding box in each frame are obtained from the sequence tracking. If it is determined that the target bounding box has not been detected in three consecutive frames, the target is confirmed to have disappeared. Based on the disappearance detection results, Kalman filtering is used to iteratively calculate the historical motion state of the target to obtain the target's potential state. By combining the potential states with a pre-established database of appearance descriptions, a temporary state vector containing position, velocity, and appearance descriptions is generated, and a predicted trajectory is generated using a linear interpolation method.

7. The method according to claim 1, characterized in that, The recovery tracking identifier includes: New video frames are acquired from the video sequence, and motion regions are detected within them to obtain target candidate regions containing regions with similar appearances; Extract the appearance feature vector for the target candidate region. If the matching score between the vector and the temporarily stored target state vector exceeds a preset threshold, it is determined to be the same target object. Based on the judgment result, the position and velocity information is obtained from the target state vector and associated with the tracking identifier to obtain the recovered target tracking identifier.

8. The method according to claim 1, characterized in that, The optimized tracking sequence includes: Based on the recovered tracking identifier, the state vector of the target object is obtained from the historical frame data and updated using the particle filter algorithm to obtain the predicted position of the target in the current frame; By combining the predicted location with the image data of the current frame, the first bounding box is calculated using an object detection algorithm; If the deviation between the first bounding box and the predicted position exceeds a preset threshold, adjust the coordinates of the first bounding box to obtain the second bounding box estimate; Based on the second bounding box estimation, the inter-frame correlation data is updated, and an optimized continuous tracking sequence is generated through a sequence matching algorithm.

Images