Target following method and device, computer equipment and readable storage medium

Abstract

The invention relates to a target following method and device, computer equipment and a computer readable storage medium. The method is applied to a robot, and specifically comprises the following steps: acquiring an image of a target object and point cloud information corresponding to a timestamp; acquiring a first image feature and a second image feature of the image; the first image feature represents an image space semantic feature; the second image feature represents an image depth feature; determining position information of the target object from the image based on a preset prompt word, the first image feature and the second image feature, and determining a candidate track of the target object based on the position information; determining a target trajectory of the target object based on the candidate trajectory and the point cloud information; the target track represents a track with visual depth; and according to the target track, following the target object. The method can improve the precision of target following.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a target following method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Target following technology, also known as target tracking technology, can be widely used in fields including security, medical assistance, industrial automation, etc., such as robot dogs, unmanned vehicles, etc. However, at present, the accuracy of target following is low, which affects the effect of target following. Therefore, how to improve the accuracy of target following is a problem that needs to be solved. Summary of the invention

[0003] Based on this, it is necessary to provide a target following method, apparatus, computer device, computer readable storage medium and computer program product to improve the accuracy of target following in response to the above technical problems.

[0004] In a first aspect, the present application provides a target following method, the method comprising:

[0005] Obtain the image of the target object and the corresponding point cloud information with timestamp;

[0006] Acquire a first image feature and a second image feature of the image; the first image feature represents a spatial semantic feature of the image; the second image feature represents a depth feature of the image;

[0007] Based on the preset prompt word, the first image feature and the second image feature, determining the position information of the target object from the image, and determining the candidate trajectory of the target object based on the position information;

[0008] Based on the candidate trajectories and point cloud information, the target trajectory of the target object is determined; the target trajectory represents the trajectory with visual depth;

[0009] Follow the target object according to the target trajectory.

[0010] In one embodiment, determining the location information of the target object from the image based on the preset prompt word, the first image feature and the second image feature includes:

[0011] Adjusting the second image feature based on a preset prompt word to obtain a third image feature;

[0012] According to the first image feature, the second image feature and the third image feature, position information of the target object is determined from the image, where the position information includes a detection frame and a mask segmentation feature.

[0013] In one embodiment, the second image feature is adjusted based on a preset prompt word to obtain a third image feature, including:

[0014] Obtain text features of the prompt word;

[0015] The text features and the second image features are processed based on the multi-attention mechanism to obtain the third image features.

[0016] In one embodiment, determining the location information of the target object from the image according to the first image feature, the second image feature and the third image feature includes:

[0017] Fusing the first image feature with the second image feature to obtain a first fused feature;

[0018] Multi-task joint processing is performed based on the first fusion feature and the third image feature to obtain a detection frame and a mask segmentation feature of the target object in the image.

[0019] In one embodiment, obtaining an image of a target object and corresponding point cloud information with a timestamp includes:

[0020] Acquire multiple images of the target object based on multiple image acquisition devices; and acquire corresponding point cloud information with time stamps based on laser radar;

[0021] Among them, multiple image acquisition devices correspond to different viewing angles; the direction of the laser radar is different from the viewing angle direction of any image acquisition device among the multiple image acquisition devices.

[0022] In one of the embodiments, the location information of the target object includes a detection frame and a mask segmentation feature of the target object;

[0023] Determine candidate trajectories of the target object based on the location information, including:

[0024] Different target objects are distinguished based on the detection frame and mask segmentation features corresponding to each target object;

[0025] For each target object, a single-view trajectory of the target object in multiple images of the same channel with different time stamps is determined, and a candidate trajectory of the target object is obtained based on multiple single-view trajectories of the target object in the multiple images.

[0026] In one embodiment, different target objects are distinguished based on the detection boxes and mask segmentation features corresponding to the target objects, including:

[0027] Based on real-time tracking of the target objects in the detection frame, a tracking identifier corresponding to each target object is determined;

[0028] Based on the detection frame and mask segmentation features, determine the color features of the target object corresponding to each tracking mark;

[0029] For multiple images with different time stamps on the same channel, the tracking marks are verified based on the similarity between the color features corresponding to the tracking marks, so that the same target object corresponds to the same tracking mark, and different target objects correspond to different tracking marks;

[0030] In one embodiment, obtaining a candidate trajectory of the target object based on multiple single-view trajectories of the target object in multiple images includes:

[0031] Based on the multi-channel images, multiple single-view trajectories of each target object are obtained, and the multiple single-view trajectories are fused with the color features corresponding to each target object to obtain a candidate trajectory of each target object.

[0032] In one embodiment, the method further comprises:

[0033] Obtain tracking instructions issued by users;

[0034] The tracking instruction represents the target object that the tracking device needs to track when there are multiple target objects;

[0035] Based on the tracking instruction, the corresponding prompt word is determined.

[0036] In a second aspect, the present application further provides a target following device, comprising:

[0037] A data acquisition module is used to obtain images of target objects and corresponding point cloud information with timestamps;

[0038] A trajectory determination module is used to obtain a first image feature and a second image feature of an image; the first image feature represents a spatial semantic feature of the image; the second image feature represents a depth feature of the image; based on a preset prompt word, the first image feature and the second image feature, the position information of the target object is determined from the image, and a candidate trajectory of the target object is determined based on the position information; based on the candidate trajectory and the point cloud information, a target trajectory of the target object is determined; the target trajectory represents a trajectory with visual depth;

[0039] The following module is used to follow the target object according to the target trajectory.

[0040] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the target following method in the first aspect when executing the computer program.

[0041] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the target following method in the first aspect.

[0042] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which implements the steps of the target following method in the first aspect when executed by a processor.

[0043] The target following method, apparatus, computer device, computer-readable storage medium and computer program product described above extract features of two different dimensions from the image of the target object to obtain image spatial semantic features and image depth features, and determine the position information of the target object by combining prompt words and the aforementioned extracted features of different dimensions, thereby making the position information of the target object more accurate; determine the candidate trajectory of the target object through the position information, and combine the candidate trajectory with the point cloud information to obtain a trajectory with visual depth, i.e., the target trajectory, so that the target trajectory can more accurately reflect the changes of the target object, i.e., by combining the corresponding advantages of the image and point cloud information, which helps to enhance the adaptability to environmental changes and the anti-interference ability, thereby improving the accuracy of target following. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 A schematic diagram of a flow chart of a target following method in one embodiment;

[0046] Figure 2 A schematic diagram of a flow chart of an intelligent following system in an embodiment;

[0047] Figure 3 A schematic diagram of a flow chart of a visual following system in one embodiment;

[0048] Figure 4 is a flow chart of a DMTP model in one embodiment;

[0049] Figure 5 is a diagram of the Vit Block module structure in one embodiment;

[0050] Figure 6 is a structural diagram of a Fusion Block module in one embodiment;

[0051] Figure 7 FIG. 4 is a block diagram of an MTA module in an embodiment;

[0052] Figure 8 is a block diagram of an MTI module in an embodiment;

[0053] Fig. 9 A schematic diagram of determining a target object based on a DMTP model in an embodiment;

[0054] Fig.10 A schematic diagram of a target tracking effect in one embodiment;

[0055] Fig.11 is a structural block diagram of a target following device in one embodiment;

[0056] Fig.12 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Unless otherwise specified, the "multiple" referred to herein can be understood as two or more.

[0058] The following is a preliminary introduction to target following related technologies:

[0059] First, visual tracking technology. Visual tracking is based on 2D or 3D visual information, and completes the tracking task by detecting and identifying the position of the target object. Commonly used methods include optical flow method, correlation filter method and deep learning method. In recent years, deep learning methods based on convolutional neural network (CNN) have shown strong performance in visual tracking. However, pure visual tracking methods are often easily affected by factors such as lighting changes, deformation of target objects, occlusion, etc., resulting in reduced tracking accuracy.

[0060] Second, LiDAR tracking technology. LiDAR obtains 3D point cloud information of the environment by emitting laser beams and receiving reflected signals. This technology has high precision and anti-interference capabilities, and is suitable for object recognition and tracking in complex environments. LiDAR is widely used in fields such as autonomous driving and drones. However, the computational complexity of LiDAR data is large, the real-time processing efficiency is low, and LiDAR has limited detection capabilities for small objects.

[0061] Third, multimodal fusion perception technology. In order to overcome the limitations of a single sensor, multimodal fusion technology has emerged, which fuses visual information with data from multiple sensors such as lidar and ultrasound to enhance the robustness of target tracking. This type of method can effectively solve the problems caused by a single perception method, such as occlusion and false detection in complex environments. However, when facing large-scale data, the real-time performance and computational efficiency of existing multimodal fusion methods are still a bottleneck.

[0062] Fourth, cross-camera tracking technology. Cross-camera tracking is an important target tracking technology, which is mainly used for object tracking between multiple cameras. Existing cross-camera tracking algorithms are generally based on the matching of target appearance features, using technologies such as ReID (re-identification) combined with deep learning and feature extraction methods. However, existing methods are still prone to mistracking and missed tracking when faced with occlusion, lighting changes, and differences in viewing angles between cameras.

[0063] However, the above technology has the following shortcomings:

[0064] First, the limitations of a single perception method. Whether it is visual tracking or LiDAR tracking, a single perception method has obvious limitations. Visual tracking is sensitive to changes in ambient lighting and is easily blocked by dynamic objects, especially in complex environments where tracking performance is unstable. Although LiDAR can solve the occlusion problem to a certain extent, when detecting target objects with fewer details (such as smaller objects), LiDAR's accuracy and detection rate are insufficient, and the data processing complexity is high.

[0065] Second, multimodal fusion processing is inefficient. In practical applications, how to efficiently and in real time process data from multiple sensors is still a difficult problem. Especially in dynamic environments, facing massive amounts of visual data, point cloud data, and other sensor information, fusion algorithms often fail to achieve a good balance between real-time performance and accuracy, resulting in system response delays and affecting tracking effects.

[0066] Third, the accuracy of cross-shot tracking is insufficient. In cross-shot scenarios, the target object may appear and disappear repeatedly in the field of view of different cameras, and the difference in viewing angles between cameras, changes in the target's appearance, and different lighting conditions will affect the stability of tracking. Existing cross-shot tracking algorithms often rely on global features of the target's appearance, but global features are not robust enough in most cases and are easily affected by environmental changes and noise.

[0067] Fourth, the tracking robustness in dynamic environments is insufficient. Object tracking in dynamic environments faces many challenges, such as rapid movement of targets, sudden appearance or disappearance, and mutual occlusion between multiple objects. These problems cause existing tracking algorithms to show great instability in dynamic environments. Especially in complex social scenarios, embodied robots need to handle multiple tasks at the same time, such as interaction and decision-making. How to stably track targets in such scenarios is still a technical difficulty.

[0068] From the above analysis, we can see that factors such as perception method, processing of sensor data, cross-lens, dynamic environment, etc. will affect the effect of target tracking. Therefore, this requires better perception capabilities and tracking algorithms to improve the stability and accuracy of tracking.

[0069] Based on the above analysis, the present application provides a target following method, which uses image and point cloud data to follow the target, wherein the image is subjected to feature extraction in different dimensions and adjusted using prompt words, that is, text information is combined with visual information, and the target object in the image is subjected to more detailed instance segmentation and recognition, thereby obtaining the target trajectory, and achieving more accurate target following according to the target trajectory. The method is specifically described below through multiple embodiments:

[0070] In one embodiment, the target following method is applied to a robot for illustration, wherein the robot can be understood in a broad sense, and the robot is not limited to the form of a human. For example, the robot can be but not limited to various robots, robot dogs, robot cats, robot birds, drones, unmanned vehicles, etc.; the robot is not limited to the application industry and field. For example, the robot can be but not limited to robots in the fields of security, medical assistance, industrial automation, etc., specifically, monitoring robots, patrol robots and other intelligent devices. The robot can be a robot that performs tasks independently, or a robot that collaborates with other devices or humans. Exemplarily, the robot can be a vehicle, and the vehicle can be but not limited to various sedans, SUVs (sport utility vehicles, or suburban utility vehicles), MPVs (multi-purpose vehicles), sports cars, trucks, off-road vehicles, motorcycles, tricycles, battery vehicles, dump trucks, tractor vehicles, agricultural vehicles, special vehicles (such as sanitation vehicles, snowplows, bulldozers, etc.), etc. Exemplarily, the robot can be a new energy electric vehicle, and the method can be used to track pedestrians, vehicles, etc.

[0071] like Figure 1 As shown, the method includes steps S101 to S104, which are specifically as follows:

[0072] Step S101: Acquire an image of a target object and corresponding point cloud information with a timestamp.

[0073] The target object may be an object to be followed, for example, in a driving vehicle scenario, the target may be a pedestrian or an animal on the road, etc. The number of targets may be one, two, or more.

[0074] The image of the target object may be an image of the target object from a single angle or multiple angles; the image may be a real-time image or an image taken over a period of time.

[0075] Among them, the point cloud information can be information related to a point set that characterizes the distribution of the target object in three-dimensional space and its surface characteristics, and the point cloud information can have a timestamp corresponding to the aforementioned image; exemplarily, the point cloud information can include sparse point clouds or dense point clouds.

[0076] Exemplarily, the robot can simultaneously acquire the image of the target object and the corresponding point cloud information with a time stamp, or can acquire them in a certain order. For example, the robot can be provided with a data acquisition device, and the robot can first acquire the image of the target object and the corresponding point cloud information with a time stamp through the data acquisition device and put them into a cache area. Of course, the data acquisition device can be set independently of the robot, and the robot can acquire the acquired image and point cloud information from the data acquisition device. When target tracking is required, the robot can first acquire the image for subsequent image analysis and processing, and then acquire the corresponding point cloud information.

[0077] Step S102: Acquire a first image feature and a second image feature of the image; the first image feature represents a spatial semantic feature of the image; the second image feature represents a depth feature of the image; based on a preset prompt word, the first image feature and the second image feature, determine the position information of the target object from the image, and determine a candidate trajectory of the target object based on the position information.

[0078] Among them, image spatial semantic features can be information extracted from an image that can represent the meaning of the image content, such as the relative position and layout pattern between different elements in the image. Image depth features can be image features obtained based on deep learning related technologies, such as image features learned based on a convolutional neural network model.

[0079] Among them, the preset prompt words are prompt words associated with the target object. Different target objects may correspond to different prompt words. The preset prompt words may be texts used to improve the accuracy of target following. For example, when the target to be followed is a yellow bicycle, the corresponding prompt words may be descriptive texts related to the yellow bicycle. Exemplarily, a vocabulary of prompt words may be pre-set, and corresponding prompt words may be selected according to the requirements of the actual following task. At the same time, the prompt words and the vocabulary may also be updated.

[0080] The position information of the target object may be information indicating the position of the target object in the image. For example, the position information may include rough position information of the target object, such as a rough outline of the target object and a general relative relationship with other elements in the image; the position information may also include detailed position information of the target object, such as a specific component of the target object, corresponding detailed position information, and a detailed relative relationship with other elements in the image. For example, the target object can be quickly locked by the rough position information, and accurate tracking can be performed on the basis of locking the target object by the detailed position information.

[0081] Among them, the candidate trajectory can be the trajectory of the target object obtained based on the image. For example, the target object is in the non-motorized vehicle lane of the road in the image at time A, and in the motor vehicle lane in the image at time B. Therefore, the candidate trajectory of the target object in the time period from time A to time B can be obtained as from the non-motorized vehicle lane to the motor vehicle lane.

[0082] Exemplarily, the robot performs feature extraction on the image to obtain features of two dimensions - first image features and second image features. Through preset prompt words, first image features and second image features, the position information of the target object can be determined from the image, and thus the candidate trajectory of the target object can be determined based on the position information.

[0083] Step S103: determining a target trajectory of the target object based on the candidate trajectories and the point cloud information; the target trajectory represents a trajectory with visual depth.

[0084] For example, since the candidate trajectories are obtained based on images, the images may have quality problems that affect target tracking, for example, they are easily affected by lighting and occlusion. Similarly, although point cloud information has three-dimensional spatial characteristics and high precision, it lacks the ability to recognize colors and details. Therefore, the robot can combine the candidate trajectories with the point cloud information instead of using only one of the images and point cloud information. This combines the advantages of both to improve the accuracy of target tracking. Specifically, the robot can match and fuse the point cloud information with the candidate trajectories to obtain the target trajectory of the target object, that is, the trajectory with visual depth.

[0085] Step S104: follow the target object according to the target trajectory.

[0086] In the above target following method, the robot extracts features of two dimensions from the image to obtain image spatial semantic features and image depth features, and determines the position information of the target object by combining the prompt words and the aforementioned extracted features, so as to make the position information of the target object more accurate; the candidate trajectory of the target object is determined through the position information, and the candidate trajectory is combined with the point cloud information to obtain a trajectory with visual depth, i.e., the target trajectory, which enables the target trajectory to more accurately reflect the changes of the target object, i.e., by combining the corresponding advantages of the image and point cloud information, which helps to enhance the adaptability to environmental changes and the anti-interference ability, thereby improving the accuracy of target following.

[0087] In one of the embodiments, the “determining the location information of the target object from the image based on a preset prompt word, a first image feature, and a second image feature” in the aforementioned embodiment may include: adjusting the second image feature based on the preset prompt word to obtain a third image feature; determining the location information of the target object from the image based on the first image feature, the second image feature, and the third image feature, the location information including a detection box and a mask segmentation feature.

[0088] Among them, the detection frame can be a rectangular area used to determine the target object in the image, and the number of detection frames can be consistent with the number of target objects; the mask segmentation feature can be a feature used to perform mask segmentation on the target object.

[0089] For example, due to the complex and changeable situations in actual target tracking, such as changes in illumination, target occlusion or more background noise, relying solely on features extracted from the image may lead to a decrease in the accuracy of detecting the target object. Based on this, the robot can assist in the detection and distinction of the target object by incorporating prompt words and providing additional contextual clues. Specifically, the robot can use the prompt word to optimize the second image feature to obtain the third image feature. For example, when detecting through the prompt word "red car", it not only pays attention to the shape of the target object, but also actively screens the color information related to the prompt word. Then, the first image feature, the second image feature and the third image feature are combined to determine the location information of the target object from the image. The location information includes the detection box and the mask segmentation feature.

[0090] In this embodiment, the robot adjusts the second image feature through the prompt word to obtain the third image feature, so that the third image feature can meet the semantics required by the prompt word, thereby making the position information determined according to the first image feature, the second image feature and the third image feature more accurate, avoiding the influence of lighting changes and other factors on the accuracy of the position information, and being able to cope with complex and changeable actual situations.

[0091] In one of the embodiments, the "adjusting the second image feature based on a preset prompt word to obtain a third image feature" in the aforementioned embodiment may include: obtaining the text feature of the prompt word; processing the text feature and the second image feature based on a multi-attention mechanism to obtain the third image feature.

[0092] Exemplarily, the robot may perform feature extraction on the prompt word to obtain text features, such as semantic features. Exemplarily, the multi-attention mechanism may focus on the text features and the second image features, for example, the text features may be preliminarily processed, such as normalization, and the second image features may be preliminarily processed, such as convolution, and then the preliminarily processed text features and the second image features may be fused to obtain the third image features.

[0093] Exemplarily, after fusing the preliminarily processed text features and the second image features, the robot can also strengthen the feature representation of the fused features, and fuse the strengthened features with the aforementioned preliminarily processed text features again to obtain a third image feature, so that the third image feature can prominently reflect the semantics of the target object implied by the prompt word.

[0094] In this embodiment, the robot obtains the text features of the prompt word and processes the text features and the second image features based on the multi-attention mechanism to obtain the third image features, which enables the third image features to prominently reflect the semantics of the target object contained in the prompt word.

[0095] In one of the embodiments, the “determining the location information of the target object from the image based on the first image feature, the second image feature and the third image feature” in the aforementioned embodiment may include: fusing the first image feature with the second image feature to obtain a first fused feature; performing multi-task joint processing based on the first fused feature and the third image feature to obtain a detection box and mask segmentation features of the target object in the image.

[0096] Exemplarily, the second image feature may include sub-features of multiple scales. The robot may fuse the first image feature with each sub-feature of the second image feature respectively to obtain multiple fused features, and then fuse the multiple fused features again to obtain the first fused feature.

[0097] Exemplarily, after fusing the plurality of fused features as mentioned above, the robot may further extract and integrate relevant features, for example, by processing using ConvFFN (Convolutional Feed-Forward Network), and finally obtaining the first fused feature.

[0098] Exemplarily, the robot may fuse the first fusion feature and the third image feature to obtain a final feature, and then may determine a detection frame and a mask segmentation feature of the target object based on the final feature.

[0099] In this embodiment, the robot obtains a first fused feature by fusing the first image feature with the second image feature, thereby integrating and strengthening relevant features such as the spatial semantic features of the target object in the image; and by performing multi-task joint processing on the first fused feature and the third image feature, since the third image feature contains the semantics of the prompt word and the second image feature, it can further integrate and strengthen the relevant features of the target object in the image, thereby obtaining a more accurate detection frame and mask segmentation features of the target object.

[0100] In one of the embodiments, the "obtaining images of the target object and corresponding point cloud information with timestamps" in the aforementioned embodiment may include: obtaining multi-channel images of the target object based on multiple image acquisition devices; and obtaining corresponding point cloud information with timestamps based on a lidar; wherein the multiple image acquisition devices correspond to different viewing angles; and the orientation of the lidar is different from the viewing angle orientation of any of the multiple image acquisition devices.

[0101] For example, the robot may be provided with multiple image acquisition devices to obtain multiple images of the target object. Of course, the multiple image acquisition devices may also be provided independently of the robot, and the robot may obtain multiple images of the target object from the multiple image acquisition devices. For the laser radar, similarly to the image acquisition device, it may be provided by the robot itself or may be independent of the robot.

[0102] For example, the viewing angle of the image acquisition device can be a dynamic viewing angle, that is, the viewing angle can be automatically adjusted according to the moving position of the robot, thereby providing more flexible and complete environmental information capture. Similarly, the direction of the laser radar can also be appropriately adjusted according to actual conditions.

[0103] In this embodiment, since the image of the target object can be multi-channel and obtained from multiple perspectives, the image can contain more information about the target object. Since there is more information about the target object, the relevant features of the target object can be extracted more accurately and comprehensively, so that the subsequent target trajectory is more accurate and comprehensive, and the target tracking accuracy is improved. At the same time, since the direction of the laser radar is different from the viewing angle direction of the image acquisition device, the laser radar is not affected when the viewing angle direction of the image acquisition device is blocked.

[0104] In one of the embodiments, the position information of the target object in the aforementioned embodiment includes a detection frame and a mask segmentation feature of the target object; the “determining the candidate trajectory of the target object based on the position information” in the aforementioned embodiment may include: distinguishing different target objects based on the detection frame and mask segmentation features corresponding to each target object; for each target object, determining a single-view trajectory of the target object in multiple images of the same channel with different time stamps, and obtaining a candidate trajectory of the target object based on multiple single-view trajectories of the target object in multiple images.

[0105] For example, since the number of target objects may be different and the target objects corresponding to the detection boxes and mask segmentation features may not be unique, it is necessary to distinguish different target objects. For example, the distinction can be achieved based on the detection boxes and mask segmentation features corresponding to each target object, for example, based on the similarity between the features.

[0106] For example, since the image may be a multi-channel image and each channel of the image may obtain a single-view trajectory, the multiple single-view trajectories may be fused to obtain a candidate trajectory of the target object.

[0107] In this embodiment, detection boxes and mask segmentation features are used to distinguish different target objects, avoiding confusion and tracking of wrong target objects. Multiple single-view trajectories are obtained for multiple images, and candidate trajectories of the target objects are obtained based on the multiple single-view trajectories. This integrates the relevant information of the target objects in the multiple images, so that the candidate trajectories can more comprehensively and accurately reflect the situation of the target objects.

[0108] In one of the embodiments, the “distinguishing different target objects based on the detection frame and mask segmentation features corresponding to each target object” in the aforementioned embodiment may include: determining the tracking identifier corresponding to each target object based on real-time tracking of the target object in the detection frame; determining the color features of the target object corresponding to each tracking identifier based on the detection frame and mask segmentation features; verifying the tracking identifier based on the similarity between the color features corresponding to each tracking identifier for multiple images of the same channel with different timestamps, so that the same target object corresponds to the same tracking identifier, and different target objects correspond to different tracking identifiers; the “obtaining candidate trajectories of the target object based on multiple single-view trajectories of the target object in multiple images” in the aforementioned embodiment may include: obtaining multiple single-view trajectories of each target object based on the multiple images, and fusing the multiple single-view trajectories with the color features corresponding to each target object to obtain candidate trajectories of each target object.

[0109] Exemplarily, the robot can add a corresponding tracking mark to each target object, for example, real-time tracking of the target object in the detection frame to determine the tracking mark corresponding to each target object. At the same time, each tracking mark can be bound to the color feature of the target object, and the color feature can be obtained based on the detection frame and mask segmentation features, so that the similarity between different color features can be used to determine whether they belong to the same target object. If they do, they should correspond to the same tracking mark. According to this principle, the tracking mark can be verified so that the same target object corresponds to the same tracking mark, and different target objects correspond to different tracking marks.

[0110] For example, after determining the color features of the target objects corresponding to the tracking marks, the re-ID features corresponding to the target objects can also be determined based on the color features. The re-ID features can be features obtained based on pedestrian re-identification technology, which can include high-level semantic features of the features of the target objects (such as pedestrians). Furthermore, different target objects can be distinguished based on the feature similarity between the re-ID features.

[0111] Exemplarily, the robot can fuse multiple single-view trajectories with color features corresponding to each target object. For example, the similarity between color features can be used to determine whether they belong to the same target object, and to avoid identifying multiple tracking marks for the same target object in multiple images. For example, if the tracking mark of target object A in the single-view trajectory corresponding to the first image is a, and the tracking mark of target object A in the single-view trajectory corresponding to the second image is b, it means that different tracking marks are identified for the same target object in different images, and the tracking marks can be unified, such as a.

[0112] In this embodiment, the tracking identifier corresponding to each target object is determined by real-time tracking of the target object in the detection frame, the color feature of the target object corresponding to each tracking identifier is determined by the detection frame and mask segmentation features, and the tracking identifier and color feature are used to distinguish different target objects, that is, the same target object corresponds to the same tracking identifier, and different target objects correspond to different tracking identifiers. Therefore, it is possible to cope with the situation where the number of target objects is not unique, accurately identify and follow different target objects, and avoid confusion. At the same time, by fusing multiple single-view trajectories with the color features corresponding to each target object, the candidate trajectory of each target object is obtained. This ensures that the situation where the number of target objects is not unique does not affect the processing of multi-channel images and cross-lens, that is, different target objects also have candidate trajectories corresponding to multiple channels of images.

[0113] In one of the embodiments, the target following method in the aforementioned embodiment further includes: obtaining a tracking instruction issued by a user; the tracking instruction represents a target object that the tracking device needs to track when there are multiple target objects; and determining a corresponding prompt word based on the tracking instruction.

[0114] The tracking instructions may be issued in the form of, but not limited to, voice, text, gestures, and selections made based on devices such as a mouse, keyboard, button, joystick, screen, etc., as long as they can indicate to the robot the target object it needs to track.

[0115] Exemplarily, the robot can determine the corresponding prompt word based on the tracking instruction issued by the user. For example, the content of the image indicates that there are multiple vehicles traveling in the same direction ahead. The user can issue a tracking instruction to the robot to indicate that the user wants to track a vehicle with a yellow body color and a license plate number ABCD.

[0116] For example, after receiving the tracking instruction, the robot can extract features of the tracking instruction, such as semantic features, to generate corresponding prompt words. Of course, the robot can also select the closest prompt word from the prompt word library according to the tracking instruction as the required prompt word.

[0117] In this embodiment, the prompt words required for target following are determined by the tracking instruction issued by the user, so that the target following meets the user's following requirements and avoids following the wrong target object when there are multiple target objects, which helps to ensure the accuracy of target following.

[0118] In one embodiment, a target following method is provided, which can be applied to an intelligent following system for 3D dynamic following of a designated target, and the system can be applied to a robot dog. Embodied 3D perception tracking uses the robot dog's egocentric vision to track a target object in a dynamic 3D environment. When performing social tasks, the embodied robot dog needs to use 3D perception to track targets of interest in its surroundings. The front robot dog can be equipped with a laser radar and a binocular wide-angle camera, wherein the laser radar is facing downward and the binocular wide-angle camera is facing 40 degrees upward. Figure 2 As shown in the figure, the main process of the intelligent following system is given, as follows:

[0119] 1. Multiple views from multiple cameras are input into the visual tracking system, i.e. images of the target object are obtained;

[0120] 2. Multiple views output stable target trajectories within the field of view during visual following;

[0121] 3. The laser radar synchronously inputs the point cloud information corresponding to the timestamp, that is, obtains the point cloud information corresponding to the timestamp;

[0122] 4. The target trajectory is matched with the point cloud of the corresponding position to give the target trajectory real depth information;

[0123] 5. The system outputs 3D target trajectory with depth.

[0124] The target trajectory in process 2 is the candidate trajectory in the above embodiment, and the 3D target trajectory in process 5 is the target trajectory in the above embodiment. Processes 4 to 5 correspond to determining the target trajectory of the target object based on the candidate trajectory and point cloud information; the target trajectory represents a trajectory with visual depth.

[0125] The visual following system described above is described in detail below. Figure 3 As shown, it mainly processes the data collected by multiple cameras to obtain the target trajectory. The specific process is as follows:

[0126] 1. Multiple wide-angle cameras input multiple video images;

[0127] 2. Multiple video images enter the DMTP model respectively, and the target detection frame and mask segmentation features corresponding to the multiple images are output, that is, the detection frame and mask segmentation features;

[0128] 3. Use the ByteTrack target tracking algorithm to track the target detection frames of multiple images in real time and obtain the tracking ID of each target. Among them, ByteTrack is a multi-target tracker. It has efficient, fast and powerful multi-target tracking capabilities, as well as its flexibility and versatility in various application scenarios. ByteTrack helps solve the tracking problem of occluded objects and improves the accuracy and robustness of tracking by associating every detection frame instead of only the detection frames with high scores.

[0129] 4. Predict the RGB features corresponding to the detection frames of multiple images through the Vits pedestrian re-identification algorithm and output the re-id features Ri corresponding to each detection frame. Person re-identification (Person re-identification, Person re-ID), also known as pedestrian re-identification, is widely considered to be a sub-problem involving image retrieval. It is a technology that uses computer vision technology to determine whether there is a specific pedestrian in an image or video sequence.

[0130] 5. Align each target in the shot by performing cosine feature matching on the spatiotemporal re-ID of a single shot to avoid duplication of IDs identified in a single shot. Exemplarily, cosine similarity can be used to compare the target feature vectors detected in different frames of a single shot. For example, when a target is detected and its feature vector is extracted in the previous frame, and the same processing is performed in the current frame, the cosine similarity of the two feature vectors can be calculated to evaluate whether they represent the same target. If the similarity is high enough, it is considered to be the same target and its trajectory is updated.

[0131] 6. The target trajectory of each shot is fused with the target re-id, mainly by matching the target re-id of each shot through cosine features to avoid identifying multiple IDs of the same object in multiple shots.

[0132] 7. Finally, output the trajectory of each target.

[0133] Among them, processes 3 to 7 correspond to determining the candidate trajectory of the target object based on the position information. The DMTP model described above is explained as follows:

[0134] Traditional target detection models mainly rely on visual features, such as the shape, color, and texture of objects. However, in complex environments, such as when there are changes in lighting, target occlusion, or more background noise, relying solely on visual features may lead to decreased detection accuracy. By incorporating prompt words, the model can use language information to provide additional contextual clues to help it detect and distinguish targets more accurately. For example, when the model detects through the prompt word "red car", it not only pays attention to the shape of the target, but also actively filters the color information related to the prompt word, thereby improving the detection accuracy. This combination effectively reduces false detections and missed detections, especially when there are multiple similar objects. The prompt word can help the model accurately lock on to a specific target and improve the robustness of the detection.

[0135] When the target exhibits multiple complex or independent parts due to occlusion, background clutter and complex shape, existing methods often produce incomplete predicted masks. This is mainly because these methods rely too much on pixel-level correlation, focus on pixel similarity and ignore semantic information. This application proposes a spatial semantic block to integrate semantic information in ViT features into video object segmentation. This application uses the cls tag (class token) from the pre-trained ViT backbone to extract semantic priors, and uses the global average polling of image patches to extract the global information of the current frame.

[0136] See also Figure 4 As shown in the figure, a possible specific implementation of the DMTP model is provided, wherein feature refers to a feature, which is described in detail as follows:

[0137] 1. First, each batch of images and text prompts enter the model at the same time;

[0138] 2. By extracting the Embedding feature of the image and entering it into the ViT Block, the spatial semantic feature feature1 is obtained, that is, the first image feature of the image is obtained;

[0139] 3. Extract the depth information feature feature2 of the image through the CNN backbone network (resnet50), that is, obtain the second image feature of the image;

[0140] 4. Extract the language feature feature3 of the text prompt word through the text encoder (Bert);

[0141] 5. Feature 1 and feature 2 are fused through the Fusion Block module to obtain feature 4;

[0142] 6. Combine feature 3 and feature 2 through the MTA Module multi-attention module to obtain feature 5;

[0143] 7. Feature5 is repeated to obtain the initial feature representation feature6;

[0144] 8. Feature 4 and feature 6 enter the MTI Module to obtain feature 7;

[0145] 9. Feature 7 predicts the detection box information of the target through the head layer;

[0146] 10.Feature7 passes through the Decoder segmentation decoder module and obtains the instance segmentation mask result through the head layer.

[0147] Among them, processes 3 to 10 correspond to determining the location information of the target object from the image based on the preset prompt word, the first image feature and the second image feature.

[0148] After introducing the overall logic of the DMTP model, the following will provide a detailed description of the specific modules in the DMTP model.

[0149] The description of the Vit Block space semantic module is as follows:

[0150] like Figure 5As shown in the figure, the structure of the Vit Block module is given. The input feature size of the module is (B, C, H, W). After dividing the image into small patches (Patches) and embedding them into a high-dimensional vector space, the size of the input feature will be converted to (B, N, D). After being processed by the ViT Block module, the size of the output feature remains unchanged, which is still (B, N, D). Among them, B represents the batch size; N represents the number of image patches (N=H / P×W / P, the size of each image patch is P×P pixels); D represents the embedding dimension of each image patch (the vector dimension after linear transformation). For example, the spatial semantic feature feature1 can include the relationship between each small patch (patch) and the specific meanings represented by B, N, D, etc.

[0151] The specific workflow of the Vit Block spatial semantics module is as follows:

[0152] 1. The image patch features enter the Embedded patches module and output feature feature1;

[0153] 2. Feature 1 is input into the Norm module, the input is normalized to obtain feature 2;

[0154] 3. Feature 2 is input into the Multi-Head Self-Attention module, which outputs the inter-correlation feature feature 3 between image blocks;

[0155] 4. Feature1 and feature3 are combined to obtain feature4;

[0156] 5. Feature 4 is input into Norm and Multi-Layer Perceptron (MLP) to output feature 5;

[0157] 6. Feature 4 and feature 5 are fused to obtain feature 6;

[0158] 7.Feature6 is input into Transformer Encoder to obtain feature feature7.

[0159] The description of the Fusion Module is as follows:

[0160] like Figure 6As shown in the figure, the structure of the Fusion Module is given. The Fusion Module mainly fuses the spatial semantic features of Vit and the depth information of the image. The input sizes are (B, C, H, W) and (N, D), and the output size is (B, E, H, W), where E is the feature dimension after fusion. The specific workflow of the Fusion Module is as follows:

[0161] 1.Multi-scale feature contains three scale features, namely feature1, feature2 and feature3;

[0162] 2. Feature 1, feature 2 and feature 3 are fused with ViT feature respectively and then fused together to obtain feature 4;

[0163] 3.Feature4 is input into the ConvFFN (Convolutional Feed-Forward Network) module to obtain feature5.

[0164] The description of the MTA Module multimodal attention module is as follows:

[0165] like Figure 7 As shown in the figure, the structure of the MTA Module multimodal attention module is given. In the MTA (Multimodal Transformer Attention) module, the main task is to fuse the output features of the text encoder and the high-level semantic features of the image. This process involves the relationship between multimodal data (text and image), and generates more expressive joint features through self-attention mechanism and interactive fusion operations. The main input of the MTA module is the features of text and image, and its output is a feature representation that fuses the information of both.

[0166] Text feature size: (B, N_text, D_text); B: batch size; N_text: text sequence length (number of tokens in text); D_text: text feature dimension (output dimension of text encoder); image features are (B, C, H, W); the output size of the MTA module is: (B, N_output, D_output), where N_output: the sequence length of the output feature, which may be related to the text or image features, or the dimension generated by the Repeat operation; D_output: the dimension of the output feature, which is related to the fused feature dimension and may be consistent with the dimension of the text or image features.

[0167] The specific workflow of the MTA Module multimodal attention module is as follows:

[0168] 1. The high-level semantic features output by the CNN Module are input into the Conv convolution module to obtain feature1;

[0169] 2. The text encoding information output by Text Encoder is normalized by Norm to obtain feature 2;

[0170] 3. Feature1 and feature2 are combined to obtain feature3;

[0171] 4. Feature 3 is enhanced by MLP (Multi-Layer Perceptron) nonlinear transformation to obtain feature 4;

[0172] 5. Feature4 and feature2 are fused to get feature5.

[0173] The description of the MTI module is as follows:

[0174] like Figure 8 As shown in the figure, the structure of the MTA Module multimodal attention module is given. The MTI module, namely the Multimodal Transformer Integration module, is the MTI Module multi-task module mentioned above. In the MTI module, image features and initialization features (initialize) are input and processed to generate the final feature representation. The main task of the MTI module is to effectively fuse image features and initialization features to generate joint features that can be used for downstream tasks.

[0175] Input size: image features: (B, C, H, W); initialization features: (B, N_init, D_init) where B: batch size, N_init: sequence length of initialization features, D_init: dimension of initialization features;

[0176] Output size: Output features: (B, N_fused, D_fused); N_fused: the sequence length of the output feature, which is consistent with the input sequence length; D_fused: the dimension of the output feature;

[0177] The specific workflow of the MTI module is as follows:

[0178] 1. Input image features into the Fusion Block module to perform preliminary fusion of different modalities (such as image features and initialization features) to obtain feature 1;

[0179] 2. Feature 1 enters the Masked Multi-Head Attention module and outputs feature 2;

[0180] 3. Feature2 and feature1 enter Add & Norm, which mainly performs residual connection and normalization operations to obtain feature3;

[0181] 4. Feature3 and initialize are simultaneously input into the Multi-Head Attention module, and feature4 is output;

[0182] 5.Finally, feature4 is subjected to Add & Norm operation to obtain the output feature feature5.

[0183] See also Fig. 9 As shown in FIG. 1 , a possible schematic diagram of using the DMTP model to determine the target detection frame and obtain the specific features required for target tracking is given. The target object is a person, so the corresponding detection frame and mask segmentation features of each person in the figure can be obtained. For example, the corresponding probability can also be given for each detection frame to indicate the probability that the object in the detection frame belongs to the target object (person). Fig.10 As shown, a schematic diagram of the following effect obtained by using the above method and the DMTP model to follow the target is given, where the target object is a person, and each person has a corresponding identifier.

[0184] After introducing the DMTP model from the perspectives of the whole and the specific modules, a possible training method for the DMTP model is given below, including:

[0185] ① Data preparation. Data collection: Collect image datasets containing rich annotation information, including instance segmentation annotations and spatial semantic annotations. Data preprocessing: Perform preprocessing operations such as standardization, cropping, and scaling on the data to adapt to the model input requirements. Data enhancement: Apply data enhancement techniques (such as rotation, flipping, and color adjustment) to improve the generalization ability of the model.

[0186] ② Model initialization. Construct the network structure: Construct the network according to the architecture design of the DMTP model (for example, combining the Transformer and spatial semantic modules with the prompt word).

[0187] Initialize weights: Initialize using pre-trained model weights.

[0188] ③ Prompt word design. Generate prompt words: Design appropriate prompt words according to task requirements to guide the model's attention mechanism. Prompt word embedding: Convert prompt words into embedding vectors for fusion with image features.

[0189] ④ Model training. Define the loss function: select a loss function suitable for instance segmentation detection, including detection loss function (BCE Loss, CIOU Loss), instance segmentation loss function (Cross-Entropy Loss, Dice Loss, Intersection over Union Loss, Focal Loss); Forward propagation: input the image and prompt word into the model, and generate the prediction result through the forward propagation of the network; Calculate the loss: calculate the loss based on the prediction result and the true label; Back propagation: update the model weights through the back propagation algorithm and optimize the loss function; Iterative training: perform multiple rounds of training on the training set, adjust the learning rate and other hyperparameters to improve model performance.

[0190] ⑤ Verification and adjustment. Model verification: Evaluate the performance of the model on the verification set and calculate indicators such as mAP (mean average precision) and IoU (intersection over union). Hyperparameter adjustment: Adjust hyperparameters such as learning rate and batch size according to the verification results to further optimize the model. Model fine-tuning: Fine-tune the model on specific scenarios or data sets to improve the generalization ability and adaptability of the model.

[0191] ⑥ Testing and evaluation. Test model: Evaluate the performance of the final trained model on the test set; Performance evaluation: Use standard evaluation indicators (such as mAP, IoU, etc.) to evaluate the performance of the model on actual data; Error analysis: Analyze the error cases of the model and identify potential improvement directions.

[0192] The target following method provided in this embodiment has the following effects:

[0193] First, the comprehensive perception capability of egocentric vision. This method can be applied to the egocentric vision system of robots, robot dogs and other equipment. Through multi-view camera arrays and lidar sensors, it can obtain all-round information about the surrounding environment and overcome the problem of limited viewing angle of fixed cameras. Compared with existing static cameras, the egocentric vision system can automatically adjust the viewing angle according to the robot's moving position, providing more flexible and complete environmental information capture. In addition, egocentric vision can effectively reduce the risk of target loss. In traditional target tracking, objects are easily missed by the tracking system due to occlusion or out of field of view. Egocentric vision can enhance the ability to continuously track target objects through multi-view collaborative perception, thereby improving the stability and continuity of tracking. This design enables robots, robot dogs and other equipment to more accurately track and identify target objects when performing complex social tasks.

[0194] Second, the precise tracking of the cross-lens tracking algorithm. In traditional multi-camera systems, cross-lens tracking usually faces problems such as inconsistent perspectives, changes in illumination, and occlusion, resulting in low matching accuracy of target objects between different cameras. The cross-lens following algorithm proposed in this application improves the accuracy of cross-lens tracking by combining ReID (re-identification) technology and local feature extraction algorithm. Specifically, the cross-lens following algorithm can accurately match and identify the target object based on its appearance features, motion trajectory, and depth information when it moves to the field of view of other cameras, ensuring that the target object can still be continuously tracked when switching between cameras. This not only solves the common problems of mistracking and missed tracking in traditional cross-lens tracking, but also improves the tracking robustness in complex environments. In addition, the cross-lens algorithm combines the ability of deep learning, which can automatically adjust the weight of feature matching while calculating in real time, thereby coping with the visual feature differences of the target object caused by changes in illumination and angle. Therefore, it can still provide high-precision target tracking in cross-camera scenarios, providing technical support for the free movement of robots, robot dogs and other equipment in a wide range of environments.

[0195] Third, efficient perception by fusion of vision and laser radar 3D point cloud. In order to enhance the perception ability of the robot dog in complex 3D environments, a vision and laser radar 3D point cloud fusion system is proposed. By combining the 2D image captured by the camera with the 3D point cloud data generated by the laser radar, the system can accurately identify the position, shape and motion trajectory of the target object. Compared with the traditional single sensor perception method, multimodal fusion technology effectively makes up for the limitations of vision and laser radar. Visual perception has the advantages of high resolution and rich color information, but is easily affected by illumination and occlusion; while laser radar can generate high-precision point cloud data in three-dimensional space, but lacks the ability to recognize color and details. By combining the advantages of both, the system can perform more comprehensive analysis and tracking of target objects in complex dynamic scenes, which not only improves the accuracy of target detection, but also enhances the system's ability to adapt to environmental changes. In addition, the fusion of vision and laser radar also improves the system's anti-interference ability. In some environments with complex lighting or a lot of noise, traditional pure visual perception systems often find it difficult to accurately track targets, but with the assistance of 3D point clouds, the system can achieve more robust tracking effects in complex environments.

[0196] Fourth, intelligent instance segmentation of the DMTP model based on text prompts. The DMTP model uses natural language processing technology to combine text information with visual information, and performs more detailed instance segmentation and recognition of targets through language prompts. For example, users can guide the system to identify and track the specified target object by inputting "red basketball". This innovative interaction method simplifies the target calibration process and improves the intelligence level of the system. The DMTP model based on text prompts can effectively cope with complex tracking tasks in multiple targets and multiple scenarios. When there are multiple targets in the environment, users can select the target to be tracked through simple language instructions (i.e. tracking instructions), and the system will intelligently judge and perform instance segmentation operations based on text prompts. This method not only improves the user's interactive experience, but also greatly enhances the system's adaptability in complex scenarios, especially in social tasks or human-computer interactions, and can quickly adjust the tracking strategy according to task requirements.

[0197] Fifth, the robustness and real-time performance of the system are improved. In a dynamic 3D environment, the target object may frequently change position, shape, or even disappear and reappear. Through cross-camera following, vision and lidar fusion, and intelligent detection based on text prompts, efficient target tracking and recognition are achieved, ensuring that robots, robot dogs and other equipment can complete long-term and stable task execution in complex environments. Especially when dealing with highly dynamic social tasks, the system can respond to environmental changes in real time and flexibly adjust tracking strategies according to task requirements. This technological breakthrough enables robots, robot dogs and other equipment to play a role in more practical applications, such as security patrols, medical care, industrial automation, etc.

[0198] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0199] Based on the same inventive concept, the embodiment of the present application also provides a target following device for implementing the target following method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more target following device embodiments provided below can refer to the limitations of the target following method above, and will not be repeated here.

[0200] In an exemplary embodiment, Fig.11 As shown, a target following device 1100 is provided, comprising:

[0201] The data acquisition module 1101 is used to obtain the image of the target object and the corresponding point cloud information with a time stamp;

[0202] The trajectory determination module 1102 is used to obtain a first image feature and a second image feature of the image; the first image feature represents a spatial semantic feature of the image; the second image feature represents a depth feature of the image; based on a preset prompt word, the first image feature and the second image feature, the position information of the target object is determined from the image, and a candidate trajectory of the target object is determined based on the position information; based on the candidate trajectory and the point cloud information, a target trajectory of the target object is determined; the target trajectory represents a trajectory with visual depth;

[0203] The following module 1103 is used to follow the target object according to the target trajectory.

[0204] In one embodiment, the trajectory determination module 1102 is also used to determine the location information of the target object from the image based on a preset prompt word, a first image feature, and a second image feature, including: adjusting the second image feature based on the preset prompt word to obtain a third image feature; determining the location information of the target object from the image based on the first image feature, the second image feature, and the third image feature, the location information including a detection box and a mask segmentation feature.

[0205] In one embodiment, the trajectory determination module 1102 is further used to adjust the second image feature based on a preset prompt word to obtain a third image feature, including: obtaining a text feature of the prompt word; processing the text feature and the second image feature based on a multiple attention mechanism to obtain the third image feature.

[0206] In one embodiment, the trajectory determination module 1102 is also used to determine the location information of the target object from the image based on the first image feature, the second image feature and the third image feature, including: fusing the first image feature with the second image feature to obtain a first fused feature; performing multi-task joint processing based on the first fused feature and the third image feature to obtain a detection frame and mask segmentation features of the target object in the image.

[0207] In one embodiment, the data acquisition module 1101 is also used to acquire images of the target object and corresponding point cloud information with timestamps, including: acquiring multi-channel images of the target object based on multiple image acquisition devices; and acquiring corresponding point cloud information with timestamps based on a laser radar; wherein the multiple image acquisition devices correspond to different viewing angles; and the orientation of the laser radar is different from the viewing angle orientation of any of the multiple image acquisition devices.

[0208] In one of the embodiments, the location information of the target object includes a detection frame and a mask segmentation feature of the target object;

[0209] The trajectory determination module 1102 is also used to determine the candidate trajectory of the target object based on the position information, including: distinguishing different target objects based on the detection frame and mask segmentation features corresponding to each target object; for each target object, determining the single-view trajectory of the target object in multiple images of the same channel with different time stamps, and obtaining the candidate trajectory of the target object based on multiple single-view trajectories of the target object in multiple images.

[0210] In one embodiment, the trajectory determination module 1102 is also used to distinguish different target objects based on the detection frame and mask segmentation features corresponding to each target object, including: determining the tracking identifier corresponding to each target object based on real-time tracking of the target object in the detection frame; determining the color features of the target object corresponding to each tracking identifier based on the detection frame and mask segmentation features; verifying the tracking identifier based on the similarity between the color features corresponding to each tracking identifier for multiple images of the same channel with different timestamps, so that the same target object corresponds to the same tracking identifier, and different target objects correspond to different tracking identifiers; obtaining candidate trajectories of the target object based on multiple single-view trajectories of the target object in multiple images, including: obtaining multiple single-view trajectories of each target object based on the multiple images, and fusing the multiple single-view trajectories with the color features corresponding to each target object to obtain candidate trajectories of each target object.

[0211] In one embodiment, the data acquisition module 1101 is further used to obtain a tracking instruction issued by a user; the tracking instruction represents a target object that the tracking device needs to track when there are multiple target objects;

[0212] The track determination module 1102 is further configured to determine a corresponding prompt word based on the tracking instruction.

[0213] Each module in the target following device 1100 can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0214] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.12 As shown. The computer device includes a processor, a memory, an input / output interface, and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, a target object following method is implemented.

[0215] Those skilled in the art will understand that Fig.12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0216] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the target following method in the above-mentioned embodiment are implemented.

[0217] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the target following method in the above-mentioned embodiment are implemented.

[0218] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of the target following method in the above-mentioned embodiment when executed by a processor.

[0219] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0220] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0221] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A target object following method, characterized in that: The method comprises: Acquire an image of the target object and corresponding point cloud information with a timestamp; Acquire a first image feature and a second image feature of the image; the first image feature represents a spatial semantic feature of the image; and the second image feature represents a depth feature of the image; Based on a preset prompt word, the first image feature, and the second image feature, determining the position information of the target object from the image, and determining a candidate trajectory of the target object based on the position information; Determining a target trajectory of the target object based on the candidate trajectory and the point cloud information; the target trajectory represents a trajectory with visual depth; The target object is followed according to the target trajectory.

2. The method according to claim 1, characterized in that The determining the location information of the target object from the image based on the preset prompt word, the first image feature and the second image feature includes: Adjusting the second image feature based on a preset prompt word to obtain a third image feature; The location information of the target object is determined from the image according to the first image feature, the second image feature and the third image feature.

3. The method according to claim 2, characterized in that The adjusting the second image feature based on the preset prompt word to obtain the third image feature includes: Acquire text features of the prompt word; The text feature and the second image feature are processed based on a multiple attention mechanism to obtain the third image feature.

4. The method according to claim 2, characterized in that: Determining the location information of the target object from the image according to the first image feature, the second image feature, and the third image feature includes: Fusing the first image feature with the second image feature to obtain a first fused feature; Multi-task joint processing is performed based on the first fusion feature and the third image feature to obtain a detection frame and a mask segmentation feature of the target object in the image.

5. The method according to claim 1, characterized in that: The step of acquiring the image of the target object and the corresponding point cloud information with a timestamp includes: Acquire multiple images of the target object based on multiple image acquisition devices; and acquire corresponding point cloud information with time stamps based on laser radar; Among them, multiple image acquisition devices correspond to different viewing angles; the direction of the laser radar is different from the viewing angle direction of any image acquisition device among the multiple image acquisition devices.

6. The method according to claim 5, characterized in that The location information of the target object includes a detection frame and a mask segmentation feature of the target object; The determining the candidate trajectory of the target object based on the position information includes: Distinguishing different target objects based on the detection frames and the mask segmentation features corresponding to the target objects; For each target object, a single-view trajectory of the target object in multiple images of the same channel with different time stamps is determined, and a candidate trajectory of the target object is obtained based on the multiple single-view trajectories of the target object in the multiple images.

7. The method according to claim 6, characterized in that The distinguishing different target objects based on the detection frame and the mask segmentation features corresponding to each target object includes: Based on real-time tracking of the target objects in the detection frame, determining a tracking identifier corresponding to each of the target objects; Determining the color features of the target object corresponding to each tracking mark based on the detection frame and the mask segmentation features; For multiple images of the same channel with different time stamps, based on the similarity between the color features corresponding to the tracking marks, the tracking marks are verified so that the same target object corresponds to the same tracking mark, and different target objects correspond to different tracking marks; The step of obtaining a candidate trajectory of the target object based on the plurality of single-view trajectories of the target object in the multi-channel images comprises: Based on the multi-channel images, a plurality of single-view trajectories of each target object are obtained, and the plurality of single-view trajectories are fused with the color features corresponding to each target object to obtain a candidate trajectory of each target object.

8. A target object following device, characterized in that: The device comprises: A data acquisition module, used to acquire an image of the target object and corresponding point cloud information with a timestamp; A trajectory determination module, used to obtain a first image feature and a second image feature of the image; the first image feature represents a spatial semantic feature of the image; the second image feature represents a depth feature of the image; based on a preset prompt word, the first image feature and the second image feature, determine the position information of the target object from the image, and determine a candidate trajectory of the target object based on the position information; based on the candidate trajectory and the point cloud information, determine a target trajectory of the target object; the target trajectory represents a trajectory with visual depth; A following module is used to follow the target object according to the target trajectory.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images