A target following method, device, storage medium and product

Abstract

The application provides a target following method and device, a storage medium and a product, and relates to the technical field of robots. The agent device to which the method is applied is configured with multiple image acquisition devices, the field angles of the multiple image acquisition devices are combined to cover a horizontal omnidirectional field of view, and there is an overlapping field of view between adjacent image acquisition devices. The method comprises the following steps: acquiring an image acquired by at least one image acquisition device; performing target detection based on the image to obtain a target detection result; if it is determined according to the first detection result that a target object to be followed enters a target overlapping field of view, a second detection result is acquired, and the target objects in the first detection result and the second detection result are associated; and based on the position of the associated target object, the agent device is controlled to move following the target object. The application embodiment solves the problems of a complex mechanical rotating holder structure, mechanical wear during rotation, a large following response delay, and easy target loss.

Description

Technical Field

[0001] This application relates to the field of robotics technology, specifically to a target following method, device, storage medium, and product. Background Technology

[0002] With the continuous development of robotics technology, robots and other intelligent devices are widely used in scenarios such as home services, intelligent monitoring, and autonomous navigation to provide services to humans. In these application scenarios, intelligent devices typically need to have environmental perception and intelligent following capabilities.

[0003] Therefore, in traditional target following schemes, due to the limited field of view of a monocular camera, the camera body needs to be rotated frequently by a mechanical rotating gimbal in order to search for and follow the target. However, the mechanical rotating gimbal has a complex structure, mechanical wear during rotation, and a large follow-up response delay, which can easily lead to target loss. Summary of the Invention

[0004] In view of this, the embodiments of this application aim to provide a target following method, device, storage medium and product to solve the problems of complex mechanical rotating gimbal structure, mechanical wear during rotation, large following response delay and easy target loss in the prior art.

[0005] In a first aspect, one embodiment of this application provides a target following method applied to an intelligent agent device. The intelligent agent device is equipped with multiple image acquisition devices, the combined field of view of the multiple image acquisition devices covers a horizontal omnidirectional field of view, and there is a field of view overlap between adjacent image acquisition devices. The method includes: acquiring images acquired by at least one image acquisition device; performing target detection based on the images to obtain target detection results; if it is determined from the first detection result that the target object to be followed has entered the target field of view overlap area, then acquiring a second detection result and associating the first detection result with the target object in the second detection result, wherein the first detection result is the target detection result corresponding to the first image acquired by the first image acquisition device, the second detection result is the target detection result corresponding to the second image acquired by the second image acquisition device, the first image acquisition device and the second image acquisition device are adjacent and there is a target field of view overlap area; and controlling the intelligent agent device to move to follow the target object based on the position of the associated target object.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, associating the first detection result with the target object in the second detection result includes: obtaining association information between the target object in the first detection result and multiple objects to be associated in the second detection result; determining the first object associated with the target object from the multiple objects to be associated based on the association information; and associating the first object in the second detection result with the target object in the first detection result.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, the association information includes at least two of the following: the distance between the position of the target object in the first detection result and the position of the object to be associated in the second detection result; the similarity between the appearance features of the target object in the first detection result and the appearance features of the object to be associated in the second detection result; and the difference between the depth of the target object in the first detection result and the depth of the object to be associated in the second detection result.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, determining the first object associated with the target object from multiple objects to be associated based on association degree information includes: constructing an association cost function based on association degree information; calculating the association cost function, and determining the optimal associated object corresponding to the target object as the first object from multiple objects to be associated.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: if it is determined from the first detection result that the target object is occluded, then predict the motion trajectory of the target object; if the motion trajectory points to the field of view area corresponding to the adjacent third image acquisition device, then initiate the target detection process for the third image acquired by the third image acquisition device.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, target detection is performed based on the image to obtain target detection results, including: performing distortion perception and feature extraction on the image to obtain image features; and performing target detection on the image based on the image features to obtain target detection results.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, distortion sensing and feature extraction are performed on the image to obtain image features, including: projecting the pixels in the image onto a spherical coordinate system to obtain the spherical coordinates corresponding to the pixels; generating radial weights based on the incident angle in the spherical coordinates; and using deformable convolution to perform distortion sensing and feature extraction on the image based on the radial weights to obtain image features.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the target detection result includes the parameter information of the object detection box; after performing target detection on the image based on image features and obtaining the target detection result, the method further includes: adjusting the parameter information of the object detection box according to the incident angle corresponding to the center point of the object detection box, so as to achieve distortion correction of the object detection box.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, before controlling the intelligent agent device to move along with the target object based on the position of the associated target object, the method further includes: acquiring depth maps corresponding to images acquired by multiple image acquisition devices respectively; performing consistency processing on the depths corresponding to the same pixel in the multiple depth maps to obtain multiple depth maps with a unified depth; generating a panoramic depth map based on the multiple depth maps with a unified depth; and determining the position of the associated target object based on the panoramic depth map.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, consistency processing is performed on the depths corresponding to the same pixel in multiple depth maps to obtain multiple depth maps with a unified depth, including: projecting multiple depth maps onto a spherical coordinate system respectively; and based on the spherical coordinate system, performing depth consistency regularization processing on the same pixel in the region corresponding to the overlapping area of ​​the field of view in multiple depth maps to obtain multiple depth maps with a unified depth.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, obtaining the depth map corresponding to the images acquired by the multiple image acquisition devices includes: using a self-supervised training depth estimation model to perform depth estimation on the images acquired by the multiple image acquisition devices to obtain the depth map corresponding to the images acquired by the multiple image acquisition devices.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, a self-supervised training depth estimation model is used to perform depth estimation on images acquired by multiple image acquisition devices to obtain depth maps corresponding to the images acquired by the multiple image acquisition devices. This includes: inputting the images acquired by the image acquisition devices into the depth estimation model, using the depth estimation model to estimate the depth of pixels in the image, and outputting a relative depth; and restoring the relative depth to the absolute depth based on the installation heights corresponding to the multiple image acquisition devices to obtain the depth map corresponding to the image.

[0017] In conjunction with the first aspect, in some implementations of the first aspect, before using a self-supervised trained depth estimation model to perform depth estimation on images acquired by multiple image acquisition devices to obtain depth maps corresponding to the images acquired by the multiple image acquisition devices, the method further includes: acquiring sample images acquired by multiple image acquisition devices to obtain multiple sample images; and performing self-supervised training on an initial depth estimation model based on the acquisition viewpoints corresponding to the multiple image acquisition devices and the multiple sample images to obtain a depth estimation model.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, after the initial depth estimation model is self-supervised trained based on the acquisition viewpoints corresponding to multiple image acquisition devices and multiple sample images to obtain the depth estimation model, the method further includes: performing stereo matching on the image regions corresponding to the overlapping areas of the field of view between adjacent image acquisition devices in the sample images acquired synchronously by adjacent image acquisition devices, and calculating the absolute depth of multiple matching pixels in the image regions; using the absolute depth of multiple matching pixels as a supervision signal to adjust the model parameters of the depth estimation model.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, determining the location of the associated target object based on the panoramic depth map includes: determining the median depth of the object detection box where the associated target object is located based on the panoramic depth map; and determining the location of the associated target object based on the median depth.

[0020] Secondly, one embodiment of this application provides a target following device configured on an intelligent agent device. The intelligent agent device is equipped with multiple image acquisition devices, the combined field of view of which covers a horizontal omnidirectional field of view, and adjacent image acquisition devices have overlapping fields of view. The device includes: an acquisition module for acquiring images acquired by at least one image acquisition device; a detection module for performing target detection based on the images to obtain target detection results; an association module for acquiring a second detection result if, based on a first detection result, the target object to be followed enters the target overlapping field of view, and associating the first detection result with the target object in the second detection result, wherein the first detection result is the target detection result corresponding to a first image acquired by a first image acquisition device, the second detection result is the target detection result corresponding to a second image acquired by a second image acquisition device, and the first and second image acquisition devices are adjacent and have overlapping fields of view; and a control module for controlling the intelligent agent device to follow the target object based on the position of the associated target object.

[0021] Thirdly, one embodiment of this application provides an intelligent agent device, including: a plurality of image acquisition devices, wherein the horizontal field of view of the plurality of image acquisition devices is combined to cover a horizontal omnidirectional field of view and there is a field of view overlap between adjacent image acquisition devices; and a processor for executing the target following method described in the first aspect.

[0022] In conjunction with the third aspect, in some implementations of the third aspect, the multiple image acquisition devices include three horizontally distributed fisheye cameras, with the optical axes of the three fisheye cameras intersecting at an angle of 120° to each other.

[0023] Fourthly, one embodiment of this application provides a computer-readable storage medium storing a computer program for performing the target following method described in the first aspect.

[0024] Fifthly, one embodiment of this application provides an electronic device, the electronic device comprising: a processor; a memory for storing processor-executable instructions; the processor being configured to execute the target following method described in the first aspect.

[0025] In a sixth aspect, one embodiment of this application provides a computer program product including instructions that, when executed on an electronic device, cause the electronic device to implement the target following method described in the first aspect.

[0026] In this application, by deploying multiple image acquisition devices and combining their field of view angles to cover the horizontal omnidirectional field of view, the horizontal blind spot in the target tracking process is physically eliminated, eliminating the need to rely on a mechanical rotating gimbal to adjust the field of view direction. Simultaneously, by utilizing the overlapping area of ​​the fields of view between adjacent image acquisition devices as a "transition zone," the target objects in the target detection results corresponding to adjacent image acquisition devices are associated. This enables seamless and smooth transfer of target object position tracking when a target object moves from the field of view of one image acquisition device to the field of view of another, ensuring the continuity of target object tracking during cross-view switching. This allows the intelligent agent device to stably and uninterruptedly follow freely moving target objects, effectively reducing the probability of target loss. Compared to monocular camera target tracking schemes that rely on mechanical rotating gimbals, the hardware structure of the multiple image acquisition devices in this application embodiment is relatively simple and there is no mechanical wear. Furthermore, since this application embodiment can achieve target tracking without waiting for the gimbal to rotate and align, the tracking response latency is reduced, effectively avoiding tracking interruptions caused by gimbal rotation lag or target loss, thereby improving the system's reliability and practicality. Attached Figure Description

[0027] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0028] Figure 1 The diagram shown is a flowchart of a target following method provided in an embodiment of this application.

[0029] Figure 2 The diagram shown is a schematic flowchart of a target detection process provided in an embodiment of this application.

[0030] Figure 3 The diagram shown is a flowchart illustrating the target object location determination process according to an embodiment of this application.

[0031] Figure 4 The diagram shown is a schematic diagram of the target following device provided in an embodiment of this application.

[0032] Figure 5 The diagram shown is a structural schematic of an intelligent agent device provided in an embodiment of this application.

[0033] Figure 6 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented even without certain specific details. In some instances, methods and means well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0036] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0037] Furthermore, the terms “first,” “second,” “third,” and “fourth” are used only for distinguishing descriptions and should not be interpreted as indicating or implying relative importance.

[0038] With the continuous advancement and maturation of robotics technology, various intelligent devices, such as robots, have been widely applied in numerous practical scenarios, including home services, intelligent monitoring systems, and autonomous navigation tasks, greatly facilitating human life and work. In these diverse application scenarios, to better accomplish tasks and meet user needs, intelligent devices typically require the ability to accurately perceive their surroundings and possess intelligent target-following capabilities to ensure efficient execution of related tasks.

[0039] Therefore, in traditional target-following schemes, the limited field of view of a monocular camera (usually less than 120°) makes it easy to lose the target. Furthermore, monocular camera-based schemes often require frequent adjustments to the camera's orientation using a mechanical gimbal to search for and continuously track the target. However, this reliance on a mechanical gimbal presents several problems: First, the relatively complex structure of a mechanical gimbal increases both the design and manufacturing difficulty and maintenance costs. Second, the frequent rotation of the gimbal's mechanical components inevitably causes wear and tear, affecting the equipment's lifespan. Additionally, the slow response speed of mechanical rotation can lead to significant tracking delays, severely impacting the real-time performance of target tracking and potentially causing target loss due to untimely responses, thus greatly reducing the system's reliability and usability.

[0040] To address the aforementioned technical issues, embodiments of this application provide a target following method, device, storage medium, and product. By deploying multiple image acquisition devices and combining their field of view angles to cover the horizontal omnidirectional field of view, the horizontal blind spot in the target following process is physically eliminated, eliminating the need to rely on a mechanical rotating gimbal to adjust the field of view direction. Simultaneously, by utilizing the overlapping area of ​​the field of view between adjacent image acquisition devices as a "transition zone," the target objects in the target detection results corresponding to adjacent image acquisition devices are associated. This enables seamless and smooth transfer of target object position tracking when a target object moves from the field of view of one image acquisition device to the field of view of another, thereby ensuring the continuity of target object tracking during cross-view switching. This allows the intelligent agent device to stably and continuously follow freely moving target objects, effectively reducing the probability of target loss. Compared to target following schemes using monocular cameras that rely on mechanically rotating gimbals, the hardware structure of the multiple image acquisition devices in this application embodiment is relatively simple and there is no mechanical wear. At the same time, since this application embodiment can achieve target following without waiting for the gimbal to rotate and align, it reduces the following response delay and effectively avoids following interruptions caused by gimbal rotation lag or target loss, thereby improving the reliability and practicality of the system.

[0041] The following is combined with Figures 1 to 3 The target following method provided in this application is described in detail.

[0042] Figure 1The diagram shown is a flowchart illustrating a target following method according to an embodiment of this application. This method can be applied to intelligent agent devices. For example, an intelligent agent device may include a mobile robot, a drone, an autonomous vehicle, or other intelligent agent device with mobility capabilities. The intelligent agent device is equipped with multiple image acquisition devices, and the combined horizontal field of view of these multiple image acquisition devices covers a horizontal omnidirectional field of view, with overlapping areas between adjacent image acquisition devices.

[0043] like Figure 1 As shown, the target following method may include the following steps.

[0044] S110, acquire an image acquired by at least one image acquisition device.

[0045] In some examples, the image acquisition device can be a device for acquiring images of the environment surrounding the intelligent agent device, such as a camera or webcam. Further, the image acquisition device can be a camera with a large field of view (e.g., a field of view ≥ 180°), such as a fisheye camera.

[0046] For example, multiple image acquisition devices can be installed at a preset height position on the intelligent agent device. The combined horizontal field of view of these multiple image acquisition devices covers a horizontal 360° omnidirectional field of view, with a certain overlap area between adjacent image acquisition devices. For instance, three fisheye cameras can be installed on the intelligent agent device at a height of 1.2 meters above the ground. Each camera has a horizontal field of view ≥185° and a vertical field of view ≥185°. The optical axes of the three fisheye cameras are 120° apart, forming a horizontal 360° omnidirectional field of view coverage and a large vertical field of view perception. A 40° horizontal overlap area is maintained between adjacent cameras. Specifically, the three fisheye cameras may include a front camera, a left camera, and a right camera. The front camera faces directly forward of the device, with its optical axis parallel to the ground. The left camera rotates 120° counterclockwise relative to the front camera, and the right camera rotates 120° clockwise relative to the front camera. In addition, these multiple image acquisition devices can be rigidly fixed, meaning they do not have mechanical rotational freedom.

[0047] Based on the multiple image acquisition devices installed as described above, images acquired by the device can be obtained from at least one of the image acquisition devices. For example, images acquired by each of the multiple image acquisition devices can be obtained, or images acquired by one or more image acquisition devices that may capture the target object to be followed can be obtained.

[0048] S120, target detection is performed based on the image to obtain the target detection result.

[0049] In some examples, the images captured by the image acquisition device may contain one or more objects, among which a target object to be followed may be included. These objects may be movable targets such as people, animals, or vehicles, or obstacle targets such as tables, chairs, cabinets, or roadblocks; the limitations are not specified here. The target object may be a specific movable target to be followed, such as a person with a specific identity.

[0050] For example, object detection can be performed on an image, such as identifying and detecting all identifiable objects in the image to obtain object detection results. These results may include relevant information corresponding to each identified object, such as the category identifier of each object and parameter information of the object detection box representing the object's boundary location.

[0051] In some specific examples, an image can be input into a trained object detection model, which then detects identifiable objects in the image and outputs a detection result containing information such as category labels and bounding box parameters. This object detection model could be, for example, a YOLO algorithm network. It should be noted that the image input to the object detection model can be the original image, i.e., retaining the complete field of view without perspective cropping.

[0052] S130, if it is determined from the first detection result that the target object to be followed has entered the target visual field overlap area, then the second detection result is obtained, and the first detection result is associated with the target object in the second detection result. The first detection result is the target detection result corresponding to the first image acquired by the first image acquisition device, and the second detection result is the target detection result corresponding to the second image acquired by the second image acquisition device. The first image acquisition device and the second image acquisition device are adjacent to each other and there is a target visual field overlap area.

[0053] In some examples, the first image acquisition device and the second image acquisition device can be any two adjacent image acquisition devices from the plurality of image acquisition devices mentioned above, and there is an overlapping area of ​​the target field of view between the two image acquisition devices. The image acquired by the first image acquisition device is the first image, and the image acquired by the second image acquisition device is the second image.

[0054] For example, each image acquisition device can independently track objects detected within its field of view. For instance, based on the target detection result (i.e., the first detection result) corresponding to the first image acquired by the first image acquisition device, the DeepSORT algorithm can be used to track each object detected in the first image, maintaining the motion trajectory information of each object. This motion trajectory information can be, for example, […]. ,in, Let k be the two-dimensional position coordinate vector of the k-th object. Let be the boundary vector of the object detection box corresponding to the k-th object. Let k be the velocity vector of the k-th object. Let be the appearance feature vector of the k-th object.

[0055] If a target object to be followed is detected in the first image acquired by the first image acquisition device, the target object can be tracked and it can be determined whether it has entered the field of view overlap area between the first image acquisition device and the adjacent image acquisition device. If it has entered, a cross-device association processing flow can be triggered. For example, if the target object enters the target field of view overlap area between the first image acquisition device and the second image acquisition device, the target detection result (i.e., the second detection result) of the second image acquired by the adjacent second image acquisition device can be obtained, and the target object in the first detection result and the second detection result can be associated, for example, with a unified identity identifier.

[0056] In order to accurately find the object that matches the target object in the first detection result in the second detection result, in some embodiments, the above step S130 may include obtaining the second detection result and associating the first detection result with the target object in the second detection result. Specifically, this may include: obtaining the degree of association information between the target object in the first detection result and multiple objects to be associated in the second detection result; determining the first object associated with the target object from the multiple objects to be associated based on the degree of association information; and associating the first object in the second detection result with the target object in the first detection result.

[0057] In some examples, the correlation information can be related information that represents the degree of correlation between the target object and the object to be correlated, such as the distance between their locations, the similarity between their appearances, or the depth difference between them in the image. The object to be correlated can be any of the identifiable objects detected in the second detection result, or it can be any object located within the overlapping region of the target's visual field among all the identifiable objects detected in the second detection result.

[0058] For example, the position and other information of the objects in the first detection result and the second detection result can be projected onto a unified global coordinate system, and then the degree of association between the target object in the first detection result and multiple objects to be associated in the second detection result can be obtained under the global coordinate system.

[0059] In specific examples, for fisheye cameras, a spherical coordinate system can be used as the global coordinate system, and the intrinsic and extrinsic parameters of each camera can be obtained through joint calibration. The camera intrinsic parameters may include, for example, various distortion coefficients and mirror parameters, used to project points in the image coordinate system to the camera coordinate system; the camera extrinsic parameters may include, for example, the transformation matrix from the camera to the device base coordinate system, used to project points in the camera coordinate system to the device base coordinate system. Furthermore, for the overlapping fields of view between adjacent cameras, feature matching can be performed on this overlapping region to determine the relative pose between the cameras. (That is, relative extrinsic parameters) to establish a global spherical coordinate system. The above camera calibration process can use a checkerboard target, combined with the objective function of minimizing spherical projection error. This objective function can be, for example, the function shown in the following formula (1), to determine the optimal extrinsic parameters corresponding to each camera.

[0060] in, For spherical projection functions, This refers to the coordinates of a point in three-dimensional space (such as the corner points of a chessboard in the device's base coordinate system). For camera internal parameters, For camera external parameters, These are the coordinates of the image observation points (such as the corner points of a checkerboard grid, which correspond to the corresponding pixel points observed on the image).

[0061] Based on this, the position and other information of the objects detected in the image can be projected from the image coordinate system to the device base coordinate system using the aforementioned calibrated camera intrinsic and extrinsic parameters. Then, the information can be projected to the global spherical coordinate system using a preset spherical projection function. The information of each object can be compared and verified in the spherical coordinate system to determine the degree of correlation.

[0062] To improve the accuracy of association, a multiple association mechanism can be used to determine the degree of association between the target object and the object to be associated. In some implementations, the association degree information includes at least two of the following: the distance between the position of the target object in the first detection result and the position of the object to be associated in the second detection result; the similarity between the appearance features of the target object in the first detection result and the appearance features of the object to be associated in the second detection result; and the difference between the depth of the target object in the first detection result and the depth of the object to be associated in the second detection result.

[0063] In some examples, the positions of the target object in the first detection result and the objects to be associated in the second detection result can be projected from the image coordinate system to the global spherical coordinate system using camera intrinsic and extrinsic parameters, and the geometric distance between them can be calculated in the global spherical coordinate system. .

[0064] In other examples, a pre-defined neural network can be used to extract the appearance features of the target object in the first detection result and the object to be associated in the second detection result, and calculate the similarity between the two (such as cosine similarity). The pre-defined neural network used could be, for example, OSNet.

[0065] In some other examples, monocular depth estimation can be combined to calculate the 3D positions of the target object in the first detection result and the object to be associated in the second detection result, so as to determine the difference between their depths and help verify their geometric consistency.

[0066] For example, the degree of association between associated objects can be determined by combining at least two of the above three types of association information. Then, the first object with the highest degree of association with the target object can be selected from the objects to be associated. The identity of the first object is unified with the identity of the target object, thereby realizing the identity of the target object can be unified during the cross-device tracking process, which facilitates continuous and uninterrupted tracking of the target object and reduces the probability of target loss.

[0067] Additionally, it should be noted that not only can cross-device association identification be performed on target objects, but also on all detected objects. The specific method is the same as above and will not be repeated here.

[0068] In this way, by comprehensively evaluating and utilizing multi-dimensional correlation information such as location distance, appearance feature similarity, and depth difference, richer and more reliable decision-making basis is provided for cross-device detection result matching. This significantly improves the accuracy and robustness of target object identity association in complex scenarios (such as when there are multiple similar targets or partial occlusion), and effectively reduces the probability of false association and identity jump.

[0069] In addition, to further improve the accuracy of the assessment of the degree of object association, in some implementations, the above-mentioned determination of the first object associated with the target object from multiple objects to be associated based on the degree of association information may specifically include: constructing an association cost function based on the degree of association information; calculating the association cost function, and determining the optimal associated object corresponding to the target object as the first object from multiple objects to be associated.

[0070] For example, the association cost function shown in formula (2) can be constructed based on the above three types of association information.

[0071] in, The target object in the first detection result. The object to be associated in the second detection result. This is the geometric distance between the position of the target object and the position of the object to be associated. Cosine similarity between the appearance features of the target object and the appearance features of the object to be associated. For the depth of the target object, The depth of the object to be associated. , , This is the weighting coefficient, which can be set according to actual needs. This represents the value of the association cost function.

[0072] Based on the aforementioned association cost function, the Hungarian Algorithm can be used to solve for the optimal association, and then the optimal associated object corresponding to the target object can be determined from multiple objects to be associated as the first object, thereby realizing the unified identification of the target object across devices.

[0073] Thus, by formalizing the association decision problem into a mathematical problem of finding the optimal solution to the association cost function, and using optimization algorithms to determine the final matching relationship, the decision-making process for cross-device target association becomes more rigorous and objective. It can find theoretically better matching schemes at the global level, improve the stability and optimality of the association results, and further improve the accuracy of the evaluation of the degree of object association.

[0074] S140, based on the position of the associated target object, control the intelligent agent device to follow the target object.

[0075] For example, after the target object is associated with other devices, a corresponding follow control command can be generated based on the three-dimensional position of the target object with the unified identification, and the follow control command can be output to the chassis motion controller to control the intelligent device to follow the target object.

[0076] In some specific examples, based on the three-dimensional position coordinates of the target object, a follow control command can be determined according to a preset follow control strategy. This preset follow control strategy may include a distance-keeping strategy, an orientation alignment strategy, and an obstacle avoidance fusion strategy. The distance-keeping strategy may include setting a follow distance and then calculating the linear velocity of the device's movement. The orientation alignment strategy may include calculating the horizontal azimuth angle of the target object and then calculating the angular velocity of the device's movement. The obstacle avoidance fusion strategy may include detecting obstacles on the device's movement path and triggering an emergency stop when the distance between the device and an obstacle is less than a preset safety distance. Based on the above follow control strategies, corresponding follow control commands can be generated. ),in, Linear velocity, ω is the angular velocity. The chassis motion controller responds to this follow control command by performing obstacle avoidance checks and safety constraints, and controls the intelligent agent device to follow the target object.

[0077] In this embodiment, by deploying multiple image acquisition devices and combining their fields of view to cover the horizontal omnidirectional field of view, the horizontal blind spot in the target tracking process is physically eliminated, eliminating the need to rely on a mechanical rotating gimbal to adjust the field of view direction. Simultaneously, by utilizing the overlapping area of ​​the fields of view between adjacent image acquisition devices as a "transition zone," the target objects in the target detection results corresponding to adjacent image acquisition devices are associated. This enables seamless and smooth transfer of target object position tracking when a target object moves from the field of view of one image acquisition device to the field of view of another, ensuring the continuity of target object tracking during cross-view switching. This allows the intelligent agent device to stably and uninterruptedly follow freely moving target objects, effectively reducing the probability of target loss. Compared to monocular camera target tracking schemes that rely on mechanical rotating gimbals, the hardware structure of the multiple image acquisition devices in this embodiment is relatively simple and there is no mechanical wear. Furthermore, since this embodiment can achieve target tracking without waiting for the gimbal to rotate and align, the tracking response latency is reduced, effectively avoiding tracking interruptions caused by gimbal rotation lag or target loss, thereby improving the system's reliability and practicality.

[0078] In addition, based on the above embodiments, in order to further reduce the cross-device field of view switching delay, in some embodiments, the above target following method may further include: if it is determined that the target object is occluded according to the first detection result, then predict the motion trajectory of the target object; if the motion trajectory points to the field of view area corresponding to the adjacent third image acquisition device, then start the target detection process of the third image acquired by the third image acquisition device.

[0079] In some examples, the third image acquisition device may be an image acquisition device adjacent to the first image acquisition device, such as a second image acquisition device or other image acquisition devices, without limitation.

[0080] For example, to conserve computing resources, real-time target detection can be performed on images acquired by all image acquisition devices, instead of detecting the target object only when it enters or is about to enter the field of view of the image acquisition device. Based on this, when the target object is obscured by an obstacle, its trajectory can be predicted using a preset filtering algorithm. If the predicted trajectory passes through or points to the field of view of an adjacent camera, that camera can be activated in advance to perform target detection, initiating the target detection process for the image acquired by that camera. The preset filtering algorithm could be, for example, a Kalman filter.

[0081] Thus, in this embodiment, when a target object disappears from the field of view of the current image acquisition device due to occlusion, the system predicts the possible field of view range of adjacent image acquisition devices based on its motion trajectory and actively triggers the target detection function of the image acquisition device. This enables rapid recapture of targets that may be temporarily lost, shortens the delay of the target object re-entering the system's perception, enhances the system's tracking and recovery capabilities in the face of brief occlusion scenarios, and further reduces the latency of cross-device field of view switching.

[0082] In addition, when a fisheye camera is used as the image acquisition device, distortion correction is required during the target detection process to address the radial distortion characteristics of the fisheye camera.

[0083] Based on this, the following is combined with Figure 2 Describe in detail the object detection process in an image.

[0084] Figure 2 The diagram shown is a schematic flowchart of a target detection process provided in an embodiment of this application. Figure 1 Extending from the illustrated embodiment Figure 2 The illustrated embodiment will be described in detail below. Figure 2 The illustrated embodiments and Figure 1 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0085] like Figure 2 As shown, the above-mentioned target detection based on the image to obtain the target detection result (step S120 above) may specifically include the following steps.

[0086] S210 performs distortion sensing and feature extraction on the image to obtain image features.

[0087] In some examples, a fisheye adaptive feature extraction network can be embedded in the object detection model to perform distortion perception and feature extraction on the image, outputting distortion-corrected image features. Based on these image features, the object detection model can then be used for object detection.

[0088] To achieve distortion perception of the image, a spherical coordinate embedding layer can be further embedded in the encoder (such as a ResNet encoder or a lighter MobileNet encoder) of the fisheye adaptive feature extraction network to utilize the radial distortion characteristics of the fisheye camera for distortion perception and feature extraction. Based on this, in some embodiments, step S210 may specifically include: projecting pixels in the image onto a spherical coordinate system to obtain the spherical coordinates corresponding to the pixels; generating radial weights based on the incident angle in the spherical coordinates; and using deformable convolution to perform distortion perception and feature extraction on the image based on the radial weights to obtain image features.

[0089] For example, each pixel in the image can be projected onto a spherical coordinate system, and the spherical coordinates of each pixel can be calculated. ,in, It is the angle between the incident ray and the camera's optical axis (i.e., the angle of incidence). This is the azimuth angle. This is the angle of incidence. It can be calculated using a spherical equidistant projection model, which is established to account for the radial distortion characteristics of fisheye cameras and represents the coordinates of image pixels. With the angle of incidence The model of the mapping relationship between them. The spherical isometric projection model can be specifically the model shown in the following formula (3).

[0090] in, pixels in the image With camera optical center The distance between them For camera focal length, The angle of incidence is calculated based on the above. Each pixel in an image can be encoded in spherical coordinates using a spherical coordinate embedding layer, and its spherical coordinates can be calculated. .

[0091] In the spherical coordinate embedding layer, the spherical coordinates corresponding to each pixel can also be used. angle of incidence Generate adaptive weights, also known as radial weights. Among them, due to the image edge regions (such as The distortion is more severe in areas with larger values, making feature extraction more difficult. Therefore, higher weights can be assigned to these areas to enhance the network's attention to them.

[0092] In addition, deformable convolution can be used in the spherical coordinate embedding layer. The sampling position of deformable convolution can be dynamically shifted according to the radial weight and the pre-calculated local distortion parameters, so that the convolution kernel can adapt to the geometric distortion of different regions of the image, thereby extracting more accurate image features.

[0093] In this way, by projecting the image onto a spherical coordinate system that better conforms to the physical model of fisheye imaging, and generating adaptive weights based on the incident angle of the pixels to guide the feature extraction of deformable convolution, a deep coupling between the neural network feature extraction process and image distortion perception is achieved. This enables feature extraction to actively adapt to the degree of distortion in different regions of the image, further optimizing the quality of feature representation under distortion conditions.

[0094] S220 performs target detection on the image based on image features and obtains the target detection results.

[0095] For example, an improved YOLO architecture target detection model can be used, with the original image acquired by the image acquisition device as the model input, and the distortion-corrected image features extracted in the aforementioned steps used for target detection. The model can output the category identifier of each object, the parameter information of the object detection box, and the azimuth angle in the spherical coordinate system, etc., as the target detection result.

[0096] Furthermore, to address the issue of severely stretched objects at the bottom (e.g., near the ground) of images captured by fisheye cameras, the parameter information of the object detection box in that area can be adaptively adjusted to correct distortion. Based on this, in some embodiments, the target detection result includes the parameter information of the object detection box. Accordingly, after step S220 above, the target following method may further include: adjusting the parameter information of the object detection box according to the incident angle corresponding to the center point of the object detection box to achieve distortion correction of the object detection box.

[0097] In some examples, the parameter information of the object detection box may include the boundary position information of the object detection box, such as the position coordinates of the four boundary points of the detection box, the width and height of the detection box, etc.

[0098] For example, the width and height of the object detection box can be adaptively adjusted according to the following formula (4) to correct the distortion of the object detection box output by the model.

[0099] in, The adjusted width of the object detection box. The width of the object detection box before adjustment. The height of the adjusted object detection box. The height of the object detection box before adjustment. This is the angle of incidence corresponding to the center point of the object detection box. In other words, it can be determined based on the angle of incidence corresponding to the center point of the object detection box. The width of the object detection box is adjusted, while the height remains unchanged.

[0100] In this way, after obtaining the preliminary detection results, the parameter information of the detection box is corrected according to the incident angle corresponding to the center point of the object detection box in the spherical coordinate system. This allows the final output object detection box to more accurately reflect the geometric contour of the object in the real world, providing more precise position information for subsequent position-based control.

[0101] In this embodiment, by explicitly considering and adapting the geometric distortion characteristics of the image in the feature extraction stage before target detection, distortion perception and feature extraction are performed on the image, thereby effectively overcoming the negative impact of nonlinear image deformation caused by specific types of image acquisition devices (such as fisheye cameras) on the performance of traditional target detection algorithms and improving the target detection accuracy of distorted images.

[0102] In addition, based on the above embodiments, in order to achieve tracking of the target object, it is also necessary to accurately determine the three-dimensional position of the target object.

[0103] Based on this, the following is combined with Figure 3 Describe in detail the process of determining the location of the target object.

[0104] Figure 3 The diagram shown is a schematic flowchart illustrating the target object location determination process according to an embodiment of this application. Figure 1 and / or Figure 2 Extending from the illustrated embodiment Figure 3 The illustrated embodiment will be described in detail below. Figure 3 The illustrated embodiments and Figure 1 and / or Figure 2 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0105] like Figure 3 As shown, before controlling the intelligent agent device to move along with the target object based on the position of the associated target object (step S140), the following steps may be included.

[0106] S310: Obtain depth maps corresponding to images acquired by multiple image acquisition devices respectively.

[0107] In some examples, a trained depth estimation model can be used to estimate the depth of images acquired by multiple image acquisition devices to obtain corresponding depth maps. The depth estimation model may also include a fisheye adaptive feature extraction network, or share a fisheye adaptive feature extraction network with the object detection model. This allows for distortion perception and feature extraction of images acquired by fisheye cameras when the image acquisition device is a fisheye camera, and then depth estimation based on the distortion-corrected image features to obtain a more accurate depth map.

[0108] In addition, to reduce model training costs, a self-supervised training depth estimation model can be used to obtain the depth map corresponding to each image. Based on this, in some embodiments, step S310 may specifically include: using a self-supervised training depth estimation model to perform depth estimation on images acquired by multiple image acquisition devices respectively, to obtain the depth maps corresponding to the images acquired by the multiple image acquisition devices respectively.

[0109] In some examples, the depth estimation model described above can be based on FisheyeDistanceNet++ and trained using the Structure-from-Motion (SfM) self-supervised paradigm, without the need for ground truth depth annotations during training.

[0110] For example, when the image acquisition device is a fisheye camera, differentiable fisheye projection and backprojection layers can be embedded during the forward propagation of the depth estimation model. The projection layer projects 3D location points onto the image plane using a pre-defined fisheye camera projection model, such as the Kannala-Brandt model. The backprojection layer backprojects points in the depth map into spherical coordinates. These fisheye projection and backprojection layers can be used to calculate photometric loss during model training. Here, by replacing the standard pinhole camera projection model with a fisheye camera projection model that better conforms to fisheye imaging geometry, the photometric reprojection loss can be accurately calculated during model training.

[0111] In addition, since the brightness distribution of images captured by fisheye cameras varies greatly in different radial regions, instance normalization (IN) can be used instead of batch normalization (BN) in the depth estimation model. That is, instance normalization is used to normalize each image or sample image independently, which is more suitable for handling illumination changes caused by fisheye distortion.

[0112] In this way, since a self-supervised training depth estimation model is used for image depth estimation, and the depth estimation model does not need to use expensive depth sensors (such as radar, depth cameras, etc.) to collect ground truth data during the self-supervised training process, the training cost of the depth estimation model can be reduced.

[0113] Furthermore, since the depth predicted by the self-supervised monocular depth estimation model has scale uncertainty, the absolute scale can be recovered by further incorporating the installation height of the image acquisition device. Based on this, in some embodiments, the aforementioned method of using a self-supervised trained depth estimation model to estimate the depth of images acquired by multiple image acquisition devices, thereby obtaining depth maps corresponding to the images acquired by the multiple image acquisition devices, can specifically include: inputting the images acquired by the image acquisition devices into the depth estimation model; using the depth estimation model to estimate the depth of pixels in the image and outputting a relative depth; and based on the installation heights corresponding to the multiple image acquisition devices, restoring the relative depth to the absolute depth to obtain the depth map corresponding to the image.

[0114] For example, after the original image acquired by the image acquisition device is input into the depth estimation model, the depth estimation model can estimate the depth of each pixel in the image, and then output the depth value corresponding to each pixel. This depth value is a relative depth and has scale uncertainty.

[0115] In some specific examples, the fixed installation height of the image acquisition device on the intelligent agent device (e.g.) can be utilized. And, assuming that the bottom region of the image corresponds to the ground, the relative depth predicted by the model is compared with the ground depth calculated according to the following formula (5). Alignment is performed to restore the relative depth to the absolute depth, resulting in the final depth map.

[0116] in, The installation height of the image acquisition device, For ground depth, The angle of incidence is denoted as .

[0117] In this way, by utilizing the prior geometric information of the known installation height of the image acquisition device, the dimensionless relative depth value output by the self-supervised training depth estimation model can be restored to the actual physical scale, thereby solving the inherent scale ambiguity problem of monocular depth estimation and enabling the generated depth map to be directly used to determine the accurate location of the target object.

[0118] Furthermore, during the training of the depth estimation model, joint training can be performed on sample images acquired by multiple image acquisition devices to make the model invariant to different viewpoints. Based on this, in some embodiments, before using the self-supervised trained depth estimation model to estimate the depth of images acquired by multiple image acquisition devices and obtain depth maps corresponding to the images acquired by the multiple image acquisition devices, the method may further include: acquiring sample images acquired by the multiple image acquisition devices to obtain multiple sample images; and performing self-supervised training on the initial depth estimation model based on the acquisition viewpoints corresponding to the multiple image acquisition devices and the multiple sample images to obtain a depth estimation model.

[0119] For example, in order to train a depth estimation model that can accurately estimate depth from monocular images and is robust to different mounting perspectives, training data needs to be prepared and a specific training process needs to be performed.

[0120] First, a large number of sample images are acquired by multiple image acquisition devices on the intelligent agent device in a real-world operating environment, forming a training dataset. Since these image acquisition devices are rigidly mounted with fixed relative poses and each covers different horizontal directions, these sample images naturally contain information about the same scene observed from multiple different perspectives. For example, in a system with three fisheye cameras, the sample image set may include image sequences acquired synchronously or asynchronously by the front, left, and right cameras in various scenes (such as a living room or a corridor).

[0121] Based on these multi-view sample images and their corresponding acquisition perspectives, the initial depth estimation model is trained under self-supervised conditions. The training process mainly revolves around constructing and optimizing a comprehensive loss function. The core of the loss function is the photometric reprojection loss. For image frames with consecutive timestamps acquired by the same camera (temporal sequence), the camera pose changes between adjacent frames are estimated during training. Using the depth map predicted for the current frame, the image of the reference frame is projected onto the viewpoint of the current frame, and then the error between the projected image and the original image of the current frame is calculated (e.g., L1 loss combined with structural similarity (SSIM)). For image pairs (spatial sequence) acquired simultaneously by different cameras (e.g., front and left cameras), since the relative poses between the cameras are known and fixed through calibration, this external parameter can be directly used to project an image and its predicted depth map onto the viewpoints of adjacent cameras, and the photometric loss is calculated with the actual acquired image. This method, which utilizes fixed multi-view geometric constraints, provides a stronger and more stable supervisory signal than monocular temporal constraints.

[0122] Furthermore, to enable the model to distinguish and adapt to the viewing angle characteristics of different image acquisition devices, the encoding information of the acquisition viewpoint is added to the network input during training. For example, a learnable embedding vector (viewpoint encoding) is assigned to the front, left, and right cameras respectively, and this encoding is input into the model along with the sample image data. This allows the model to learn the distortion patterns or observation characteristics unique to different camera viewpoints simultaneously when learning depth estimation, thereby enhancing the model's generalization ability to images acquired from different acquisition viewpoints during inference.

[0123] Besides photometric loss, other auxiliary losses can be introduced during training to improve the quality of depth estimation. For example, geometric consistency loss can be introduced, utilizing a pre-trained optical flow network to estimate pixel correspondences between image frames, which helps reduce ambiguity in depth estimation for dynamic objects and occluded regions. Furthermore, considering the drastic depth changes in indoor environments, a scale regression network can be introduced into the model to predict the global depth scale factor. The depth output is decomposed into: .

[0124] Furthermore, to minimize the scale uncertainty of the depth map predicted by the depth estimation model, the traditional pose estimation network (such as PoseNet) used during training to estimate the camera pose changes between adjacent frames can be replaced by using odometry or an inertial measurement unit (IMU) to obtain the true scale pose changes between adjacent frames, thus providing a true scale supervision signal. Simultaneously, a scale alignment term as shown in formula (6) is added to the loss function.

[0125] in, For scale loss, For the predicted camera pose, The actual camera pose obtained from the IMU.

[0126] Finally, by optimizing the combined loss function using gradient descent on a large-scale multi-view sample image dataset, the parameters of the initial depth estimation model are continuously adjusted, eventually converging into a stable depth estimation model. This model can receive a single image captured by any image acquisition device and output the corresponding depth map, where each pixel value represents the relative depth of that point with respect to the camera. This training process is completely self-supervised, does not rely on ground truth labeled data, and greatly reduces the cost of model acquisition.

[0127] Furthermore, this application embodiment also provides another absolute scale recovery method by incorporating the overlapping field of view between adjacent image acquisition devices. Based on this, in some embodiments, after performing self-supervised training on the initial depth estimation model based on the acquisition viewpoints corresponding to multiple image acquisition devices and multiple sample images to obtain the depth estimation model, the method may further include: performing stereo matching on the image regions corresponding to the overlapping field of view between adjacent image acquisition devices in the sample images synchronously acquired by adjacent image acquisition devices, and calculating the absolute depth of multiple matching pixels in the image regions; using the absolute depth of the multiple matching pixels as a supervision signal to adjust the model parameters of the depth estimation model.

[0128] For example, the overlapping area of ​​the field of view between adjacent image acquisition devices can be used to perform stereo matching on the original sample images acquired simultaneously from two channels, obtaining multiple pixels with matching features, i.e., matching pixels. Based on pre-calibrated camera intrinsic parameters and relative extrinsic parameters, the absolute depth of the spatial point corresponding to each matching pixel is calculated through triangulation, obtaining the absolute depth ground truth of multiple sparse matching pixels. Using this sparse absolute depth as a "pseudo-ground truth" (i.e., supervision signal), the model parameters of the monocular depth estimation model are fine-tuned online through a knowledge distillation framework, so that the depth map output by the model is aligned from relative depth to absolute physical scale, obtaining an absolute scale depth map.

[0129] In this way, by using the overlapping area of ​​the cross-device field of view for stereo matching and calculating the absolute depth of the matched pixels in the area as a supervision signal to adjust the parameters of the depth estimation model, the adjusted self-supervised monocular depth estimation model can directly output a depth map with absolute scale.

[0130] S320 performs consistency processing on the depths corresponding to the same pixel in multiple depth maps to obtain multiple depth maps with the same depth.

[0131] In some examples, to generate accurate and depth-consistent panoramic depth maps, it is necessary to perform consistency processing on the depth values ​​corresponding to the same pixel in depth maps from different image acquisition devices, thereby obtaining multiple depth maps with uniform depth in overlapping areas. This step is crucial because the depth maps corresponding to the images acquired by each image acquisition device are obtained independently. For example, they are all independently estimated using depth estimation models. Even if the models are jointly trained from multiple views, the predicted depth values ​​for the same object located in the overlapping areas of the two cameras' views may have slight deviations due to differences in viewpoints, occlusion boundaries, or noise. If these inconsistent depth maps are directly fused, ghosting, breaks, or depth jumps will occur in the overlapping areas, thereby affecting the accuracy and tracking stability of subsequent target localization and obstacle avoidance.

[0132] In addition, the above-mentioned consistency processing methods include, but are not limited to, regularization processing and weighted fusion processing.

[0133] To achieve depth uniformity, each depth map can first be projected onto a unified global coordinate system, and then depth consistency processing can be performed within that global coordinate system. Based on this, in some embodiments, consistency processing is performed on the depths corresponding to the same pixel in multiple depth maps to obtain multiple depth maps with uniform depth. This includes: projecting multiple depth maps onto a spherical coordinate system; and, based on the spherical coordinate system, performing depth consistency regularization processing on the same pixel in the region corresponding to the overlapping area of ​​the viewpoint in the multiple depth maps to obtain multiple depth maps with uniform depth.

[0134] For example, the acquired depth map can be back-projected into a global spherical coordinate system using the calibrated intrinsic and extrinsic parameters of each image acquisition device. Specifically, taking a fisheye camera as an example, a uniform extrinsic parameter can be used to perform the transformation according to the following formula (7).

[0135] in, For fisheye back projection function, For pixels in the depth map The depth value, and For camera extrinsic parameters (from camera coordinate system to device base coordinate system). The projection point is set in the device's base coordinate system. Then, this projection point is projected onto a spherical coordinate system to obtain spherical coordinates. In this way, the depth maps corresponding to images acquired from different image acquisition devices are mapped to the same global spherical coordinate system with angles as coordinates.

[0136] In the overlapping area of ​​the field of view of adjacent image acquisition devices (such as the 40° area where the field of view of the front and left cameras overlap), the depth consistency regularization calculation is performed according to the following formula (8).

[0137] in, For the first Pixels in an image acquired by an image acquisition device The corresponding depth value, For the first Pixels in an image acquired by an image acquisition device The corresponding depth value, the first The image acquisition device and the first There is an overlapping field of view between the image acquisition devices. , This represents a loss due to deep consistency.

[0138] For example, the aforementioned depth consistency loss term can be introduced during the training of the depth estimation model. By minimizing this loss term, the model is forced to learn a property: for the same 3D spatial location point, the predicted depth value should tend to be consistent regardless of which camera's perspective it is viewed from. In this way, the trained depth estimation model itself tends to output multiple depth maps with inherent depth consistency in the overlapping areas of the field of view.

[0139] Of course, during the depth estimation model inference phase (i.e., actual deployment and operation), even if the model already has a tendency for consistent depth estimation, further post-processing can be performed to further ensure the uniformity of depth values. For example, before generating a panoramic depth map, multiple depth values ​​from different cameras mapped to the same location in the spherical coordinate system can be weighted and fused, and the fused depth values ​​can be assigned to the corresponding pixels in different depth maps, thereby obtaining multiple depth maps with uniform depth.

[0140] In this way, by utilizing the overlapping area of ​​the field of view between adjacent image acquisition devices, the depth corresponding to the same pixel in the depth map is uniformly regularized in a unified global spherical coordinate system, which can achieve a unified depth expression between different depth maps and provide a stable and reliable data reference for subsequent target tracking.

[0141] The S330 generates a panoramic depth map based on multiple depth maps with a unified depth.

[0142] In some examples, by obtaining multiple depth maps with a uniform depth, a panoramic depth map covering a horizontal 360° omnidirectional field of view can be generated.

[0143] For example, the spherical point cloud corresponding to each depth map can be projected onto the cylindrical panoramic depth map according to the following formula (9). Wherein, the resolution of the cylindrical panoramic depth map can be... (Horizontal × Vertical), each column corresponds to a 1° azimuth angle. A cylindrical panoramic depth map can be represented as... .

[0144] in, The horizontal azimuth of the spherical point cloud. Let be the vertical pitch angle of the spherical point cloud. The focal length of the cylindrical projection. The horizontal coordinates of the cylindrical panoramic depth map. The vertical coordinates of the cylindrical panoramic depth map.

[0145] S340 determines the location of the associated target object based on the panoramic depth map.

[0146] For example, after obtaining the panoramic depth map, the depth corresponding to the target object can be extracted from the panoramic depth map, and the actual three-dimensional position coordinates of the target object can be calculated by combining the depth value with the pixel position of the target object in the image.

[0147] To improve the effectiveness of the extracted depth values ​​corresponding to the target objects, in some embodiments, step S340 may specifically include: determining the median depth of the object detection box where the associated target object is located based on the panoramic depth map; and determining the position of the associated target object based on the median depth.

[0148] For example, the object detection bounding box corresponding to the target object output by the object detection model can be used as an example. The median depth is taken from all depth values ​​corresponding to the detection box in the panoramic depth map. This is used as the depth corresponding to the target object to exclude invalid edge values.

[0149] Then, the actual three-dimensional position coordinates of the target object can be calculated according to the following formula (10). .

[0150] in, and You can retrieve the bottom pixels of the object detection box. The corresponding spherical coordinates are adapted to the object standing on the ground (corresponding to the feet).

[0151] In this embodiment of the application, by performing depth consistency processing on the depth maps corresponding to images acquired from multiple different viewpoints, a seamless panoramic depth map covering the omnidirectional field of view is generated, thereby providing the entire system with a unified perception base containing three-dimensional spatial information, and improving the accuracy and reliability of target object position estimation.

[0152] In addition, based on the above embodiments, the following follow control strategy can also be adopted: based on the position of the associated target object, the intelligent agent device can be controlled to follow the target object to move.

[0153] For example, based on the actual three-dimensional position coordinates of the associated target object The following control parameters can be calculated.

[0154] For example, setting the following distance The linear velocity of the controlled device is calculated according to the following formula (11). .

[0155] in, The depth corresponding to the target object. The set following distance, This refers to the proportional gain coefficient of the equipment chassis motion controller. The differential gain coefficient of the equipment chassis motion controller. The rate of change of depth. This is calculated by measuring the linear velocity of the control device's movement. It can perform distance control.

[0156] In addition, the angular velocity of the control device is calculated according to the following formula (12). .

[0157] in, The horizontal azimuth angle corresponding to the target object. This is the angular velocity control gain coefficient. It is calculated by controlling the angular velocity of the moving device. It can control the device to face the target object, achieving directional alignment.

[0158] Furthermore, before generating control commands based on control parameters, a "safety channel" can be defined on the panoramic depth map, centered on the intelligent agent device and along its current or target direction. The width of this channel is slightly larger than the width of the device itself (e.g., robot width + a set margin). Real-time checks are performed to ensure that no depth value within the safety channel is below a safety threshold. The pixel region (i.e., the obstacle). If a depth value exists for a certain region... If the target object is outside the "safe passage," obstacle avoidance behavior is triggered, such as modifying the angular velocity parameter to avoid the obstacle, or setting the linear velocity parameter to zero in an emergency. Additionally, if the target object is outside the "safe passage," obstacle avoidance behavior can be triggered first, modifying the angular velocity parameter to avoid the obstacle, thus prioritizing the continuity of following. In this way, integrated control of following and obstacle avoidance can be achieved solely through the vision system, without the need for additional ultrasonic or lidar sensors.

[0159] Based on the above embodiments, a specific example is given below to better illustrate the entire solution.

[0160] In some specific examples, this application provides a target following system that may include a hardware perception layer, a visual processing layer, a spatial understanding layer, and a decision control layer.

[0161] For example, the hardware perception layer may include three horizontally uniformly distributed fisheye cameras (horizontal field of view ≥185°, vertical field of view ≥185°), an IMU, and a wheeled odometer. The optical axes of the three fisheye cameras are at an angle of 120° to each other, and the installation height is 1.2m above the ground, forming a 360° horizontal surround coverage and a large vertical field of view perception, with a 40° horizontal overlap between adjacent cameras.

[0162] The visual processing layer is responsible for image preprocessing and target detection, including distortion correction based on the isometric projection model, target detection processing with YOLO-fisheye adaptation, and cross-camera feature extraction.

[0163] The spatial understanding layer is responsible for depth estimation and target association, including monocular depth prediction processing, spherical spatial fusion processing, and cross-camera target association processing.

[0164] The decision control layer is responsible for path planning and motion control, including real-time obstacle avoidance based on panoramic depth maps and calculating the device's following speed (such as linear velocity) based on the target's 3D position. With angular velocity ).

[0165] Based on the aforementioned target following system, this application also provides a target following method, including an initialization phase, a perception phase, a fusion phase, a tracking phase, and a control phase.

[0166] Specifically, in the initialization phase, the calibration parameters of the three cameras can be loaded, and the depth estimation model, object detection model, and tracker can be initialized. In the perception phase, the three fisheye cameras simultaneously acquire images, and each camera independently performs distortion perception detection and monocular depth estimation in parallel. In the fusion phase, cross-camera object association is performed, object identification is unified, and cylindrical panoramic depth maps and detection results are generated. In the tracking phase, the 3D position and motion state of the target object are updated, and occlusion and cross-camera switching are handled. In the control phase, follow control commands are calculated. It performs obstacle avoidance checks and safety constraints, and outputs control commands to the chassis motion controller.

[0167] The performance indicators achievable by the aforementioned target following system and method include: a field of view coverage of up to 360° horizontally and ≥180° vertically, with no mechanical rotation blind spots; absolute relative error in depth accuracy <0.12 (within the indoor range of 1-5 meters), and scale-invariant logarithmic error <14.5; in terms of real-time performance, three cameras process in parallel with an end-to-end latency of <50ms (20fps); cross-camera switching latency <100ms; and an effective following range of 0.5m to 10m.

[0168] In addition, the embedded platform deployment of the target following system can be optimized in the following ways.

[0169] For example, a lightweight model is adopted. Specifically, MobileNetV3 is used as the encoder instead of ResNet; the depth estimation resolution of the depth estimation model is reduced from 640×480 during training to 320×240 during inference (which can be recovered using bilinear interpolation); and the network architecture of the object detection model adopts a variant of YOLOv8-Nano.

[0170] For example, FP16 quantization is performed on the depth estimation model and the object detection model; Jetson's DLA (DeepLearning Accelerator) is used to offload the inference load; the three-camera inference pipeline is parallelized and multi-threaded stream processing is used.

[0171] In addition, a circular buffer is used to manage the image queue; a spherical projection lookup table is pre-computed and resident in video memory; the depth map and the original image (RGB image) share the feature extraction network (e.g., using a multi-task learning approach).

[0172] Based on this, the performance metrics that can be achieved after the above optimizations include: single-path depth estimation inference time <25ms / frame; total time for three-path depth estimation plus fusion <50ms (20fps); GPU memory usage <4GB.

[0173] It should be noted that any parts not fully described in the above examples can be referred to the aforementioned embodiments, and will not be repeated here.

[0174] Thus, this embodiment eliminates the dependence on mechanical rotation mechanisms by using equidistant projection transformation and spherical space calculation, solving the problem of target detection failure caused by radial distortion at the edge of fisheye images; it achieves seamless tracking of target objects when they are occluded or when switching between fields of view by performing cross-camera target association through the overlapping area of ​​the three-camera field of view (40° overlap area for each pair of cameras); and it recovers absolute-scale depth under radar-free conditions by combining a self-supervised monocular depth estimation model with camera height constraints and cross-camera triangulation, providing accurate three-dimensional position information for follow control.

[0175] The above text combined Figures 1 to 3 The present application describes in detail the implementation of the target following method, which is illustrated below. Figure 4 This application describes in detail the embodiments of the target following device. It should be understood that the descriptions of the target following method embodiments correspond to the descriptions of the target following device embodiments; therefore, any parts not described in detail can be referred to the foregoing method embodiments.

[0176] Figure 4 The diagram shown is a schematic representation of a target following device according to an embodiment of this application. This device can be configured on an intelligent agent device. For example, the intelligent agent device may include a mobile robot, a drone, an autonomous vehicle, or other intelligent agent device with mobility capabilities. The intelligent agent device is equipped with multiple image acquisition devices. The combined horizontal field of view of the multiple image acquisition devices covers an omnidirectional horizontal field of view, and there is an overlap in the field of view between adjacent image acquisition devices.

[0177] like Figure 4 As shown, the target following device 400 provided in this application embodiment includes: Acquisition module 401 is used to acquire images acquired by at least one image acquisition device; Detection module 402 is used to perform target detection based on the image and obtain the target detection result; The association module 403 is used to obtain a second detection result if it is determined from the first detection result that the target object to be followed has entered the target visual field overlap area, and associate the first detection result with the target object in the second detection result. The first detection result is the target detection result corresponding to the first image acquired by the first image acquisition device, and the second detection result is the target detection result corresponding to the second image acquired by the second image acquisition device. The first image acquisition device and the second image acquisition device are adjacent to each other and there is a target visual field overlap area. The control module 404 is used to control the intelligent agent device to follow the target object based on the position of the associated target object.

[0178] In one embodiment of this application, the association module 403 is further configured to: obtain association degree information between the target object in the first detection result and multiple objects to be associated in the second detection result; determine the first object associated with the target object from the multiple objects to be associated based on the association degree information; and associate the first object in the second detection result with the target object in the first detection result.

[0179] In one embodiment of this application, the association information includes at least two of the following: the distance between the position of the target object in the first detection result and the position of the object to be associated in the second detection result; the similarity between the appearance features of the target object in the first detection result and the appearance features of the object to be associated in the second detection result; and the difference between the depth of the target object in the first detection result and the depth of the object to be associated in the second detection result.

[0180] In one embodiment of this application, the association module 403 is further configured to: construct an association cost function based on association degree information; calculate the association cost function, and determine the optimal association object corresponding to the target object as the first object from multiple objects to be associated.

[0181] In one embodiment of this application, the target following device 400 further includes: a prediction module, configured to predict the motion trajectory of the target object if it is determined from the first detection result that the target object is occluded; and a start module, configured to start the target detection process of the third image acquired by the third image acquisition device if the motion trajectory points to the field of view area corresponding to the adjacent third image acquisition device.

[0182] In one embodiment of this application, the detection module 402 is further configured to: perform distortion perception and feature extraction on the image to obtain image features; and perform target detection on the image based on the image features to obtain target detection results.

[0183] In one embodiment of this application, the detection module 402 is further configured to: project pixels in the image onto a spherical coordinate system to obtain spherical coordinates corresponding to the pixels; generate radial weights based on the incident angle in the spherical coordinates; and perform distortion perception and feature extraction on the image using deformable convolution based on the radial weights to obtain image features.

[0184] In one embodiment of this application, the target detection result includes parameter information of the object detection box. Accordingly, the detection module 402 is further configured to: adjust the parameter information of the object detection box according to the incident angle corresponding to the center point of the object detection box, so as to realize distortion correction of the object detection box.

[0185] In one embodiment of this application, the control module 404 is further configured to: acquire depth maps corresponding to images acquired by multiple image acquisition devices respectively; perform consistency processing on the depths corresponding to the same pixel in the multiple depth maps to obtain multiple depth maps with a unified depth; generate a panoramic depth map based on the multiple depth maps with a unified depth; and determine the position of the associated target object based on the panoramic depth map.

[0186] In one embodiment of this application, the control module 404 is further configured to: project multiple depth maps onto a spherical coordinate system; and, based on the spherical coordinate system, perform depth consistency regularization processing on the same pixel point in the region corresponding to the overlapping area of ​​the field of view in the multiple depth maps to obtain multiple depth maps with a unified depth.

[0187] In one embodiment of this application, the control module 404 is further configured to: use a self-supervised training depth estimation model to perform depth estimation on images acquired by multiple image acquisition devices respectively, and obtain a depth map corresponding to the images acquired by the multiple image acquisition devices respectively.

[0188] In one embodiment of this application, the control module 404 is further configured to: input the image acquired by the image acquisition device into the depth estimation model, use the depth estimation model to estimate the depth of the pixels in the image, and output the relative depth; based on the installation heights corresponding to the multiple image acquisition devices, restore the relative depth to the absolute depth, and obtain a depth map corresponding to the image.

[0189] In one embodiment of this application, the control module 404 is further configured to: acquire sample images acquired by multiple image acquisition devices respectively, and obtain multiple sample images; and perform self-supervised training on the initial depth estimation model based on the acquisition viewpoints corresponding to the multiple image acquisition devices and the multiple sample images, to obtain a depth estimation model.

[0190] In one embodiment of this application, the control module 404 is further configured to: perform stereo matching on the image region corresponding to the field of view overlap area between adjacent image acquisition devices in the sample images synchronously acquired by adjacent image acquisition devices, and calculate the absolute depth of multiple matching pixels in the image region; and adjust the model parameters of the depth estimation model using the absolute depth of multiple matching pixels as a supervision signal.

[0191] In one embodiment of this application, the control module 404 is further configured to: determine the median depth of the object detection box where the associated target object is located based on the panoramic depth map; and determine the position of the associated target object based on the median depth.

[0192] Below, for reference Figure 5 This describes an intelligent agent device according to embodiments of the present application. Figure 5 The diagram shown is a structural schematic of an intelligent agent device provided in an exemplary embodiment of this application.

[0193] like Figure 5 As shown, the intelligent agent device 500 includes multiple image acquisition devices 501 and a processor 502.

[0194] The horizontal field of view of multiple image acquisition devices 501 is combined to cover the horizontal omnidirectional field of view, and there is an overlapping area of ​​field of view between adjacent image acquisition devices 501; the processor 502 is used to execute the target following method provided in the above embodiments.

[0195] In some embodiments, the plurality of image acquisition devices 501 include three horizontally distributed fisheye cameras, the optical axes of the three fisheye cameras being 120° apart.

[0196] It should be noted that any parts not fully described in the above embodiments can be referred to the foregoing embodiments, and will not be repeated here.

[0197] Below, for reference Figure 6 This describes an electronic device according to embodiments of the present application. Figure 6 The diagram shown is a structural schematic of an electronic device provided in an exemplary embodiment of this application.

[0198] like Figure 6 As shown, the electronic device 600 includes one or more processors 601 and memory 602.

[0199] The processor 601 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 600 to perform desired functions.

[0200] The memory 602 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 601 may execute the program instructions to implement the target following methods of the various embodiments of this application described above and / or other desired functions. Various content, such as images, target detection results, and following control instructions, may also be stored in the computer-readable storage medium.

[0201] In one example, the electronic device 600 may also include an input device 603 and an output device 604, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0202] The input device 603 may include, for example, a keyboard, a mouse, etc.

[0203] The output device 604 can output various information to the outside, including images, target detection results, follow control commands, etc. The output device 604 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0204] Of course, for the sake of simplicity, Figure 6 Only some of the components of the electronic device 600 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 600 may include any other suitable components depending on the specific application.

[0205] In addition to the methods and apparatus described above, embodiments of this application may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the target following methods according to various embodiments of this application as described above.

[0206] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0207] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the target following methods according to various embodiments of this application described above.

[0208] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0209] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0210] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0211] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0212] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0213] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A target following method, characterized in that, An application to an intelligent agent device, wherein the intelligent agent device is equipped with multiple image acquisition devices, the combined field of view of the multiple image acquisition devices covers a horizontal omnidirectional field of view and there is a visual overlap area between adjacent image acquisition devices, the method comprising: Acquire at least one image acquired by the image acquisition device; Target detection is performed based on the image to obtain the target detection result; If the target object to be followed is determined to enter the target visual field overlap area based on the first detection result, then the second detection result is obtained, and the first detection result is associated with the target object in the second detection result. The first detection result is the target detection result corresponding to the first image acquired by the first image acquisition device, and the second detection result is the target detection result corresponding to the second image acquired by the second image acquisition device. The first image acquisition device and the second image acquisition device are adjacent to each other and there is a target visual field overlap area. Based on the location of the associated target object, the intelligent agent device is controlled to move to follow the target object.

2. The method according to claim 1, characterized in that, Associating the first detection result with the target object in the second detection result includes: Obtain the degree of association between the target object in the first detection result and multiple objects to be associated in the second detection result; Based on the association information, a first object associated with the target object is determined from the plurality of objects to be associated. Associate and identify the first object in the second detection result with the target object in the first detection result.

3. The method according to claim 2, characterized in that, The correlation information includes at least two of the following: The distance between the location of the target object in the first detection result and the location of the object to be associated in the second detection result; The similarity between the appearance features of the target object in the first detection result and the appearance features of the object to be associated in the second detection result; The difference between the depth of the target object in the first detection result and the depth of the object to be associated in the second detection result.

4. The method according to claim 2 or 3, characterized in that, The step of determining the first object associated with the target object from the plurality of objects to be associated based on the association degree information includes: Construct an association cost function based on the association degree information; Calculate the association cost function, and determine the optimal associated object corresponding to the target object from the plurality of objects to be associated as the first object.

5. The method according to claim 1, characterized in that, Also includes: If it is determined from the first detection result that the target object is occluded, then the motion trajectory of the target object is predicted; If the motion trajectory points to the field of view area corresponding to the adjacent third image acquisition device, then the target detection process for the third image acquired by the third image acquisition device is initiated.

6. The method according to claim 1, characterized in that, The target detection based on the image, to obtain the target detection result, includes: Distortion sensing and feature extraction are performed on the image to obtain image features; Based on the image features, target detection is performed on the image to obtain the target detection result.

7. The method according to claim 6, characterized in that, The process of performing distortion sensing and feature extraction on the image to obtain image features includes: Project the pixels in the image onto a spherical coordinate system to obtain the spherical coordinates corresponding to the pixels; Generate radial weights based on the incident angle in the spherical coordinates; Based on the radial weights, deformable convolution is used to perform distortion sensing and feature extraction on the image to obtain the image features.

8. The method according to claim 7, characterized in that, The target detection result includes parameter information of the object detection bounding box; after performing target detection on the image based on the image features to obtain the target detection result, the method further includes: The parameters of the object detection box are adjusted based on the incident angle corresponding to the center point of the object detection box to achieve distortion correction of the object detection box.

9. The method according to claim 1, characterized in that, Before controlling the intelligent agent device to move to follow the target object based on the associated position of the target object, the method further includes: Obtain depth maps corresponding to the images acquired by the multiple image acquisition devices respectively; Consistency processing is performed on the depths corresponding to the same pixel in multiple depth maps to obtain multiple depth maps with the same depth. A panoramic depth map is generated based on multiple depth maps of the same depth. The location of the associated target object is determined based on the panoramic depth map.

10. The method according to claim 9, characterized in that, The process of performing consistency processing on the depths corresponding to the same pixel in multiple depth maps to obtain multiple depth maps with uniform depth includes: The multiple depth maps are projected onto a spherical coordinate system; Based on the spherical coordinate system, depth consistency regularization is performed on the same pixel point in the region corresponding to the overlapping region of the field of view in multiple depth maps to obtain multiple depth maps with uniform depth.

11. The method according to claim 9, characterized in that, The step of obtaining the depth map corresponding to the images acquired by the multiple image acquisition devices includes: Using a self-supervised training depth estimation model, depth estimation is performed on images acquired by multiple image acquisition devices to obtain depth maps corresponding to the images acquired by the multiple image acquisition devices.

12. The method according to claim 11, characterized in that, The step of using a self-supervised trained depth estimation model to estimate the depth of images acquired by multiple image acquisition devices to obtain depth maps corresponding to the images acquired by the multiple image acquisition devices includes: The image acquired by the image acquisition device is input into the depth estimation model, and the depth estimation model is used to estimate the depth of the pixels in the image, and outputs the relative depth. Based on the installation heights of the multiple image acquisition devices, the relative depth is restored to the absolute depth to obtain a depth map corresponding to the image.

13. The method according to claim 11, characterized in that, Before using a self-supervised trained depth estimation model to perform depth estimation on the images acquired by the multiple image acquisition devices respectively, and obtaining the depth maps corresponding to the images acquired by the multiple image acquisition devices respectively, the method further includes: Multiple sample images are obtained by acquiring sample images acquired by the multiple image acquisition devices respectively. Based on the acquisition viewpoints corresponding to the multiple image acquisition devices and the multiple sample images, the initial depth estimation model is trained under self-supervised conditions to obtain the depth estimation model.

14. The method according to claim 13, characterized in that, After obtaining the depth estimation model by performing self-supervised training on the initial depth estimation model based on the acquisition viewpoints corresponding to the multiple image acquisition devices and the multiple sample images, the method further includes: Stereo matching is performed on the image regions corresponding to the overlapping areas of the field of view between the adjacent image acquisition devices in the sample images acquired synchronously by adjacent image acquisition devices, and the absolute depth of multiple matching pixels in the image regions is calculated. The absolute depth of the multiple matching pixels is used as a supervision signal to adjust the model parameters of the depth estimation model.

15. The method according to claim 9, characterized in that, Determining the location of the associated target object based on the panoramic depth map includes: The median depth of the object detection box containing the associated target object is determined based on the panoramic depth map; Based on the median depth, the location of the associated target object is determined.

16. An intelligent agent device, characterized in that, include: Multiple image acquisition devices, wherein the combined horizontal field of view of the multiple image acquisition devices covers the horizontal omnidirectional field of view and there is an overlapping area of ​​field of view between adjacent image acquisition devices; A processor for executing the target following method according to any one of claims 1 to 15.

17. The device according to claim 16, characterized in that, The plurality of image acquisition devices include three horizontally distributed fisheye cameras, the optical axes of the three fisheye cameras being 120° apart.

18. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the target following method according to any one of claims 1 to 15.

19. A computer program product, characterized in that, The computer program product includes instructions that, when executed on an electronic device, cause the electronic device to perform the target following method according to any one of claims 1 to 15.

Images