Target tracking method, device and equipment based on multiple cameras and storage medium

Abstract

The invention provides a target tracking method and device based on multiple cameras, equipment and a storage medium. The method is applied to a vehicle, and a plurality of cameras are installed on the vehicle. Comprising the steps that according to fisheye images collected by cameras at the current moment, an environment image of the environment where a vehicle is located is obtained; wherein the environment image is an all-round cylindrical surface image; identifying a target object from the environment image, and determining a key point of the target object; wherein the key points are coordinate points representing the appearance characteristics of the target object; and according to the coordinate information of the key point in the environment image, determining the position information of the target object in the environment where the vehicle is located. According to the method, the positioning precision of the obstacles around the vehicle is improved, and accurate tracking of the obstacles is realized.

Description

Technical Field

[0001] The present application relates to autonomous driving technology, and in particular to a target tracking method, device, equipment and storage medium based on multiple cameras. Background Art

[0002] With the development of autonomous driving technology, people have higher and higher requirements for vehicles to perceive surrounding obstacles, especially in parking scenarios, which require vehicles to accurately locate obstacles to achieve safe and accurate parking.

[0003] It is difficult for current vehicles to obtain the true position of surrounding obstacles, and the tracking accuracy is low, which cannot meet people's requirements for vehicle obstacle avoidance functions. Summary of the invention

[0004] The present application provides a target tracking method, device, equipment and storage medium based on multiple cameras to improve the tracking accuracy of obstacles.

[0005] In a first aspect, the present application provides a target tracking method based on multiple cameras, which is applied to a vehicle, wherein multiple cameras are installed on the vehicle; the method comprises:

[0006] According to the fisheye images collected by each camera at the current moment, an environmental image of the environment where the vehicle is located is obtained; wherein the environmental image is a cylindrical image of the surrounding view;

[0007] Identify the target object from the environment image and determine the key points of the target object; wherein the key points are coordinate points that characterize the appearance features of the target object;

[0008] The position information of the target object in the environment where the vehicle is located is determined according to the coordinate information of the key point in the environment image.

[0009] In a second aspect, the present application provides a target tracking device based on multiple cameras, which is applied to a vehicle, and the vehicle is equipped with multiple cameras; the device includes:

[0010] An image acquisition unit, used to obtain an environmental image of the environment where the vehicle is located according to the fisheye images collected by each camera at the current moment; wherein the environmental image is a cylindrical image of the surrounding view;

[0011] An object recognition unit, used to recognize a target object from the environment image and determine key points of the target object; wherein the key points are coordinate points that characterize the appearance features of the target object;

[0012] A position determination unit is used to determine the position information of the target object in the environment where the vehicle is located according to the coordinate information of the key point in the environment image.

[0013] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0014] The memory stores computer-executable instructions;

[0015] The processor executes the computer-executable instructions stored in the memory to implement the method according to the first aspect.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0017] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0018] The present application provides a multi-camera-based target tracking method, device, equipment and storage medium, which obtains a cylindrical environmental image through fisheye images collected by multiple cameras. The target object to be located is determined from the cylindrical environmental image, and multiple key points on the target object are determined. According to the coordinate information of these key points, the position information of the target object is determined to achieve the positioning of the target object. By determining the cylindrical environmental image, the distortion phenomenon caused by splicing the surround view images into a bird's-eye view is avoided. By processing the coordinate information of the key points and optimizing the positioning algorithm of the target object, the size, shape and other external features of the target object can be determined, thereby improving the tracking accuracy of the target object, thereby improving the accuracy and safety of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0020] Figure 1 A schematic diagram of a flow chart of a target tracking method based on multiple cameras provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of an environment image provided by an embodiment of the present application;

[0022] Figure 3 A schematic diagram of a flow chart of a target tracking method based on multiple cameras provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of key points provided in an embodiment of the present application;

[0024] Figure 5A schematic diagram of a preset camera coordinate system provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of a flow chart of a target tracking method based on multiple cameras provided in an embodiment of the present application;

[0026] Figure 7 A structural block diagram of a multi-camera target tracking device provided in an embodiment of the present application;

[0027] Figure 8 A structural block diagram of a multi-camera target tracking device provided in an embodiment of the present application;

[0028] Fig. 9 A structural block diagram of an electronic device provided in an embodiment of the present application;

[0029] Fig.10 A structural block diagram of an electronic device provided in an embodiment of the present application.

[0030] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0031] In order to make the objectives, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0032] It should be clear that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0033] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0034] In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0035] It should be noted that due to space limitations, this application specification does not list all optional implementation methods. After reading this application specification, those skilled in the art should be able to think that as long as the technical features do not contradict each other, any combination of technical features can constitute an optional implementation method. Each embodiment is described in detail below.

[0036] Fisheye cameras are usually used as visual sensors for the assisted driving and parking functions of vehicles. Multiple fisheye cameras can be regarded as surround view cameras. The images collected by the surround view cameras are usually spliced ​​into a bird's-eye view, which is used as the data input of the obstacle recognition model to obtain the location information of the obstacles.

[0037] However, the bird's-eye view has a limited recognition range and can usually only locate objects within a range of 10 meters, and cannot achieve positioning within a range of 10 to 20 meters. In addition, the bird's-eye view mosaic has the defect of projection distortion for three-dimensional obstacles such as vehicles, making it difficult to obtain the true ground position of the obstacle, and thus unable to provide high-precision positioning results for subsequent timing tracking and smoothing solutions. If the accuracy of the initial obstacle data is not considered and a timing neural network deep learning solution is used, a large amount of data needs to be invested, and the collection and annotation costs are higher than the method of the embodiment of the present application.

[0038] The present application provides a multi-camera based target tracking method, device, equipment and storage medium, which are intended to solve the above technical problems in the prior art.

[0039] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0040] Figure 11 is a flow chart of a target tracking method based on multiple cameras provided in an embodiment of the present application. The method can be performed by a target tracking device based on multiple cameras. The method is applied to a vehicle, and multiple cameras are installed on the vehicle. Figure 1 As shown, the method comprises the following steps:

[0041] S101. Obtain an environment image of the environment where the vehicle is located according to the fisheye images collected by each camera at the current moment; wherein the environment image is a cylindrical image of the surrounding view.

[0042] Exemplarily, a plurality of cameras are installed on the vehicle. For example, a camera may be installed in each of the four directions of the vehicle, namely, a front view camera, a rear view camera, a left view camera, and a right view camera. The surround view camera can be used to collect images in real time, and the images collected by the surround view camera are fisheye images. The fisheye images collected by each camera are obtained in real time. For example, if four cameras are installed on the vehicle as surround view cameras, four fisheye images can be obtained. Based on the multiple fisheye images obtained at the current moment, an environmental image corresponding to the current moment can be obtained, and the environmental image can represent the environment in which the vehicle is located.

[0043] In this embodiment, the camera on the vehicle is a surround fisheye visual sensor, which converts the fisheye image into a cylindrical image, that is, the environment image is a cylindrical image. For example, multiple fisheye images can be horizontally spliced ​​into one image as the environment image. In this embodiment, it is not necessary to splice the fisheye images into a bird's-eye view.

[0044] In this embodiment, an environmental image of the vehicle's environment is obtained based on the fisheye images captured by each camera, including: stitching the fisheye images captured by each camera according to a preset camera stitching order to obtain an environmental image of the vehicle's environment.

[0045] Specifically, at any moment, each camera can capture a fisheye image, and the fisheye images captured by multiple cameras at that moment are combined to obtain the environment image at that moment. For multiple cameras on the vehicle, a camera stitching order is preset, and the camera stitching order refers to the order in which the images captured by the cameras are stitched. According to the preset camera stitching order, the fisheye images captured by each camera are stitched, and the stitched image is determined as the environment image. For example, there are four cameras installed on the vehicle, starting from the camera at the front position, the images are stitched in a clockwise order to obtain an environment image with four images connected horizontally.

[0046] An image sequence of images in a time sequence can be obtained. For example, the time sequence is {t0, t1, t2, t3...}. Taking each image at a certain time t0, assuming there are four cameras, the fisheye image {p0, p1, p2, p3} can be obtained. For a certain moment in the time sequence, the four images are converted into a surround environment image according to the preset camera stitching order.

[0047] Figure 2 Schematic diagram of the environment image. Figure 2 In the figure, the environment image is stitched from four fisheye images, which are image A, image B, image C, and image D. Image A may be an image taken by a front-view camera on a vehicle, image B may be an image taken by a rear-view camera on a vehicle, image C may be an image taken by a left-view camera on a vehicle, and image D may be an image taken by a right-view camera on a vehicle. The preset camera stitching order may be to stitch the image taken by the front-view camera at the upper left corner, the image taken by the rear-view camera at the upper right corner, the image taken by the left-view camera at the lower left corner, and the image taken by the right-view camera at the lower right corner.

[0048] The beneficial effect of such a setting is that the original fisheye image captured by the camera is converted into a cylindrical stitching image, there is no need to generate a bird's-eye view, deformation of pedestrians and vehicles is avoided, and the accuracy of target positioning is improved.

[0049] S102, identifying a target object from the environment image, and determining key points of the target object; wherein the key points are coordinate points that characterize the appearance features of the target object.

[0050] For example, obstacles such as pedestrians and other vehicles may be included in the environment image, and the obstacles in the environment image may be used as target objects. In this embodiment, the positioning scene may be a parking lot, and the obstacles may be other vehicles in the parking lot.

[0051] The environmental image can be processed by image recognition, obstacles in the environment can be identified from the environmental image, and the identified obstacles can be used as target objects. For example, other vehicles close to the vehicle can be identified from the environmental image as target objects. The recognition algorithm of the target object can be preset, for example, it can be identified based on shape, size and other external features. In this embodiment, the preset recognition algorithm is not specifically limited. If there are multiple obstacles in the environment, multiple target objects can be identified. For example, there is a car parked on the left side of the vehicle and a car parked on the right side of the vehicle, then both the left vehicle and the right vehicle can be target objects.

[0052] After the target object is determined, one or more coordinate points can be determined from the environment image as key points of the target object. Key points can be used to characterize the appearance features of the target object, for example, the contour shape, size and other features of the target object. After the target object is identified, coordinate points can be selected from the contour of the target object as key points. The center point of the target object can also be used as a key point.

[0053] S103: Determine the position information of the target object in the environment where the vehicle is located according to the coordinate information of the key point in the environment image.

[0054] Exemplarily, the coordinate information of the key point in the environment image is determined, and the coordinate information is a two-dimensional coordinate. According to the coordinate information of the key point, the position information of the target object in the environment where the vehicle is located is determined, that is, the position of the target object in the actual space is determined. The coordinate information of the key point is the coordinate in the image taken by the camera, and the position information of the target object is the position coordinate of the target object relative to the vehicle. Therefore, the coordinate system can be transformed according to the coordinate information of the key point. For example, the camera coordinate system can be transformed to the vehicle coordinate system. In this embodiment, the camera coordinate system is the world coordinate system.

[0055] In this embodiment, a formula for calculating the coordinate information of the key points can be preset. For example, the orientation and other information of the target object can be determined based on the coordinate information of the key points. The coordinate information is converted into a coordinate system and combined with the orientation and other information to obtain accurate position information. For another example, multiple key points of the target object can be determined, and the multiple key points can be screened. According to the coordinate information of the key points retained after the screening, the position of the target object relative to the vehicle is determined, so as to track the target object.

[0056] In an embodiment of the present application, a cylindrical environmental image is obtained by using fisheye images collected by multiple cameras. The target object to be located is determined from the cylindrical environmental image, and multiple key points on the target object are determined. Based on the coordinate information of these key points, the position information of the target object is determined to achieve the positioning of the target object. By determining the cylindrical environmental image, the distortion phenomenon caused by splicing the surround image into a bird's-eye view is avoided. By processing the coordinate information of the key points and optimizing the positioning algorithm of the target object, the size, shape and other external features of the target object can be determined, thereby improving the tracking accuracy of the target object, thereby improving the accuracy and safety of autonomous driving.

[0057] Figure 3 A flowchart of a multi-camera target tracking method provided in an embodiment of the present application is provided. This embodiment is an optional embodiment based on the above embodiment.

[0058] In this embodiment, the position information of the target object in the environment where the vehicle is located is determined based on the coordinate information of the key points in the environmental image, including: determining two key points from the key points of the target object based on the coordinate information of the key points in the environmental image, namely the first target point and the second target point; determining the position information of the target object in the environment where the vehicle is located based on the coordinate information of the first target point and the second target point.

[0059] like Figure 3 As shown, the method comprises the following steps:

[0060] S301. Obtain an environment image of the environment where the vehicle is located according to the fisheye images collected by each camera at the current moment; wherein the environment image is a surrounding cylindrical image.

[0061] Exemplarily, this step may refer to the above-mentioned step S101 and will not be described in detail.

[0062] S302: Identify the target object from the environment image and determine the key points of the target object; wherein the key points are coordinate points that characterize the appearance features of the target object.

[0063] Exemplarily, this step may refer to the above-mentioned step S102 and will not be described in detail.

[0064] S303 . Determine two key points from the key points of the target object according to the coordinate information of the key points in the environment image, namely a first target point and a second target point.

[0065] Exemplarily, the target object corresponds to multiple key points, and the coordinate information of each key point is determined. According to the coordinate information of the key points, the key points are screened, and two key points are selected from the key points of the target object as the first target point and the second target point, respectively. The selection can be made according to the size of the coordinates. For example, the key point with the smallest ordinate can be selected as the first target point, and the key point with the second smallest ordinate can be selected as the second target point.

[0066] In this embodiment, the key point is located on a target frame of the target object, the target frame is used to surround the target object, and the coordinate information includes a horizontal coordinate and a vertical coordinate; according to the coordinate information of the key point in the environmental image, two key points are determined from the key points of the target object, namely a first target point and a second target point, including: determining the coordinate information of the center point in the environmental image; determining the difference in the horizontal coordinates between the center point and each key point according to the horizontal coordinate of the center point and the horizontal coordinates of each key point; determining the first target point from the key points of the target object according to the difference in the horizontal coordinates between the center point and each key point, and the size of the vertical coordinate of each key point; determining the second target point from the key points of the target object according to the layout position of the first target point on the target frame.

[0067] Specifically, the target object on the environment image can be surrounded by a target frame of a preset shape, for example, the target frame can be a bounding box of a rectangular frame or a cubic frame. The key points can all be located on the target frame, for example, the key points can be vertices of the target frame, or points on the edge of the target frame. Since the image is two-dimensional data, the coordinate information of each key point includes a horizontal coordinate and a vertical coordinate.

[0068] Determine the coordinate information of the center point in the environment image, for example, the center point is the coordinate origin. The coordinate information of the center point also includes a horizontal coordinate and a vertical coordinate. Determine the size of the horizontal coordinate of the center point and the size of the horizontal coordinate of each key point. For each key point, calculate the difference between the horizontal coordinate of the center point and the horizontal coordinate of the key point. That is, each key point corresponds to a difference. The first target point can be determined based on the size of the difference. For example, the key point corresponding to the smallest difference can be determined as the first target point. The size of the vertical coordinates of each key point can also be compared, the key points are sorted according to the size of the vertical coordinates, and the key point corresponding to the smallest vertical coordinate is determined as the first target point.

[0069] The first target point may also be selected in combination with the size of the ordinate and the size of the difference. For example, the key points may be sorted in ascending order according to the size of the ordinate, and then the key point whose abscissa is closest to the center point in the ascending order is selected as the first target point.

[0070] After obtaining the first target point, the second target point can be determined according to the layout of the first target point on the target frame. For example, a key point adjacent to the first target point can be determined as the second target point.

[0071] The beneficial effect of this setting is that for the target object located in the image, the closer the grounding point is to the bottom edge of the image, the higher the calculated depth accuracy. Therefore, in order to improve the positioning accuracy of the above-mentioned target object, the target point can be selected according to the horizontal and vertical coordinates of the key points, which not only improves the positioning accuracy, but also reduces the subsequent calculation amount and improves the tracking efficiency.

[0072] In this embodiment, according to the layout position of the first target point in the target frame, the second target point is determined from the key points of the target object, including: according to the layout position of the first target point in the target frame, the key points adjacent to the first target point are determined as adjacent points; wherein the adjacent points and the first target point are located on the same side of the target frame; the distance information between the first target point and the adjacent points is determined; and according to the distance information, the second target point is determined from the adjacent points.

[0073] Specifically, the key points are arranged on the target frame. After obtaining the first target point, the key points adjacent to the first target point are determined according to the position of the first target point on the target frame as adjacent points. In this embodiment, the key points are all located at the vertices of the target frame, and the adjacent points may refer to the key points located on the same side as the first target point. It is worth noting that the target frame in this embodiment is a cubic frame, and the first target point and the second target point are restricted to be points on the bottom surface. Therefore, after determining the first target point, there are only two adjacent points.

[0074] Figure 4 A schematic diagram of the key points. Figure 4 There are eight key points in , namely a0, a1, a2, a3, a4, a5, a6, and a7, among which a0 to a3 are the key points of the bottom surface. If a0 is determined as the first target point, the adjacent points can be determined to be a1 and a2.

[0075] According to the coordinate information of the first target point and the adjacent points, the distance information between the first target point and each adjacent point is determined. The two distance information are compared, and a key point is determined from the adjacent points as the second target point. For example, the adjacent point corresponding to the longer distance information can be determined as the second target point. In other words, the key point located on the same long side as the first target point is determined as the second target point.

[0076] The beneficial effect of such a setting is that another point located on the long side connected to the first target point is used as the second target point, thereby improving the efficiency of determining the second target point and ensuring that both the first target point and the second target point are grounding points, thereby improving the efficiency and accuracy of tracking.

[0077] S304: Determine the position information of the target object in the environment where the vehicle is located according to the coordinate information of the first target point and the second target point.

[0078] Exemplarily, the coordinate information of the first target point and the second target point is determined, and the coordinate information of the first target point and the second target point is calculated to obtain the position information of the target object in the environment where the vehicle is located. For example, the position of the target object in the camera coordinate system can be determined based on the coordinate information of the first target point and the second target point. Then, the position of the target object in the camera coordinate system is converted into the position of the target object in the vehicle body coordinate system, that is, the position information of the target object is obtained.

[0079] The target object is located in the environment image. Specifically, the target object may be located in a surround view image in the environment image. Different surround view images are captured by different cameras. Different cameras may correspond to different camera coordinate systems. The surround view image where the target object is located is determined by determining the position of the target object in the camera coordinate system corresponding to the surround view image.

[0080] In this embodiment, the position information of the target object in the environment where the vehicle is located is determined based on the coordinate information of the first target point and the second target point, including: determining the angle information of the target object based on the coordinate information of the first target point and the second target point; wherein the angle information represents the orientation of the target object relative to the vehicle; determining the position information of the target object in a preset camera coordinate system based on the angle information of the target object and the coordinate information of the first target point; determining the position information of the target object in the environment where the vehicle is located based on the position information of the target object in the preset camera coordinate system.

[0081] Specifically, according to the coordinate information of the first target point and the second target point, the angle information of the target object is calculated, and the angle information is the azimuth of the target object, that is, the azimuth of the target object in the parking route of the vehicle. The calculation formula of the angle information can be:

[0082] θ = arctan(y2-y1, x2-x1);

[0083] Among them, θ represents the angle information, (x1, y1) represents the coordinate information of the first target point, and (x2, y2) represents the coordinate information of the second target point.

[0084] According to the angle information of the target object and the coordinate information of the first target point, the position information of the target object in the preset camera coordinate system is determined. The target object is transferred to the camera coordinate system, thereby converting the camera coordinate system and the vehicle coordinate system. That is, according to the position information of the target object in the preset camera coordinate system, the position information of the target object in the vehicle coordinate system is determined as the final positioning position information.

[0085] The beneficial effect of such a setting is that the position of the target object in the parking route of the vehicle is determined, and the position information is determined according to the position and coordinates, so as to realize the accurate positioning of the target object. By converting the coordinate system, the actual position of the target object can be obtained, which is convenient for the vehicle to park safely.

[0086] In this embodiment, the position information of the target object in a preset camera coordinate system is determined according to the angle information of the target object and the coordinate information of the first target point, including: determining an initial rectangular frame in the preset camera coordinate system; wherein the initial rectangular frame represents the shape of a top view of the target object; rotating the initial rectangular frame according to the angle information of the target object to obtain a rotated rectangular frame; translating the rotated rectangular frame according to the coordinate information of the first target point to obtain a translated rectangular frame; and determining the position information of the target object in the preset camera coordinate system according to the coordinate information of the vertices of the translated rectangular frame in the preset camera coordinate system.

[0087] Specifically, the target object is transferred to the camera coordinate system in the form of a rectangular frame. An initial rectangular frame can be generated in the preset camera coordinate system according to the coordinate information of each key point of the target object, and the initial rectangular frame represents the shape of the top view of the target object. For example, the size of the rectangular frame can be determined according to the information of the key points. A rectangular frame of a default size can also be preset. If a rectangular frame of a default size is preset, the initial rectangular frame can be directly generated in the preset camera coordinate system. The center point of the initial rectangular frame can be placed at the origin of the camera coordinate system, or a vertex of the initial rectangular frame can be placed at the origin of the camera coordinate system. In this embodiment, the vertex at the lower left corner of the initial rectangular frame can be used as the origin.

[0088] According to the angle information of the target object, the initial rectangular frame is rotated, and the rotated initial rectangular frame is determined as the rotated rectangular frame. For example, the rotation can be performed with the center point of the initial rectangular frame as the rotation center. After obtaining the rotated rectangular frame, the rotated rectangular frame can be translated according to the coordinate information of the first target point, and the translated rotated rectangular frame can be determined as the translated rectangular frame. For example, the vertex corresponding to the first target point in the rotated rectangular frame can be determined, and the vertex can be translated to the coordinate information of the first target point to obtain the translated rectangular frame. After obtaining the translated rectangular frame, the coordinate information of each vertex of the translated rectangular frame in the preset camera coordinate system can be determined as the position information of the target object in the preset camera coordinate system.

[0089] Figure 5 A schematic diagram of the preset camera coordinate system. Figure 5 In the figure, a0 is the first target point, the solid line frame is the initial rectangular frame, and a0 is located at the lower left corner of the initial rectangular frame. The initial rectangular frame is rotated and then translated to the lower right to obtain the translated rectangular frame.

[0090] The beneficial effect of such a setting is that the rectangular frame is rotated and translated in the camera coordinate system so that the position and orientation of the rectangular frame in the camera coordinate system are consistent with the target object in the image, thereby improving the positioning accuracy of the target object.

[0091] In this embodiment, the position information of the target object in the preset camera coordinate system includes the coordinate information of the vertices of the translation rectangular box in the preset camera coordinate system; based on the position information of the target object in the preset camera coordinate system, the position information of the target object in the environment where the vehicle is located is determined, including: based on the coordinate information of the vertices of the translation rectangular box in the preset camera coordinate system, determining the coordinate information of the vertices of the translation rectangular box in the vehicle's own vehicle coordinate system; determining the coordinate information of the vertices of the translation rectangular box in the vehicle's own vehicle coordinate system as the position information of the target object in the environment where the vehicle is located.

[0092] Specifically, the position information of the target object in the preset camera coordinate system may include the coordinate information of the four vertices of the translation rectangular box in the preset camera coordinate system. The coordinate information of the vertices of the translation rectangular box in the preset camera coordinate system is converted to obtain the coordinate information of the vertices in the vehicle coordinate system. The coordinate information of the four vertices in the vehicle coordinate system is determined, and the position information of the target object in the environment where the vehicle is located is determined based on the coordinate information of the four vertices in the vehicle coordinate system. For example, the mean coordinates of the coordinate information of the four vertices in the vehicle coordinate system can be used as the position information of the target object in the environment where the vehicle is located.

[0093] A conversion method of a coordinate system may be preset, and conversion calculation may be performed according to preset parameters of the camera. Different cameras may correspond to different preset parameters. In this embodiment, the conversion method of the preset coordinate system is not specifically limited.

[0094] The beneficial effect of such a setting is that the camera coordinate system, that is, the world coordinate system, can be converted to the vehicle coordinate system, which is conducive to obtaining the actual position of the target object and improving the accuracy of positioning and tracking.

[0095] In this embodiment, the coordinate information of the vertices of the translation rectangle in the vehicle's own vehicle coordinate system is represented as:

[0096] transX=y·sin(angle)+x·cos(angle)+T0;

[0097] transY=y·cos(angle)-x·sin(angle)+T1;

[0098] Among them, X is the horizontal coordinate of the vertices of the translation rectangle box in the vehicle's own vehicle coordinate system, Y is the vertical coordinate of the vertices of the translation rectangle box in the vehicle's own vehicle coordinate system, x is the horizontal coordinate of the vertices of the translation rectangle box in the preset camera coordinate system, y is the vertical coordinate of the vertices of the translation rectangle box in the preset camera coordinate system, angle is the preset angle parameter, and T0 and T1 are both preset times.

[0099] Specifically, different cameras can be preset with different angles. Angle is a preset parameter. Angle is the maximum rotation angle between a single camera in the surround view camera system installed on the vehicle body and the vehicle body coordinate system. (x, y) are the vertex coordinates of the translation rectangle. A 3×3 matrix T is preset. T0 and T1 are parameters obtained based on the matrix T. T0 and T1 can be matrices of size 3×1. For example, T0 is the component of the matrix T in the x direction, and T1 is the component of the matrix T in the y direction. The above formula is used to transform the coordinate system, which is simple and convenient to calculate, uses fewer variables, reduces the system error caused by the introduction of more variables, and improves the calculation accuracy.

[0100] In an embodiment of the present application, a cylindrical environmental image is obtained by using fisheye images collected by multiple cameras. The target object to be located is determined from the cylindrical environmental image, and multiple key points on the target object are determined. Based on the coordinate information of these key points, the position information of the target object is determined to achieve the positioning of the target object. By determining the cylindrical environmental image, the distortion phenomenon caused by splicing the surround image into a bird's-eye view is avoided. By processing the coordinate information of the key points and optimizing the positioning algorithm of the target object, the size, shape and other external features of the target object can be determined, thereby improving the tracking accuracy of the target object, thereby improving the accuracy and safety of autonomous driving.

[0101] Figure 6 A flowchart of a multi-camera target tracking method provided in an embodiment of the present application is provided. This embodiment is an optional embodiment based on the above embodiment.

[0102] In this embodiment, the position information of the target object in the environment where the vehicle is located is determined based on the coordinate information of the key points in the environmental image, including: determining two key points from the key points of the target object based on the coordinate information of the key points in the environmental image, namely the first target point and the second target point; determining the position information of the target object in the environment where the vehicle is located based on the coordinate information of the first target point and the second target point.

[0103] like Figure 6 As shown, the method comprises the following steps:

[0104] S601. Obtain an environment image of the environment where the vehicle is located according to the fisheye images collected by each camera at the current moment; wherein the environment image is a surrounding cylindrical image.

[0105] Exemplarily, this step may refer to the above-mentioned step S101 and will not be described in detail.

[0106] S602, input the current environment image into a preset target recognition model to obtain a target frame in the current environment image; wherein the preset target recognition model is a neural network model, which is used to recognize the target object in the image; the target frame is used to surround the target object.

[0107] Exemplarily, a target recognition model is pre-built, and the target recognition model is a neural network model, which can be used to recognize obstacles such as vehicles in an image. In this embodiment, the model structure of the target recognition model is not specifically limited. The current environmental image is obtained, and the environmental image is input into the target recognition model. The target recognition model recognizes the target object in the image and obtains a target frame of the target object. The target frame can be a cubic frame surrounding the target object. If multiple target objects are identified, each target object corresponds to a target frame.

[0108] S603: Determine the key points of the target object on the target frame according to preset key point detection conditions.

[0109] Exemplarily, the key points of the target object are determined based on the obtained target frame, and the key points are located on the target frame, for example, on the edge or vertex of the target frame. Key point detection conditions can be pre-set, and key points are determined from the target frame based on the key point detection conditions. For example, the preset key point detection condition can be to determine all vertices on the target frame as key points.

[0110] In this embodiment, the target frame is a cubic frame; according to preset key point detection conditions, the key points of the target object on the target frame are determined, including: obtaining the vertices of the target frame as the key points of the target object in a preset key point order.

[0111] Specifically, the number of key points and the order of acquisition can be preset. For example, the number of key points is preset to be eight, that is, the eight vertices of the target frame. The preset order can be to start from the bottom surface of the target frame, and sequentially acquire four vertices according to the size of the vertical coordinate, and then start from the top surface of the target frame, and sequentially acquire four vertices according to the size of the vertical coordinate, so as to obtain eight key points.

[0112] The beneficial effect of such a setting is that by presetting the detection conditions of key points, key points can be quickly obtained, thereby improving the efficiency of target positioning.

[0113] S604: Determine the position information of the target object in the environment where the vehicle is located according to the coordinate information of the key point in the environment image.

[0114] For example, the position information of the target object obtained from the environment image is the position information at the current moment. During the parking or driving process of the vehicle, the position of the vehicle is constantly changing, and the position information of the target object is also dynamically changing relative to the vehicle. Therefore, the target object can be tracked in time to ensure the parking and driving safety of the vehicle.

[0115] In this embodiment, the method also includes: obtaining an environmental image of the environment in which the vehicle is located at the next moment; obtaining at least one bounding box in the environmental image at the next moment based on a preset target recognition model; determining the target object from the environmental image at the next moment according to the target box in the environmental image at the current moment and the bounding boxes in the environmental image at the next moment; determining new key points of the target object from the environmental image at the next moment, and determining new position information of the target object in the environment in which the vehicle is located according to the coordinate information of the new key points in the environmental image at the next moment.

[0116] Specifically, the surround view camera can collect surround view images in real time, and can obtain the environment image at each moment in real time. That is, after obtaining the location information of the target object at the current moment, the surround view image at the next moment can be obtained in real time, thereby obtaining the environment image at the next moment. In other words, after obtaining the environment image at the initial moment t0, the environment image at the moment t1 can be obtained, and after obtaining the environment image at the moment t1, the environment image at the moment t2 can be obtained.

[0117] After obtaining the environment image at the next moment, the environment image at the next moment is input into a preset target recognition model. The target recognition model recognizes each object in the image, and each object corresponds to a bounding box, that is, multiple bounding boxes can be obtained, and the bounding boxes can be cubic boxes. Each cubic box encloses an object, and the target object just determined is found out from it. The current moment in this embodiment is the previous moment of the next moment, that is, the target object determined at the current moment is found from the environment image at the next moment.

[0118] The target object can be determined from the environment image at the next moment based on the target frame in the environment image at the current moment and each cube frame in the environment image at the next moment. For example, the target frame in the environment image at the current moment can be compared with each cube frame in the environment image at the next moment, and the object in the cube frame with the highest similarity can be determined as the target object in the environment image at the next moment. The same object at different moments can be given a unique object ID to avoid object confusion and improve the accuracy of positioning and tracking.

[0119] According to the cube frame of the target object in the environmental image at the next moment, the new key points of the target object are determined. For example, the eight vertices of the cube frame can be used as new key points. According to the coordinate information of the new key points in the environmental image at the next moment, the new position information of the target object in the environment where the vehicle is located is determined, that is, the positioning and tracking of the target object is achieved. For example, according to the coordinate information of the new key points, two key points can be determined from the new key points, namely the new first target point and the new second target point, and then according to the coordinate information of the new first target point and the new second target point, the new position information of the target object is determined.

[0120] The beneficial effect of such a setting is that the target object can be tracked in time, the target object can be positioned in real time, and omissions in tracking can be avoided, thereby improving the safety of vehicle driving.

[0121] In this embodiment, a target object is determined from the environmental image at the next moment based on a target frame in the environmental image at the current moment and each bounding frame in the environmental image at the next moment, including: determining a confidence level corresponding to a bounding frame in the environmental image at the next moment; wherein the confidence level represents the possibility that an object in the bounding frame is of a preset category; determining a candidate frame from the bounding frame in the environmental image at the next moment based on the confidence level; determining an overlap rate between the target frame in the environmental image at the current moment and the candidate frame in the environmental image at the next moment; and if the overlap rate corresponding to the candidate frame satisfies a preset object screening condition, determining the object in the candidate frame as the target object.

[0122] Specifically, for each bounding box at the next moment, the confidence corresponding to the bounding box is calculated, and the confidence can represent the possibility that the object in the bounding box is a preset category. The preset category can represent different car models, and the bounding box is a cubic box. For example, the possibility that the object in the cubic box is a small car, a large car, or other obstacles can be determined. According to the confidence corresponding to the cubic box, it can be determined what category of obstacle the object in the cubic box may be.

[0123] Determine the category of the target object. For example, the confidence of the target frame at the current moment can be determined to obtain the category of the target object. According to the category of the object in the cube frame at the next moment and the category of the target object, one or more cube frames are determined from multiple cube frames at the next moment as candidate frames. For example, the cube frame corresponding to the category of the target object can be determined as the candidate frame, that is, the category of the object in the candidate frame is guaranteed to be consistent with the category of the target object. For example, if the target object is a small car, the objects in the candidate frame are also small cars.

[0124] Between adjacent moments, the driving distance of the vehicle is very small. Therefore, there are many repeated scenes in the two environmental images at adjacent moments, and the positions of the same object in the two adjacent environmental images are close. Determine the overlap rate between the target frame in the environmental image at the current moment and each candidate frame in the environmental image at the next moment. For example, superimpose the environmental image at the current moment and the environmental image at the next moment, determine whether there is overlap between the target frame in the environmental image at the current moment and each candidate frame in the environmental image at the next moment, and determine the area of ​​the overlap. The larger the overlap area, the higher the overlap rate.

[0125] An object screening condition is preset, and if the overlap rate corresponding to the candidate frame satisfies the preset object screening condition, the object in the candidate frame is determined as the target object; if the overlap rate corresponding to the candidate frame does not satisfy the preset object screening condition, the object in the candidate frame is determined not to be the target object. For example, the preset object screening condition may be to determine the object in the candidate frame with the largest overlap rate as the target object.

[0126] The beneficial effect of this setting is that the target object at consecutive moments is determined by combining confidence with IOU (Intersection over Union) matching, which improves the accuracy of object tracking, thereby improving the positioning accuracy and ensuring the safety of vehicle driving.

[0127] In this embodiment, it also includes: if the overlapping rates corresponding to the candidate boxes do not meet the preset object screening conditions, determining the distance value between the target box and the candidate box; if the distance value corresponding to the candidate box is within the preset distance threshold, determining the object in the candidate box as the target object.

[0128] Specifically, if the overlap rates corresponding to the candidate boxes do not meet the preset object screening conditions, the distance values ​​between the target box and each candidate box can be further determined. For example, the distance between the center point of the target box and the center point of the candidate box can be determined.

[0129] A distance threshold is preset. Different distance thresholds can be set for different preset categories. Determine the category of the target object and find the distance threshold corresponding to the category. Compare the distance value between the target frame and the candidate frame with the preset distance threshold. If the distance value is within the distance threshold, the object in the candidate frame corresponding to the distance value is determined as the target object. Alternatively, the object in the candidate frame corresponding to the smallest distance value can be determined as the target object.

[0130] The beneficial effect of such a setting is that the distance threshold is classified according to the category of the detection target, thereby ensuring the tracking of the target object, avoiding losing the target object, and improving the efficiency and accuracy of tracking.

[0131] In this embodiment, the environment image includes surround view images corresponding to multiple cameras; the method also includes: projecting the cube frame in the surround view image corresponding to the camera in the environment image at the next moment to the surround view image corresponding to the adjacent camera to obtain the projection position; wherein the projection position represents the two-dimensional coordinate of the cube frame projected to the surround view image corresponding to the adjacent camera; according to the projection position and the bottom surface position of each cube frame in the surround view image corresponding to the adjacent camera, determining the redundant frame from the surround view image corresponding to the adjacent camera, and performing subtraction processing on the objects in the redundant frame in the surround view image corresponding to the adjacent camera; wherein the objects in the redundant frame are objects that appear repeatedly in the environment image.

[0132] Specifically, the environment image is composed of multiple surround view images, so the environment image includes multiple surround view images. The scenes captured by each camera may be repeated, that is, the same objects may exist in different surround view images, that is, there may be repeated objects in the environment image. The repeated objects can be eliminated to improve the efficiency and accuracy of positioning.

[0133] For any camera, the fisheye image collected by the camera in the next moment's environment image is converted into a surround view image, and the cube frame in the surround view image is projected into the surround view image corresponding to the adjacent camera to obtain the projection position, which represents the two-dimensional coordinates of the cube frame projected into the surround view image corresponding to the adjacent camera. That is, the 3D cube frame is converted into the 2D coordinates of the image. For example, the cube frame in the image corresponding to the front view camera can be projected into the image corresponding to the left view camera.

[0134] According to the projection position and the bottom surface position of each cube frame in the surround view image corresponding to the adjacent camera, a redundant frame is determined from the surround view image corresponding to the adjacent camera. The redundant frame is an extra cube frame, and the object in the redundant frame is an object that appears repeatedly in the environment image. The confidence between the projection position and the bottom surface position of each cube frame in the surround view image corresponding to the adjacent camera can be calculated, and the confidence can be characterized as the possibility of the same object. For example, the overlap rate or similarity between the projection position and the bottom surface position can be calculated, and the higher the overlap rate or the higher the similarity, the higher the confidence.

[0135] If the confidence exceeds the preset confidence threshold, it means that the cube frame corresponding to the bottom position is a redundant frame, and the objects in the redundant frames in the surround images corresponding to the adjacent cameras can be eliminated, that is, repeated objects can be screened out.

[0136] The beneficial effect of this setting is that the 3D coordinates of the object are back-projected to the image coordinate system of the adjacent camera to obtain the coordinates of the above object in the adjacent image, so as to perform redundant positioning screening and fusion strategy for the same object in the image, and improve the fusion of redundant targets in the current tracking frame. For example, the redundant targets of the front-view camera and the left-view camera are projected and transformed from the world 3D coordinates to the image 2D coordinates. After the image matching is completed, the redundant targets are eliminated to facilitate the provision of accurate positioning information of obstacles, so as to accurately track the obstacles.

[0137] In this embodiment, the method also includes: if it is determined that the target object is stationary, obtaining the motion information of the vehicle; wherein the motion information represents the moving distance and moving direction of the vehicle; based on the motion information of the vehicle and the position information of the target object in the environment where the vehicle is located at the current moment, determining the position information of the target object in the environment where the vehicle is located at the next moment.

[0138] Specifically, by identifying and locating the target object in each frame of the environment image, the continuous position information of the target object can be obtained. However, there may be problems such as the direction of the target object jumping during calculation, resulting in low positioning accuracy.

[0139] To solve the above problems, different positioning optimization schemes can be adopted according to the different driving states of the target object. The driving state can be divided into two categories: static and moving. If the target object is static, when the vehicle moves, the distance between the vehicle and the target object changes with the movement of the vehicle, and the direction of the target object also changes with the movement of the vehicle.

[0140] When the target object is stationary, for example, the target object is parked in a parking lot, and the ego vehicle is leaving the parking lot, the ego vehicle can obtain its own motion information in real time, and the motion information can represent the moving distance and moving direction of the vehicle. The position information of the target object in the environment where the vehicle is located at the current moment can be the position information of the target object determined at the initial moment, or it can be the latest position information of the determined target object. According to the motion information of the ego vehicle and the position information of the target object, the new position information of the target object is updated in real time. For example, the direction and position of the stationary obstacle can be recursively deduced between time series based on the information such as the steering and movement of the ego vehicle transmitted from the chassis.

[0141] For a target object in motion, the position information can be continuously confirmed based on the environment images at different times, and Kalman, SG (Savitzky-Golay, mean square error filtering) and other filters can be used to optimize the combined position information to obtain a target object with a stable temporal position orientation. In this embodiment, the filtering process is not specifically limited.

[0142] The beneficial effect of such a setting is that the stationary obstacle is recursively positioned using the vehicle's steering and movement information according to the vehicle status information transmitted from the chassis, thereby reducing measurement calculations and improving tracking accuracy.

[0143] In an embodiment of the present application, a cylindrical environmental image is obtained by using fisheye images collected by multiple cameras. The target object to be located is determined from the cylindrical environmental image, and multiple key points on the target object are determined. Based on the coordinate information of these key points, the position information of the target object is determined to achieve the positioning of the target object. By determining the cylindrical environmental image, the distortion phenomenon caused by splicing the surround image into a bird's-eye view is avoided. By processing the coordinate information of the key points and optimizing the positioning algorithm of the target object, the size, shape and other external features of the target object can be determined, thereby improving the tracking accuracy of the target object, thereby improving the accuracy and safety of autonomous driving.

[0144] Figure 7 The structure block diagram of a target tracking device based on multiple cameras provided by an embodiment of the present invention. For the convenience of explanation, only the parts related to the embodiment of the present invention are shown. The device is applied to a vehicle, and multiple cameras are installed on the vehicle. Figure 7The target tracking device 700 based on multiple cameras includes: an image obtaining unit 701, an object recognition unit 702 and a position determination unit 703.

[0145] The image acquisition unit 701 is used to obtain an environmental image of the environment where the vehicle is located according to the fisheye images collected by each camera at the current moment; wherein the environmental image is a cylindrical image of the surrounding view;

[0146] The object recognition unit 702 is used to recognize the target object from the environment image and determine the key points of the target object; wherein the key points are coordinate points that characterize the appearance features of the target object;

[0147] The position determination unit 703 is used to determine the position information of the target object in the environment where the vehicle is located according to the coordinate information of the key point in the environment image.

[0148] Figure 8 A structural block diagram of a multi-camera target tracking device provided in an embodiment of the present disclosure is shown in FIG. Figure 8 As shown, the multi-camera based target tracking device 800 includes an image acquisition unit 801 , an object recognition unit 802 , and a position determination unit 803 , wherein the position determination unit 803 includes a target point determination module 8031 ​​and a position determination module 8032 .

[0149] A target point determination module 8031 ​​is used to determine two key points from the key points of the target object according to the coordinate information of the key points in the environment image, namely a first target point and a second target point;

[0150] The position determination module 8032 is used to determine the position information of the target object in the environment where the vehicle is located according to the coordinate information of the first target point and the second target point.

[0151] In one example, the key point is located on a target frame of a target object, the target frame is used to surround the target object, and the coordinate information includes a horizontal coordinate and a vertical coordinate; the target point determination module 8031 ​​includes:

[0152] A coordinate determination submodule, used to determine the coordinate information of the center point in the environment image;

[0153] A difference determination submodule, used for determining the difference between the abscissa of the central point and each of the key points according to the abscissa of the central point and the abscissa of each of the key points;

[0154] A first determination submodule, configured to determine a first target point from the key points of the target object according to a difference between the horizontal coordinates of the central point and each of the key points, and a size of the vertical coordinate of each of the key points;

[0155] The second determination submodule is used to determine a second target point from key points of the target object according to a layout position of the first target point on the target frame.

[0156] In one example, the second determination submodule is specifically used to:

[0157] According to the layout position of the first target point in the target frame, determine a key point adjacent to the first target point as an adjacent point; wherein the adjacent point and the first target point are located on the same side in the target frame;

[0158] Determine the distance information between the first target point and the adjacent point;

[0159] A second target point is determined from the adjacent points according to the distance information.

[0160] In one example, the location determination module 8032 includes:

[0161] An angle determination submodule, used to determine angle information of the target object according to the coordinate information of the first target point and the second target point; wherein the angle information represents the orientation of the target object relative to the vehicle;

[0162] a third determination submodule, configured to determine position information of the target object in a preset camera coordinate system according to the angle information of the target object and the coordinate information of the first target point;

[0163] The fourth determination submodule is used to determine the position information of the target object in the environment where the vehicle is located according to the position information of the target object in a preset camera coordinate system.

[0164] In one example, the third determination submodule is specifically used for:

[0165] According to the coordinate information of the key points, an initial rectangular frame is determined in the preset camera coordinate system; wherein the initial rectangular frame represents the shape of the top view of the target object;

[0166] According to the angle information of the target object, the initial rectangular frame is rotated to obtain a rotated rectangular frame;

[0167] According to the coordinate information of the first target point, the rotated rectangular frame is translated to obtain a translated rectangular frame;

[0168] The position information of the target object in the preset camera coordinate system is determined according to the coordinate information of the vertices of the translation rectangular frame in the preset camera coordinate system.

[0169] In one example, the position information of the target object in the preset camera coordinate system includes the coordinate information of the vertices of the translation rectangular box in the preset camera coordinate system; the fourth determination submodule is specifically used to:

[0170] Determine the coordinate information of the vertices of the translation rectangular box in the vehicle coordinate system according to the coordinate information of the vertices of the translation rectangular box in the preset camera coordinate system;

[0171] The coordinate information of the vertices of the translation rectangular frame in the vehicle coordinate system is determined as the position information of the target object in the environment where the vehicle is located.

[0172] In one example, the coordinate information of the vertices of the translation rectangle in the vehicle's own vehicle coordinate system is represented as:

[0173] transX=y·sin(angle)+x·cos(angle)+T0;

[0174] transY=y·cos(angle)-x·sin(angle)+T1;

[0175] Among them, X is the horizontal coordinate of the vertices of the translation rectangle box in the vehicle's own vehicle coordinate system, Y is the vertical coordinate of the vertices of the translation rectangle box in the vehicle's own vehicle coordinate system, x is the horizontal coordinate of the vertices of the translation rectangle box in the preset camera coordinate system, y is the vertical coordinate of the vertices of the translation rectangle box in the preset camera coordinate system, angle is the preset angle parameter, and T0 and T1 are both preset parameters.

[0176] In one example, the object identification unit 802 includes:

[0177] A target frame determination module, used to input the current environment image into a preset target recognition model to obtain a target frame in the current environment image; wherein the preset target recognition model is a neural network model, used to recognize the target object in the image; the target frame is used to surround the target object;

[0178] The key point determination module is used to determine the key points of the target object on the target frame according to preset key point detection conditions.

[0179] In one example, the target box is a cubic box; the key point determination module is specifically used for:

[0180] The vertices of the target frame are obtained according to a preset key point sequence as the key points of the target object.

[0181] In one example, it also includes:

[0182] An acquisition unit, used to acquire an environmental image of the environment where the vehicle is located at a next moment;

[0183] A recognition unit, configured to obtain at least one bounding box in the environment image at the next moment based on a preset target recognition model;

[0184] a target determination unit, configured to determine the target object from the environment image at the next moment according to the target frame in the environment image at the current moment and each enclosing frame in the environment image at the next moment;

[0185] The position updating unit is used to determine the new key point of the target object from the environmental image at the next moment, and determine the new position information of the target object in the environment where the vehicle is located based on the coordinate information of the new key point in the environmental image at the next moment.

[0186] In one example, the target determination unit includes:

[0187] A confidence determination module, used to determine the confidence corresponding to the bounding box in the environment image at the next moment; wherein the confidence represents the possibility that the object in the bounding box is a preset category;

[0188] A candidate determination module, configured to determine a candidate box from a bounding box in the environment image at the next moment according to the confidence level;

[0189] An overlap rate determination module, used to determine an overlap rate between a target frame in the environment image at the current moment and a candidate frame in the environment image at the next moment;

[0190] The object screening module is used to determine the object in the candidate frame as the target object if the overlap rate corresponding to the candidate frame meets a preset object screening condition.

[0191] In one example, it also includes:

[0192] a distance value determining unit, configured to determine a distance value between the target frame and the candidate frame if the overlap rates corresponding to the candidate frames do not satisfy a preset object screening condition;

[0193] The distance comparison unit is used to determine the object in the candidate frame as the target object if the distance value corresponding to the candidate frame is within a preset distance threshold.

[0194] In one example, the environment image includes surround view images corresponding to multiple cameras; the device also includes:

[0195] A projection unit, used to project the bounding box in the surround view image corresponding to the camera in the environment image at the next moment into the surround view image corresponding to the adjacent camera to obtain a projection position; wherein the projection position represents the two-dimensional coordinates of the bounding box projected into the surround view image corresponding to the adjacent camera;

[0196] A subtraction unit is used to determine a redundant frame from the surround image corresponding to the adjacent camera according to the projection position and the bottom surface position of each bounding frame in the surround image corresponding to the adjacent camera, and to perform subtraction processing on the objects in the redundant frame in the surround image corresponding to the adjacent camera; wherein the objects in the redundant frame are objects that appear repeatedly in the environment image.

[0197] In one example, it also includes:

[0198] An information acquisition unit, configured to acquire the motion information of the vehicle if it is determined that the target object is stationary; wherein the motion information represents the moving distance and moving direction of the vehicle;

[0199] A position optimization unit is used to determine the position information of the target object in the environment where the vehicle is located at the next moment according to the movement information of the vehicle and the position information of the target object in the environment where the vehicle is located at the current moment.

[0200] In one example, the image acquisition unit 801 is specifically configured to:

[0201] According to a preset camera stitching sequence, the fisheye images collected by each camera are stitched together to obtain an environmental image of the environment where the vehicle is located at the current moment.

[0202] Fig. 9 A structural block diagram of an electronic device provided in an embodiment of the present application, such as Fig. 9 As shown, the electronic device includes: a memory 91 and a processor 92; the memory 91 is a memory for storing instructions executable by the processor 92.

[0203] The processor 92 is configured to execute the method provided in the above embodiment.

[0204] The electronic device further includes a receiver 93 and a transmitter 94. The receiver 93 is used to receive instructions and data sent by other devices, and the transmitter 94 is used to send instructions and data to external devices.

[0205] Fig.10 It is a block diagram of a terminal device according to an exemplary embodiment. The device may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0206] Device 1000 may include one or more of the following components: a processing component 1002 , a memory 1004 , a power component 1006 , a multimedia component 1008 , an audio component 1010 , an input / output (I / O) interface 1012 , a sensor component 1014 , and a communication component 1016 .

[0207] The processing component 1002 generally controls the overall operation of the device 1000, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1002 may include one or more processors 1020 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 1002 may include one or more modules to facilitate interaction between the processing component 1002 and other components. For example, the processing component 1002 may include a multimedia module to facilitate interaction between the multimedia component 1008 and the processing component 1002.

[0208] The memory 1004 is configured to store various types of data to support the operation of the device 1000. Examples of such data include instructions for any application or method operating on the device 1000, contact data, phone book data, messages, pictures, videos, etc. The memory 1004 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0209] The power supply component 1006 provides power to the various components of the device 1000. The power supply component 1006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 1000.

[0210] The multimedia component 1008 includes a screen that provides an output interface between the device 1000 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1008 includes a front camera and / or a rear camera. When the device 1000 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0211] The audio component 1010 is configured to output and / or input audio signals. For example, the audio component 1010 includes a microphone (MIC), and when the device 1000 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1004 or sent via the communication component 1016. In some embodiments, the audio component 1010 also includes a speaker for outputting audio signals.

[0212] I / O interface 1012 provides an interface between processing component 1002 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0213] The sensor assembly 1014 includes one or more sensors for providing various aspects of the status assessment of the device 1000. For example, the sensor assembly 1014 can detect the open / closed state of the device 1000, the relative positioning of components, such as the display and keypad of the device 1000, the sensor assembly 1014 can also detect the position change of the device 1000 or a component of the device 1000, the presence or absence of user contact with the device 1000, the orientation or acceleration / deceleration of the device 1000, and the temperature change of the device 1000. The sensor assembly 1014 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 1014 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1014 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0214] The communication component 1016 is configured to facilitate wired or wireless communication between the device 1000 and other devices. The device 1000 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1016 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1016 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0215] In an exemplary embodiment, the apparatus 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described methods.

[0216] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1004 including instructions, and the instructions can be executed by the processor 1020 of the device 1000 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0217] A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a terminal device, enables the terminal device to execute the above-mentioned terminal device's multi-camera based target tracking method.

[0218] The present application also discloses a computer program product, including a computer program, which implements the method described in this embodiment when executed by a processor.

[0219] Various embodiments of the systems and techniques described above in the present application can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs, which can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor, which can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0220] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or electronic device.

[0221] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0222] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0223] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data electronic device), or a computing system that includes middleware components (e.g., an application electronic device), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0224] The computer system may include a client and an electronic device. The client and the electronic device are generally far away from each other and usually interact through a communication network. The relationship between the client and the electronic device is generated by computer programs running on corresponding computers and having a client-electronic device relationship with each other. The electronic device may be a cloud electronic device, also known as a cloud computing electronic device or a cloud host, which is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The electronic device may also be an electronic device of a distributed system, or an electronic device combined with a blockchain. It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0225] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0226] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A target tracking method based on multiple cameras, characterized in that: The method is applied to a vehicle, and a plurality of cameras are installed on the vehicle; the method comprises: According to the fisheye images collected by each camera at the current moment, an environmental image of the environment where the vehicle is located is obtained; wherein the environmental image is a cylindrical image of the surrounding view; Identify the target object from the environment image and determine the key points of the target object; wherein the key points are coordinate points that characterize the appearance features of the target object; The position information of the target object in the environment where the vehicle is located is determined according to the coordinate information of the key point in the environment image.

2. The method according to claim 1, characterized in that Determining the position information of the target object in the environment where the vehicle is located according to the coordinate information of the key point in the environment image includes: According to the coordinate information of the key point in the environment image, two key points are determined from the key points of the target object, namely a first target point and a second target point; The position information of the target object in the environment where the vehicle is located is determined according to the coordinate information of the first target point and the second target point.

3. The method according to claim 2, characterized in that The key point is located on a target frame of the target object, the target frame is used to surround the target object, and the coordinate information includes a horizontal coordinate and a vertical coordinate; according to the coordinate information of the key point in the environment image, two key points are determined from the key points of the target object, namely a first target point and a second target point, including: Determine the coordinate information of the center point in the environment image; Determine the difference between the abscissas of the central point and the key points according to the abscissas of the central point and the key points; Determine a first target point from the key points of the target object according to the difference between the abscissas of the central point and each of the key points, and the size of the ordinates of each of the key points; According to the layout position of the first target point on the target frame, a second target point is determined from key points of the target object.

4. The method according to claim 3, characterized in that Determining a second target point from key points of the target object according to the layout position of the first target point in the target frame includes: According to the layout position of the first target point in the target frame, determine a key point adjacent to the first target point as an adjacent point; wherein the adjacent point and the first target point are located on the same side in the target frame; Determine the distance information between the first target point and the adjacent point; A second target point is determined from the adjacent points according to the distance information.

5. The method according to claim 2, characterized in that: Determining the position information of the target object in the environment where the vehicle is located according to the coordinate information of the first target point and the second target point includes: Determine angle information of the target object according to coordinate information of the first target point and the second target point; wherein the angle information represents the orientation of the target object relative to the vehicle; Determining position information of the target object in a preset camera coordinate system according to the angle information of the target object and the coordinate information of the first target point; The position information of the target object in the environment where the vehicle is located is determined according to the position information of the target object in a preset camera coordinate system.

6. The method according to claim 5, characterized in that Determining the position information of the target object in a preset camera coordinate system according to the angle information of the target object and the coordinate information of the first target point includes: Determining an initial rectangular frame in the preset camera coordinate system; wherein the initial rectangular frame represents the shape of the top view of the target object; According to the angle information of the target object, the initial rectangular frame is rotated to obtain a rotated rectangular frame; According to the coordinate information of the first target point, the rotated rectangular frame is translated to obtain a translated rectangular frame; The position information of the target object in the preset camera coordinate system is determined according to the coordinate information of the vertices of the translation rectangular frame in the preset camera coordinate system.

7. The method according to claim 6, characterized in that The position information of the target object in the preset camera coordinate system includes coordinate information of the vertices of the translation rectangular frame in the preset camera coordinate system; Determining the position information of the target object in the environment where the vehicle is located according to the position information of the target object in a preset camera coordinate system includes: Determine the coordinate information of the vertices of the translation rectangular box in the vehicle coordinate system according to the coordinate information of the vertices of the translation rectangular box in the preset camera coordinate system; The coordinate information of the vertices of the translation rectangular frame in the vehicle coordinate system is determined as the position information of the target object in the environment where the vehicle is located.

8. The method according to claim 7, characterized in that The coordinate information of the vertices of the translation rectangle in the vehicle's own vehicle coordinate system is represented as: transX=y·sin(angle)+x·cos(angle)+T0; transY=y·cos(angle)-x·sin(angle)+T1; Among them, X is the horizontal coordinate of the vertices of the translation rectangle box in the vehicle's own vehicle coordinate system, Y is the vertical coordinate of the vertices of the translation rectangle box in the vehicle's own vehicle coordinate system, x is the horizontal coordinate of the vertices of the translation rectangle box in the preset camera coordinate system, y is the vertical coordinate of the vertices of the translation rectangle box in the preset camera coordinate system, angle is the preset angle parameter, and T0 and T1 are both preset parameters.

9. The method according to any one of claims 1 to 8, characterized in that Identifying a target object from the environment image and determining key points of the target object include: Inputting the current environment image into a preset target recognition model to obtain a target frame in the current environment image; wherein the preset target recognition model is a neural network model for recognizing a target object in an image; and the target frame is used to surround the target object; According to preset key point detection conditions, key points of the target object on the target frame are determined.

10. The method according to claim 9, characterized in that The target frame is a cubic frame; Determining the key points of the target object on the target frame according to preset key point detection conditions includes: The vertices of the target frame are obtained according to a preset key point sequence as the key points of the target object.

11. The method according to claim 10, characterized in that Also includes: Acquire an environmental image of the environment where the vehicle is located at the next moment; Based on a preset target recognition model, obtaining at least one bounding box in the environment image at the next moment; Determine the target object from the environment image at the next moment according to the target frame in the environment image at the current moment and the enclosing frames in the environment image at the next moment; Determine a new key point of the target object from the environment image at the next moment, and determine new position information of the target object in the environment where the vehicle is located based on coordinate information of the new key point in the environment image at the next moment.

12. The method according to claim 11, characterized in that Determining the target object from the environment image at the next moment according to the target frame in the environment image at the current moment and each enclosing frame in the environment image at the next moment includes: Determine the confidence level corresponding to the bounding box in the environment image at the next moment; wherein the confidence level represents the possibility that the object in the bounding box is of a preset category; Determine a candidate box from the bounding box in the environment image at the next moment according to the confidence level; Determine an overlap rate between the target frame in the environment image at the current moment and the candidate frame in the environment image at the next moment; If the overlap rate corresponding to the candidate frame meets the preset object screening condition, the object in the candidate frame is determined as the target object.

13. The method according to claim 12, characterized in that Also includes: If the overlap rates corresponding to the candidate frames do not satisfy the preset object screening condition, determining the distance value between the target frame and the candidate frame; If the distance value corresponding to the candidate frame is within a preset distance threshold, the object in the candidate frame is determined as the target object.

14. The method according to claim 11, characterized in that The environment image includes surround images corresponding to multiple cameras; the method further includes: Projecting the bounding box in the surround view image corresponding to the camera in the environment image at the next moment into the surround view image corresponding to the adjacent camera to obtain a projection position; wherein the projection position represents the two-dimensional coordinate of the bounding box projected into the surround view image corresponding to the adjacent camera; According to the projection position and the bottom surface position of each bounding box in the surround image corresponding to the adjacent camera, a redundant box is determined from the surround image corresponding to the adjacent camera, and objects in the redundant box in the surround image corresponding to the adjacent camera are subtracted; wherein the objects in the redundant box are objects that appear repeatedly in the environmental image.

15. A target tracking device based on multiple cameras, characterized in that: The device is applied to a vehicle, and a plurality of cameras are installed on the vehicle; the device comprises: An image acquisition unit, used to obtain an environmental image of the environment where the vehicle is located according to the fisheye images collected by each camera at the current moment; wherein the environmental image is a cylindrical image of the surrounding view; An object recognition unit, used to recognize a target object from the environment image and determine key points of the target object; wherein the key points are coordinate points that characterize the appearance features of the target object; A position determination unit is used to determine the position information of the target object in the environment where the vehicle is located according to the coordinate information of the key point in the environment image.

16. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 14.

17. A computer-readable storage medium / computer program product, characterized in that: The computer-readable storage medium stores computer-executable instructions, the computer program product includes a computer program, and the computer-executable instructions / the computer program are used to implement the method according to any one of claims 1 to 14 when executed by a processor.

Images