Object association method, computer device, computer readable storage medium and vehicle

Abstract

The present application relates to the technical field of automatic driving, and particularly provides an object association method, a computer device, a storage medium and a vehicle, aiming at solving the problem of accurately identifying objects around the vehicle. The method of the present application comprises: according to the image features of the to-be-identified objects in the continuous multiple frames of images collected by the same vehicle-mounted visual sensor, associating the to-be-identified objects in the continuous multiple frames of images to form single-sensor-level to-be-identified objects; according to the image features of each single-sensor-level to-be-identified object, associating the single-sensor-level to-be-identified objects corresponding to different vehicle-mounted visual sensors to form cross-sensor-level to-be-identified objects; and according to the image features of each cross-sensor-level to-be-identified object, associating the cross-sensor-level to-be-identified objects to form the final to-be-identified objects. In the above manner, the same to-be-identified object can be prevented from being split into multiple different objects, thereby improving the accuracy of object identification.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, specifically providing an object association method, a computer device, a computer-readable storage medium, and a vehicle. Background Technology

[0002] In autonomous driving, onboard sensors are needed to identify objects around the vehicle. Based on the objects' positions and / or speeds, the vehicle's trajectory is planned and controlled to follow that trajectory. To improve the accuracy and efficiency of object recognition, multiple onboard sensors are typically used for object identification, and the objects from these sensors are then correlated to obtain the final identified object. However, due to differences in image acquisition precision and / or angle among different onboard sensors, the image features extracted from the same object captured by different sensors may vary. This can lead to discrepancies in the object features extracted from the images of the same object, potentially splitting the same object into multiple distinct objects after correlation, thus affecting the accuracy of object recognition and reducing vehicle safety.

[0003] Accordingly, a new technical solution is needed in this field to solve the above problems. Summary of the Invention

[0004] To overcome the above-mentioned deficiencies, the present invention is proposed to provide an object recognition method, computer device, computer-readable storage medium, and vehicle that solves or at least partially solves the technical problem of how to accurately identify objects around a vehicle.

[0005] Firstly, a method for associating objects is provided, the method comprising:

[0006] Based on the image features of the object to be identified in a series of consecutive frames of images acquired by the same vehicle-mounted vision sensor, the object to be identified in the series of consecutive frames of images is associated to form at least one single-sensor level object to be identified corresponding to the same vehicle-mounted vision sensor.

[0007] Based on the image features of each object to be identified at the single sensor level, the objects to be identified at the single sensor level corresponding to different vehicle vision sensors are associated to form at least one object to be identified across sensor levels.

[0008] Based on the image features of each cross-sensor level object to be identified, the cross-sensor level objects to be identified are associated to form the final objects to be identified.

[0009] In one technical solution of the above object association method, the step of "associating the objects to be identified in the continuous multi-frame images based on the image features of the objects to be identified in the same vehicle-mounted vision sensor to form at least one single-sensor-level object to be identified corresponding to the same vehicle-mounted vision sensor" specifically includes:

[0010] Based on the image features of each object to be identified in the current frame and the image features of each object to be identified in the previous frame, the feature similarity between each object to be identified in the current frame and each object to be identified in the previous frame is determined respectively.

[0011] The object to be identified in the current frame is associated with the object to be identified in the previous frame based on the feature similarity, so as to form at least one object to be identified at the single sensor level.

[0012] In one technical solution of the above object association method, the step of "determining the feature similarity between each object to be identified in the current frame and each object to be identified in the previous frame based on the image features of each object to be identified in the current frame and the image features of each object to be identified in the previous frame" specifically includes:

[0013] Based on the object position information in the image features, coordinate system transformation is performed on the object to be identified in the current frame and the object to be identified in the previous frame, so as to transform the object to be identified in the current frame and the object to be identified in the previous frame onto the BEV image with the current vehicle as the origin of the BEV (Birds Eye Views) coordinate system.

[0014] Based on the coordinates of the object to be identified in the current frame and the object to be identified in the previous frame on the BEV image, the object distance between the object to be identified in the current frame and the current vehicle, and the object distance between the object to be identified in the previous frame and the current vehicle are calculated respectively.

[0015] Based on the object distance, determine the first feature similarity between each object to be identified in the current frame and each object to be identified in the previous frame;

[0016] And / or, the step of "determining the feature similarity between each object to be identified in the current frame and each object to be identified in the previous frame based on the image features of each object to be identified in the current frame and the image features of each object to be identified in the previous frame" specifically includes:

[0017] Based on the object bounding boxes in the image features, calculate the intersection-union ratio (IUU) between the bounding box of each object to be identified in the current frame and the bounding box of each object to be identified in the previous frame.

[0018] Based on the intersection-union ratio, the second feature similarity between each object to be identified in the current frame and each object to be identified in the previous frame is determined.

[0019] In one technical solution of the above object association method, the step of "associating the object to be identified in the current frame with the object to be identified in the previous frame based on the feature similarity to form at least one object to be identified at the single sensor level" specifically includes:

[0020] For each object to be identified in the current frame, the object to be identified in the current frame is associated with each object to be identified in the previous frame in the following ways:

[0021] Determine whether the first feature similarity between the object to be identified in the current frame and the object to be identified in the previous frame is greater than a preset first similarity threshold;

[0022] If so, the object to be identified in the current frame is associated with the object to be identified in the previous frame to form a single sensor-level object to be identified.

[0023] If not, then continue to determine whether the second feature similarity between the object to be identified in the current frame and the object to be identified in the previous frame is greater than the preset second similarity threshold.

[0024] If the similarity is greater than the second similarity threshold, the object to be identified in the current frame is associated with the object to be identified in the previous frame to form a single sensor-level object to be identified; if the similarity is less than or equal to the second similarity threshold, the object to be identified in the current frame is not associated with the object to be identified in the previous frame and the object to be identified in the current frame is treated as a single sensor-level object to be identified.

[0025] Alternatively, the step of "associating the object to be identified in the current frame with the object to be identified in the previous frame based on the feature similarity to form at least one object to be identified at the single sensor level" specifically includes:

[0026] For each object to be identified in the current frame, the object to be identified in the current frame is associated with each object to be identified in the previous frame in the following ways:

[0027] Determine whether the first feature similarity between the object to be identified in the current frame and the object to be identified in the previous frame is greater than a preset first similarity threshold or whether the second feature similarity between the object to be identified in the current frame and the object to be identified in the previous frame is greater than a preset second similarity threshold;

[0028] If yes, then the object to be identified in the current frame is associated with the object to be identified in the previous frame to form a single sensor-level object to be identified; if no, then the object to be identified in the current frame is not associated with the object to be identified in the previous frame and the object to be identified in the current frame is treated as a single sensor-level object to be identified.

[0029] In one technical solution of the above object association method, the step of "associating the objects to be identified at the single sensor level corresponding to different vehicle vision sensors based on the image features of each single sensor level object to be identified, so as to form at least one cross-sensor level object to be identified" specifically includes:

[0030] Based on the image features of each object to be identified at the single sensor level, the feature similarity between the objects to be identified at the single sensor level corresponding to different vehicle vision sensors is determined.

[0031] Based on the feature similarity, the objects to be identified at the single sensor level corresponding to different vehicle vision sensors are associated to form at least one object to be identified across the sensor level.

[0032] In one technical solution of the above object association method, the step of "determining the feature similarity between the single-sensor level objects to be identified according to the image features of each single-sensor level object to be identified" specifically includes:

[0033] Based on the object position information in the image features, coordinate system transformation is performed on each of the single-sensor level objects to be identified, so as to transform each of the single-sensor level objects to be identified onto the BEV image with the current vehicle as the origin of the BEV (Birds Eye Views) coordinate system;

[0034] Based on the coordinates of each object to be identified at the single sensor level on the BEV image, calculate the object distance between each object to be identified at the single sensor level and the current vehicle.

[0035] Based on the object distance, the third feature similarity between the objects to be identified at the single sensor level corresponding to different vehicle vision sensors is determined respectively.

[0036] And / or, the step of "determining the feature similarity between objects to be identified at the single sensor level corresponding to different vehicle vision sensors based on the image features of each single sensor level object to be identified" specifically includes:

[0037] Based on the object bounding boxes in the image features, calculate the intersection-over-union ratio of the bounding boxes of the objects to be identified at the single sensor level for different vehicle vision sensors;

[0038] Based on the intersection-union ratio, the fourth feature similarity between the objects to be identified at the single sensor level corresponding to different vehicle vision sensors is determined respectively.

[0039] And / or, the step of "determining the feature similarity between objects to be identified at the single sensor level corresponding to different vehicle vision sensors based on the image features of each single sensor level object to be identified" specifically includes:

[0040] The Person Re-Identification technique is employed, and ReID features of each object to be identified at the single sensor level are obtained based on the image features of each object to be identified at the single sensor level.

[0041] Based on the ReID features, the fifth feature similarity between the objects to be identified at the single sensor level corresponding to different vehicle vision sensors is determined respectively.

[0042] And / or, the step of "associating the objects to be identified at the single sensor level corresponding to different vehicle vision sensors based on the feature similarity to form at least one object to be identified across the sensor level" specifically includes:

[0043] For each object to be identified at the single sensor level, determine whether there are other objects to be identified at the single sensor level with a feature similarity greater than a preset third similarity threshold.

[0044] If so, the current single-sensor level object to be identified is associated with the other single-sensor level objects to be identified to form a cross-sensor level object to be identified;

[0045] If not, the object to be identified at the current single-sensor level will be treated as a separate object to be identified at a cross-sensor level.

[0046] In one technical solution of the above object association method, the step of "determining the feature similarity between the single-sensor level objects to be identified according to the image features of each single-sensor level object to be identified" further includes:

[0047] The third feature similarity and / or the fourth feature similarity and / or the fifth feature similarity are weighted and summed, and the result is used as the final feature similarity.

[0048] In one technical solution of the above object association method, the step of "associating the cross-sensor level objects based on the image features of each cross-sensor level object to form the final object to be identified" specifically includes:

[0049] Based on the two-dimensional information of the object in the image features of each cross-sensor level object to be identified, the two-dimensional similarity between the objects to be identified at different cross-sensor levels is calculated respectively. The objects to be identified at different cross-sensor levels are associated based on the two-dimensional similarity to form the final objects to be identified.

[0050] And / or, based on the object's 3D information in the image features of each cross-sensor level object to be identified, calculate the 3D similarity between objects to be identified at different cross-sensor levels, and associate the objects to be identified at different cross-sensor levels based on the 3D similarity to form the final object to be identified.

[0051] In a second aspect, a computer device is provided, comprising a processor and a storage device, the storage device being adapted to store a plurality of program codes, the program codes being adapted to be loaded and executed by the processor to perform the object association method described in any of the above-described technical solutions.

[0052] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, the program codes being adapted to be loaded and run by a processor to perform the object association method described in any of the above-described object association methods.

[0053] In a fourth aspect, a vehicle is provided that includes the computer equipment described in the above-described computer equipment technical solution.

[0054] Solution 1. An object association method, characterized in that the method includes:

[0055] Based on the image features of the object to be identified in a series of consecutive frames of images acquired by the same vehicle-mounted vision sensor, the object to be identified in the series of consecutive frames of images is associated to form at least one single-sensor level object to be identified corresponding to the same vehicle-mounted vision sensor.

[0056] Based on the image features of each object to be identified at the single sensor level, the objects to be identified at the single sensor level corresponding to different vehicle vision sensors are associated to form at least one object to be identified across sensor levels.

[0057] Based on the image features of each cross-sensor level object to be identified, the cross-sensor level objects to be identified are associated to form the final objects to be identified.

[0058] Solution 2. The object association method according to Solution 1, characterized in that the step of "associating the objects to be identified in the continuous multi-frame images based on the image features of the objects to be identified in the same vehicle-mounted vision sensor to form at least one single-sensor-level object to be identified corresponding to the same vehicle-mounted vision sensor" specifically includes:

[0059] Based on the image features of each object to be identified in the current frame and the image features of each object to be identified in the previous frame, the feature similarity between each object to be identified in the current frame and each object to be identified in the previous frame is determined respectively.

[0060] The object to be identified in the current frame is associated with the object to be identified in the previous frame based on the feature similarity, so as to form at least one object to be identified at the single sensor level.

[0061] Solution 3. The object association method according to Solution 2, characterized in that the step of "determining the feature similarity between each object to be identified in the current frame and each object to be identified in the previous frame based on the image features of each object to be identified in the current frame and the image features of each object to be identified in the previous frame" specifically includes:

[0062] Based on the object position information in the image features, coordinate system transformation is performed on the object to be identified in the current frame and the object to be identified in the previous frame, so as to transform the object to be identified in the current frame and the object to be identified in the previous frame onto the BEV image with the current vehicle as the origin of the BEV (Birds Eye Views) coordinate system.

[0063] Based on the coordinates of the object to be identified in the current frame and the object to be identified in the previous frame on the BEV image, the object distance between the object to be identified in the current frame and the current vehicle, and the object distance between the object to be identified in the previous frame and the current vehicle are calculated respectively.

[0064] Based on the object distance, determine the first feature similarity between each object to be identified in the current frame and each object to be identified in the previous frame;

[0065] And / or,

[0066] The step of "determining the feature similarity between each object to be identified in the current frame and each object to be identified in the previous frame based on the image features of each object to be identified in the current frame and the image features of each object to be identified in the previous frame" specifically includes:

[0067] Based on the object bounding boxes in the image features, calculate the intersection-union ratio (IUU) between the bounding box of each object to be identified in the current frame and the bounding box of each object to be identified in the previous frame.

[0068] Based on the intersection-union ratio, the second feature similarity between each object to be identified in the current frame and each object to be identified in the previous frame is determined.

[0069] Solution 4. The object association method according to Solution 3, characterized in that the step of "associating the object to be identified in the current frame with the object to be identified in the previous frame based on the feature similarity to form at least one object to be identified at the single sensor level" specifically includes:

[0070] For each object to be identified in the current frame, the object to be identified in the current frame is associated with each object to be identified in the previous frame in the following ways:

[0071] Determine whether the first feature similarity between the object to be identified in the current frame and the object to be identified in the previous frame is greater than a preset first similarity threshold;

[0072] If so, the object to be identified in the current frame is associated with the object to be identified in the previous frame to form a single sensor-level object to be identified.

[0073] If not, then continue to determine whether the second feature similarity between the object to be identified in the current frame and the object to be identified in the previous frame is greater than the preset second similarity threshold.

[0074] If the similarity is greater than the second similarity threshold, the object to be identified in the current frame is associated with the object to be identified in the previous frame to form a single sensor-level object to be identified; if the similarity is less than or equal to the second similarity threshold, the object to be identified in the current frame is not associated with the object to be identified in the previous frame and the object to be identified in the current frame is treated as a single sensor-level object to be identified.

[0075] or,

[0076] The step of "associating the object to be identified in the current frame with the object to be identified in the previous frame based on the feature similarity to form at least one object to be identified at the single sensor level" specifically includes:

[0077] For each object to be identified in the current frame, the object to be identified in the current frame is associated with each object to be identified in the previous frame in the following ways:

[0078] Determine whether the first feature similarity between the object to be identified in the current frame and the object to be identified in the previous frame is greater than a preset first similarity threshold or whether the second feature similarity between the object to be identified in the current frame and the object to be identified in the previous frame is greater than a preset second similarity threshold;

[0079] If yes, then the object to be identified in the current frame is associated with the object to be identified in the previous frame to form a single sensor-level object to be identified; if no, then the object to be identified in the current frame is not associated with the object to be identified in the previous frame and the object to be identified in the current frame is treated as a single sensor-level object to be identified.

[0080] Solution 5. The object association method according to Solution 1, characterized in that the step of "associating the objects to be identified at the single sensor level corresponding to different vehicle vision sensors based on the image features of each single sensor level object to be identified, so as to form at least one cross-sensor level object to be identified" specifically includes:

[0081] Based on the image features of each object to be identified at the single sensor level, the feature similarity between the objects to be identified at the single sensor level corresponding to different vehicle vision sensors is determined.

[0082] Based on the feature similarity, the objects to be identified at the single sensor level corresponding to different vehicle vision sensors are associated to form at least one object to be identified across the sensor level.

[0083] Solution 6. The object association method according to Solution 5, characterized in that the step of "determining the feature similarity between the objects to be identified at the single sensor level corresponding to different vehicle vision sensors based on the image features of each object to be identified at the single sensor level" specifically includes:

[0084] Based on the object position information in the image features, coordinate system transformation is performed on each of the single-sensor level objects to be identified, so as to transform each of the single-sensor level objects to be identified onto the BEV image with the current vehicle as the origin of the BEV (Birds Eye Views) coordinate system;

[0085] Based on the coordinates of each object to be identified at the single sensor level on the BEV image, calculate the object distance between each object to be identified at the single sensor level and the current vehicle.

[0086] Based on the object distance, the third feature similarity between the objects to be identified at the single sensor level corresponding to different vehicle vision sensors is determined respectively.

[0087] And / or,

[0088] The step of "determining the feature similarity between objects to be identified at the single sensor level corresponding to different vehicle vision sensors based on the image features of each single sensor-level object to be identified" specifically includes:

[0089] Based on the object bounding boxes in the image features, calculate the intersection-over-union ratio of the bounding boxes of the objects to be identified at the single sensor level for different vehicle vision sensors;

[0090] Based on the intersection-union ratio, the fourth feature similarity between the objects to be identified at the single sensor level corresponding to different vehicle vision sensors is determined respectively.

[0091] And / or,

[0092] The step of "determining the feature similarity between objects to be identified at the single sensor level corresponding to different vehicle vision sensors based on the image features of each single sensor-level object to be identified" specifically includes:

[0093] The Person Re-Identification technique is employed, and ReID features of each object to be identified at the single sensor level are obtained based on the image features of each object to be identified at the single sensor level.

[0094] Based on the ReID features, the fifth feature similarity between the objects to be identified at the single sensor level corresponding to different vehicle vision sensors is determined respectively.

[0095] And / or,

[0096] The step of "associating the objects to be identified at the single-sensor level corresponding to different vehicle vision sensors based on the feature similarity to form at least one object to be identified across the sensor level" specifically includes:

[0097] For each object to be identified at the single sensor level, determine whether there are other objects to be identified at the single sensor level with a feature similarity greater than a preset third similarity threshold.

[0098] If so, the current single-sensor level object to be identified is associated with the other single-sensor level objects to be identified to form a cross-sensor level object to be identified;

[0099] If not, the object to be identified at the current single-sensor level will be treated as a separate object to be identified at a cross-sensor level.

[0100] Solution 7. The object association method according to Solution 6, characterized in that the step of "determining the feature similarity between the objects to be identified at the single sensor level corresponding to different vehicle vision sensors based on the image features of each object to be identified at the single sensor level" further includes:

[0101] The third feature similarity and / or the fourth feature similarity and / or the fifth feature similarity are weighted and summed, and the result is used as the final feature similarity.

[0102] Solution 8. The object association method according to Solution 1, characterized in that the step of "associating the cross-sensor level objects based on the image features of each cross-sensor level object to form the final object to be identified" specifically includes:

[0103] Based on the two-dimensional information of the object in the image features of each cross-sensor level object to be identified, the two-dimensional similarity between the objects to be identified at different cross-sensor levels is calculated respectively. The objects to be identified at different cross-sensor levels are associated based on the two-dimensional similarity to form the final objects to be identified.

[0104] And / or,

[0105] Based on the 3D information of the object in the image features of each cross-sensor level object to be identified, the 3D similarity between the objects to be identified at different cross-sensor levels is calculated respectively. The objects to be identified at different cross-sensor levels are associated based on the 3D similarity to form the final object to be identified.

[0106] Scheme 9. A computer device comprising a processor and a storage device, the storage device being adapted to store a plurality of program codes, characterized in that the program codes are adapted to be loaded and run by the processor to perform the object association method as described in any one of Schemes 1 to 8.

[0107] Scheme 10. A computer-readable storage medium storing a plurality of program codes, characterized in that the program codes are adapted to be loaded and run by a processor to perform the object association method of any one of Schemes 1 to 8.

[0108] Option 11. A vehicle, characterized in that the vehicle includes the computer equipment described in Option 9.

[0109] The above-described technical solutions of the present invention have at least one or more of the following beneficial effects:

[0110] In implementing the technical solution of the present invention, firstly, based on the image features of the object to be identified in a series of consecutive frames of images acquired by the same vehicle vision sensor, the object to be identified in the series of consecutive frames of images is associated to form at least one single-sensor level object to be identified corresponding to the same vehicle vision sensor; then, based on the image features of each single-sensor level object to be identified, the objects to be identified at the single-sensor level corresponding to different vehicle vision sensors are associated to form at least one cross-sensor level object to be identified; finally, based on the image features of each cross-sensor level object to be identified, the cross-sensor level objects to be identified are associated to form the final object to be identified.

[0111] Image features of an object to be identified in multiple consecutive frames of images acquired by the same vehicle-mounted vision sensor can represent the historical temporal feature information of the object. By using this historical temporal feature information to associate the objects to be identified and form single-sensor-level objects to be identified, the correlation between the objects to be identified can be better established. This effectively avoids the situation where the same object to be identified is split into multiple different objects due to differences in image acquisition accuracy and / or image acquisition angle of different vehicle-mounted vision sensors, which may lead to certain differences in image features extracted from images of the same object to be identified by different vehicle-mounted vision sensors. This improves the accuracy of object recognition. Attached Figure Description

[0112] The disclosure of this invention will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. Wherein:

[0113] Figure 1 This is a schematic flowchart of the main steps of an object association method according to an embodiment of the present invention;

[0114] Figure 2 This is a schematic flowchart of the main steps of a method for forming a single-sensor-level object to be identified according to an embodiment of the present invention;

[0115] Figure 3 This is a schematic flowchart illustrating the main steps of forming an object to be identified across sensor levels according to an embodiment of the present invention. Detailed Implementation

[0116] Some embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0117] In the description of this invention, "processor" can include hardware, software, or a combination of both. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Non-transitory computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B.

[0118] See appendix Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of an object association method according to an embodiment of the present invention. Figure 1 As shown, the object association method in this embodiment of the invention mainly includes the following steps S101 to S103.

[0119] Step S101: Based on the image features of the object to be identified in a series of consecutive frames of images acquired by the same vehicle-mounted vision sensor, the object to be identified in the series of consecutive frames of images is associated to form at least one single-sensor level object to be identified corresponding to the same vehicle-mounted vision sensor.

[0120] Vehicle-mounted vision sensors are sensors installed on vehicles to acquire images. The image features of the object to be identified include, but are not limited to, the object's position information, size, and orientation angle. The object's position information can be the object's two-dimensional coordinates and three-dimensional coordinates.

[0121] Images captured by vehicle-mounted vision sensors may contain multiple different objects to be identified. By associating the objects to be identified in consecutive frames of images, the same objects to be identified in these images can be linked together to form a single-sensor-level object to be identified.

[0122] For example, multiple consecutive frames of images acquired by the vehicle's vision sensor A all contain the object O to be identified. A1 O A2 and O A3 By associating the objects to be identified in multiple consecutive frames of images, the objects to be identified in each frame can be identified. A1 These are linked together to form a single-sensor level object to be identified, O′. A1 It can identify the object O in each frame of the image. A2 These are linked together to form a single-sensor level object to be identified, O′. A2 The object to be identified in each frame of the image OA3 These are linked together to form a single-sensor level object to be identified, O′. A3 .

[0123] Step S102: Based on the image features of the object to be identified at each single sensor level, associate the objects to be identified at the single sensor level corresponding to different vehicle vision sensors to form at least one object to be identified across sensor levels.

[0124] The objects to be identified at different single sensor levels may be the same object or they may not be the same object. By associating the objects to be identified at the single sensor level corresponding to different vehicle vision sensors, the same objects to be identified at these single sensor levels can be linked together to form a cross-sensor level object to be identified.

[0125] For example, a car is equipped with three onboard vision sensors A, B, and C. The system identifies the object O in multiple frames of images captured by onboard vision sensor A. A From the image features, the single-sensor level object O′ to be identified corresponding to the vehicle vision sensor A can be obtained. A Similarly, the single-sensor level object O′ to be identified corresponding to the vehicle-mounted vision sensor B can also be obtained. B The object to be identified, O′, corresponds to the single-sensor level of the vehicle-mounted vision sensor C. C To identify object O′ A O′ B O′ C By performing correlation, a cross-sensor level object O′ can be obtained. ABC .

[0126] Step S103: Associate the objects to be identified across sensor levels based on the image features of each object across sensor levels to form the final objects to be identified.

[0127] Different objects to be identified across different sensor levels may be the same object or they may not be the same object. By associating objects to be identified across different sensor levels, the same objects among these objects can be linked together to form the final object to be identified.

[0128] Since the image features of the object to be identified in multiple consecutive frames of images acquired by the same vehicle vision sensor can represent the historical temporal feature information of the object to be identified, the object to be identified can be associated using this historical temporal feature information to form a single sensor-level object to be identified. This can better establish the correlation between the objects to be identified. Therefore, the method described in steps S101 to S103 can effectively avoid the problem of splitting the same object to be identified into multiple different objects when there are certain differences in the image features extracted from images of the same object to be identified by different vehicle vision sensors, thereby improving the recognition accuracy of the object to be identified.

[0129] The following provides further explanation of steps S101 to S103.

[0130] See appendix Figure 2 In order to accurately associate the objects to be identified in multiple consecutive frames of images acquired by the same vehicle vision sensor, in one embodiment of step S101 above, the objects to be identified can be associated through the following steps S1011 to S1012.

[0131] Step S1011: Based on the image features of each object to be identified in the current frame and the image features of each object to be identified in the previous frame, determine the feature similarity between each object to be identified in the current frame and each object to be identified in the previous frame.

[0132] Feature similarity refers to the similarity between image features of objects to be identified. The higher the feature similarity, the higher the degree of similarity between the objects to be identified, and the lower the feature similarity, the lower the degree of similarity between the objects to be identified.

[0133] Since the image features of the objects to be identified can include various features such as the object's position, size, and direction cosine, different methods can be used to determine the feature similarity between the objects in this embodiment based on different feature information. The specific methods for determining the feature similarity between the objects to be identified based on the object position information and object bounding boxes in the image features are described below.

[0134] 1. Determine feature similarity based on object location information.

[0135] In this embodiment, the feature similarity between objects to be identified can be determined based on the object location information in the image features and through the following steps 11 to 13.

[0136] Step 11: Based on the object position information in the image features, perform coordinate system transformation on the object to be identified in the current frame and the object to be identified in the previous frame respectively, so as to transform the object to be identified in the current frame and the object to be identified in the previous frame onto the BEV image with the current vehicle as the origin of the BEV (Birds Eye Views) coordinate system.

[0137] In this embodiment, conventional coordinate system transformation methods in the field of image processing technology can be used to transform the object to be identified onto the BEV image based on the object position information of the object to be identified. This embodiment will not elaborate on the above coordinate system transformation methods.

[0138] Step 12: Based on the coordinates of the object to be identified in the current frame and the object to be identified in the previous frame on the BEV image, calculate the object distance between the object to be identified in the current frame and the current vehicle, and the object distance between the object to be identified in the previous frame and the current vehicle.

[0139] In this embodiment, the distance between the object to be identified and the current vehicle can be calculated using the following methods:

[0140] The coordinates of the object to be identified on the BEV image include the coordinates of the X-axis and the Y-axis in the image coordinate system of the BEV image. After obtaining the coordinates of the object to be identified on the BEV image, the coordinates of the X-axis and the Y-axis can be normalized first, and then the distance between the object to be identified and the current vehicle can be calculated by the following formula (1).

[0141]

[0142] The parameters in formula (1) have the following meanings: d1 represents the distance between the object to be identified and the current vehicle, x nor1 The x-axis coordinates are normalized, and the y-axis coordinates are normalized. nor1 This represents the Y-axis coordinate after normalization.

[0143] In addition, in some other implementations, the Euclidean distance calculation method can be used to calculate the object distance between the object to be identified and the current vehicle based on the coordinates of the object to be identified on the BEV image.

[0144] Step 13: Based on the object distance, determine the first feature similarity between each object to be identified in the current frame and each object to be identified in the previous frame.

[0145] The closer the object distances of two objects to be identified, the higher the similarity between them. In this embodiment, the distance deviation between the object distances of the two objects to be identified can be calculated first, and then the distance deviation can be inverted. The result of the inversion operation is used as the first feature similarity, so that the first feature similarity is positively correlated with the similarity between the objects to be identified (the larger the first feature similarity, the higher the similarity between the objects to be identified, and vice versa). The inversion operation is a method of operation that makes the result negatively correlated with the distance deviation, that is, the smaller the distance deviation, the larger the result obtained by inverting the distance deviation. For example, in some embodiments, after calculating the distance deviation between the object distances of the two objects to be identified, the distance deviation can be inverted using the following formula (2).

[0146]

[0147] In formula (2), the parameter d1′ represents the result of the operation after inverting the distance deviation d1.

[0148] By using the methods described in steps 11 to 13 above, the feature similarity between objects to be identified can be accurately determined based on the object location information in the image features.

[0149] 2. Determine feature similarity based on object bounding boxes.

[0150] In this embodiment, the feature similarity between objects to be identified can be determined based on the object bounding boxes in the image features and through the following steps 21 to 22.

[0151] Step 21: Based on the object bounding boxes in the image features, calculate the intersection over union (IoU) ratio between the bounding box of the object to be identified in the current frame and the bounding box of the object to be identified in the previous frame.

[0152] Step 22: Based on the intersection-union ratio, determine the second feature similarity between each object to be identified in the current frame and each object to be identified in the previous frame.

[0153] The higher the intersection-union ratio (IUR) of two objects to be identified, the higher the similarity between the two objects. The second feature similarity determined by the IUR is positively correlated with the similarity between the objects to be identified (the higher the second feature similarity, the higher the similarity between the objects to be identified, and vice versa).

[0154] By using the methods described in steps 21 to 22 above, the feature similarity between objects to be identified can be accurately determined based on the object bounding boxes in the image features.

[0155] The above is a detailed explanation of the method for determining the feature similarity between objects to be identified in step S1011. Step S1012 will be explained below.

[0156] Step S1012: Associate the object to be identified in the current frame with the object to be identified in the previous frame based on feature similarity to form at least one object to be identified at the single sensor level.

[0157] Since the magnitude of feature similarity can represent the degree of similarity between objects to be identified, associating objects to be identified based on feature similarity can more accurately link the same objects together, forming objects to be identified at the single sensor level.

[0158] In this embodiment, objects with high feature similarity can be grouped together to form single-sensor-level objects to be identified. As described in step S1011, different methods can be used to determine the feature similarity between objects based on different feature information, such as first feature similarity and second feature similarity. Accordingly, in this embodiment, objects can be associated based on first feature similarity, second feature similarity, or both simultaneously.

[0159] Specifically, in some implementations, for each object to be identified in the current frame, the following steps 31 to 33 can be used to associate the object to be identified in the current frame with each object to be identified in the previous frame based on the first feature similarity or the second feature similarity.

[0160] Step 31: Determine whether the first feature similarity between the object to be identified in the current frame and the object to be identified in the previous frame is greater than a preset first similarity threshold or whether the second feature similarity between the object to be identified in the current frame and the object to be identified in the previous frame is greater than a preset second similarity threshold; if the first feature similarity is greater than the preset first similarity threshold or the second feature similarity is greater than the preset second similarity threshold, then proceed to step 32; otherwise, proceed to step 33.

[0161] It should be noted that those skilled in the art can flexibly set the specific values ​​of the preset first similarity threshold and the second similarity threshold according to actual needs, and this embodiment does not impose specific limitations on this.

[0162] Step 32: Associate the object to be identified in the current frame with the object to be identified in the previous frame to form a single sensor-level object to be identified, that is, associate the two objects to be identified to form a single sensor-level object to be identified.

[0163] Step 33: Do not associate the object to be identified in the current frame with the object to be identified in the previous frame, and treat the object to be identified in the current frame as a single sensor-level object to be identified.

[0164] Through steps 31 to 33 above, the object to be identified can be associated based on either the first feature similarity or the second feature similarity. In other embodiments, to improve the accuracy of the association of the object to be identified, the object can be associated simultaneously based on both the first feature similarity and the second feature similarity. Specifically, in these embodiments, for each object to be identified in the current frame, the object to be identified in the current frame can be associated with each object to be identified in the previous frame through the following steps 41 to 45.

[0165] Step 41: Determine whether the first feature similarity between the object to be identified in the current frame and the object to be identified in the previous frame is greater than the preset first similarity threshold; if it is greater, proceed to step 42; if it is less than or equal to, proceed to step 43.

[0166] Step 42: Associate the object to be identified in the current frame with the object to be identified in the previous frame to form a single sensor-level object to be identified.

[0167] Step 43: Determine whether the second feature similarity between the object to be identified in the current frame and the object to be identified in the previous frame is greater than the preset second similarity threshold; if it is greater, proceed to step 44; if it is less than or equal to, proceed to step 45.

[0168] Step 44: Associate the object to be identified in the current frame with the object to be identified in the previous frame to form a single sensor-level object to be identified.

[0169] Step 45: Do not associate the object to be identified in the current frame with the object to be identified in the previous frame, and treat the object to be identified in the current frame as a single sensor-level object to be identified.

[0170] By employing the methods described in steps 41 to 45, once it is determined that the object to be identified can be associated based on the first feature similarity, subsequent steps 43 to 45 are no longer necessary. This not only reduces the resource consumption of the computer and other equipment used to execute the method but also improves the efficiency of associating the object to be identified. Furthermore, if it cannot be determined whether the object to be identified can be associated based on the first feature similarity, a further determination can be made based on the second feature similarity, thereby improving the accuracy of associating the object to be identified.

[0171] The above is about Figure 1 Further explanation of step S101 in the illustrated embodiment will continue below. Figure 1 Step S102 in the illustrated embodiment will be further explained.

[0172] See appendix Figure 3 In order to accurately associate objects to be identified at the single sensor level corresponding to different vehicle vision sensors to form at least one object to be identified across sensor levels, in one embodiment of the above step S102, the objects to be identified can be associated through the following steps S1021 to S1022.

[0173] Step S1021: Based on the image features of the object to be identified at each single sensor level, determine the feature similarity between the objects to be identified at the single sensor level corresponding to different vehicle vision sensors.

[0174] Feature similarity refers to the similarity between image features of "objects to be identified at the single sensor level". The higher the feature similarity, the higher the similarity between "objects to be identified at the single sensor level", and the lower the feature similarity, the lower the similarity between "objects to be identified at the single sensor level".

[0175] The image features of a "single-sensor level object to be identified" can also include various feature information such as the object's position, size, and direction cosine. Therefore, in this embodiment, different methods can be used to determine the feature similarity between objects to be identified based on different feature information. The specific method for determining the feature similarity between objects to be identified in this embodiment will be described below.

[0176] 1. Determine feature similarity based on object location information.

[0177] In this embodiment, the feature similarity between objects to be identified at different single sensor levels can be determined based on the object location information in the image features and through the following steps 51 to 53.

[0178] Step 51: Based on the object position information in the image features, perform coordinate system transformation on each single-sensor level object to be identified, so as to transform each single-sensor level object to be identified onto the BEV image with the current vehicle as the origin of the BEV (BirdsEye Views) coordinate system. In this embodiment, conventional coordinate system transformation methods in the field of image processing technology can be used to transform the single-sensor level object to be identified onto the BEV image based on the object position information. This embodiment will not elaborate on the above coordinate system transformation method.

[0179] Step 52: Based on the coordinates of the object to be identified at each single sensor level on the BEV image, calculate the object distance between the object to be identified at each single sensor level and the current vehicle.

[0180] In this embodiment, the distance between the object to be identified and the current vehicle at the single-sensor level can be calculated using the following methods:

[0181] The coordinates of the object to be identified at the single sensor level on the BEV image include the coordinates of the X-axis and the Y-axis in the image coordinate system of the BEV image. After obtaining the coordinates of the object to be identified at the single sensor level on the BEV image, the coordinates of the X-axis and the Y-axis can be normalized first, and then the distance between the object to be identified at the single sensor level and the current vehicle can be calculated by the following formula (3).

[0182]

[0183] The parameters in formula (3) have the following meanings: d2 represents the distance between the object to be identified at the single sensor level and the current vehicle, x nor2 The x-axis coordinates are normalized, and the y-axis coordinates are normalized. nor2 This represents the Y-axis coordinate after normalization.

[0184] In addition, in some other implementations, the Euclidean distance calculation method can also be used to calculate the object distance between the object to be identified at the single sensor level and the current vehicle based on the coordinates of the object to be identified at the single sensor level on the BEV image.

[0185] Step 53: Based on the object distance, determine the third feature similarity between the objects to be identified at the single sensor level corresponding to different vehicle vision sensors.

[0186] The closer the object distances of two objects to be identified at the single-sensor level, the higher the similarity between them. In this embodiment, the distance deviation between the object distances of the two objects to be identified at the single-sensor level can be calculated first, and then the distance deviation can be inverted. The result of the inversion operation is used as the third feature similarity, so that the third feature similarity is positively correlated with the similarity between the objects to be identified at the single-sensor level (the larger the third feature similarity, the higher the similarity between the objects to be identified, and vice versa). The inversion operation is a calculation method that makes the calculation result negatively correlated with the distance deviation, that is, the smaller the distance deviation, the larger the calculation result obtained by inverting the distance deviation. For example, in some embodiments, after calculating the distance deviation between the object distances of the two objects to be identified at the single-sensor level, the distance deviation can be inverted using the following formula (4).

[0187]

[0188] In formula (4), the parameter d2′ represents the result obtained by inverting the distance deviation d2.

[0189] By using the methods described in steps 51 to 53 above, the feature similarity between objects to be identified at different single sensor levels can be accurately determined based on the object location information in the image features.

[0190] 2. Determine feature similarity based on object bounding boxes.

[0191] In this embodiment, the feature similarity between objects to be identified at different single sensor levels can be determined based on the object bounding boxes in the image features and through the following steps 61 to 62.

[0192] Step 61: Based on the object bounding boxes in the image features, calculate the intersection over union (IoU) of the bounding boxes of the objects to be identified at the single sensor level for different vehicle vision sensors.

[0193] Step 62: Based on the cross-union ratio, determine the fourth feature similarity between the objects to be identified at the single sensor level corresponding to different vehicle vision sensors.

[0194] The higher the cross-union ratio (CUI) of two single-sensor level objects to be identified, the higher the similarity between the two single-sensor level objects to be identified. The fourth feature similarity determined by the CUI is positively correlated with the similarity between the single-sensor level objects to be identified (the higher the fourth feature similarity, the higher the similarity between the objects to be identified, and vice versa).

[0195] By using the methods described in steps 61 to 62 above, the feature similarity between objects to be identified at the single sensor level can be accurately determined based on the object bounding boxes in the image features.

[0196] Third, determine the feature similarity based on the ReID (Person Re-Identification) features of the image features.

[0197] In this embodiment, the feature similarity between objects to be identified at different single sensor levels can be determined based on the ReID features of the image features and through the following steps 71 to 72.

[0198] Step 71: Employ Person Re-Identification (PRI) technology and, based on the image features of the object to be identified at each single sensor level, obtain the ReID features of the object to be identified at each single sensor level.

[0199] Re-identification technology can map all the image features of the object to be identified at the single sensor level into the same dimensional feature space. That is, the ReID features of the object to be identified at the single sensor level are features in the same dimensional feature space.

[0200] Step 72: Based on the ReID features, determine the fifth feature similarity between the objects to be identified at the single sensor level corresponding to different vehicle vision sensors.

[0201] A higher fifth feature similarity indicates a higher degree of similarity between objects to be identified at the single-sensor level; conversely, a lower fifth feature similarity indicates a lower degree of similarity. In this embodiment, conventional feature similarity calculation methods from the image processing technology field can be used to calculate the fifth feature similarity based on ReID features. For example, the Euclidean distance between the ReID features of different objects to be identified can be calculated, and then the Euclidean distance can be inverted to obtain the fifth feature similarity. The inversion operation is a method that makes the calculation result negatively correlated with the Euclidean distance; that is, the smaller the Euclidean distance, the larger the result obtained by inverting the Euclidean distance. For example, the inversion operation can calculate the reciprocal of the Euclidean distance.

[0202] By using the methods described in steps 71 to 72 above, the feature similarity between objects to be identified at different single sensor levels can be accurately determined based on the ReID features in the image features.

[0203] Furthermore, in some preferred embodiments, after determining the third, fourth, or fifth feature similarity between objects to be identified at different single-sensor levels using the above method, the third and / or fourth and / or fifth feature similarities can be weighted and calculated, and the calculation result can be used as the final feature similarity. Specifically, different weights can be set for the third, fourth, and fifth feature similarities, and then the third and / or fourth and / or fifth feature similarities can be weighted and calculated according to these weights. In some embodiments, the final feature similarity can be obtained by weighting and calculating according to the weights corresponding to the third, fourth, and fifth feature similarities using the following formula (5).

[0204] S final =k1×S1+k2×S2+k3×S3 (5)

[0205] The meanings of the parameters in formula (5) are as follows: S final The final feature similarity is represented by S1 and k1, which represent the third feature similarity and its weight, respectively; S2 and k2 represent the fourth feature similarity and its weight, respectively; and S3 and k3 represent the fifth feature similarity and its weight, respectively. It should be noted that those skilled in the art can flexibly set the specific values ​​of the weights k1, k2, and k3 according to actual needs. For example, in some preferred embodiments, k1 = k2 = k3 = 1.

[0206] Step S1022: Associate the objects to be identified at the single sensor level corresponding to different vehicle vision sensors based on feature similarity to form at least one object to be identified across sensor levels.

[0207] Since the magnitude of feature similarity can represent the degree of similarity between objects to be identified at the single sensor level, associating objects to be identified at the single sensor level based on feature similarity can more accurately link the same objects to be identified at the single sensor level, forming objects to be identified across the sensor level.

[0208] In some implementations, for each object to be identified at the single sensor level, the objects to be identified at the single sensor level corresponding to different vehicle vision sensors can be associated through the following steps 81 to 83 to form at least one object to be identified across sensor levels.

[0209] Step 81: Determine whether there are other single-sensor level objects to be identified that have a feature similarity greater than the preset third similarity threshold with the current single-sensor level object to be identified; if they exist, proceed to step 82; if they do not exist, proceed to step 83.

[0210] Those skilled in the art can flexibly set the specific value of the preset third similarity threshold according to actual needs, and the embodiments of the present invention do not impose specific limitations on this.

[0211] Furthermore, as can be seen from the aforementioned step S1021, different feature similarities can be determined using specific methods for different feature similarities between objects to be identified, such as third feature similarity, fourth feature similarity, fifth feature similarity, and final feature similarity. In this embodiment, one or more of the above feature similarities are used to perform step 81.

[0212] For example, when using one of the above feature similarity steps in step 81, step 81 can be executed based on the final feature similarity, that is, to determine whether there are other single-sensor level objects to be identified whose "final feature similarity" with the current single-sensor level object to be identified is greater than the preset third similarity threshold; if there are, proceed to step 82; if not, proceed to step 83.

[0213] For example, when using multiple execution steps 81 in the above feature similarity method, step 81 can be executed simultaneously based on the third feature similarity and the fourth feature similarity. Specifically, it determines whether there are other single-sensor level objects to be identified whose third feature similarity and fourth feature similarity are greater than their respective third similarity thresholds. If such objects exist, proceed to step 82; otherwise, proceed to step 83. The third similarity threshold corresponding to the third feature similarity can be the same as or different from the third similarity threshold corresponding to the fourth feature similarity.

[0214] Step 82: Associate the current single-sensor level object to be identified with the other single-sensor level objects to be identified to form a cross-sensor level object to be identified.

[0215] Step 83: Treat the object to be identified at the current single sensor level as a separate object to be identified across sensor levels.

[0216] Through steps 81 to 83 above, the single-sensor level objects to be identified corresponding to different vehicle vision sensors can be accurately associated based on the feature similarity between the objects to be identified at the single-sensor level, thus forming cross-sensor level objects to be identified.

[0217] The above is about Figure 1 Further explanation of step S102 in the illustrated embodiment will continue below. Figure 1 Step S103 in the illustrated embodiment will be further explained.

[0218] To accurately associate objects across sensor levels to form the final object to be identified, in some embodiments of step S103 above, the two-dimensional similarity and / or three-dimensional similarity between objects at different sensor levels can be calculated based on the two-dimensional and / or three-dimensional information of the objects in the image features of the objects at different sensor levels. The objects at different sensor levels are then associated based on the two-dimensional and / or three-dimensional similarity to form the final object to be identified. The two-dimensional information of the objects includes at least the two-dimensional coordinates of the objects to be identified, and the three-dimensional information of the objects includes at least the three-dimensional coordinates of the objects to be identified.

[0219] Specifically, in some implementations, the two-dimensional similarity between objects at different sensor levels can be calculated based on the two-dimensional information of the objects in the image features of each object across sensor levels. The objects at different sensor levels are then associated based on this two-dimensional similarity to form a final object to be identified. Specifically, if the two-dimensional similarity between two objects at different sensor levels is greater than a preset two-dimensional similarity threshold, these two objects are associated to form a single final object to be identified; otherwise, they are not associated, and each object is treated as a separate final object to be identified.

[0220] In other implementations, the 3D similarity between objects at different sensor levels can be calculated based on the 3D information of the objects in the image features of each object across sensor levels. The objects at different sensor levels are then associated based on this 3D similarity to form a final object to be identified. Specifically, if the 3D similarity between two objects at different sensor levels is greater than a preset 3D similarity threshold, these two objects are associated to form a single final object; otherwise, they are not associated, and each object is treated as a separate final object.

[0221] In other implementations, the two-dimensional and three-dimensional similarities between objects at different sensor levels can be calculated based on the two-dimensional and three-dimensional information of the object in the image features of each object across sensor levels. These similarities are then used to associate the objects across different sensor levels to form a final object to be identified. Specifically, if the two-dimensional similarity between two objects across sensor levels is greater than a preset two-dimensional similarity threshold and the three-dimensional similarity is greater than a preset three-dimensional similarity threshold, then these two objects are associated to form a single final object to be identified; otherwise, they are not associated, and each object is treated as a separate final object to be identified.

[0222] It should be noted that those skilled in the art can flexibly set the specific values ​​of the preset two-dimensional similarity threshold and the preset three-dimensional similarity threshold according to actual needs, and this embodiment does not impose specific limitations on this.

[0223] This concludes the further explanation of step S103 above.

[0224] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effects of the present invention, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders, and these variations are all within the scope of protection of the present invention.

[0225] Those skilled in the art will understand that all or part of the processes in the method of the above embodiment of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0226] Furthermore, the present invention also provides a computer device. In one embodiment of the computer device according to the present invention, the computer device includes a processor and a storage device. The storage device can be configured to store a program for executing the object association method of the above-described method embodiments, and the processor can be configured to execute the program in the storage device. The program includes, but is not limited to, the program for executing the object association method of the above-described method embodiments. For ease of explanation, only the parts related to the embodiments of the present invention are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. This computer device may be a control device device comprising various electronic devices.

[0227] Furthermore, the present invention also provides a computer-readable storage medium. In one embodiment of the computer-readable storage medium according to the present invention, the computer-readable storage medium can be configured to store a program that performs the object association method of the above-described method embodiments. This program can be loaded and run by a processor to implement the above-described object association method. For ease of explanation, only the parts related to the embodiments of the present invention are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. The computer-readable storage medium can be a storage device device comprising various electronic devices. Optionally, in the embodiments of the present invention, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0228] Furthermore, the present invention also provides a vehicle. In one embodiment of the vehicle according to the present invention, the vehicle may include the computer equipment described in the above-described computer equipment embodiments. In this embodiment, the vehicle may be an autonomous vehicle, an unmanned vehicle, or the like. Moreover, according to the type of power source, the vehicle in this embodiment may be a gasoline vehicle, an electric vehicle, a hybrid vehicle using a mixture of electric and gasoline power, or a vehicle using other new energy sources, etc.

[0229] The technical solution of the present invention has been described above with reference to one embodiment shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions resulting from such changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A method for associating objects, characterized in that, The method includes: Based on the image features of the object to be identified in a series of consecutive frames of images acquired by the same vehicle-mounted vision sensor, the object to be identified in the series of consecutive frames of images is associated to form at least one single-sensor level object to be identified corresponding to the same vehicle-mounted vision sensor. Based on the image features of each object to be identified at the single sensor level, the objects to be identified at the single sensor level corresponding to different vehicle vision sensors are associated to form at least one object to be identified across sensor levels. Based on the image features of each cross-sensor level object to be identified, the cross-sensor level objects to be identified are associated to form the final objects to be identified; in, The association of the objects to be identified in the consecutive multi-frame images includes: Based on the object position information in the image features, coordinate system transformation is performed on the object to be identified in the current frame and the object to be identified in the previous frame, so as to transform the object to be identified in the current frame and the object to be identified in the previous frame onto the BEV image with the current vehicle as the origin of the BEV coordinate system. Based on the coordinates of the object to be identified in the current frame and the object to be identified in the previous frame on the BEV image, calculate the object distance between the object to be identified in the current frame and the current vehicle, and the object distance between the object to be identified in the previous frame and the current vehicle. Based on the object distance, determine the first feature similarity between each object to be identified in the current frame and each object to be identified in the previous frame; The object to be identified in the current frame is associated with the object to be identified in the previous frame based on the first feature similarity, so as to form at least one object to be identified at the single sensor level.

2. The object association method of claim 1, wherein, The association of the objects to be identified in the consecutive multi-frame images further includes: Based on the object bounding boxes in the image features, calculate the intersection-union ratio (IUU) between the bounding box of each object to be identified in the current frame and the bounding box of each object to be identified in the previous frame. Based on the intersection-union ratio, the second feature similarity between each object to be identified in the current frame and each object to be identified in the previous frame is determined respectively; Based on the first feature similarity and the second feature similarity, the object to be identified in the current frame is associated with the object to be identified in the previous frame to form at least one object to be identified at the single sensor level.

3. The object association method of claim 2, wherein, The step of associating the object to be identified in the current frame with the object to be identified in the previous frame based on the first feature similarity and the second feature similarity includes associating each object to be identified in the current frame with each object to be identified in the previous frame in the following ways: Determine whether the first feature similarity between the object to be identified in the current frame and the object to be identified in the previous frame is greater than a preset first similarity threshold; If so, the object to be identified in the current frame is associated with the object to be identified in the previous frame to form a single sensor-level object to be identified. If not, then continue to determine whether the second feature similarity between the object to be identified in the current frame and the object to be identified in the previous frame is greater than the preset second similarity threshold. If the similarity is greater than the second similarity threshold, the object to be identified in the current frame is associated with the object to be identified in the previous frame to form a single sensor-level object to be identified; if the similarity is less than or equal to the second similarity threshold, the object to be identified in the current frame is not associated with the object to be identified in the previous frame and the object to be identified in the current frame is treated as a single sensor-level object to be identified. or, The step of associating the object to be identified in the current frame with the object to be identified in the previous frame based on the first feature similarity and the second feature similarity includes associating each object to be identified in the current frame with each object to be identified in the previous frame in the following ways: Determine whether the first feature similarity between the object to be identified in the current frame and the object to be identified in the previous frame is greater than a preset first similarity threshold or whether the second feature similarity between the object to be identified in the current frame and the object to be identified in the previous frame is greater than a preset second similarity threshold; If yes, then the object to be identified in the current frame is associated with the object to be identified in the previous frame to form a single sensor-level object to be identified; if no, then the object to be identified in the current frame is not associated with the object to be identified in the previous frame and the object to be identified in the current frame is treated as a single sensor-level object to be identified.

4. The object association method of claim 1, wherein, The step of "associating objects at the single-sensor level corresponding to different vehicle vision sensors based on the image features of each single-sensor level object to form at least one cross-sensor level object to be identified" specifically includes: Based on the image features of each object to be identified at the single sensor level, the feature similarity between the objects to be identified at the single sensor level corresponding to different vehicle vision sensors is determined. Based on the feature similarity, the objects to be identified at the single sensor level corresponding to different vehicle vision sensors are associated to form at least one object to be identified across the sensor level.

5. The object association method of claim 4, wherein, The step of "determining the feature similarity between objects to be identified at the single sensor level corresponding to different vehicle vision sensors based on the image features of each single sensor level object to be identified" specifically includes: Based on the object position information in the image features, coordinate system transformation is performed on each of the single-sensor level objects to be identified, so as to transform each of the single-sensor level objects to be identified onto the BEV image with the current vehicle as the origin of the BEV coordinate system. Based on the coordinates of each object to be identified at the single sensor level on the BEV image, calculate the object distance between each object to be identified at the single sensor level and the current vehicle. Based on the object distance, the third feature similarity between the objects to be identified at the single sensor level corresponding to different vehicle vision sensors is determined respectively. And / or, The step of "determining the feature similarity between objects to be identified at the single sensor level corresponding to different vehicle vision sensors based on the image features of each single sensor level object to be identified" specifically includes: Based on the object bounding boxes in the image features, calculate the intersection-over-union ratio of the bounding boxes of the objects to be identified at the single sensor level for different vehicle vision sensors; Based on the intersection-union ratio, the fourth feature similarity between the objects to be identified at the single sensor level corresponding to different vehicle vision sensors is determined respectively. And / or, The step of "determining the feature similarity between objects to be identified at the single sensor level corresponding to different vehicle vision sensors based on the image features of each single sensor level object to be identified" specifically includes: Re-identification technology is employed, and ReID features of each object to be identified at the single sensor level are obtained based on the image features of each object to be identified at the single sensor level. Based on the ReID features, the fifth feature similarity between the objects to be identified at the single sensor level corresponding to different vehicle vision sensors is determined respectively. And / or, The step of "associating the objects to be identified at the single-sensor level corresponding to different vehicle vision sensors based on the feature similarity to form at least one object to be identified across the sensor level" specifically includes: For each object to be identified at the single sensor level, determine whether there are other objects to be identified at the single sensor level with a feature similarity greater than a preset third similarity threshold. If so, the current single-sensor level object to be identified is associated with the other single-sensor level objects to be identified to form a cross-sensor level object to be identified; If not, the object to be identified at the current single-sensor level will be treated as a separate object to be identified at a cross-sensor level.

6. The object association method of claim 5, wherein, The step of "determining the feature similarity between the objects to be identified at the single sensor level corresponding to different vehicle vision sensors based on the image features of each single sensor level object to be identified" further includes: The third feature similarity and / or the fourth feature similarity and / or the fifth feature similarity are weighted and summed, and the result is used as the final feature similarity.

7. The object association method of claim 1, wherein, The step of "associating the objects to be identified across the sensor level based on the image features of each object across the sensor level to form the final objects to be identified" specifically includes: Based on the two-dimensional information of the object in the image features of each cross-sensor level object to be identified, the two-dimensional similarity between the objects to be identified at different cross-sensor levels is calculated respectively. The objects to be identified at different cross-sensor levels are associated based on the two-dimensional similarity to form the final objects to be identified. And / or, Based on the 3D information of the object in the image features of each cross-sensor level object to be identified, the 3D similarity between the objects to be identified at different cross-sensor levels is calculated respectively. The objects to be identified at different cross-sensor levels are associated based on the 3D similarity to form the final object to be identified.

8. A computer device comprising a processor and a storage device, said storage device being adapted to store a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by the processor to perform the object association method according to any one of claims 1 to 7.

9. A computer readable storage medium having stored therein a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by a processor to perform the object association method according to any one of claims 1 to 7.

10. A vehicle characterized by comprising: The vehicle includes the computer equipment as described in claim 8.

Images