An object recognition method, apparatus and storage medium

Abstract

Embodiments of the present application relate to the technical field of automatic driving, and provide an object identification method and device and a storage medium. The method comprises: acquiring an image set to be identified, the image set comprising a plurality of images arranged in time sequence; acquiring a candidate object set, the candidate object set comprising candidate objects identified in each image; restoring each candidate object to a geographical space respectively to obtain a restoration result of each candidate object; and determining whether each candidate object is a same target object according to the restoration result of each candidate object. The method can quickly and accurately track objects.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to an object recognition method, apparatus, and storage medium. Background Technology

[0002] Currently, in the field of autonomous driving, radar is used to provide information about objects captured during driving and to track those objects.

[0003] Because real-time images are two-dimensional and tracking during driving requires extremely high timeliness, location tracking based on real-time images cannot simultaneously satisfy the need for quick and accurate determination of whether objects identified in different images are the same object. Summary of the Invention

[0004] This application provides an object recognition method, apparatus, and storage medium that can quickly and accurately track objects.

[0005] In a first aspect, embodiments of this application provide an object recognition method from the perspective of an object recognition device, the method comprising:

[0006] Obtain a set of images to be identified, the set of images comprising multiple frames arranged in chronological order;

[0007] Obtain a candidate object set, which includes candidate objects identified in each frame of the image;

[0008] Each candidate object is restored to its geographic space, and the restoration result of each candidate object is obtained;

[0009] Determine whether each candidate object is the same target object based on the restoration results of each candidate object.

[0010] Secondly, embodiments of this application also provide an object recognition apparatus for implementing the object recognition method described in the first aspect, the object recognition apparatus comprising:

[0011] An input / output module is used to acquire a set of images to be identified, the set of images including multiple frames of images arranged in chronological order;

[0012] The processing module is used to obtain a set of candidate objects, which includes candidate objects identified in each frame of the image; restore each candidate object to geospatial data to obtain the restoration result of each candidate object; and determine whether each candidate object is the same target object based on the restoration result of each candidate object.

[0013] In some implementations, the candidate object set includes a first candidate object and a second candidate object, wherein the first candidate object and the second candidate object are temporally adjacent; the processing module is specifically used for:

[0014] Obtain the first detection box of the first candidate object and the second detection box of the second candidate object;

[0015] Obtain the target intersection and target union, wherein the target intersection is the intersection of the first detection box and the second detection box, and the target union is the union of the first detection box and the second detection box;

[0016] The target intersection-union ratio is obtained based on the target intersection and the target union.

[0017] The target intersection-union ratio is used to determine whether each candidate object is the same target object.

[0018] In some implementations, the image set includes a first image and a second image, the first image and the second image being temporally adjacent and each including multiple detection boxes; the processing module is specifically used for:

[0019] Construct an intersection-union matrix based on the multiple detection boxes included in the first image and the second image;

[0020] The first target matrix is ​​obtained based on the intersection-union ratio matrix;

[0021] Solve for the linear sum distribution of the first target matrix to obtain a first target set. The first target set includes multiple first matching results, and the first matching result is the matching degree between the first detection box and the second detection box.

[0022] The target crossover ratio is determined from the first target set, wherein the target crossover ratio is the crossover ratio with a matching degree greater than a first threshold.

[0023] In some implementations, the image set includes a first image and a second image, the first image and the second image being temporally adjacent and each including multiple detection boxes; the processing module is specifically used for:

[0024] Construct an intersection-union matrix based on the multiple detection boxes included in the first image and the second image;

[0025] The second target matrix is ​​obtained based on the intersection-union ratio matrix;

[0026] Solve for the linear sum distribution of the second target matrix to obtain the second target set. The second target set includes multiple second matching results. The second matching result is the target distance between the first candidate object and the second candidate object mapped to the same road surface.

[0027] The target crossover ratio is determined from the second target set, wherein the target crossover ratio is the crossover ratio where the target spacing is less than a second threshold.

[0028] In some implementations, the processing module is specifically used for:

[0029] Determine a set of target points, the set of target points including multiple target points, the target points being points in the target region of the image;

[0030] Obtain the distance between each target point and the geographic space;

[0031] Based on the distance between each target point and the geographic space, each target point is restored to its spatial location to obtain the restoration result. The restoration result includes the set of target pixels of each candidate object in the geographic space. The set of target pixels refers to the set of points of each pixel in each candidate object after being mapped to the spatial coordinate system.

[0032] In some embodiments, the processing module is further configured to:

[0033] Determine the object type of the target object in each frame image;

[0034] Point cloud data is acquired through the input / output module;

[0035] The location information of the target object within different time windows is determined based on the point cloud data, wherein one time window corresponds to at least one frame of image;

[0036] The matching result is determined based on the object type and the location information of the target object in different time windows. The matching result is the matching result of the target object in different frame images.

[0037] In some implementations, after determining the matching result based on the object type and the location information of the target object within different time windows, the processing module is further configured to:

[0038] If the matching result includes a third matching result, then the second position of the target object in the first image is determined according to the correspondence and the first position information; wherein, the third matching result is that there is no object in the first image that matches the object type and the first position information; the correspondence is the coordinate transformation relationship between the image coordinate system and the point cloud coordinate system;

[0039] The target object is supplemented at the second position in the first image.

[0040] Thirdly, embodiments of this application also provide a processing device, including a processor and a memory, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, it executes the steps of any of the object recognition methods provided in the first aspect of this application.

[0041] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the object recognition methods provided in the first aspect of this application.

[0042] Compared with existing technologies, in this embodiment, after acquiring the image set to be identified, a candidate object set is obtained from the image set. Since the candidate object set includes candidate objects identified in each frame of the image, the restoration results of each candidate object can be obtained after restoring each candidate object to the geographic space. Based on the restoration results of each candidate object, it can be determined whether each candidate object is the same target object. Therefore, this embodiment, based on planar images, identifies candidate objects from multiple consecutive frames of images and then restores them to the real geographic space. This allows the geographic locations of each candidate object within different time windows to be reproduced in the real geographic space, ultimately effectively identifying multiple candidate objects belonging to the same target object. Therefore, this solution can track objects quickly and accurately. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of a network architecture for the communication system in this application;

[0045] Figure 2 This is a flowchart illustrating one of the object recognition methods in this application;

[0046] Figure 3a This is a schematic diagram illustrating the tracking of multiple candidate objects in consecutive frames of images in this application;

[0047] Figure 3b This is a schematic diagram illustrating the tracking of multiple candidate objects in consecutive frames of images in this application;

[0048] Figure 3c This is a schematic diagram illustrating the tracking of multiple candidate objects in consecutive frames of images in this application;

[0049] Figure 4 This is a schematic diagram illustrating the relationship between the first and second detection frames in this application;

[0050] Figure 5 This is a schematic diagram of one structure of the object recognition device in this application;

[0051] Figure 6 This is a schematic diagram of the structure of a physical device that implements the object recognition method in this application;

[0052] Figure 7 This is a schematic diagram of the structure of a mobile phone for the object recognition method in this application. Detailed Implementation

[0053] The terms "target," "candidate," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. The division of modules in the embodiments of this application is merely a logical division; in actual applications, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not performed. Additionally, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interface, and the indirect coupling or communication connection between modules may be electrical or other similar forms; none of these are limited in the embodiments of this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed among multiple circuit modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of this application.

[0054] This application provides an object recognition method, apparatus, and storage medium, which can be used for image processing during image transmission. For example, it can be applied to videos recorded by dashcams, or to application scenarios such as detecting changes in objects to be identified. This application does not limit these applications.

[0055] In some implementations, this solution is applied to, for example... Figure 1 The communication system shown may include a server and multiple object identification devices. Figure 1Only one object identification device (a) is shown in the diagram. Each object identification device has a client installed. The clients installed on each object identification device can be the same or different versions, and can also be clients with the same or different operating systems. This application embodiment does not limit the version of the client or the type of operating system corresponding to the client. A detailed description follows.

[0056] Server-side: It can be used to provide image recognition, tracking and detection and other business services to various object recognition devices. For example, the server can recognize images uploaded from object recognition devices and then feed back the recognition results to the object recognition devices.

[0057] Object recognition device: It can acquire images or videos in real time, preprocess the images, and send the preprocessed images to the server in real time so that the server can recognize the preprocessed images.

[0058] In some implementations, based on Figure 1 The communication system shown describes a method where the object recognition device captures initial images or videos in real time via a camera, preprocesses the initial images, and then sends the preprocessed images to the server in real time. Upon receiving the preprocessed images from the object recognition device, the server can then perform recognition on the images uploaded by the object recognition device (e.g., obstacle recognition, road condition recognition, traffic sign recognition), and then feed the recognition results back to the object recognition device.

[0059] In some embodiments, when this solution is applied to an autonomous driving system, the autonomous driving system may include a server and at least one object recognition device. The object recognition device may be a server, an in-vehicle terminal, or a mobile terminal communicating with the in-vehicle terminal, or an application installed on the in-vehicle terminal or the mobile terminal; this application embodiment does not limit this. A detailed description follows. This application embodiment uses an in-vehicle terminal as an example.

[0060] In some implementations, the vehicle-mounted terminal acquires initial images or videos in real time via a camera, and acquires point cloud data synchronized in the time domain with the initial images or videos via radar. Then, based on the initial images or videos and the point cloud data, the identified target objects are annotated in three dimensions, such as for vehicle recognition, obstacle recognition, road condition recognition, and traffic sign recognition.

[0061] First, the relevant terms appearing in the embodiments of this application will be explained:

[0062] Point clouds can represent the spatial outline and specific location of objects. They allow us to see the shapes of streets and buildings, and the distance of objects from the camera is also known. Furthermore, point clouds are independent of viewpoint, allowing for arbitrary rotation and observation from different angles and directions. Different point clouds can be directly fused as long as they are within the same coordinate system. In 3D vision applications, such as those used in this embodiment to identify environmental features along a driving road, point clouds are primarily processed.

[0063] A point cloud is a collection of points that provides three-dimensional spatial data, unlike images which rely on perspective geometry to infer 3D data. A point cloud is a dataset of points in a coordinate system, where each point includes its 3D coordinates (X, Y, Z), color, classification value, intensity value, and time. Point clouds are generally classified into two types based on their composition: ordered point clouds and unordered point clouds.

[0064] Ordered point clouds: These are point clouds typically reconstructed from depth maps. They are arranged row by row in a matrix, from top left to bottom right, although some points may be invalid. Because of their sequential arrangement, it's easy to find neighboring points. Ordered point clouds are convenient in certain processing tasks, but they are often unavailable.

[0065] Unordered point clouds: Unordered point clouds are collections of points arranged in no particular order, and changing the order of the points has no effect. Unordered point clouds are a relatively common form of point cloud, while ordered point clouds can also be treated as unordered point clouds.

[0066] 1. Methods for obtaining point clouds:

[0067] Point clouds are obtained through 3D imaging sensors, such as binocular cameras, 3D scanners, and RGB-D cameras. Currently, mainstream RGB-D cameras include Microsoft's Kinect series, Intel's RealSense series, and structure sensors (which require an iPad). Point clouds can be created from scanned RGB-D images using the scanning camera's intrinsic parameters. This is achieved through camera calibration, using the camera's intrinsic parameters to calculate real-world points (x, y). Therefore, RGB-D images are grid-aligned images, while point clouds represent a sparser structure. Furthermore, methods for obtaining point clouds also include LiDAR laser detection and measurement, primarily through spaceborne, airborne, and ground-based methods.

[0068] 2. Contents of point clouds:

[0069] Point clouds obtained based on laser measurement principles include three-dimensional coordinates (XYZ) and laser reflection intensity. The intensity information is related to the target's surface material, roughness, incident angle, instrument emission energy, and laser wavelength. Point clouds obtained based on photogrammetry principles include three-dimensional coordinates (XYZ) and color information (RGB). Point clouds obtained by combining laser measurement and photogrammetry principles include three-dimensional coordinates (XYZ), laser reflection intensity, and color information (RGB).

[0070] 3. Point cloud properties:

[0071] The properties of point clouds include spatial resolution, point accuracy, surface normal vectors, etc.

[0072] Point cloud storage formats include: pts, LAS, PCD, .xyz, and .pcap, etc.

[0073] 4. There are multiple ways to represent 3D point clouds.

[0074] Different representations correspond to different processing methods. A relatively easy approach is to project it into a two-dimensional image or convert it into three-dimensional voxels, thus transforming the disordered spatial points into a regular data arrangement. Alternatively, the original points can be used as the representation without any transformation, which has the advantage of preserving as much original information as possible. Furthermore, point clouds, as a spatially disordered set of points, can be considered graph data in a universal sense. Point clouds also have another representation called a mesh, which can also be viewed as points with locally connected relationships, i.e., a graph. Treating point clouds as graph data allows for processing using the emerging graph convolution technique. It is worth mentioning that there is no clear boundary between the representation of original points and graph representation (in fact, there are some differences between original point clouds and meshes, but from the perspective of semantic understanding methods, this difference can be temporarily ignored, and a mesh can be seen as having an added connection relationship).

[0075] 5. 3D object detection can be divided into three categories: multi-view method, projection-based method, and point-based method.

[0076] Projection-based methods

[0077] (1) Point-based method

[0078] Most point-based methods attempt to minimize information loss during feature extraction and are currently the best performing group of methods.

[0079] Compared to the multi-view methods mentioned above, the segmentation-based method performs better for complex scenes and occluded objects.

[0080] 6. LiDAR and depth-sensing cameras:

[0081] In this embodiment of the application, distance measurement can be performed using lidar and depth-sensing cameras.

[0082] Among them, lidar is a miniature device that detects the distance to an object and records depth information. It is a radar system that uses laser beams to detect the position, speed and other characteristics of a target. It consists of a laser transmitter, an optical receiver, a turntable and an information processing system. It can obtain parameters such as the target's distance, azimuth, height, speed, attitude and even shape, so as to detect, track and identify it.

[0083] LiDAR mainly has the following two functions:

[0084] (1) Calculate the approximate distance between the object and the mobile phone.

[0085] Distance is measured by emitting a laser beam invisible to the naked eye.

[0086] (2) Record the depth information of objects within the projection range.

[0087] Depth-sensing camera:

[0088] Camera calibration is the process of transforming from the world coordinate system to the image coordinate system, which is essentially the process of calculating the final projection matrix PP. The camera calibration process consists of two parts:

[0089] Step 1: Transform from the world coordinate system to the camera coordinate system. This step is a transformation from 3D point to 3D point, including parameters such as RR and tt (camera extrinsic parameters);

[0090] The second step is to transform from the camera coordinate system to the image coordinate system. This step is the transformation from three-dimensional points to two-dimensional points, including parameters such as KK (camera intrinsics).

[0091] Camera intrinsics: Determine the projection relationship of the camera from three-dimensional space to two-dimensional images.

[0092] Camera extrinsic parameters: determine the relative positional relationship between the camera coordinates and the world coordinate system.

[0093] It should be specifically noted that the server involved in this application embodiment can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The object recognition device involved in this application embodiment can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, personal digital assistant, etc., but is not limited to these. The object recognition device and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application embodiment.

[0094] The object recognition method provided in the embodiments of this application will now be described.

[0095] See Figure 2 , Figure 2 A flowchart of an object recognition method according to an embodiment of this application is shown. The following describes an object recognition method provided by an embodiment of this application, which includes:

[0096] 101. Obtain the set of images to be identified.

[0097] The image set includes multiple frames of images arranged in chronological order.

[0098] In some implementations, the image set can be acquired at a preset frame rate to maintain the continuity of the position of the same candidate object in each frame. The preset frame rate should not be slower than 10fps to avoid the same object being too far apart in the positions of consecutive frames, thus making it impossible to associate them.

[0099] 102. Obtain the set of candidate objects.

[0100] The candidate object set includes candidate objects identified in each frame of the image. A single frame may contain at least one candidate object. Each frame may contain the same or different candidate objects. Therefore, the candidate object set includes a first candidate object and a second candidate object, which are temporally adjacent.

[0101] In this embodiment, the aim is to determine whether a first candidate object and a second candidate object in different frame images correspond to the same target object, i.e., the tracked target, through tracking. This embodiment uses a first candidate object and a second candidate object detected by a detection box in different frame images as an example, and the first candidate object and the second candidate object can be temporally adjacent.

[0102] For example, other objects such as vehicle 1, vehicle 2, vehicle 3, and vehicle 4 are detected in the third frame image, and also in the fourth frame image. The third and fourth frames are temporally adjacent. If vehicle 1 is selected as the target object to be tracked, then vehicle 1 detected in the third frame image is the first candidate object, and vehicle 1 detected in the fourth frame image is the second candidate object.

[0103] In some implementations, for example Figures 3a-3c The image shows the tracking of detected objects across three consecutive frames. Specifically, the pixel coordinates of the detected vehicle bounding boxes are as follows:

[0104] (Arranged according to x, y, width, height)

[0105] like Figure 3a As shown, there are 4 car types: [(186., 114., 71., 60.), (93.0, 95.0, 74.0, 69.0), (175.,118., 22., 22.), (283., 106., 52., 45.)].

[0106] like Figure 3b As shown, there are 4 car types: [(185., 114., 71., 63.), (97., 98., 72., 67.), (176.,120., 19., 22.), (286., 107., 54., 47.)].

[0107] like Figure 3c As shown: [(189., 114., 73., 63.), (107., 98., 69., 65.), (182.,120., 17., 22.), (292., 107., 57., 48.), (270., 120., 13., 18.)] There are a total of 5 car types.

[0108] 103. Restore each candidate object to its geographic space to obtain the restoration result of each candidate object.

[0109] The restoration result is obtained by identifying candidate objects through multiple consecutive frames, and then restoring the identified candidate objects to geospatial representation. Based on the restoration results from multiple frames to geospatial representation, it is determined whether they are the same target object (e.g., vehicle, pedestrian, or other object), ultimately achieving the tracking of the target object.

[0110] In some implementations, restoring each candidate object to geospatial space means restoring each target point to its spatial location based on the distance between each target point and the geospatial space, obtaining the restoration result. The restoration result includes a set of target pixels for each candidate object in geospatial space, where the set of target pixels refers to the set of points in each candidate object mapped to a spatial coordinate system. After obtaining the set of target pixels, the spatial positions of each point in the initial point cloud data can be mapped to the target object bounding boxes, i.e., mapping the spatial positions of each point in the initial point cloud data to the target object bounding boxes. The target object bounding boxes are annotation boxes used to label the candidate objects.

[0111] Once the target object in the first image is identified, object recognition can begin.

[0112] 104. Determine whether each candidate object is the same target object based on the restoration results of each candidate object.

[0113] In this embodiment of the application, the following steps (1)-(4) can be used to determine whether each candidate object is the same target object:

[0114] (1) Obtain the first detection box of the first candidate object and the second detection box of the second candidate object.

[0115] Understandably, the first detection box is used to detect the first candidate object in the first image, and the second detection box is used to detect the second candidate object in the second image. For example... Figure 4 As shown.

[0116] (2) Obtain the intersection and union of the targets.

[0117] Wherein, the target intersection is the intersection of the first detection box and the second detection box, and the target union is the union of the first detection box and the second detection box.

[0118] For example Figures 3a-3c The image shows the tracking of detected objects across three consecutive frames. Each bounding box detects one object (taking a car as an example), and the ID of the same car is the same in consecutive frames.

[0119] like Figure 4 As shown, Figure 4 (a) in the figure represents the intersection of the two frames. Figure 4 (b) in the figure represents the union of the two frames.

[0120] (3) Obtain the target intersection-union ratio based on the target intersection and the target union.

[0121] In some implementations, the intersection-union ratio (IOU) of two rectangles is calculated as follows:

[0122]

[0123] In this embodiment of the application, the following two methods can be used to determine whether each candidate object is the same target object:

[0124] Method 1:

[0125] In some implementations, the image set includes a first image and a second image, the first image and the second image being temporally adjacent and each including multiple detection boxes; obtaining the target intersection-union ratio based on the target intersection and the target union includes:

[0126] Construct an intersection-union matrix based on the multiple detection boxes included in the first image and the second image;

[0127] The first target matrix is ​​obtained based on the intersection-union ratio matrix;

[0128] Solve for the linear sum distribution of the first target matrix to obtain a first target set. The first target set includes multiple first matching results, and the first matching result is the matching degree between the first detection box and the second detection box.

[0129] The target crossover ratio is determined from the first target set, wherein the target crossover ratio is the crossover ratio with a matching degree greater than a first threshold.

[0130] By using this matrix, we can solve the linear sum distribution problem and obtain the matching degree corresponding to Ax-By.

[0131] The results need further filtering; for example, only matches with an IoU greater than 0.5 are considered valid. Applying the above process to multiple frames of images consecutively can achieve tracking of detected objects, which will not be elaborated further.

[0132] The specific steps for obtaining the association results of the same object in two images through intersection-union comparison are as follows:

[0133] Image A contains the detected bounding box Ax, for example, Ax includes three bounding boxes A1, A2, and A3; image B contains the detected bounding box By, for example, By includes four bounding boxes (B1, B2, B3, and B4). An intersection-over-union (IoU) matrix is ​​constructed based on Ax and By. One way to express the IoU matrix is ​​as follows:

[0134] Formula 1

[0135] The first target matrix can be obtained from the intersection-union matrix:

[0136] cost_matrix = 1- Formula 2

[0137] Method 2:

[0138] The step of obtaining the target intersection-union ratio based on the target intersection and the target union includes:

[0139] Construct an intersection-union matrix based on the multiple detection boxes included in the first image and the second image;

[0140] The second target matrix is ​​obtained based on the intersection-union ratio matrix;

[0141] Solve for the linear sum distribution of the second target matrix to obtain the second target set. The second target set includes multiple second matching results. The second matching result is the target distance between the first candidate object and the second candidate object mapped to the same road surface.

[0142] The target crossover ratio is determined from the first target set, wherein the target crossover ratio is the crossover ratio where the target spacing is less than a second threshold.

[0143] (4) Determine whether each candidate object is the same target object based on the target intersection-union ratio.

[0144] In this embodiment of the application, when determining the target object, it can be based on a certain intersection-union ratio, or it can be the point closest to the bottom of the image projected onto the geographic space, distinguished according to distance, depending on the effect. This embodiment of the application does not limit the specific implementation method.

[0145] Using projection requires a prerequisite: that all detected objects are on the road surface, and that the camera's extrinsic parameters be pre-calibrated, including the camera's installation height and its rotation matrix with respect to space. Using the following formula (Formula 3) and the calibrated camera height, the position (real coordinates) of a point on the road surface relative to the vehicle's front / back and left / right sides can be calculated.

[0146] Formula 3

[0147] Simultaneously, by replacing the specific elements in the cost_matrix of the first target matrix with the distances between objects in the two frames, and similarly solving the linear sum distribution problem, the distance-based matching can be obtained. The matching results also need to be filtered; for example, a distance less than 1m is considered a valid filter.

[0148] by Figures 3a-3c For example, we can obtain Figure 3a and Figure 3b The iou_matrix elements of the detection results are shown in Table 1 below:

[0149] Table 1

[0150]

[0151] Then, based on cost_matrix = 1 - iou_matrix, solve the linear sum distribution problem to obtain the matching result: ( Figure 3b , Figure 3a )

[0152] (1,1),(2,2),(3,3),(4,4)

[0153] Similarly, Figure 3b and Figure 3c The iou_matrix is ​​shown in Table 2 below:

[0154] Table 2

[0155]

[0156] Then, based on cost_matrix = 1 - iou_matrix, solve the linear sum distribution problem to obtain the matching result: ( Figure 3c , Figure 3b )

[0157] (1,1),(2,2),(3,3),(4,4),(5,-1)

[0158] In some embodiments of this application, the method further includes:

[0159] Determine the object type of the target object in each frame image;

[0160] Acquire point cloud data;

[0161] The location information of the target object within different time windows is determined based on the point cloud data, wherein one time window corresponds to at least one frame of image;

[0162] The matching result is determined based on the object type and the location information of the target object in different time windows. The matching result is the matching result of the target object in different frame images.

[0163] The location information obtained by the LiDAR and the type information obtained by the camera are matched and output together.

[0164] In some implementations, for areas where image information is discontinuous (e.g., within a single frame), LiDAR is used to supplement the positional information, thereby maintaining positional continuity. Specifically, after determining the matching result based on the object type and the target object's positional information within different time windows, the method further includes:

[0165] If the matching result includes a third matching result, then the second position of the target object in the first image is determined according to the correspondence and the first position information; wherein, the third matching result is that there is no object in the first image that matches the object type and the first position information; the correspondence is the coordinate transformation relationship between the image coordinate system and the point cloud coordinate system;

[0166] The target object is supplemented at the second position in the first image.

[0167] It can be seen that by supplementing the target object at the second position in the first image, the continuity between the positions of candidate objects belonging to the same target object can be maintained.

[0168] In this embodiment, after obtaining the set of images to be identified, a set of candidate objects is obtained from the set of images. Since the set of candidate objects includes candidate objects identified in each frame of the image, the restoration results of each candidate object can be obtained by restoring each candidate object to the geographic space. Based on the restoration results of each candidate object, it can be determined whether each candidate object is the same target object. Therefore, this embodiment, based on planar images, identifies candidate objects from multiple consecutive frames of images and then restores them to the real geographic space. This allows the geographic locations of each candidate object within different time windows to be reproduced in the real geographic space, ultimately effectively identifying multiple candidate objects belonging to the same target object. Therefore, this solution can track objects quickly and accurately.

[0169] Figures 1 to 5 Any technical feature mentioned in the embodiments corresponding to any one of the above also applies to the embodiments of this application. Figures 4 to 7 The corresponding implementation examples will not be repeated hereafter.

[0170] The above describes an object recognition method in the embodiments of this application. The object recognition device and server that execute the above object recognition method are described below.

[0171] See Figure 4 ,like Figure 4 The diagram shows the structure of an object recognition device 40, which can be used to preprocess images to be transmitted, such as performing dimensionality reduction processing on images captured by a dashcam. The object recognition device 40 in this embodiment can achieve the corresponding... Figure 1- The steps of the object recognition method executed by the object recognition device in any of the embodiments corresponding to Figure 3. The function implemented by device 40 can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware. The device 40 may include an input / output module 401 and a processing module 402. The functional implementation of the input / output module 401 and the processing module 402 can be found in [reference]. Figures 1-7 The operations performed in any of the corresponding embodiments will not be described in detail here.

[0172] In some implementations, the candidate object set includes a first candidate object and a second candidate object, wherein the first candidate object and the second candidate object are temporally adjacent; the processing module 402 is specifically used for:

[0173] Obtain the first detection box of the first candidate object and the second detection box of the second candidate object;

[0174] Obtain the target intersection and target union, wherein the target intersection is the intersection of the first detection box and the second detection box, and the target union is the union of the first detection box and the second detection box;

[0175] The target intersection-union ratio is obtained based on the target intersection and the target union.

[0176] The target intersection-union ratio is used to determine whether each candidate object is the same target object.

[0177] In some embodiments, the image set includes a first image and a second image, the first image and the second image being temporally adjacent and each including multiple detection boxes; the processing module 402 is specifically used for:

[0178] Construct an intersection-union matrix based on the multiple detection boxes included in the first image and the second image;

[0179] The first target matrix is ​​obtained based on the intersection-union ratio matrix;

[0180] Solve for the linear sum distribution of the first target matrix to obtain a first target set. The first target set includes multiple first matching results, and the first matching result is the matching degree between the first detection box and the second detection box.

[0181] The target crossover ratio is determined from the first target set, wherein the target crossover ratio is the crossover ratio with a matching degree greater than a first threshold.

[0182] In some embodiments, the image set includes a first image and a second image, the first image and the second image being temporally adjacent and each including multiple detection boxes; the processing module 402 is specifically used for:

[0183] Construct an intersection-union matrix based on the multiple detection boxes included in the first image and the second image;

[0184] The second target matrix is ​​obtained based on the intersection-union ratio matrix;

[0185] Solve for the linear sum distribution of the second target matrix to obtain the second target set. The second target set includes multiple second matching results. The second matching result is the target distance between the first candidate object and the second candidate object mapped to the same road surface.

[0186] The target crossover ratio is determined from the second target set, wherein the target crossover ratio is the crossover ratio where the target spacing is less than a second threshold.

[0187] In some embodiments, the processing module 402 is specifically used for:

[0188] Determine a set of target points, the set of target points including multiple target points, the target points being points in the target region of the image;

[0189] Obtain the distance between each target point and the geographic space;

[0190] Based on the distance between each target point and the geographic space, each target point is restored to its spatial location to obtain the restoration result. The restoration result includes the set of target pixels of each candidate object in the geographic space. The set of target pixels refers to the set of points of each pixel in each candidate object after being mapped to the spatial coordinate system.

[0191] In some embodiments, the processing module 402 is further configured to:

[0192] Determine the object type of the target object in each frame image;

[0193] Point cloud data is acquired through the input / output module 401;

[0194] The location information of the target object within different time windows is determined based on the point cloud data, wherein one time window corresponds to at least one frame of image;

[0195] The matching result is determined based on the object type and the location information of the target object in different time windows. The matching result is the matching result of the target object in different frame images.

[0196] In some implementations, after determining the matching result based on the object type and the location information of the target object within different time windows, the processing module 402 is further configured to:

[0197] If the matching result includes a third matching result, then the second position of the target object in the first image is determined according to the correspondence and the first position information; wherein, the third matching result is that there is no object in the first image that matches the object type and the first position information; the correspondence is the coordinate transformation relationship between the image coordinate system and the point cloud coordinate system;

[0198] The target object is supplemented at the second position in the first image.

[0199] In this embodiment, after the input / output module 401 acquires the image set to be identified, the processing module 402 obtains a candidate object set from the image set. Since the candidate object set includes candidate objects identified in each frame of the image, after restoring each candidate object to the geographic space, the restoration result of each candidate object can be obtained. Based on the restoration result of each candidate object, it can be determined whether each candidate object is the same target object. Therefore, this embodiment, based on planar images, identifies candidate objects from multiple consecutive frames of images and then restores them to the real geographic space. This allows the geographic locations of each candidate object within different time windows to be reproduced in the real geographic space, ultimately effectively identifying multiple candidate objects belonging to the same target object. Therefore, this solution can track objects quickly and accurately.

[0200] The object identification device 40 for executing the object identification method in this application embodiment has been described above from the perspective of modular functional entities. The object identification device for executing the object identification method in this application embodiment will now be described below from the perspective of hardware processing. It should be noted that in this application embodiment... Figure 4 In the illustrated embodiment, the physical device corresponding to the input / output module 401 can be an input / output unit, transceiver, radio frequency circuit, communication module, and output interface, etc., and the physical device corresponding to the processing module 402 can be a processor. Figure 4 The object recognition device 40 shown can have, for example, Figure 6 The structure shown, when Figure 4 The object recognition device 40 shown has, for example, Figure 6 When the structure shown is used, Figure 6 The processor and transceiver in the device can perform the same or similar functions as the input / output module 401 and processing module 402 provided in the aforementioned embodiment of the object recognition device 40. Figure 6 The memory storage processor in the memory needs to call the computer program when executing the above object identification method.

[0201] This application also provides another object recognition device, such as... Figure 7 As shown, for ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. The object recognition device can be any object recognition device, including mobile phones, tablets, personal digital assistants (PDAs), point-of-sale (POS) devices, in-vehicle computers, etc. Taking a mobile phone as an example:

[0202] Figure 7 This is a block diagram showing a portion of the structure of a mobile phone related to the object recognition device provided in the embodiments of this application. (Reference) Figure 7 The mobile phone includes: a radio frequency (RF) circuit 710, a memory 720, an input unit 730, a display unit 740, a sensor 780, an audio circuit 760, a wireless-fidelity (Wi-Fi) module 7100, a processor 780, and a power supply 790, among other components. Those skilled in the art will understand that... Figure 7 The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0203] The following is combined Figure 7 A detailed introduction to each component of a mobile phone:

[0204] The RF circuit 710 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with the processor 780; additionally, it transmits uplink data to the base station. Typically, the RF circuit 710 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, the RF circuit 710 can also communicate wirelessly with networks and other devices. The aforementioned wireless communications may use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Messaging Service (SMS).

[0205] The memory 720 can be used to store software programs and modules. The processor 780 executes various mobile phone functions and data processing by running the software programs and modules stored in the memory 720. The memory 720 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 720 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0206] The input unit 730 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 730 may include a touch panel 731 and other input devices 732. The touch panel 731, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 731), and drive the corresponding connected devices according to a pre-set program. Optionally, the touch panel 731 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 780, and can also receive and execute commands sent by the processor 780. In addition, the touch panel 731 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 731, the input unit 730 may also include other input devices 732. Specifically, other input devices 732 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0207] The display unit 740 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 740 may include a display panel 741, which may optionally be configured as a Liquid Crystal Display (LCD), Organic Light-Emitting Diode (OLED), or similar display panel 741. Further, a touch panel 731 may cover the display panel 741. When the touch panel 731 detects a touch operation on or near it, it transmits the information to the processor 780 to determine the type of touch event. Subsequently, the processor 780 provides corresponding visual output on the display panel 741 based on the type of touch event. Although in Figure 7 In this embodiment, the touch panel 731 and the display panel 741 are two separate components to realize the input and output functions of the mobile phone. However, in some embodiments, the touch panel 731 and the display panel 741 can be integrated to realize the input and output functions of the mobile phone.

[0208] The mobile phone may also include at least one sensor 780, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 741 according to the ambient light level, and the proximity sensor can turn off the display panel 741 and / or the backlight when the phone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, taps), etc. Other sensors that may be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0209] Audio circuit 760, speaker 761, and microphone 762 provide an audio interface between the user and the mobile phone. Audio circuit 760 converts received audio data into electrical signals and transmits them to speaker 761, where speaker 761 converts them into sound signals for output. On the other hand, microphone 762 converts collected sound signals into electrical signals, which are received by audio circuit 760, converted into audio data, and then processed by processor 780 before being transmitted via RF circuit 710 to, for example, another mobile phone, or the audio data can be output to memory 720 for further processing.

[0210] Wi-Fi is a short-range wireless transmission technology. Through the Wi-Fi module 7100, mobile phones can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 7 The Wi-Fi module 7100 is shown, but it is understood that it is not a necessary component of the mobile phone and can be omitted as needed without changing the nature of the application.

[0211] The processor 780 is the control center of the mobile phone, connecting various parts of the phone through various interfaces and lines. It executes software programs and / or modules stored in the memory 720, and calls data stored in the memory 720 to perform various functions and process data, thereby providing overall monitoring of the phone. Optionally, the processor 780 may include one or more processing units; preferably, the processor 780 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 780.

[0212] The mobile phone also includes a power supply 790 (such as a battery) that supplies power to various components. The power supply can be logically connected to the processor 780 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0213] Although not shown, mobile phones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0214] In this embodiment of the application, the processor 780 included in the mobile phone also has the function of controlling and executing the above-mentioned... Figure 7 The method flow executed by the object recognition device 40 shown is illustrated. The steps performed by the object recognition device in the above embodiments can be based on this... Figure 7 The mobile phone structure is shown. For example, the processor 722 performs the following operations by calling instructions from memory 732:

[0215] The input unit 730 acquires a set of images to be identified, the set of images including multiple frames of images arranged in chronological order;

[0216] The processing module is used to acquire a candidate object set, which includes candidate objects identified in each frame of the image; to restore each candidate object to its geographic space, obtaining the restoration result for each candidate object; and to determine whether each candidate object is the same target object based on the restoration result.

[0217] The restoration result can also be output via RF circuit 710.

[0218] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0219] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0220] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, apparatuses, or modules, and may be electrical, mechanical, or other forms.

[0221] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0222] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0223] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0224] The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0225] The technical solutions provided in the embodiments of this application have been described in detail above. Specific examples have been used in the embodiments of this application to illustrate the principles and implementation methods of the embodiments of this application. The description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of the embodiments of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the embodiments of this application. Therefore, the content of this specification should not be construed as a limitation on the embodiments of this application.

Claims

1. An object recognition method, characterized in that, The method includes: A set of images to be identified is obtained, the set of images including multiple frames of images arranged in chronological order; the set of images includes a first image and a second image, the first image and the second image being temporally adjacent and each including multiple detection boxes; A candidate object set is obtained, which includes candidate objects identified in each frame image; the candidate object set includes a first candidate object and a second candidate object, which are temporally adjacent; Each candidate object is restored to its geographic space to obtain the restoration result of each candidate object; wherein, the process includes: determining a set of target points, the set of target points including multiple target points, the target points being points in the target region of the image; obtaining the distance between each target point and the geographic space; and restoring each target point to its corresponding position in the geographic space based on the distance between each target point and the geographic space to obtain the restoration result, the restoration result including: a set of target pixel points of each candidate object in the geographic space; the set of target pixel points refers to the set of points of each pixel point in each candidate object after being mapped to a spatial coordinate system; Determine whether each candidate object is the same target object based on the restoration results of each candidate object; wherein the process includes: obtaining a first detection box of the first candidate object and a second detection box of the second candidate object; obtaining a target intersection and a target union, wherein the target intersection is the intersection of the first detection box and the second detection box, and the target union is the union of the first detection box and the second detection box; obtaining a target intersection-union ratio based on the target intersection and the target union; and determining whether each candidate object is the same target object based on the target intersection-union ratio. The step of obtaining the target intersection-union ratio (CIU) based on the target intersection and the target union includes: constructing a CIU matrix based on multiple detection boxes included in both the first image and the second image; obtaining a second target matrix based on the CIU matrix; solving for the linear sum distribution of the second target matrix to obtain a second target set, the second target set including multiple second matching results, the second matching results being the target distance between the first candidate object and the second candidate object mapped to the same road surface; and determining the target CIU from the second target set, the target CIU being the CIU where the target distance is less than a second threshold.

2. The method according to claim 1, characterized in that, The step of obtaining the target intersection-union ratio based on the target intersection and the target union includes: Construct an intersection-union matrix based on the multiple detection boxes included in the first image and the second image; The first target matrix is ​​obtained based on the intersection-union ratio matrix; Solve for the linear sum distribution of the first target matrix to obtain a first target set. The first target set includes multiple first matching results, and the first matching result is the matching degree between the first detection box and the second detection box. The target crossover ratio is determined from the first target set, wherein the target crossover ratio is the crossover ratio with a matching degree greater than a first threshold.

3. The method according to claim 1, characterized in that, The method further includes: Determine the object type of the target object in each frame image; Acquire point cloud data; The location information of the target object within different time windows is determined based on the point cloud data, wherein one time window corresponds to at least one frame of image; The matching result is determined based on the object type and the location information of the target object in different time windows. The matching result is the matching result of the target object in different frame images.

4. The method according to claim 3, characterized in that, After determining the matching result based on the object type and the location information of the target object within different time windows, the method further includes: If the matching result includes a third matching result, then the second position of the target object in the first image is determined according to the correspondence and the first position information; wherein, the third matching result is that there is no object in the first image that matches the object type and the first position information; the correspondence is the coordinate transformation relationship between the image coordinate system and the point cloud coordinate system; The target object is supplemented at the second position in the first image.

5. An object recognition device, characterized in that, The object recognition device includes: An input / output module is used to acquire a set of images to be identified, the set of images including multiple frames of images arranged in time sequence; the set of images includes a first image and a second image, the first image and the second image being temporally adjacent and each including multiple detection boxes; The processing module is used to obtain a candidate object set, which includes candidate objects identified in each frame image; the candidate object set includes a first candidate object and a second candidate object, which are temporally adjacent. Each candidate object is restored to its geographic location to obtain a restoration result for each candidate object. This process includes: determining a set of target points, which includes multiple target points, each target point being a point within a target region of an image; obtaining the distance between each target point and the geographic space; restoring each target point to its spatial location based on the distance between the target point and the geographic space, obtaining the restoration result, which includes a set of target pixels for each candidate object in the geographic space, the target pixel set referring to the set of points mapped to a spatial coordinate system for each pixel in each candidate object; determining whether each candidate object is the same target object based on the restoration result of each candidate object; this process includes: obtaining a first detection box for the first candidate object and a second detection box for the second candidate object; obtaining a target intersection and a target union, where the target intersection is the intersection of the first detection box and the second detection box, and the target union is the union of the first detection box and the second detection box; obtaining a target intersection-union ratio (DIR) based on the target intersection and the target union; and determining whether each candidate object is the same target object based on the DIR. The step of obtaining the target intersection-union ratio (CIU) based on the target intersection and the target union includes: constructing a CIU matrix based on multiple detection boxes included in both the first image and the second image; obtaining a second target matrix based on the CIU matrix; solving for the linear sum distribution of the second target matrix to obtain a second target set, the second target set including multiple second matching results, the second matching results being the target distance between the first candidate object and the second candidate object mapped to the same road surface; and determining the target CIU from the second target set, the target CIU being the CIU where the target distance is less than a second threshold.

6. A processing apparatus, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the method as described in any one of claims 1 to 4 when it invokes the computer program in the memory.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the method of any one of claims 1 to 4.

Images