Object recognition method, device, movable platform and storage medium

Abstract

An object recognition method, device, mobile platform and storage medium, the method comprising: obtaining point cloud data and image data of an object to be recognized, and obtaining a first fusion feature according to the point cloud data and the image data; and obtaining a first image feature according to the image data (S101); obtaining a first size coefficient according to the first fusion feature; and obtaining a second size coefficient according to the first image feature; the first size coefficient and the second size coefficient represent the probability that the object to be recognized belongs to an object within a target size range (S102); obtaining a second fusion feature according to the first fusion feature and the first size coefficient (S103); and obtaining a second image feature according to the first image feature and the second size coefficient; and recognizing the object to be recognized according to the second fusion feature and the second image feature (S104). This embodiment realizes effective recognition of objects within the target size range.

Description

Technical Field

[0001] The present application relates to the technical field of object recognition, and in particular to an object recognition method, device, movable platform and storage medium. Background Art

[0002] With the development of technology, mobile platforms such as unmanned vehicles and unmanned aerial vehicles have gradually developed. Among them, during the movement of the mobile platform, it is necessary to sense the environment around the mobile platform and obtain information about objects in the environment to control the mobile platform to work safely and reliably. For example, obstacle avoidance is an issue that mobile platforms such as unmanned vehicles and unmanned aerial vehicles need to focus on. The key to achieving obstacle avoidance lies in the accurate identification of objects around the mobile platform. Therefore, how to accurately identify the information of objects around the mobile platform has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] In view of this, one of the objectives of the present application is to provide an object recognition method, device, movable platform and storage medium.

[0004] In a first aspect, an embodiment of the present application provides an object recognition method, comprising:

[0005] Acquire point cloud data and image data of the object to be identified, and acquire a first fusion feature according to the point cloud data and the image data; and acquire a first image feature according to the image data;

[0006] Acquire a first size coefficient according to the first fusion feature; and acquire a second size coefficient according to the first image feature; the first size coefficient and the second size coefficient represent the probability that the object to be identified belongs to an object within a target size range;

[0007] Acquire a second fusion feature according to the first fusion feature and the first size coefficient; and acquire a second image feature according to the first image feature and the second size coefficient;

[0008] The object to be identified is identified according to the second fusion feature and the second image feature.

[0009] In a second aspect, an embodiment of the present application provides an object recognition device, including a processor and a memory storing a computer program;

[0010] When executing the computer program, the processor implements the following steps:

[0011] Acquire point cloud data and image data of the object to be identified, and acquire a first fusion feature according to the point cloud data and the image data; and acquire a first image feature according to the image data;

[0012] Acquire a first size coefficient according to the first fusion feature; and acquire a second size coefficient according to the first image feature; the first size coefficient and the second size coefficient represent the probability that the object to be identified belongs to an object within a target size range;

[0013] Acquire a second fusion feature according to the first fusion feature and the first size coefficient; and acquire a second image feature according to the first image feature and the second size coefficient;

[0014] The object to be identified is identified according to the second fusion feature and the second image feature.

[0015] In a third aspect, an embodiment of the present application provides a movable platform, including:

[0016] Body;

[0017] a power system, installed in the body, for providing power to the movable platform; and

[0018] An object recognition device as described in the first aspect.

[0019] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0020] An object recognition method, device, movable platform and storage medium provided in an embodiment of the present application, after acquiring point cloud data and image data of an object to be recognized, obtain a first fusion feature according to the point cloud data and the image data, and obtain a first image feature according to the image data; then obtain a first size coefficient according to the first fusion feature, and obtain a second size coefficient according to the first image feature, the first size coefficient and the second size coefficient representing the probability that the object to be recognized belongs to an object within a target size range; then obtain a second fusion feature according to the first fusion feature and the first size coefficient, and obtain a second image feature according to the first image feature and the second size coefficient; finally, recognize the object to be recognized according to the second fusion feature and the second image feature. In this embodiment, the first size coefficient and the second size coefficient are determined to increase the attention to objects in the target size range, and then the first size coefficient and the second size coefficient are used to increase the information about objects in the target size range in the second fusion feature and the second image feature, so that the second fusion feature and the second image feature contain more semantic information about objects in the target size range, so that small objects can be accurately identified based on the second fusion feature and the second image feature, thereby improving the recognition accuracy of objects; further, this embodiment combines point cloud data and image data to perform object recognition, point cloud data can provide spatial information to assist image data, and image data can provide color information to assist point cloud data, and the two types of data complement each other, which is conducive to further improving the accuracy of object recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0022] Figure 1A It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0023] Figure 1B is a schematic structural diagram of an unmanned vehicle provided by an embodiment of the present application;

[0024] Figure 2 is a flowchart of an object recognition method provided by an embodiment of the present application;

[0025] Figure 3A as well as Figure 3B is a schematic diagram of different structures of an object recognition model provided by an embodiment of the present application;

[0026] Figure 4 is a structural schematic diagram of an object recognition device provided by an embodiment of the present application;

[0027] Figure 5 It is a structural schematic diagram of a movable platform provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] With the development of technology, mobile platforms such as unmanned vehicles and unmanned aerial vehicles have gradually developed. Among them, during the movement of the mobile platform, it is necessary to sense the environment around the mobile platform and obtain information about objects in the environment to control the mobile platform to work safely and reliably. For example, obstacle avoidance is an issue that mobile platforms such as unmanned vehicles and unmanned aerial vehicles need to focus on. The key to achieving obstacle avoidance lies in the accurate identification of objects around the mobile platform. Therefore, how to accurately identify the information of objects around the mobile platform has become a technical problem that needs to be solved urgently.

[0030] The inventors have discovered that in the object recognition method in the related art, for large objects with larger sizes, the related sensors can obtain more detection data from the large objects, and then can extract more feature information from the detection data. More feature information can make the object recognition method in the related art relatively accurate in recognizing large objects, but for small objects with smaller sizes, the detection data obtained by the related sensors from the small objects is much less than the detection data of large objects, so that the feature information extracted from the detection data of small objects is also relatively small, and then the object recognition method in the related art has lower recognition accuracy for small objects. From the perspective of business scenarios, such as in some extreme scenarios in the field of vehicle driving, the recognition defects of the object recognition method in the related art for small objects may have serious consequences for the safe driving of vehicles.

[0031] Based on this, an embodiment of the present application provides an object recognition method. After obtaining point cloud data and image data of an object to be recognized, a first fusion feature is obtained based on the point cloud data and the image data, and a first image feature is obtained based on the image data; then a first size coefficient is obtained based on the first fusion feature, and a second size coefficient is obtained based on the first image feature, the first size coefficient and the second size coefficient characterizing the probability that the object to be recognized belongs to an object within a target size range; then a second fusion feature is obtained based on the first fusion feature and the first size coefficient, and a second image feature is obtained based on the first image feature and the second size coefficient; finally, the object to be recognized is recognized based on the second fusion feature and the second image feature. In this embodiment, the first size coefficient and the second size coefficient are determined to increase the attention to objects in the target size range, and then the first size coefficient and the second size coefficient are used to increase the information about objects in the target size range in the second fusion feature and the second image feature, so that the second fusion feature and the second image feature contain more semantic information about objects in the target size range, so that small objects can be accurately identified based on the second fusion feature and the second image feature, thereby improving the recognition accuracy of objects; further, this embodiment combines point cloud data and image data to perform object recognition, point cloud data can provide spatial information to assist image data, and image data can provide color information to assist point cloud data, and the two types of data complement each other, which is conducive to further improving the accuracy of object recognition.

[0032] The object recognition method of this embodiment can be implemented by an object recognition device. In one possible implementation, the object recognition device can be a computer chip or integrated circuit with data processing capabilities, such as a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC) or a readily available field-programmable gate array (FPGA). The object recognition device can be installed in a movable platform; the movable platform of the embodiment of the present application may include: a car, an unmanned aerial vehicle, an unmanned ship or a robot, wherein the car can be an unmanned vehicle or a manned vehicle, and the unmanned aerial vehicle can be a four-rotor drone, a six-rotor drone or an eight-rotor drone. In another implementation, the object recognition device can also be a movable platform, etc.; the movable platform at least includes a car, an unmanned aerial vehicle, an unmanned ship or a robot, wherein the car can be an unmanned vehicle or a manned vehicle.

[0033] In an exemplary application scenario, see Figure 1A as well as Figure 1B , the movable platform is an unmanned vehicle, and the unmanned vehicle includes the object recognition device as an example for explanation, Figure 1A shows a driving scene of the unmanned vehicle 100, Figure 1BThe structure diagram of the unmanned vehicle 100 is shown. The unmanned vehicle 100 may be equipped with a laser radar 10 for acquiring point cloud data, and the unmanned vehicle 100 may also be equipped with a shooting device 20 for acquiring image data. Exemplarily, the number of the laser radar 10 and the shooting device 20 may be one or more. It is understandable that the installation positions of the laser radar 10 and the shooting device 20 may be specifically set according to the actual application scenario. Exemplarily, one of the laser radars 10 and one of the shooting devices 20 may be installed on the front of the unmanned vehicle 100. During the driving of the unmanned vehicle 100, the laser radar 10 collects point cloud data of objects around the unmanned vehicle 100 and transmits it to the object recognition device 30 in the unmanned vehicle 100, and the shooting device 20 collects image data of objects around the unmanned vehicle 100 and transmits it to the object recognition device 30 in the unmanned vehicle 100. The object recognition device 30 obtains the point cloud data and image data of objects around the unmanned vehicle 100, and performs object recognition based on the object recognition method of the embodiment of the present application to obtain a recognition result. In a first possible implementation, after obtaining the recognition result, the unmanned vehicle 100 may use the recognition result to make obstacle avoidance decisions or perform route planning; in a second possible implementation, the recognition result may be displayed on an interface of the unmanned vehicle 100 or an interface of a terminal that is communicatively connected to the unmanned vehicle 100, so that a user can understand the driving conditions of the unmanned vehicle 100 and the road conditions around the unmanned vehicle 100; in a third possible implementation, the recognition result may be transmitted to other components in the unmanned vehicle 100, so that the other components control the unmanned vehicle 100 to operate safely and reliably based on the recognition result.

[0034] Next, the object recognition method provided in the embodiment of the present application is described: Figure 2 , Figure 2 A flowchart of an object recognition method provided in an embodiment of the present application is provided. The method can be implemented by an object recognition device; the object recognition device can be a movable platform, or the object recognition device is installed as a chip in the movable platform; the method includes:

[0035] In step S101, point cloud data and image data of an object to be identified are obtained, and a first fusion feature is obtained based on the point cloud data and the image data; and a first image feature is obtained based on the image data.

[0036] In step S102, a first size coefficient is obtained according to the first fusion feature; and a second size coefficient is obtained according to the first image feature; the first size coefficient and the second size coefficient represent the probability that the object to be identified belongs to an object within a target size range.

[0037] In step S103, a second fusion feature is obtained according to the first fusion feature and the first size coefficient; and a second image feature is obtained according to the first image feature and the second size coefficient.

[0038] In step S104, the object to be identified is identified according to the second fusion feature and the second image feature.

[0039] The point cloud data and image data of the object to be identified are obtained by spatial sampling by the sensor of the movable platform. Specifically, the point cloud data can be obtained by using a laser radar configured on the movable platform or a camera with a depth information acquisition function; and / or, the image data can be obtained by using a camera configured on the movable platform.

[0040] The laser radar is used to emit a laser pulse sequence into the space where the movable platform is located, and then receive the laser pulse sequence reflected from the object to be identified, and generate point cloud data based on the reflected laser pulse sequence. In one example, the laser radar can determine the reception time of the reflected laser pulse sequence, for example, by detecting the rising edge time and / or falling edge time of the electrical signal pulse to determine the reception time of the laser pulse sequence. In this way, the laser radar can calculate TOF (Time of flight) using the reception time information and emission time of the laser pulse sequence, so as to determine the distance from the object to be identified to the laser radar. The laser radar is a self-luminous sensor, which does not rely on light source illumination, is less affected by ambient light, and can work normally even in a lightless closed environment, so as to subsequently generate a high-precision three-dimensional model, and has a wide range of applicability.

[0041] The shooting device with the depth information acquisition function includes but is not limited to a binocular vision sensor or a structured light depth camera, etc.; the binocular vision sensor acquires two images of the target scene from different positions based on the parallax principle, and acquires three-dimensional geometric information by calculating the position deviation between corresponding points of the two images, thereby generating point cloud data; the structured light depth camera projects light with certain structural features into space and then acquires the light. This light with a certain structure will acquire different image phase information due to different depth areas of the object to be identified, and then converts it into depth information to obtain point cloud data.

[0042] The image data may be a color image, a grayscale image, or an infrared image, etc., and the shooting device for collecting the image data includes, but is not limited to, a visible light camera, a grayscale camera, and an infrared camera, etc. The shooting device may capture an image sequence at a specified frame rate. The shooting device may have adjustable shooting parameters. Under different shooting parameters, the shooting device may capture different images despite being subjected to exactly the same external conditions (e.g., position, illumination). Shooting parameters may include exposure (e.g., exposure time, shutter speed, aperture, film speed), gain, gamma, region of interest, binning / subsampling, pixel clock, offset, trigger, ISO, etc. Parameters related to exposure may control the amount of light reaching the image sensor in the shooting device. For example, the shutter speed may control the amount of time for light to reach the image sensor, while the aperture may control the amount of light reaching the image sensor within a given time. Parameters related to gain may control the amplification of the signal from the optical sensor. ISO may control the sensitivity level of the camera to available light.

[0043] In an exemplary embodiment, the movable platform is equipped with a laser radar and a visible light camera. In one example, the laser radar and the visible light camera can operate at the same frame rate. In another example, the laser radar and the visible light camera can also operate at different frame rates, and the frame rates of the laser radar and the visible light camera are sufficient to acquire point cloud data and image data within a preset time period.

[0044] Exemplarily, during the movement of the movable platform, the laser radar can collect point cloud data of the object to be identified in real time and transmit it to the object identification device, and the shooting device can collect image data of the object to be identified in real time and transmit it to the object identification device. In this embodiment, object identification is performed by combining point cloud data and image data. Point cloud data can provide spatial information to assist image data, and image data can provide color information to assist point cloud data. The two types of data complement each other, which is conducive to further improving the accuracy of object identification.

[0045] After acquiring the point cloud data and image data of the object to be identified, the object recognition device acquires a first fusion feature according to the point cloud data and the image data, and acquires a first image feature according to the image data; then acquires a first size coefficient according to the first fusion feature, and acquires a second size coefficient according to the first image feature, the first size coefficient and the second size coefficient characterizing the probability that the object to be identified belongs to an object within a target size range. This embodiment determines information of objects belonging to the target size range (i.e., the first size coefficient and the second size coefficient) from the first fusion feature and the first image feature, and increases the attention to objects within the target size range through the determined first size coefficient and the second size coefficient. Further, a second fusion feature is acquired according to the first fusion feature and the first size coefficient, and a second image feature is acquired according to the first image feature and the second size coefficient, so that semantic information about objects within the target size range in the second fusion feature and the second image feature is increased through the first size coefficient and the second size coefficient, so that small objects can be accurately identified based on the second fusion feature and the second image feature, thereby improving the accuracy of object recognition.

[0046] Among them, the target size range of this embodiment may include: a size range that is smaller than a preset target size threshold. Considering that the definition of the size of small objects is different for different types of movable platforms, the preset target size threshold can be flexibly configured according to business needs, and this embodiment does not limit this.

[0047] As for point cloud data, point cloud data is unstructured data and needs to be processed into a format that can be used for data analysis. For example, the point cloud data is processed to obtain the point cloud density corresponding to each voxel of the point cloud data. The point cloud data can be processed by three-dimensional gridding of the point cloud. For example, the point cloud data is gridded to obtain multiple voxels of the point cloud data. The ratio of the number of point clouds contained in each voxel in the point cloud data to the number of all point clouds in the point cloud data constitutes the point cloud density of the voxel. Among them, since the point cloud density characterizes the number of point clouds contained in the voxel, if the point cloud density is larger, it means that the probability that the voxel corresponds to an object is greater. Therefore, the point cloud density corresponding to each voxel of the point cloud data can be used as the feature information of the object. Processing irregular point clouds into regular representations can better express the contour information of the object to be identified.

[0048] For image data, pixel values ​​in the image data may be used for data analysis, and the pixel values ​​include but are not limited to RGB values ​​or grayscale values.

[0049] In some embodiments, the object recognition device may obtain a first fusion feature based on the point cloud density of each voxel in the point cloud data and the pixel value in the image data. The first fusion feature fuses the feature information of the point cloud data and the feature information of the image data, and can better reflect the characteristics of the object to be recognized from a three-dimensional perspective and a two-dimensional perspective, which is conducive to improving the accuracy of object recognition.

[0050] In a possible implementation, the point cloud data and the image data may be spliced ​​to obtain spliced ​​data and feature extraction may be performed on the spliced ​​data to obtain the first fusion feature. In one example, feature extraction may be performed on the spliced ​​data using a pre-trained object recognition model to obtain the first fusion feature.

[0051] Among them, the point cloud data can be rasterized into a three-dimensional grid of H*W*C (where H and W represent length and width respectively, and C represents the depth of the three-dimensional grid), each grid represents a voxel, and the value of the voxel is the point cloud density of the voxel. The image data (taking RGB image as an example) can be represented as H*W*3 data (where H and W represent length and width respectively, and 3 represents RGB3 channels), and then the point cloud data and the image data can be spliced ​​into spliced ​​data of a size of H*W*(C+3). The spliced ​​data includes the position information (obtained from the point cloud data) and color information (obtained from the image data) of the object to be identified, and can better reflect the characteristics of the object to be identified from three-dimensional and two-dimensional perspectives, thereby helping to improve the accuracy of object recognition.

[0052] In another possible implementation, in order to further improve the accuracy of object recognition, the object recognition device can determine the first projection position of the point cloud data in the two-dimensional space based on the projection relationship from the point cloud to the image, and then obtain the pixel value at the first projection position in the image data, and generate point cloud data including the pixel value according to the pixel value at the first projection position and the point cloud data; taking the image data as an RGB image as an example, the object recognition device can assign the RGB value at the first projection position in the image data to the corresponding point cloud, thereby generating a color point cloud; in this embodiment, the point cloud data integrates the color data in the image, further strengthening the correspondence between the image data and the point cloud data, so that the point cloud data including the pixel value can better reflect the characteristics of the object to be identified, thereby helping to improve the accuracy of object recognition. Among them, the projection relationship from the point cloud to the image can be obtained based on the external parameters of the point cloud coordinate system to the camera coordinate system and the internal parameters of the camera. Suppose the coordinates of a point in the point cloud in the point cloud coordinate system are P, the external parameters of the point cloud coordinate system to the camera coordinate system are RT, the internal parameters of the camera are K, and the first projection position of the point in the two-dimensional space is p, then p=RT*K*P.

[0053] Next, the object recognition device may obtain the first fusion feature based on the point cloud data including the pixel value and the image data. For example, the object recognition device may splice the point cloud data including the pixel value and the image data to obtain spliced ​​data, and perform feature extraction on the spliced ​​data to obtain the first fusion feature. In one example, feature extraction may be performed on the spliced ​​data using a pre-trained object recognition model to obtain the first fusion feature.

[0054] The point cloud data can be rasterized into a three-dimensional grid of H*W*C, each grid represents a voxel, the value of the voxel is the point cloud density of the voxel and the pixel value of the point cloud (such as RGB value), the image data (taking RGB image as an example) can be represented as H*W*3 data, and the point cloud data including the pixel value and the image data can be spliced ​​into spliced ​​data with a size of H*W*(C+3), the spliced ​​data includes the position information of the object to be identified, the correspondence information between the position and color (obtained from the point cloud data including the pixel value) and the color information (obtained from the image data), and can comprehensively reflect the characteristics of the object to be identified from three-dimensional and two-dimensional perspectives, thereby helping to improve the accuracy of object recognition.

[0055] In some embodiments, the object recognition device performs feature extraction on the image data to obtain the first image feature; in one example, feature extraction can be performed on the image data using a pre-trained object recognition model to obtain the first image feature.

[0056] After acquiring the first fusion feature and the first image feature, the object recognition device acquires a first size coefficient from the first fusion and a second size coefficient from the first image feature, the first size coefficient and the second size coefficient characterizing the probability that the object to be recognized belongs to an object within a target size range. This embodiment determines information about objects within a target size range from the first fusion feature and the first image feature, and increases attention to objects within the target size range by determining the first size coefficient and the second size coefficient, thereby improving the recognition accuracy of small objects.

[0057] In a possible implementation, the first fusion feature and the first image feature can be represented in the form of a feature map, and the object recognition device can obtain a first size coefficient at each position in the feature map including the first fusion feature, the first size coefficient represents the probability that each position in the feature map including the first fusion feature belongs to an object within a target size range, and the first fusion feature fuses point cloud information and image information, so that the first size coefficient obtained from the first fusion feature also reflects the point cloud information and image information, and therefore has better semantic information; and the object recognition device can obtain a second size coefficient at each position in the feature map including the first image feature, the second size coefficient represents the probability that each position in the feature map including the first image feature belongs to an object within a target size range.

[0058] In one example, the object recognition device may use a pre-trained object recognition model to obtain a first size coefficient from the first fusion feature, and obtain a second size coefficient from the first image feature.

[0059] After obtaining the first size coefficient and the second size coefficient, the object recognition device obtains the second fusion feature according to the first fusion feature and the first size coefficient, and obtains the second image feature according to the first image feature and the second size coefficient, and finally performs target recognition on the object to be recognized according to the second fusion feature and the second image feature. This embodiment realizes increasing the information about objects in the target size range in the second fusion feature and the second image feature through the first size coefficient and the second size coefficient, so that the second fusion feature and the second image feature can fuse more semantic information about objects in the target size range, so that small objects can be accurately recognized based on the second fusion feature and the second image feature, thereby improving the accuracy of object recognition; of course, in the case where the point cloud data and the image data also include data collected from large objects, the second fusion feature and the second image feature obtained based on the point cloud data and the image data also include feature information extracted from large objects, so large objects can also be accurately recognized.

[0060] In one example, the second fusion feature may be the sum of the first fusion feature and the first size coefficient; and / or, the second image feature may be the sum of the first image feature and the second size coefficient, so that the second fusion feature and the second image feature can fuse more semantic information about objects in the target size range. In other examples, for example, the second fusion feature or the second image feature may also be obtained by other calculation methods. For example, in some other possible embodiments, the second fusion feature is the product of the first fusion feature and the first size coefficient, or the second image feature is the product of the first image feature and the second size coefficient, which may be specifically set according to the actual application scenario.

[0061] Exemplarily, the first fusion feature, the first image feature, the second fusion feature and the second image feature can be represented in the form of a feature map. The object recognition device can obtain the first size coefficient at each position in the feature map including the first fusion feature, and obtain the second size coefficient at each position in the feature map including the first image feature; that is, each position in the feature map including the first fusion feature corresponds to the first size coefficient, and each position in the feature map including the first image feature corresponds to the second size coefficient; therefore, the value of each position in the feature map including the first fusion feature can be added to the first size coefficient of the position to obtain a feature map including the second fusion feature; and the value of each position in the feature map including the first image feature can be added to the second size coefficient of the position to obtain a feature map including the second image feature.

[0062] When the object to be identified is identified based on the second fusion feature and the second image feature, in order to further improve the accuracy of object identification, the object identification device may further fuse the two. Specifically, the object identification device determines the second projection position of the second fusion feature in the two-dimensional space based on the projection relationship from the point cloud to the image, and then obtains the image feature at the second projection position in the second image feature, obtains the third fusion feature based on the image feature and the corresponding second fusion feature, and finally identifies the object to be identified based on the third fusion feature to obtain the identification result. In this embodiment, the third fusion feature is obtained by fusing the second fusion feature and the second image feature. The third fusion feature fuses image information and point cloud information and further fuses more information about objects in the target size range, and has good semantic information. Using the third fusion feature for object identification is conducive to improving the accuracy of object identification.

[0063] The projection relationship from the point cloud to the image can be obtained based on the external parameters from the point cloud coordinate system to the camera coordinate system and the internal parameters of the camera. The recognition result at least includes: the confidence and / or state information of the object to be recognized; the confidence is used to characterize the probability that the object to be recognized is an obstacle. The state information includes at least one of the following: size information, position information and orientation information of the object.

[0064] In some embodiments, the point cloud data and the image data may be input into a pre-trained object recognition model, and the point cloud data and the image data may be processed using the object recognition model to further perform object recognition.

[0065] In some embodiments, the object recognition method of the embodiment of the present application can be implemented by a trained object recognition model. The object recognition model can be pre-installed in the object recognition device so that the object recognition process can be performed through the object recognition model.

[0066] The training process of the object recognition model can be: first, a model is represented by modeling, then the model is evaluated by constructing an evaluation function, and finally, the evaluation function is optimized according to sample data and optimization methods to adjust the model to the optimal level.

[0067] Modeling is the process of converting actual problems into problems that can be understood by computers, that is, converting actual problems into a way that computers can represent them. Modeling generally refers to the process of estimating the objective function of a model based on a large amount of sample data.

[0068] The goal of evaluation is to judge the quality of the built model. For the model built in the first step, evaluation is an indicator used to indicate the quality of the model. This involves some evaluation indicators and the design of some evaluation functions. There are targeted evaluation indicators in machine learning. For example, after the modeling is completed, a loss function needs to be designed for the model to evaluate the output error of the model.

[0069] The goal of optimization is the evaluation function. That is, the evaluation function is optimized using the optimization method to find the model with the highest evaluation. For example, the minimum value (optimal solution) of the output error of the loss function can be found through optimization methods such as gradient descent, and the parameters of the model can be adjusted to the optimal value.

[0070] It can be understood that before training a model, we first determine a suitable parameter estimation method, and then use this parameter estimation method to estimate the various parameters in the objective function of the model, and then determine the final mathematical expression of the objective function.

[0071] The object recognition model in the related art has been able to achieve very good results in object recognition and can recognize objects with high accuracy. However, the inventors have found that in actual business scenarios related to mobile platforms, object recognition models often focus on larger targets such as people, vehicles, roads or trees, and the sample data of these larger targets often has a large amount of data. During the training process, the model will tend to be global optimal. The features of larger targets are often more obvious and easier to be noticed. The features of small objects are relatively subtle and difficult to get the attention of the model, which makes the model tend to extract the features of large objects. This bias ultimately leads to the model being able to recognize large objects well, but not paying enough attention to the recognition of small objects, resulting in defects in the object recognition model in the related art. From the perspective of business scenarios, such as in some extreme scenarios in the field of vehicle driving, subtle defects in the object recognition model may have serious consequences for the safe driving of the vehicle. From a technical perspective, it is extremely challenging and difficult to further address defects on the basis of the existing high accuracy, because in the field of machine learning, as mentioned above, there are many links involved from modeling to training, such as the selection and processing of sample data, the design of data features, the design of models, the design of loss functions or the design of optimization methods, etc. The slightest difference in any link may lead to defects in the model.

[0072] Based on this, see Figure 3AThe embodiment of the present application provides an object recognition model for object recognition, the object recognition model includes a first feature extraction network, a second feature extraction network, a first size coefficient extraction network, a second size coefficient extraction network and an object prediction network; the first feature extraction network is used to obtain a first fusion feature according to the point cloud data and the image data; the second feature extraction network is used to obtain a first image feature according to the image data; the first size coefficient extraction network is used to obtain a first size coefficient according to the first fusion feature, and to obtain a second fusion feature according to the first fusion feature and the first size coefficient; the second size coefficient extraction network is used to obtain a second size coefficient according to the first image feature, and to obtain a second image feature according to the first image feature and the second size coefficient; the object prediction network is used to identify the object to be identified according to the second fusion feature and the second image feature to obtain a recognition result. The recognition result at least includes: the confidence and / or state information of the object to be identified; the confidence is used to characterize the probability that the object to be identified is an obstacle. The state information includes at least one of the following: size information, position information and orientation information of the object to be identified.

[0073] Among them, the first feature extraction network can perform feature extraction on the spliced ​​data (the spliced ​​data is formed by splicing the point cloud data and the image data, or the spliced ​​data is formed by splicing the aforementioned point cloud data including the pixel value and the image data) to obtain the first fusion feature.

[0074] The second feature extraction network is used to perform feature extraction on the image data to obtain the first image feature.

[0075] The first size coefficient extraction network and the second size coefficient extraction network both include convolution layers, the first size coefficient is obtained by performing a convolution operation on the first fusion feature through the first size coefficient extraction network; and the second size coefficient is obtained by performing a convolution operation on the first image feature through the second size coefficient extraction network. It is understandable that the number of convolution layers can be specifically set according to the actual application scenario, for example, the number of convolution layers is at least 2, the first size coefficient is obtained by performing at least two convolution operations on the first fusion feature through the first size coefficient extraction network; and the second size coefficient is obtained by performing at least two convolution operations on the first image feature through the second size coefficient extraction network. The first size coefficient and the second size coefficient represent the probability that the object to be identified belongs to an object within the target size range.

[0076] Among them, the first fusion feature fuses point cloud information and image information, so that the first size coefficient obtained from the first fusion feature also reflects the point cloud information and image information, and therefore has good semantic information. Furthermore, this embodiment realizes determining the information of objects belonging to the target size range from the first fusion feature and the first image feature (i.e., the first size coefficient and the second size coefficient), and increases the attention to objects in the target size range by determining the first size coefficient and the second size coefficient, which is conducive to improving the recognition accuracy of the model for small objects.

[0077] In some embodiments, the first size coefficient extraction network is further used to obtain a second fusion feature based on the sum of the first fusion feature and the first size coefficient; the second size coefficient extraction network is further used to obtain a second image feature based on the sum of the first image feature and the second size coefficient, but is not limited thereto. For example, the second fusion feature or the second image feature may also be obtained by other calculation methods. For example, in some other possible embodiments, the second fusion feature is the product of the first fusion feature and the first size coefficient, or the second image feature is the product of the first image feature and the second size coefficient, which may be specifically set according to the actual application scenario.

[0078] For example, see Figure 3B , the first size coefficient extraction network includes a first size coefficient extraction subnetwork and a first fusion subnetwork, the first size coefficient extraction subnetwork is used to perform a convolution operation on the first fusion feature to obtain a first size coefficient, and the first fusion subnetwork is used to obtain a second fusion feature according to the first fusion feature and the first size coefficient. The second size coefficient extraction network includes a second size coefficient extraction subnetwork and a second fusion subnetwork, the second size coefficient extraction subnetwork is used to perform a convolution operation on the first image feature to obtain a second size coefficient, and the second fusion subnetwork is used to obtain a second image feature according to the first image feature and the second size coefficient. In this embodiment, the first size coefficient and the second size coefficient are determined to increase the attention to objects in the target size range, so that the first size coefficient and the second size coefficient are used to increase the information about objects in the target size range in the second fusion feature and the second image feature, and the second fusion feature and the second image feature contain more semantic information about objects in the target size range, so that small objects can be accurately identified based on the second fusion feature and the second image feature.

[0079] The object prediction network is used to obtain a third fusion feature based on the second fusion feature and the second image feature, and then perform target recognition on the object to be recognized based on the third fusion feature. Specifically, the object prediction network can obtain the third fusion feature based on the aforementioned method, and perform target recognition on the object to be recognized based on the third fusion feature to obtain a recognition result. The recognition result at least includes: the confidence and / or state information of the object to be recognized; the confidence is used to characterize the probability that the object to be recognized is an obstacle. The state information includes at least one of the following: size information, position information, and orientation information.

[0080] Next, the training process of the object recognition model is described. In this embodiment, sample data for training can be prepared in advance. The sample data may include: point cloud sample data and image sample data, and the point cloud sample data and image sample data include data of objects within the target size range and data of objects that do not fall within the target size range, so that the trained object recognition model can accurately recognize both small objects and large objects.

[0081] In this embodiment, model training can be supervised training or unsupervised training. In some examples, supervised training can be used to improve the training speed, and the true value can be marked in the sample data. Through supervised training, the speed and accuracy of model training can be improved. Among them, the object state information may include one or more information, and the specific information can be configured according to business needs. As an example, the true value includes: the confidence of objects belonging to the target size range (characterizing the probability that the object to be identified is a small object), the object confidence (characterizing the probability that the object to be identified is an obstacle), and the state information of the object; the state information may include at least one of the following: size information, position information, and orientation information.

[0082] In some examples, the sample data may be data obtained after feature engineering of the original data. Feature engineering refers to the process of finding some physically meaningful features from the original data to participate in model training, which involves data cleaning, data dimensionality reduction, feature extraction, feature normalization, feature evaluation and screening, feature dimensionality reduction or feature encoding.

[0083] For example, point cloud data is unstructured data and needs to be processed into a format that can be input into an object recognition model. For example, the point cloud data is processed to obtain the point cloud density corresponding to each voxel of the point cloud data, and the point cloud density corresponding to each voxel of the point cloud data is used as the input of the object recognition model. For image data, the pixel value of the image data can be used as the input of the object recognition model.

[0084] In some embodiments, in order to improve the accuracy of object recognition, the sample data input into the object recognition model in this embodiment includes: spliced ​​sample data obtained according to the point cloud sample data and the image sample data, and the image sample data.

[0085] In one example, the point cloud sample data can be rasterized into a three-dimensional grid of H*W*C (where H and W represent length and width, respectively, and C represents the depth of the three-dimensional grid), each grid represents a voxel, and the value of the voxel is the point cloud density of the voxel. The image sample data (taking RGB image as an example) can be represented as H*W*3 data (where H and W represent length and width, respectively, and 3 represents RGB3 channels), and then the point cloud sample data and the image sample data can be spliced ​​into spliced ​​sample data of size H*W*(C+3). The spliced ​​sample data includes the location information (obtained from the point cloud data) and color information (obtained from the image data) of the object, which can better reflect the characteristics of the object from a three-dimensional perspective and a two-dimensional perspective, thereby helping to improve the accuracy of object recognition.

[0086] In another example, point cloud sample data including pixel values ​​can be generated according to the point cloud sample data and the image sample data (the generation method can refer to the aforementioned method of generating point cloud data including the pixel values, which will not be repeated here), wherein the point cloud sample data can be rasterized into a three-dimensional grid of H*W*C, each grid represents a voxel, and the value of the voxel is the point cloud density of the voxel and the pixel value of the point cloud (such as RGB value), and the image sample data (taking RGB image as an example) can be represented as H*W*3 data, and the point cloud sample data including pixel values ​​and the image data can be spliced ​​into spliced ​​sample data of a size of H*W*(C+3).

[0087] Using the above sample data, the object recognition model can be obtained by training the machine learning model using the sample data. The machine learning model can be a neural network model, such as a neural network model based on deep learning. The specific structural design of the object recognition model is one of the important aspects of the training process. In this embodiment, the structure of the object recognition model at least includes: a first feature extraction network, a second feature extraction network, a first size coefficient extraction network, a second size coefficient extraction network, and an object prediction network.

[0088] Among them, the embodiment of the present application respectively extracts information about objects within the target size range through the first size coefficient extraction network and the second size coefficient extraction network to obtain the first size coefficient and the second size coefficient, and then enhances the information about objects within the target size range on the basis of the first fusion feature and the first image feature, respectively, to obtain the second fusion feature and the second image feature including more semantic information about objects within the target size range, thereby enhancing the model's recognition of objects within the target size range.

[0089] In one possible implementation, the first feature extraction network and the second feature extraction network are the main networks, the first size coefficient extraction network is the branch network of the first feature extraction network, and the second size coefficient extraction network is the branch network of the second feature extraction network. Therefore, on the basis of extracting the first fusion feature and the first image feature, the first size coefficient and the second size coefficient can be further extracted to obtain the second fusion feature and the second image feature. Compared with using two independent neural networks, the execution overhead is correspondingly reduced and the model execution efficiency is improved.

[0090] In an exemplary embodiment, in the field of autonomous driving, three-dimensional object detection is a core issue, and the use of sensors such as lidar for detection may make it difficult to detect small objects. In this embodiment, a neural network in deep learning is used to detect the position and confidence of three-dimensional objects. By adding network branches (the first size coefficient extraction network and the second size coefficient extraction network) to the neural network and improving the training strategy of the neural network in the deep learning algorithm, the network branches are used to extract the feature information of small objects, ultimately making the model small object detection more friendly.

[0091] Another important aspect of the training process is to design a suitable loss function based on business needs. The loss function is also called the cost function. In the scenario of supervised model training, the sample data is labeled with the true value, and the loss function is used to estimate the error between the model's predicted value and the true value. The loss function is crucial to the recognition accuracy of the model. It is difficult to design a loss function based on the existing sample data and model requirements. In some examples, some existing loss functions such as logarithmic loss function, square loss function, exponential loss function, 0 / 1 loss function, etc. can be used to form the loss function of the corresponding scenario.

[0092] Based on the requirements of this embodiment, as an example, the loss function used by the object recognition model during training includes at least: a first loss function for optimizing the first size coefficient extraction network, a second loss function for optimizing the second size coefficient extraction network, a third loss function for describing state differences, and a fourth loss function for describing confidence differences. The first loss function and the second loss function enable the model to focus on objects within the target size range, so that the model can distinguish small objects more clearly.

[0093] Among them, the optimization goal of the first loss function includes: if the sample data indicates an object belonging to the target size range, increase the first size coefficient; the sample data includes point cloud sample data and image sample data. More specifically, the first size coefficient extraction network is used to extract the first size coefficient from the first fusion feature, and further obtain the second fusion feature based on the first size coefficient and the first fusion feature; the first size coefficient extraction network is used to predict the probability that the second fusion feature belongs to the target size range. Therefore, the optimization goal of the first loss function specifically includes: if the second fusion feature obtained based on the sample data indicates an object belonging to the target size range, increase the second fusion feature, and if the second fusion feature obtained based on the sample data indicates an object that does not belong to the target size range, reduce the second fusion feature.

[0094] In an example, taking the second fusion feature represented in the form of a feature map as an example, the optimization goal of the first loss function specifically includes: if a certain position in the feature map including the second fusion feature obtained based on the sample data indicates an object belonging to the target size range, increase the feature value at the position, otherwise, reduce the feature value at the position, so that the object recognition model can distinguish small objects more clearly.

[0095] The first loss function can be used to describe the difference between the confidence prediction value of the object belonging to the target size range obtained by the object recognition model from the sample data and the true confidence value of the object belonging to the target size range corresponding to the sample data; it can be explained by taking a formula as an example: in, represents the true value of the annotation, that is, the true value of the confidence that the object to be identified belongs to the target size range; fseg(x k1 ) represents the confidence prediction value of the object belonging to the target size range predicted by the object recognition model from the sample data.

[0096] Among them, the optimization goal of the second loss function includes: if the image sample data indicates an object that belongs to the target size range, increase the second size coefficient. More specifically, the second size coefficient extraction network is used to extract the second size coefficient from the first image feature, and further obtain the second image feature based on the second size coefficient and the first image feature; the second size coefficient extraction network is used to predict the probability that the second image feature belongs to the target size range. Therefore, the optimization goal of the second loss function specifically includes: if the second image feature obtained based on the image sample data indicates an object that belongs to the target size range, increase the second image feature, and if the second image feature obtained based on the sample data indicates an object that does not belong to the target size range, reduce the second image feature.

[0097] In one example, taking the second image feature represented in the form of a feature map as an example, the optimization goal of the second loss function specifically includes: if a certain position in the feature map including the second image feature obtained based on the image sample data indicates an object belonging to the target size range, increase the feature value at the position, otherwise, reduce the feature value at the position, so that the object recognition model can distinguish small objects more clearly.

[0098] The second loss function is used to describe the difference between the confidence prediction value of the object belonging to the target size range obtained by the object recognition model from the image sample data and the true confidence value of the object belonging to the target size range corresponding to the image sample data. This can be explained by taking a formula as an example: in, represents the true value of the annotation, that is, the true value of the confidence that the object to be identified belongs to the target size range; fseg(x k2 ) represents the confidence prediction value of the object belonging to the target size range predicted by the object recognition model from the image sample data.

[0099] The optimization goal of the third loss function includes: reducing the difference between the predicted value of the state information of the object obtained by the object recognition model from the sample data and the true value of the state information of the object corresponding to the sample data. The object state information includes at least one of the following: size information, position information and orientation information of the object.

[0100] The third loss function is used to describe the difference between the predicted size, predicted position and / or predicted orientation of the object obtained by the object recognition model from the sample data and the real size, real position and / or real orientation of the object corresponding to the sample data; the sample data includes point cloud sample data and image sample data. This can be explained by taking a formula as an example: in, represents the true value of the annotation, that is, the true size, true position and / or true orientation of the object annotated in the sample data; floc(x i ) represents the predicted size, predicted position and / or predicted orientation of the object obtained by the object recognition model from the sample data.

[0101] The optimization goal of the fourth loss function includes: reducing the difference between the confidence prediction value of the object obtained by the object recognition model from the sample data and the true confidence value of the object corresponding to the sample data. The confidence of the object represents the probability that the predicted object is an obstacle.

[0102] The fourth loss function is used to describe the difference between the confidence prediction value of the object obtained by the object recognition model from the sample data and the true confidence value of the object corresponding to the sample data. This can be explained by taking a formula as an example: in, represents the true value of the annotation, that is, the true value of the confidence of the object annotated in the sample data, fpred(x i ) represents the confidence prediction value of the object obtained by the object recognition model from the sample data.

[0103] In summary, the loss function used in the object recognition model of this embodiment during the training process may include:

[0104] The specific formula of the above loss function is only for illustration. In practical applications, the specific mathematical description of the function can be flexibly configured as needed, and whether to add a regularization term can be determined as needed. This embodiment does not limit this.

[0105] During the training process, it is necessary to use an optimization method to optimize the evaluation function and find the model with the highest evaluation. For example, the minimum value (optimal solution) of the output error of the loss function can be found through optimization methods such as gradient descent, and the parameters of the model can be adjusted to the optimal value, that is, the optimal coefficients of each network layer in the model are solved. In some examples, the solution process can be to calculate the gradient of adjusting the model parameters by calculating the output of the model and the error value of the loss function. As an example, the back propagation function can be called to calculate the gradient, and the calculation result of the loss function can be back-propagated to the object recognition model so that the object recognition model updates the model parameters.

[0106] In some examples, the solution of the above loss function can be solved by an independent solver. In other examples, taking the object recognition model as a neural network model as an example, a network branch can be set on the basis of the backbone network to calculate the loss function of the network. As an example, the loss function can be divided into the above four functions: a first loss function for optimizing the first size coefficient extraction network, a second loss function for optimizing the second size coefficient extraction network, a third loss function for describing state differences, and a fourth loss function for describing confidence differences. These loss functions will uniformly guide the update of the parameters of the neural network to enable it to have better prediction performance.

[0107] Through the above training process, an object recognition model is obtained after the training is completed. The obtained object recognition model can also be tested using test samples to verify the recognition accuracy of the object recognition model. The object recognition model finally obtained can be set in an object recognition device. Exemplarily, the object recognition device can be a movable platform, or the object recognition device is installed in the movable platform as a chip.

[0108] During the movement of the movable platform, point cloud data can be acquired through a laser radar or a shooting device with a depth information acquisition function configured on the movable platform, as well as image data acquired through a shooting device configured on the movable platform. Then, an object recognition device in the movable platform acquires stitching data based on the point cloud data and the image data, and then inputs the stitching data and the image data into the object recognition model, thereby acquiring a recognition result output by the object recognition model, wherein the recognition result includes confidence and status information of the object. Furthermore, in order to facilitate the subsequent processing based on the recognition results, the recognition result is data with a confidence greater than a preset threshold, and for data with a confidence not greater than the preset threshold, it indicates that it is not an obstacle, and there is no need to further process the data with a confidence not greater than the preset threshold. The preset threshold can be specifically set according to the actual application scenario; as an example, for an input object to be identified, a series of candidate boxes can be identified, each candidate box may correspond to a possible object, and based on the confidence of the object to be identified, the probability that the candidate box corresponds to an obstacle can be determined, and the confidences corresponding to the candidate boxes are sorted. After sorting, they are screened according to the set threshold. Those greater than the set threshold can be considered to be an object, thereby obtaining the final recognition result.

[0109] After obtaining the recognition result, in a first possible implementation manner, the movable platform may use the recognition result to make an obstacle avoidance decision or perform route planning.

[0110] In a second possible implementation, the recognition result can be displayed on the interface of the mobile platform or the interface of the terminal connected to the mobile platform for communication, so that the user can understand the driving condition of the mobile platform and the road conditions around the mobile platform; further, the point cloud data and image data can be displayed on the interface of the mobile platform or the interface of the terminal connected to the mobile platform for communication, and the recognition result can be further displayed on the point cloud data and image data, so that the user can watch it in combination with the specific scene, so that the user can further understand the actual driving condition.

[0111] In a third possible implementation manner, the recognition result may be transmitted to other components in the movable platform, so that the other components control the movable platform to operate safely and reliably based on the recognition result.

[0112] Accordingly, see Figure 4 The embodiment of the present application further provides an object recognition device 30, which may be a movable platform, or the object recognition device 30 may be installed as a chip in the movable platform; the object recognition device 30 includes a processor 31 and a memory 32 storing a computer program;

[0113] The processor 31 implements the following steps when executing the computer program:

[0114] Acquire point cloud data and image data of the object to be identified, and acquire a first fusion feature according to the point cloud data and the image data; and acquire a first image feature according to the image data;

[0115] Acquire a first size coefficient according to the first fusion feature; and acquire a second size coefficient according to the first image feature; the first size coefficient and the second size coefficient represent the probability that the object to be identified belongs to an object within a target size range;

[0116] Acquire a second fusion feature according to the first fusion feature and the first size coefficient; and acquire a second image feature according to the first image feature and the second size coefficient;

[0117] The object to be identified is identified according to the second fusion feature and the second image feature.

[0118] In one embodiment, the point cloud data and image data of the object to be identified are obtained by spatial sampling by a sensor of a movable platform.

[0119] In one embodiment, the processor 31 is further used to: obtain a first fusion feature according to the point cloud density of each voxel in the point cloud data and the pixel value in the image data; wherein the voxel is obtained by gridding the point cloud data.

[0120] In one embodiment, the first fusion feature and the first image feature are represented in the form of a feature map.

[0121] The first size coefficient represents the probability that each position in the feature map including the first fusion feature belongs to an object of a target size range; and the second size coefficient represents the probability that each position in the feature map including the first image feature belongs to an object of a target size range.

[0122] In one embodiment, the processor 31 is also used to: determine a first projection position of the point cloud data in two-dimensional space based on a projection relationship from the point cloud to the image; obtain a pixel value at the first projection position in the image data; generate point cloud data including the pixel value based on the pixel value and the point cloud data; and obtain the first fusion feature based on the point cloud data including the pixel value and the image data.

[0123] In one embodiment, the projection relationship from the point cloud to the image is obtained based on the external parameters of the point cloud coordinate system to the camera coordinate system and the intrinsic parameters of the camera.

[0124] In one embodiment, the processor 31 is also used to: splice the point cloud data and the image data to obtain spliced ​​data and input it into the object recognition model; perform feature extraction on the splicing information through the first feature extraction network in the object recognition model to obtain the first fusion feature.

[0125] In one embodiment, the first image feature is obtained by performing feature extraction on the image data.

[0126] In one embodiment, the processor 31 is further used to: input the image data into an object recognition model, perform feature extraction on the image data through a second feature extraction network in the object recognition model, and obtain the first image feature.

[0127] In one embodiment, the first size coefficient is obtained from the first fused feature by a first size coefficient extraction network in an object recognition model; and the second size coefficient is obtained from the first image feature by a second size coefficient extraction network in the object recognition model.

[0128] In one embodiment, the first size coefficient extraction network and the second size coefficient extraction network both include convolutional layers.

[0129] In one embodiment, the first size coefficient is obtained by performing a convolution operation on the first fusion feature through the first size coefficient extraction network; and the second size coefficient is obtained by performing a convolution operation on the first image feature through the second size coefficient extraction network.

[0130] In one embodiment, the loss function used by the object recognition model during training includes at least: a first loss function for optimizing the first size coefficient extraction network and a second loss function for optimizing the second size coefficient extraction network.

[0131] In one embodiment, the optimization objective of the first loss function includes: if the sample data indicates an object belonging to a target size range, increasing the first size coefficient; the sample data includes point cloud sample data and image sample data; and, the optimization objective of the second loss function includes: if the image sample data indicates an object belonging to a target size range, increasing the second size coefficient.

[0132] In one embodiment, the first loss function is used to describe the difference between a confidence prediction value of an object belonging to the target size range obtained by the object recognition model from sample data and a true confidence value of the object belonging to the target size range corresponding to the sample data; the sample data includes point cloud sample data and image sample data.

[0133] The second loss function is used to describe the difference between the confidence prediction value of the object belonging to the target size range obtained by the object recognition model from the image sample data and the true confidence value of the object belonging to the target size range corresponding to the image sample data.

[0134] In one embodiment, the second fusion feature is the sum of the first fusion feature and the first size coefficient; and / or the second image feature is the sum of the first image feature and the second size coefficient.

[0135] In one embodiment, the processor 31 is also used to: determine the second projection position of the second fusion feature in the two-dimensional space based on the projection relationship from the point cloud to the image; obtain the image feature at the second projection position in the second image feature; obtain the third fusion feature based on the image feature and the corresponding second fusion feature, and identify the object to be identified based on the third fusion feature.

[0136] In one embodiment, the third fusion feature is an average of the image feature and the corresponding second fusion feature; or, the third fusion feature is a larger one between the image feature and the corresponding second fusion feature.

[0137] In one embodiment, the processor 31 is further used to: identify the object to be identified based on the second fusion feature and the second image feature, and generate a recognition result; the recognition result at least includes: the confidence and / or status information of the object to be identified; the confidence is used to characterize the probability that the object to be identified is an obstacle.

[0138] In one embodiment, the state information includes at least one of the following: size information, position information, and orientation information.

[0139] In one embodiment, the confidence and state information of the object to be identified are obtained by processing the second fusion feature and the second image feature using an object prediction network in an object recognition model.

[0140] In one embodiment, the loss function used by the object recognition model during training includes at least a third loss function for describing state differences and a fourth loss function for describing confidence differences.

[0141] The state difference includes: the predicted size, predicted position and / or predicted orientation of the object obtained by the object recognition model from the sample data, and the difference between the actual size, actual position and / or actual orientation of the object corresponding to the sample data; the sample data includes point cloud sample data and image sample data.

[0142] The confidence difference includes: the difference between the confidence prediction value of the object obtained by the object recognition model from the sample data and the true confidence value of the object corresponding to the sample data.

[0143] In one embodiment, the recognition result is data with a confidence level greater than a preset threshold.

[0144] In one embodiment, the processor 31 is further used to: identify the object to be identified according to the second fusion feature and the second image feature to generate a recognition result, and the recognition result is used to make obstacle avoidance decisions or mobile route planning for the movable platform.

[0145] In one embodiment, the processor 31 is also used to: identify the object to be identified based on the second fusion feature and the second image feature to generate a recognition result; the recognition result is used to be displayed on the interface of the movable platform or the interface of a terminal device that is communicatively connected to the movable platform.

[0146] In one embodiment, the point cloud data is acquired using a laser radar configured on a movable platform or a photographing device with a depth information acquisition function; and / or, the image data is acquired using a photographing device configured on a movable platform.

[0147] In one embodiment, the movable platform includes: an unmanned aerial vehicle, a car, an unmanned ship or a robot.

[0148] For the device embodiment, since it basically corresponds to the method embodiment, the relevant part refers to the partial description of the method embodiment. The various embodiments described here can be implemented using a computer readable medium such as computer software, hardware or any combination thereof. For hardware implementation, the embodiments described here can be implemented by using at least one of a specific application integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a processor, a controller, a microcontroller, a microprocessor, and an electronic unit designed to perform the functions described here. For software implementation, an embodiment such as a process or function can be implemented with a separate software module that allows the execution of at least one function or operation. The software code can be implemented by a software application (or program) written in any appropriate programming language, and the software code can be stored in a memory and executed by a controller.

[0149] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprises a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0150] Accordingly, see Figure 5 The embodiment of the present application also provides a movable platform 100, including: a body 101; a power system 102, installed in the body 101, for providing power for the movable platform 100; and the above-mentioned object recognition device 30.

[0151] Optionally, the movable platform 100 is a vehicle, a drone, an unmanned ship or a movable robot.

[0152] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by a processor of the device to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0153] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a terminal, enables the terminal to execute the above method.

[0154] The method and device provided in the embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An object recognition method, characterized in that: include: Acquire point cloud data and image data of the object to be identified, and acquire a first fusion feature according to the point cloud data and the image data; and, acquiring a first image feature according to the image data; Acquire a first size coefficient according to the first fusion feature; and acquire a second size coefficient according to the first image feature; the first size coefficient and the second size coefficient represent the probability that the object to be identified belongs to an object within a target size range; Acquire a second fusion feature according to the first fusion feature and the first size coefficient; and acquire a second image feature according to the first image feature and the second size coefficient; The object to be identified is identified according to the second fusion feature and the second image feature.

2. The method according to claim 1, characterized in that The point cloud data and image data of the object to be identified are obtained by spatial sampling by the sensor of the movable platform.

3. The method according to claim 1, characterized in that The acquiring a first fusion feature according to the point cloud data and the image data comprises: A first fusion feature is obtained according to the point cloud density of each voxel in the point cloud data and the pixel value in the image data; wherein the voxel is obtained by gridding the point cloud data.

4. The method according to claim 1, characterized in that: The first fusion feature and the first image feature are represented in the form of a feature map; The first size coefficient represents the probability that each position in the feature map including the first fused feature belongs to an object within a target size range; The second size coefficient represents the probability that each position in the feature map including the first image feature belongs to an object of a target size range.

5. The method according to claim 1, characterized in that The acquiring a first fusion feature according to the point cloud data and the image data comprises: Based on the projection relationship from the point cloud to the image, determining a first projection position of the point cloud data in the two-dimensional space; Acquire a pixel value at the first projection position in the image data; Generate point cloud data including the pixel value according to the pixel value and the point cloud data; The first fusion feature is obtained according to the point cloud data including the pixel value and the image data.

6. The method according to claim 5, characterized in that The projection relationship from the point cloud to the image is obtained based on the external parameters of the point cloud coordinate system to the camera coordinate system and the internal parameters of the camera.

7. The method according to claim 1, characterized in that The acquiring a first fusion feature according to the point cloud data and the image data comprises: Splicing the point cloud data and the image data to obtain spliced ​​data and inputting the spliced ​​data into an object recognition model; The first fusion feature is obtained by performing feature extraction on the splicing information through a first feature extraction network in the object recognition model.

8. The method according to claim 1, characterized in that The first image feature is obtained by performing feature extraction on the image data.

9. The method according to claim 1 or 8, characterized in that: The acquiring a first image feature according to the image data comprises: The image data is input into an object recognition model, and features are extracted from the image data through a second feature extraction network in the object recognition model to obtain the first image features.

10. The method according to claim 1, characterized in that The first size coefficient is obtained from the first fused feature by a first size coefficient extraction network in the object recognition model; And, the second size coefficient is obtained from the first image feature through a second size coefficient extraction network in the object recognition model.

11. The method according to claim 10, characterized in that The first size coefficient extraction network and the second size coefficient extraction network both include convolutional layers.

12. The method according to claim 10 or 11, characterized in that: The first size coefficient is obtained by performing a convolution operation on the first fusion feature through the first size coefficient extraction network; And, the second size coefficient is obtained by performing a convolution operation on the first image feature through the second size coefficient extraction network.

13. The method according to claim 10, characterized in that The loss function used by the object recognition model during training includes at least: a first loss function for optimizing the first size coefficient extraction network and a second loss function for optimizing the second size coefficient extraction network.

14. The method according to claim 13, characterized in that The optimization objective of the first loss function includes: if the sample data indicates an object belonging to a target size range, increasing the first size coefficient; the sample data includes point cloud sample data and image sample data; And, the optimization objective of the second loss function includes: if the image sample data indicates an object belonging to a target size range, increasing the second size coefficient.

15. The method according to claim 13 or 14, characterized in that The first loss function is used to describe the difference between the confidence prediction value of the object belonging to the target size range obtained by the object recognition model from the sample data and the true confidence value of the object belonging to the target size range corresponding to the sample data; the sample data includes point cloud sample data and image sample data; The second loss function is used to describe the difference between the confidence prediction value of the object belonging to the target size range obtained by the object recognition model from the image sample data and the true confidence value of the object belonging to the target size range corresponding to the image sample data.

16. The method according to claim 1, characterized in that The second fusion feature is the sum of the first fusion feature and the first size coefficient; And / or, the second image feature is the sum of the first image feature and the second size coefficient.

17. The method according to claim 1, characterized in that The identifying the object to be identified according to the second fusion feature and the second image feature includes: Based on the projection relationship from the point cloud to the image, determining a second projection position of the second fused feature in the two-dimensional space; Acquire image features at the second projection position in the second image features; A third fusion feature is obtained according to the image feature and the corresponding second fusion feature, and the object to be identified is identified according to the third fusion feature.

18. The method according to claim 17, characterized in that The third fusion feature is the average of the image feature and the corresponding second fusion feature; Alternatively, the third fusion feature is the larger one between the image feature and the corresponding second fusion feature.

19. The method according to claim 1, characterized in that Also includes: Identify the object to be identified according to the second fusion feature and the second image feature, and generate a recognition result; The recognition result includes at least: the confidence and / or status information of the object to be recognized; The confidence level is used to represent the probability that the object to be identified is an obstacle.

20. The method according to claim 19, characterized in that The state information includes at least one of the following: size information, position information, and orientation information.

21. The method according to claim 19, characterized in that The confidence and state information of the object to be identified are obtained by processing the second fusion feature and the second image feature using an object prediction network in an object recognition model.

22. The method according to claim 21, characterized in that The loss function used by the object recognition model during training includes at least a third loss function for describing state differences and a fourth loss function for describing confidence differences; The state difference includes: the difference between the predicted size, predicted position and / or predicted orientation of the object obtained by the object recognition model from the sample data and the real size, real position and / or real orientation of the object corresponding to the sample data; the sample data includes point cloud sample data and image sample data; The confidence difference includes: the difference between the confidence prediction value of the object obtained by the object recognition model from the sample data and the true confidence value of the object corresponding to the sample data.

23. The method according to claim 19, characterized in that The recognition result is data with a confidence level greater than a preset threshold.

24. The method according to claim 1, characterized in that Also includes: The object to be identified is identified according to the second fusion feature and the second image feature to generate an identification result, and the identification result is used to make obstacle avoidance decisions or mobile route planning for the movable platform.

25. The method according to claim 1, characterized in that Also includes: identifying the object to be identified according to the second fusion feature and the second image feature to generate an identification result; The recognition result is displayed on an interface of the mobile platform or an interface of a terminal device that is communicatively connected to the mobile platform.

26. The method according to claim 1, characterized in that The point cloud data is acquired by using a laser radar configured on a movable platform or a photographing device with a depth information acquisition function; and / or, the image data is acquired by using a photographing device configured on a movable platform.

27. The method according to claim 26, characterized in that The movable platform includes: an unmanned aerial vehicle, a car, an unmanned ship or a robot.

28. An object recognition device, characterized in that: comprising a processor and a memory storing a computer program; When executing the computer program, the processor implements the following steps: Acquire point cloud data and image data of the object to be identified, and acquire a first fusion feature according to the point cloud data and the image data; and acquire a first image feature according to the image data; Acquire a first size coefficient according to the first fusion feature; and acquire a second size coefficient according to the first image feature; the first size coefficient and the second size coefficient represent the probability that the object to be identified belongs to an object within a target size range; Acquire a second fusion feature according to the first fusion feature and the first size coefficient; and acquire a second image feature according to the first image feature and the second size coefficient; The object to be identified is identified according to the second fusion feature and the second image feature.

29. The object recognition device according to claim 28, characterized in that: The point cloud data and image data of the object to be identified are obtained by spatial sampling by the sensor of the movable platform.

30. The object recognition device according to claim 28, characterized in that: The processor is also used to obtain a first fusion feature according to the point cloud density of each voxel in the point cloud data and the pixel value in the image data; wherein the voxel is obtained by gridding the point cloud data.

31. The object recognition device according to claim 28, characterized in that: The first fusion feature and the first image feature are represented in the form of a feature map; The first size coefficient represents the probability that each position in the feature map including the first fused feature belongs to an object within a target size range; The second size coefficient represents the probability that each position in the feature map including the first image feature belongs to an object of a target size range.

32. The object recognition device according to claim 28, characterized in that: The processor is further configured to: Based on the projection relationship from the point cloud to the image, determining a first projection position of the point cloud data in the two-dimensional space; Acquire a pixel value at the first projection position in the image data; Generate point cloud data including the pixel value according to the pixel value and the point cloud data; The first fusion feature is obtained according to the point cloud data including the pixel value and the image data.

33. The object recognition device according to claim 32, characterized in that: The projection relationship from the point cloud to the image is obtained based on the external parameters of the point cloud coordinate system to the camera coordinate system and the internal parameters of the camera.

34. The object recognition device according to claim 28, characterized in that: The processor is further configured to: Splicing the point cloud data and the image data to obtain spliced ​​data and inputting the spliced ​​data into an object recognition model; The first fusion feature is obtained by performing feature extraction on the splicing information through a first feature extraction network in the object recognition model.

35. The object recognition device according to claim 28, characterized in that: The first image feature is obtained by performing feature extraction on the image data.

36. The object recognition device according to claim 28 or 35, characterized in that: The processor is also used to: input the image data into an object recognition model, perform feature extraction on the image data through a second feature extraction network in the object recognition model, and obtain the first image feature.

37. The object recognition device according to claim 28, characterized in that: The first size coefficient is obtained from the first fused feature by a first size coefficient extraction network in the object recognition model; And, the second size coefficient is obtained from the first image feature through a second size coefficient extraction network in the object recognition model.

38. The object recognition device according to claim 37, characterized in that: The first size coefficient extraction network and the second size coefficient extraction network both include convolutional layers.

39. The object recognition device according to claim 37 or 38, characterized in that: The first size coefficient is obtained by performing a convolution operation on the first fusion feature through the first size coefficient extraction network; And, the second size coefficient is obtained by performing a convolution operation on the first image feature through the second size coefficient extraction network.

40. The object recognition device according to claim 37, characterized in that: The loss function used by the object recognition model during training includes at least: a first loss function for optimizing the first size coefficient extraction network and a second loss function for optimizing the second size coefficient extraction network.

41. The object recognition device according to claim 40, characterized in that: The optimization objective of the first loss function includes: if the sample data indicates an object belonging to a target size range, increasing the first size coefficient; the sample data includes point cloud sample data and image sample data; And, the optimization objective of the second loss function includes: if the image sample data indicates an object belonging to a target size range, increasing the second size coefficient.

42. The object recognition device according to claim 40 or 41, characterized in that: The first loss function is used to describe the difference between the confidence prediction value of the object belonging to the target size range obtained by the object recognition model from the sample data and the true confidence value of the object belonging to the target size range corresponding to the sample data; the sample data includes point cloud sample data and image sample data; The second loss function is used to describe the difference between the confidence prediction value of the object belonging to the target size range obtained by the object recognition model from the image sample data and the true confidence value of the object belonging to the target size range corresponding to the image sample data.

43. The object recognition device according to claim 28, characterized in that: The second fusion feature is the sum of the first fusion feature and the first size coefficient; And / or, the second image feature is the sum of the first image feature and the second size coefficient.

44. The object recognition device according to claim 28, characterized in that: The processor is further configured to: Based on the projection relationship from the point cloud to the image, determining a second projection position of the second fused feature in the two-dimensional space; Acquire image features at the second projection position in the second image features; A third fusion feature is obtained according to the image feature and the corresponding second fusion feature, and the object to be identified is identified according to the third fusion feature.

45. The object recognition device according to claim 44, characterized in that: The third fusion feature is the average of the image feature and the corresponding second fusion feature; Alternatively, the third fusion feature is the larger one between the image feature and the corresponding second fusion feature.

46. ​​The object recognition device according to claim 28, characterized in that: The processor is further configured to: identify the object to be identified according to the second fusion feature and the second image feature, and generate an identification result; The recognition result at least includes: the confidence and / or state information of the object to be recognized; the confidence is used to characterize the probability that the object to be recognized is an obstacle.

47. The object recognition device according to claim 46, characterized in that: The state information includes at least one of the following: size information, position information, and orientation information.

48. The object recognition device according to claim 46, characterized in that: The confidence and state information of the object to be identified are obtained by processing the second fusion feature and the second image feature using an object prediction network in an object recognition model.

49. The object recognition device according to claim 48, characterized in that: The loss function used by the object recognition model during training includes at least a third loss function for describing state differences and a fourth loss function for describing confidence differences; The state difference includes: the difference between the predicted size, predicted position and / or predicted orientation of the object obtained by the object recognition model from the sample data and the real size, real position and / or real orientation of the object corresponding to the sample data; the sample data includes point cloud sample data and image sample data; The confidence difference includes: the difference between the confidence prediction value of the object obtained by the object recognition model from the sample data and the true confidence value of the object corresponding to the sample data.

50. The object recognition device according to claim 46, characterized in that: The recognition result is data with a confidence level greater than a preset threshold.

51. The object recognition device according to claim 28, characterized in that: The processor is further used to: identify the object to be identified according to the second fusion feature and the second image feature to generate a recognition result, and the recognition result is used to make obstacle avoidance decisions or mobile route planning for the movable platform.

52. The object recognition device according to claim 28, characterized in that: The processor is also used to: identify the object to be identified according to the second fusion feature and the second image feature to generate an identification result; the identification result is used to display on the interface of the movable platform or the interface of a terminal device that is communicatively connected to the movable platform.

53. The object recognition device according to claim 28, characterized in that: The point cloud data is acquired by using a laser radar configured on a movable platform or a photographing device with a depth information acquisition function; and / or, the image data is acquired by using a photographing device configured on a movable platform.

54. The object recognition device according to claim 53, characterized in that: The movable platform includes: an unmanned aerial vehicle, a car, an unmanned ship or a robot.

55. A movable platform, characterized in that: include: Body; A power system, installed in the body, for providing power to the movable platform; as well as, An object recognition device as claimed in any one of claims 28 to 54.

56. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 27 is implemented.

Images