Object recognition method for automatic driving, electronic equipment and storage medium

Abstract

The invention provides an object recognition method for automatic driving, and the method comprises the steps: obtaining image features from an image sample, and splicing the image features and text features into multi-modal features; performing cross-correlation operation on the multi-modal features serving as convolution kernels and visual features of the multi-view images to obtain correlation features; predicting a target initial position feature of the entity object in the 3D space according to the correlation feature, and performing feature interaction on the image feature and the correlation feature in combination with the target initial position feature to obtain an object feature corresponding to the entity object; and training an object recognition model according to the object features corresponding to the different entity objects recognized in the plurality of image samples. According to the embodiment of the invention, the features related to the target are enhanced, and the response of unrelated backgrounds is reduced, so that the perception precision of the shielded target is improved, the high efficiency of the model is ensured, and the problem that an existing 3D target detection scheme excessively pursues comprehensive detection performance and ignores robust perception under the shielding condition is solved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to an object recognition method and device, electronic device, and storage medium for autonomous driving. Background Art

[0002] With the development of artificial intelligence technology and modern manufacturing, autonomous driving technology has gradually entered people's daily lives, subtly changing people's travel methods. Autonomous driving technology has great application prospects in both military and civilian applications. In military applications, it can not only be used as an unmanned transportation platform, but also as an unmanned blasting vehicle, unmanned combat platform, unmanned patrol and surveillance vehicle, etc. In civilian applications, in addition to bringing convenience to human life, it can also reduce the incidence of traffic accidents and improve road traffic efficiency, but it also causes a sharp increase in energy consumption and frequent traffic accidents. The update and iteration of autonomous driving technology is the key to solving the above problems. Therefore, autonomous driving technology has become the focus of attention in the automotive industry and academia.

[0003] In the algorithm system of autonomous driving, the position, size, direction and other information of objects in 3D space are perceived to provide basic information input for downstream prediction and regulation, which is crucial for the intelligence of cars and robots. Considering the high hardware construction cost of radar sensors, the more economical surround vision 3D target detection is widely used in the perception modules of technical products such as autonomous driving, assisted driving and robots. After many technical iterations, the current visual 3D detection perception accuracy is continuously improved, which can provide more accurate spatial position information of surrounding targets for downstream tasks.

[0004] However, there are many kinds of interference information in actual scenes, which may cause extreme cases that are critical to autonomous driving safety but difficult to detect accurately, such as occluded targets. Due to the lack of partial area information of the target, existing 3D target detection methods have difficulty perceiving such targets, which in turn affects downstream prediction and planning schemes, and ultimately leads to safety issues. Therefore, it is necessary to design a perception framework that enhances occluded target detection and reduces the safety hazards caused by occluded targets.

[0005] The key to occluded target detection is the loss of target information caused by occlusion, so how to identify occluded areas and perform targeted information completion is the biggest challenge. In addition, since occlusion will reduce the quality of the visual representation of the object, compared with unoccluded targets, there are fewer available discriminative features and lack of sufficient contextual clues. At the same time, considering the algorithm iteration efficiency and inference latency, how to quickly complete the perception of occluded targets and respond to the input required by downstream tasks in a timely manner is also a core factor in achieving a robust and efficient visual 3D target detection framework. Summary of the invention

[0006] The embodiments of the present application provide an object recognition method, device, electronic device and storage medium for autonomous driving to partially or completely solve the above-mentioned technical problems.

[0007] In a first aspect, an embodiment of the present application provides an object recognition method for autonomous driving, comprising:

[0008] Cut out an image region corresponding to the entity object from the image sample, and splice the image features corresponding to the image region and the text features of the object category of the entity object into a multimodal feature corresponding to the image sample;

[0009] The multimodal features of the acquired image samples are used as convolution kernels, and cross-correlation operations are performed with the visual features of the multi-view images perceived in autonomous driving to obtain correlation features that enhance target perception.

[0010] Predicting target initial position features of the entity object in the 3D space according to the correlation features, performing feature interaction on image features and correlation features corresponding to the image area in combination with the target initial position features, to obtain object features corresponding to the entity object;

[0011] The object recognition model is trained according to the object features corresponding to different entity objects recognized in multiple image samples.

[0012] In a second aspect, an object recognition device for autonomous driving includes:

[0013] A multimodal feature splicing module, used to crop an image region corresponding to an entity object from an image sample, and splice the image features corresponding to the image region and the text features of the object category of the entity object into a multimodal feature corresponding to the image sample;

[0014] A correlation feature acquisition module is used to use the multimodal features of the acquired image samples as convolution kernels to perform cross-correlation operations with the visual features of the multi-view images perceived in autonomous driving to obtain correlation features for enhanced target perception;

[0015] An object feature acquisition module is used to predict the target initial position feature of the entity object in the 3D space according to the correlation feature, and perform feature interaction on the image feature and the correlation feature corresponding to the image area in combination with the target initial position feature to obtain the object feature corresponding to the entity object;

[0016] The training module is used to train the object recognition model according to the object features corresponding to different entity objects recognized in multiple image samples.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the above methods when executing the computer program.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the above is implemented.

[0019] Compared with the prior art, this application has the following advantages:

[0020] According to this embodiment, the image area corresponding to the entity object is first cropped from the image sample, and then the image features corresponding to the image area and the text features of the object category of the entity object are spliced ​​into the multimodal features corresponding to the image sample, and then the multimodal features are used as the convolution kernel to perform cross-correlation operations with the visual features of the multi-view images perceived in autonomous driving to obtain correlation features for enhanced target perception, and then the target initial position features of the entity object in the 3D space are predicted based on the correlation features, and the image features and correlation features corresponding to the image distinction are interacted with each other in combination with the target initial position features to obtain the object features corresponding to the entity object, and finally the object recognition model is trained based on the object features corresponding to different entity objects identified in multiple image samples. The above scheme can not only introduce auxiliary visual and language multimodal large models to provide more prior information for the perception of occluded targets, solve the problem of insufficient context clues when detecting occluded objects and thus being unable to accurately identify targets, but also introduce relevant learning enhancement features to solve the problem that 3D target detection schemes overly pursue comprehensive detection performance and ignore robust perception under occlusion, ensure the efficiency of the model, enhance the feature representation of targets (especially partially occluded objects), and enhance the detection performance of occluded targets. In addition, the representation quality of unoccluded targets is further enhanced, improving the accuracy and robustness of overall perception.

[0021] Among them, the image encoder in the multimodal large model extracts the image features corresponding to each image area to supplement and enhance the visual features of the subsequent detection of occluded objects. The text encoder of the multimodal large model extracts the text features of the object category of image samples in different visibility intervals, providing humans with prior knowledge of different categories of targets to supplement the information of occluded objects from another dimension to identify the target. By supplementing the prior knowledge level of the two dimensions of image and text, it is equivalent to supplementing the contextual clues of the occluded objects, which can not only enhance the target-related features lost due to occlusion, but also further enhance the representation quality of unoccluded targets.

[0022] In addition, after constructing the multimodal features, this method uses a more computationally efficient cross-correlation operation to integrate the prior knowledge of multimodal features into the learned multi-view features. Through the cross-correlation operation, the model utilizes the prior knowledge of multimodal features to enhance the features related to the target while reducing the response of irrelevant background, thereby improving the perception accuracy of occluded targets and ensuring the efficiency of the model.

[0023] Finally, this method uses a pseudo-occlusion processing method. By selecting physical objects whose visibility exceeds the set threshold from the image area cropped from the image samples, the physical objects whose visibility exceeds the set threshold are occluded, which increases the number of training occluded targets and balances the visibility distribution of objects in the training data. This solves the problem that there are few occluded targets in the scene and it is difficult to fully train the model for this situation, enhances the perception ability of occluded targets, and makes the model more robust in perception.

[0024] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application and should not be regarded as limiting the scope of the present application.

[0026] Figure 1 A schematic diagram of an application scenario of the method provided in this application;

[0027] Figure 2 A flowchart of an object recognition method for autonomous driving according to an embodiment of the present application;

[0028] Figure 3 is a structural block diagram of an object recognition device for autonomous driving according to an embodiment of the present application; and

[0029] Figure 4 A block diagram of an electronic device used to implement an embodiment of the present application. DETAILED DESCRIPTION

[0030] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the concept or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary in nature and not restrictive.

[0031] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0032] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, without paying creative work, this specification can also be applied to other similar scenarios based on these drawings. Unless it is obvious from the voice environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0033] It should be understood that the "system", "device", "unit" and / or "module" used herein are a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0034] As shown in this specification and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0035] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0036] First of all, before specifically introducing the technical solution of the embodiment of the present disclosure, the technical background or technical evolution context on which the embodiment of the present disclosure is based is introduced. Generally speaking, in the field of autonomous driving, most of the existing 3D target detection solutions are committed to improving the overall average perception accuracy, ignoring the study of extreme cases such as occlusion detection. For example, the existing evaluation indicators mean translation error (mATE), mean scale error (mASE), mean orientation error (mAOE), and mean average precision (mAP) will average all scenes and various categories of targets in the scenes. This approach cannot locate the impact of a few extreme cases. For example, failure to detect partially obscured pedestrians near the autonomous driving car may have little impact on the average performance, but this may lead to erroneous downstream predictions and planning controls, thereby endangering the safety of autonomous driving.

[0037] For occluded target detection, existing solutions generally consider more historical frame information and use the target state when it is not occluded to assist the current frame in detection. However, more input data will increase the complexity and computational cost of the perception model, affecting actual deployment and rapid response. At the same time, due to the rapid changes in actual scenes, it is difficult to accurately match the appearance information of the corresponding target in the current frame and the historical frame. In addition, if the target is occluded from the beginning (such as a ghost), this time-series fusion solution usually does not work.

[0038] Based on the above background, the applicant provides an object recognition method for autonomous driving through the embodiments of the present application to solve all or part of the above technical problems.

[0039] The following introduces the technical solutions involved in the embodiments disclosed in this application in combination with the scenarios to which the embodiments disclosed in this application are applied.

[0040] Figure 1 The figure is a schematic diagram of an exemplary application scenario for implementing the method of the embodiment of the present application.

[0041] like Figure 1As shown, image samples are collected through multiple cameras facing different positions at multiple positions around the autonomous driving car; the image area corresponding to the physical object is cropped from the image sample, and the image features corresponding to the image area are extracted through an image encoder, and the image samples are divided into different visibility intervals according to the visibility of the image samples; the text category features of the image text in different visibility intervals are extracted using a text encoder of a multimodal large model; the text features and image features of the image samples in the same visibility interval are spliced ​​into corresponding multimodal features; the multimedia features of the image samples are used as convolution kernels to be cross-correlated with the visual features of the multi-view images perceived in the autonomous driving operation to obtain correlation features for enhanced target perception; predict a heat map representing the position of the entity object based on the correlation features to obtain the initialization features of the heat map, perform weighted fusion of the initialization features with the original random initialization features of the model to obtain the target initial position features of the entity object in the 3D space, perform feature interaction on the image features and correlation features corresponding to the image area in combination with the target initial position features to obtain the object features corresponding to the entity object; train an object recognition model based on the object features corresponding to different entity objects identified in multiple image samples, construct a contrast loss function, and use the contrast loss function and the entity object corresponding to the image area where the image sample is cropped to train a contrast enhancement model.

[0042] In the embodiment of the present application, the input of the model is a surround multi-camera image input generated by multiple cameras around the autonomous vehicle; the output of the model is the category information of the targets (vehicles, pedestrians, etc.) around the vehicle in the detected scene and their spatial position, orientation, size, and speed.

[0043] For the above information of non-self-vehicle objects, the detected non-self-vehicle dynamic objects can be annotated through the detection frame. For example, the information detected by the detection frame includes spatial position, orientation, size, and speed, wherein the spatial position reflects the position of the dynamic object, and the coordinate system may include any one or more of the world coordinate system, camera coordinate system, image coordinate system, and pixel coordinate system of the detection frame. Size refers to the length and width of the detection frame, which reflects the size of the dynamic object. Orientation refers to the moving direction of the detection frame, which reflects the movement direction of the dynamic object. For example, when the dynamic object is stationary, the orientation of the detection frame may be the direction in which the front of the dynamic object (such as the front of a vehicle, the chest of a pedestrian / animal) is facing. Time is the time corresponding to when the dynamic object is at the current position or speed, or the time when the vehicle detects that the dynamic object is in the current state. The speed can be calculated by the distance and time traveled in the virtual detection area.

[0044] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The several specific embodiments listed can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0045] like Figure 2 The flowchart of the object recognition method for autonomous driving according to an embodiment of the present application may include:

[0046] Step S201 , cropping an image region corresponding to an entity object from an image sample, and concatenating image features corresponding to the image region and text features of an object category of the entity object into a multimodal feature corresponding to the image sample.

[0047] Image samples are obtained from training datasets and test datasets in pre-collected open source datasets with training value. They include pictures taken by on-board cameras on autonomous vehicles and object categories of marked entity objects, such as pictures of trucks, cars, buses, trailers, etc. and text labels such as "car", "trailer", "bus", and "truck". Among them, the image area corresponding to the entity object is cropped according to the category of the object entity, that is, the area of ​​the object that needs attention is found and cropped, in order to reduce the time, computing resources and interference factors for processing irrelevant backgrounds, improve recognition accuracy and processing speed, and facilitate the extraction of subsequent image encoders, so that the multimodal features used as the combination features of image and text prior knowledge are more standardized. In order to better characterize the degree of occlusion of the entity objects in the image area, the image samples can be further marked with visibility. In an optional implementation, the image encoder of the multimodal large model can be used to extract the image features corresponding to each image area, and the image samples can be divided into different visibility intervals according to the visibility of the image samples. The visibility of the image samples characterizes the degree of occlusion of the entity objects in the image area; the text encoder of the multimodal large model is used to extract the text features of the object category of the image samples in different visibility intervals; the text features and image features of the image samples in the same visibility interval are spliced ​​into corresponding multimodal features.

[0048] The image feature corresponding to the image region is expressed as Where K represents the number of samples of each category of entity objects, N represents the number of categories, and D represents the feature dimension of the original information. Because the image samples contain the marked visibility, the image samples are divided into image features corresponding to each category of entity objects in different visibility intervals based on the visibility annotation. The image features corresponding to each category of entity objects are expressed as Where M is four different visibilities;

[0049] Then, the text encoder of the multimodal large model extracts the text features of the object categories of the image samples in different visibility intervals as a supplement to the prior knowledge level of the text dimension. In an optional real-time manner, the text features of the object categories of the image samples in different visibility intervals are represented as

[0050] In an optional embodiment, the text features and image features of image samples in the same visibility interval are fused through concatenation and linear mapping operations to obtain an image-text multimodal prototype, i.e., a multimodal feature. The multimodal feature is expressed as

[0051] It can be understood that the multimodal features obtained in this step contain the prior knowledge for processing autonomous driving occlusion detection. After obtaining the multimodal features, the prior knowledge of the multimodal features is integrated into the learned multi-view features, which can solve the problem of insufficient contextual clues in the current detection of occluded objects and difficulty in identifying targets.

[0052] In step S202, the multimodal features of the acquired image samples are used as convolution kernels to perform cross-correlation operations with visual features of multi-view images perceived during autonomous driving to obtain correlation features for enhanced target perception.

[0053] In the embodiment of the present application, multi-view features can be obtained and cross-correlated with multi-modal features, so that the model can use the prior knowledge of multi-modal features to enhance the features related to the target, while reducing the response of irrelevant background, thereby improving the perception accuracy of occluded targets. Among them, multi-view images refer to a collection of images of the same scene taken from different angles and different positions. Specifically, there are multiple cameras with different positions and orientations at multiple positions around the self-driving car, including a front camera, a left front camera, a right front camera, a rear camera, a left rear camera, and a right rear camera. The image inputs of the above cameras constitute a surround multi-camera image. The learned multi-view features can be obtained from the surround multi-camera image, which is expressed as Where H represents the length of the image, W represents the width of the image, and D represents the dimension of the original information.

[0054] In an optional embodiment, since the multimodal features of the image samples have been obtained before, their feature dimension is D; and because the multi-view features have been learned, their feature dimension is also D, each multimodal feature in the image sample set is used to perform a 1×1 convolution operation with the multi-view features, and the output is reduced in dimension to obtain the correlation feature with enhanced target perception, which is expressed as Because the characteristics of the target are contained in the multimodal features, the features related to the target can be enhanced while reducing the response of irrelevant background, thereby improving the perception accuracy of the occluded target. This process can be formulated as:

[0055] F corr =Conv 1×1 (P,F b )

[0056] In an optional embodiment, the multi-view features are converted into Bird's Eye View (BEV) features.

[0057] Step S203, predicting the target initial position feature of the entity object in the 3D space according to the correlation feature, performing feature interaction on the image feature and the correlation feature corresponding to the image area in combination with the target initial position feature, and obtaining the object feature corresponding to the entity object.

[0058] In an embodiment of the present application, taking into account the prior knowledge brought by the correlation features and the multi-dimensional accurate features contained therein, the target initial position features of the entity object in the 3D space are predicted using the correlation features, and the correlation features are used again in the two-way mixed sampling (i.e., the image features corresponding to the image area and the correlation features) to solve the problem of missing image features corresponding to the occluded image area, and obtain the object features corresponding to the entity object that integrates the two features, thereby enhancing the feature learning ability of the entity object.

[0059] In an optional embodiment, predicting the initial position features of the entity object in the 3D space based on the correlation features includes: predicting a heat map representing the position of the entity object based on the correlation features; using the feature corresponding to the highest probability point in the heat map as the initial position feature of the entity object in the 3D space; and weightedly fusing the initial position features with the randomly initialized reference position features according to the weights assigned by the attention module to obtain the target initial position features of the entity object in the 3D space.

[0060] Among them, since the correlation features contain strong discriminative semantic features, they are used to assist in the initialization of the target query, that is, the initial position features of the entity object in the 3D space. The specific steps are as follows: Use the correlation features enhanced by target perception to predict the heat map representing the location of the entity object. The heat map is expressed as Find the feature corresponding to the point with the highest probability in the heat map as the initial position feature of the entity object in 3D space. The initial position feature is expressed as: Where k represents the number of occlusion-aware queries, that is, the number of places where objects may exist. The initial position feature Q is converted into t The original randomly initialized reference position features in the model are fused to obtain the target initial position features of the entity object in 3D space. This process can be formulated as:

[0061]

[0062] Q = CrossAttn (query = Q l ,key=Q t ,value=Q t )

[0063] Among them, (i, j) is the index of the row and column of the local maximum position of the feature map, CrossAttn represents the multi-head attention operation, query and key represent the feature vectors for calculating the multi-head attention weights, and value represents the vector of input features. l Represents the original randomly initialized reference position features in the model.

[0064] In an optional embodiment, feature interaction is performed on image features and correlation features corresponding to the image area in combination with target initial position features to obtain object features corresponding to the entity object, including: sampling the projection position obtained by projecting the target initial position features of the entity object in 3D space onto feature dimensions corresponding to the image features corresponding to the image area and feature dimensions corresponding to the correlation features to obtain 2D plane features of the target position containing image features and 2D plane features of the target position containing correlation features; fusing the 2D plane features of the target position containing image features and the 2D plane features of the target position containing correlation features to obtain 2D plane fusion features; and using the 2D plane fusion features as the object features corresponding to the entity object.

[0065] Among them, since the multi-view features only contain semantic visual features, and the correlation features include enhanced information of the multimodal prototype, the feature interaction adopts two-way mixed sampling, and the correlation features are used again to alleviate the problem of missing original visual features due to occlusion. In other words, the initial position features of the target of the entity object in the 3D space are simultaneously projected to the feature dimensions corresponding to the image features corresponding to the image area and the feature dimensions corresponding to the correlation features, and then sampled, and the features corresponding to the 2D position are extracted to obtain the corresponding 2D plane position. The feature learning ability of the target is enhanced by splicing and fusing the information obtained from the two samplings.

[0066] In an optional embodiment, projecting the position feature of the target object onto the feature dimension corresponding to the image feature corresponding to the image region and the feature dimension corresponding to the correlation feature for sampling further includes: sampling the position containing context information around the projection position. In this way, the context information is obtained by learning the deviation around the projection position, and then the two samples are spliced ​​and fused by deformable convolution.

[0067] The above process can be formulated as:

[0068]

[0069] Where (i, j) represents the row and column index of the sampling position, p ij represents the projected position, Δp ij and Δ′p ij Represents the locations containing contextual information around the projected location.

[0070] Step S204: training an object recognition model according to object features corresponding to different entity objects recognized in the plurality of image samples.

[0071] In order to enhance the generalization ability and improve the robustness, the object recognition model is trained using object features corresponding to different entity objects identified by multiple image samples.

[0072] In an optional embodiment, training an object recognition model based on object features corresponding to different physical objects identified in multiple image samples includes: constructing a contrast loss function, using the contrast loss function and physical objects corresponding to image areas cropped from image samples to train a contrast enhancement model, wherein the training direction of the loss function is to reduce the difference between object features of physical objects of the same object category with different visibilities, and at the same time increase the difference between object features of physical objects of different object categories.

[0073] Considering that the correspondence of visual and semantic features to a feature space can enhance the discriminability of the visual features themselves, contrastive learning is used to enhance the model's ability to distinguish object categories. By constructing a loss function based on mutual information (InfoNCE Loss) to assist contrastive learning supervision, given the positive and negative classification mapping and language embedding of the target query, the model is trained for contrast enhancement using the contrastive loss function and the entity objects corresponding to the image area cropped from the image sample. This training model can bring together the feature representations of objects of the same category with different degrees of occlusion, while expanding the discriminability of different categories. The constructed loss function can be formulated as:

[0074]

[0075] in, represents a given positive classification mapping, Represents a given negative classification map, P l Text features representing the object category of an image sample.

[0076] In an optional embodiment, the method further includes selecting a physical object whose visibility exceeds a set threshold from an image area cropped from the image sample, and performing occlusion processing on the physical object whose visibility exceeds the set threshold; redetermining the visibility corresponding to the image sample after the occlusion processing and using the redetermined visibility to update the visibility of the mark corresponding to the image sample.

[0077] Since the number of occluded targets in actual autonomous driving scenarios is small, it is difficult to fully train the model's robust perception capabilities for such situations, so this embodiment proposes an occlusion perception training strategy. After obtaining the surround-view multi-camera images, the data is enhanced through pseudo-occlusion processing. According to the visibility mark of each target in the data set, objects with higher visibility are randomly occluded and marked by replacing the pixel value with the pixel mean of the area described by the object, so as to achieve the purpose of pseudo-occluding some targets. This step can balance the visibility distribution of objects in the training data, enhance the perception ability of occluded targets, and perform targeted enhancement on the training model samples.

[0078] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides an object recognition device for autonomous driving. Figure 3 The object recognition device for autonomous driving according to an embodiment of the present application is shown. The object recognition device for autonomous driving may include:

[0079] A multimodal feature splicing module 301 is used to crop an image region corresponding to an entity object from an image sample, and splice the image features corresponding to the image region and the text features of the object category of the entity object into a multimodal feature corresponding to the image sample;

[0080] A correlation feature acquisition module 302 is used to use the multimodal features of the acquired image samples as convolution kernels to perform cross-correlation operations with visual features of multi-view images perceived during autonomous driving to obtain correlation features for enhanced target perception;

[0081] The object feature acquisition module 303 is used to predict the target initial position feature of the entity object in the 3D space according to the correlation feature, and perform feature interaction on the image feature and the correlation feature corresponding to the image area in combination with the target initial position feature to obtain the object feature corresponding to the entity object;

[0082] The training module 304 is used to train the object recognition model according to the object features corresponding to different entity objects recognized in the multiple image samples.

[0083] In an optional embodiment, the multimodal feature splicing module 301 includes:

[0084] An image feature extraction submodule, for extracting image features corresponding to each image region using an image encoder of a multimodal large model, and dividing the image samples into different visibility intervals according to their visibility, wherein the visibility of the image samples represents the degree of occlusion of the entity objects in the image region;

[0085] A text feature extraction submodule, for extracting text features of object categories of image samples in different visibility intervals using a text encoder of a multimodal large model;

[0086] The image feature and text feature splicing submodule is used to splice the text features and image features of image samples in the same visibility interval into corresponding multimodal features.

[0087] In an optional embodiment, the object feature acquisition module 303 includes:

[0088] A heat map prediction submodule, used to predict a heat map representing the location of the entity object according to the correlation feature;

[0089] The initial position feature acquisition submodule is used to use the feature corresponding to the point with the highest probability in the heat map as the initial position feature of the entity object in the 3D space;

[0090] The target initial position feature acquisition submodule is used to weightedly fuse the initial position feature with the randomly initialized reference position feature according to the weight assigned by the attention module to obtain the target initial position feature of the entity object in the 3D space.

[0091] In an optional embodiment, the object feature acquisition module 303 includes:

[0092] A 2D plane feature sampling submodule is used to sample the projection position obtained by projecting the target initial position feature of the entity object in the 3D space onto the feature dimension corresponding to the image feature corresponding to the image area and the feature dimension corresponding to the correlation feature, so as to obtain the 2D plane feature of the target position including the image feature and the 2D plane feature of the target position including the correlation feature;

[0093] The 2D plane fusion feature acquisition submodule is used to fuse the 2D plane features of the target position containing image features and the 2D plane features of the target position containing correlation features to obtain 2D plane fusion features; and use the 2D plane fusion features as the object features corresponding to the entity object.

[0094] In an optional embodiment, the object feature acquisition module 303 further includes: sampling positions containing context information around the projection position.

[0095] In an optional embodiment, the training module 304 includes: constructing a contrast loss function, using the contrast loss function and the entity object corresponding to the image area cropped from the image sample to train a contrast enhancement model, wherein the training direction of the loss function is to reduce the difference between object features of entity objects of the same object category with different visibilities, and at the same time increase the difference between object features of entity objects of different object categories.

[0096] In an optional embodiment, the object recognition device for autonomous driving further includes:

[0097] The pseudo occlusion module is used to select entity objects whose visibility exceeds a set threshold from the image area cropped by the image sample, and perform occlusion processing on the entity objects whose visibility exceeds the set threshold; re-determine the visibility corresponding to the image sample after the occlusion processing and use the re-determined visibility to update the visibility of the corresponding mark of the image sample.

[0098] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0099] Figure 4 FIG. 1 is a block diagram of an electronic device used to implement an embodiment of the present application. Figure 4 As shown, the electronic device includes: a memory 410 and a processor 420. The memory 410 stores a computer program that can be run on the processor 420. When the processor 420 executes the computer program, the method in the above embodiment is implemented. The number of the memory 410 and the processor 420 can be one or more.

[0100] The electronic device also includes:

[0101] The communication interface 430 is used to communicate with external devices and perform data exchange transmission.

[0102] If the memory 410, the processor 420 and the communication interface 430 are implemented independently, the memory 410, the processor 420 and the communication interface 430 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0103] Optionally, in a specific implementation, if the memory 410, the processor 420 and the communication interface 430 are integrated on a chip, the memory 410, the processor 420 and the communication interface 430 can communicate with each other through an internal interface.

[0104] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0105] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0106] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0107] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0108] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting description, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0109] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0110] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0111] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0112] Any process or method described in the flow chart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0113] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or used in combination with these instruction execution systems, devices or apparatuses.

[0114] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0115] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0116] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope recorded in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for object recognition for autonomous driving, characterized in that: The method comprises: Cut out an image region corresponding to the entity object from the image sample, and splice the image features corresponding to the image region and the text features of the object category of the entity object into a multimodal feature corresponding to the image sample; The multimodal features of the acquired image samples are used as convolution kernels, and cross-correlation operations are performed with the visual features of the multi-view images perceived in autonomous driving to obtain correlation features that enhance target perception. Predicting target initial position features of the entity object in the 3D space according to the correlation features, performing feature interaction on image features and correlation features corresponding to the image area in combination with the target initial position features, to obtain object features corresponding to the entity object; The object recognition model is trained according to the object features corresponding to different entity objects recognized in multiple image samples.

2. The method according to claim 1, characterized in that The step of splicing the image features corresponding to the image region and the text features of the object category of the entity object into the multimodal features corresponding to the image sample includes: Using an image encoder of a multimodal large model to extract image features corresponding to each image region, the image samples are divided into different visibility intervals according to the visibility of the image samples, and the visibility of the image samples represents the degree of occlusion of the entity object in the image region; Use a multimodal large model text encoder to extract text features of object categories of image samples in different visibility intervals; The text features and image features of image samples in the same visibility range are concatenated into corresponding multimodal features.

3. The method according to claim 1, characterized in that The predicting of the target initial position feature of the entity object in the 3D space according to the correlation feature comprises: Predicting a heat map representing a location of the entity object according to the correlation feature; The feature corresponding to the point with the highest probability in the heat map is used as the initial position feature of the entity object in the 3D space; The initial position feature and the randomly initialized reference position feature are weightedly fused according to the weights assigned by the attention module to obtain the target initial position feature of the entity object in the 3D space.

4. The method according to claim 1, characterized in that: The performing feature interaction on the image features and the correlation features corresponding to the image region in combination with the target initial position features to obtain the object features corresponding to the entity object comprises: Sampling the projection position obtained by projecting the target initial position feature of the entity object in the 3D space onto the feature dimension corresponding to the image feature corresponding to the image area and the feature dimension corresponding to the correlation feature, to obtain a 2D plane feature of the target position including the image feature and a 2D plane feature of the target position including the correlation feature; Fusing the 2D plane feature of the target position containing the image feature and the 2D plane feature of the target position containing the correlation feature to obtain a 2D plane fusion feature; The 2D plane fusion feature is used as the object feature corresponding to the entity object.

5. The method according to claim 4, characterized in that The sampling of the projection position obtained by projecting the target initial position feature of the entity object in the 3D space onto the feature dimension corresponding to the image feature corresponding to the image area and the feature dimension corresponding to the correlation feature also includes: Positions containing context information around the projected position are sampled.

6. The method according to claim 1, characterized in that The training of the object recognition model according to the object features corresponding to the different entity objects recognized in the multiple image samples includes: A contrast loss function is constructed, and a contrast enhancement model is trained using the contrast loss function and the entity objects corresponding to the image area cropped from the image sample, wherein the training direction of the loss function is to reduce the difference between the object features of entity objects of the same object category but with different visibilities, while increasing the difference between the object features of entity objects of different object categories.

7. The method according to claim 1, characterized in that The method further comprises: Selecting entity objects whose visibility exceeds a set threshold from the image area cropped by the image sample, and performing occlusion processing on the entity objects whose visibility exceeds the set threshold; The visibility corresponding to the image sample after the occlusion processing is re-determined, and the visibility of the mark corresponding to the image sample is updated using the re-determined visibility.

8. An object recognition device for autonomous driving, characterized in that: The device comprises: A multimodal feature splicing module, used to crop an image region corresponding to an entity object from an image sample, and splice the image features corresponding to the image region and the text features of the object category of the entity object into a multimodal feature corresponding to the image sample; A correlation feature acquisition module is used to use the multimodal features of the acquired image samples as convolution kernels to perform cross-correlation operations with the visual features of the multi-view images perceived in autonomous driving to obtain correlation features for enhanced target perception; An object feature acquisition module is used to predict the target initial position feature of the entity object in the 3D space according to the correlation feature, and perform feature interaction on the image feature and the correlation feature corresponding to the image area in combination with the target initial position feature to obtain the object feature corresponding to the entity object; The training module is used to train the object recognition model according to the object features corresponding to different entity objects recognized in multiple image samples.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images