Target detection method and device, vehicle control unit and readable storage medium

Abstract

The embodiment of the invention provides a target detection method and device, a vehicle control unit and a readable storage medium, and relates to the technical field of target detection, and the method comprises the steps: obtaining a first feature corresponding to a current environment image, and second features corresponding to a plurality of historical environment images; aligning the content of the second feature with the content of the first feature to obtain a correction feature corresponding to the second feature; fusing the first feature with each correction feature to obtain a target feature; and performing target detection based on the target features to obtain target object information. Since the current shielded obstacle may not be shielded at the historical moment, the target feature may include the feature information of the current shielded obstacle in the historical environment image, and the target object information obtained by performing target detection on the target feature may include the information of the current shielded obstacle. The accuracy of target detection on a shielded obstacle can be improved.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a target detection method, device, vehicle controller and readable storage medium. Background Art

[0002] With the advancement of technology, more and more vehicles are equipped with autonomous driving and assisted driving functions. During the process of autonomous driving and assisted driving, it is necessary to identify obstacles around the vehicle (such as pedestrians, other vehicles, etc.), including some obscured obstacles.

[0003] In related technologies, the first method usually adopted is to construct specific data sets for occlusion scenes, such as the KITTI data set, the Caltech data set, the CityPersons data set, the VehicleOcclusion data set, and the CrowdHuman data set, and then train the target detection model through these data sets so that the target detection model can better identify the occluded obstacles. The second method is to optimize the convolutional neural network algorithm, such as using stacked transformers and multi-task perception modules to enhance the convolutional neural network algorithm, and improve the target detection model's ability to identify occluded obstacles based on the target detection model trained by the enhanced convolutional neural network algorithm.

[0004] However, in the related technologies, obstacle detection depends on the currently acquired environmental image. For obstacles that are severely obscured in the environmental image, it is still difficult to accurately identify them using the above method, resulting in poor recognition ability for obscured obstacles. Summary of the invention

[0005] In view of this, the present invention aims to propose a target detection method, device, vehicle controller and readable storage medium to solve the problem of poor accuracy in detecting obscured obstacles in the related art.

[0006] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0007] A target detection method, the method comprising:

[0008] Obtaining a first feature corresponding to the current environment image and second features corresponding to a plurality of historical environment images respectively;

[0009] Aligning the content of the second feature with the content of the first feature to obtain a correction feature corresponding to the second feature;

[0010] Fusion the first feature with each of the correction features to obtain a target feature;

[0011] Target detection is performed based on the target features to obtain target object information.

[0012] A target detection device, comprising:

[0013] An acquisition module, used to acquire a first feature corresponding to a current environment image, and second features corresponding to a plurality of historical environment images respectively;

[0014] an alignment module, used to align the content of the second feature with the content of the first feature to obtain a correction feature corresponding to the second feature;

[0015] A fusion module, used for fusing the first feature with each of the correction features to obtain a target feature;

[0016] The detection module is used to perform target detection based on the target features to obtain target object information.

[0017] A whole vehicle controller comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned target detection method when executing the computer program.

[0018] A readable storage medium stores a program or instruction, and the program or instruction implements the above-mentioned target detection method when executed by a processor.

[0019] A vehicle comprises the above-mentioned vehicle controller.

[0020] Compared with the prior art, the target detection method, device, vehicle controller and readable storage medium described in the present invention have the following advantages:

[0021] In summary, an embodiment of the present invention provides a target detection method, including: obtaining a first feature corresponding to a current environment image, and second features corresponding to a plurality of historical environment images; aligning the content of the second feature with the content of the first feature to obtain a correction feature corresponding to the second feature; fusing the first feature with each correction feature to obtain a target feature; performing target detection based on the target feature to obtain target object information. In an embodiment of the present invention, the second feature of the historical environment image can be aligned with the first feature of the current environment image, and the correction feature obtained after the alignment can be fused with the first feature to obtain the target feature. Since the currently obscured obstacle may not have been obscured at the historical moment, the target feature may contain feature information of the currently obscured obstacle in the historical environment image, so that the target object information obtained by performing target detection on the target feature can contain information of the currently obscured obstacle, which can improve the accuracy of target detection of the obscured obstacle. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0023] Figure 1 A flowchart of a target detection method provided by an embodiment of the present invention;

[0024] Figure 2 A flowchart of another target detection method provided by an embodiment of the present invention;

[0025] Figure 3 A schematic diagram of a feature pyramid network provided by an embodiment of the present invention;

[0026] Figure 4 A schematic diagram of a timing fusion enhancement module processing flow provided by an embodiment of the present invention;

[0027] Figure 5 A structural block diagram of a target detection device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0029] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0030] Reference Figure 1 , Figure 1 A flowchart of the steps of a target detection method provided by an embodiment of the present invention.

[0031] Step 101 : obtaining a first feature corresponding to a current environment image and second features corresponding to a plurality of historical environment images respectively.

[0032] In order to realize assisted driving and automatic driving functions, vehicles need to obtain surrounding road condition information at all times, such as obstacles on the road the vehicle is traveling on. Usually, a camera module is installed on the vehicle, which can capture the current environment image of the vehicle's current scene in real time. For example, the current environment image in front of the vehicle can be obtained through the camera module installed in front of the vehicle, and the current environment image behind the vehicle can be obtained through the camera module installed behind the vehicle.

[0033] In an embodiment of the present invention, the current environment image around the vehicle can be obtained by photographing the environment around the vehicle with a camera module mounted on the vehicle itself. It should be noted that the camera module can be a visible light camera module or a camera module based on non-visible light, such as an infrared camera module, a thermal imaging camera module, etc., which can photograph the current environment image.

[0034] In addition, in an embodiment of the present invention, the current environment image can be a visible light image taken by a conventional camera, an infrared image taken by an infrared camera, or a point cloud image taken by a lidar camera, etc. The embodiment of the present invention does not specifically limit the specific form and acquisition method of the current environment image.

[0035] After obtaining the current environment image, feature extraction may be performed on the current environment image to obtain a corresponding first feature, which may include but is not limited to a feature vector, a feature, etc., and is not specifically limited in the embodiment of the present invention.

[0036] Specifically, in the embodiment of the present invention, the method for extracting features of scene effects may include but is not limited to Scale-Invariant Feature Transform (SIFT), Local Binary Patterns (LBP), etc., and the embodiment of the present invention does not make specific limitations.

[0037] It should be noted that the feature extraction of the current environment image can be single-scale feature extraction or multi-scale feature extraction. For example, the first feature containing multi-scale features can be obtained based on the Feature Pyramid Networks (FPN), or the current environment image can be firstly subjected to multi-scale transformation to obtain current scene sub-images of multiple scales, and then feature extraction is performed on each current scene sub-image to obtain the first sub-feature corresponding to each current scene sub-image, thereby obtaining a first feature composed of multiple first sub-features.

[0038] In the embodiment of the present invention, the historical environment image is compared with the current environment image. A historical environment image is an environment image acquired at a historical moment before the acquisition time of the current environment image. For example, the vehicle can acquire environment images at intervals of 1 second. If the current moment is the 5th second, the current environment image is the image around the vehicle captured at the 5th second, a historical environment image can be the image around the vehicle captured at the 4th second, and a historical environment image can be the image around the vehicle captured at the 3rd second.

[0039] The above method for generating the first feature can also be used to generate a corresponding second feature according to each historical environment image. It should be noted that the specific number of the above multiple historical environment images (for example, 3) can be set by the technician according to actual needs, equipment computing power, etc., and the embodiment of the present invention does not make a specific limitation.

[0040] Step 102: align the content of the second feature with the content of the first feature to obtain a correction feature corresponding to the second feature.

[0041] During the process of vehicle target detection, since the vehicle and objects outside the vehicle often move, it is easy for objects around the vehicle to be blocked by other objects at a certain moment, thereby affecting the target detection result. In an embodiment of the present invention, the first feature of the current environment image can be fused with the second feature of the historical environment image to obtain a target feature containing the environment image features at multiple moments. Since the objects blocked by the environment at different moments are different, the objects blocked in the current environment image may not be blocked in the historical environment image. Therefore, the fused target feature can include object features that are not in the current environment image, and then the blocked objects in the current environment image can be detected by performing target detection through the target feature.

[0042] First, the second feature and the first feature may be aligned in content. Specifically, the offset of the scene in the second feature and the first feature may be determined based on the second feature and the first feature. For example, the offset between the two may be determined based on the average distance between a plurality of similar feature elements (such as feature points) in the first feature and the second feature. The offset of the scene in the second feature and the first feature may also be determined based on the distance between similar pixels in the current environment image and the historical environment image. The above-mentioned offset may also be determined based on other methods, which are not specifically limited in the embodiments of the present invention.

[0043] Then, according to the offset corresponding to each second feature, the second feature is corrected to obtain a corrected feature whose content is aligned with the first feature, wherein the above image alignment means that the positions of as many identical features as possible in the corrected feature and the first feature are kept consistent in the image.

[0044] Step 103: Fusing the first feature with each of the correction features to obtain a target feature.

[0045] In the embodiment of the present invention, the first feature may be fused with each correction feature to obtain a target feature.

[0046] Specifically, the first feature and the correction feature may be fused by feature concatenation, neural network fusion, or the like. For example, for the feature concatenation method, the feature vector corresponding to the first feature may be fused with the feature vectors of each correction feature by matrix multiplication to obtain the target feature. The CONCATENATE function may also be used to fuse the first feature and the correction feature to obtain the target feature. The embodiment of the present invention does not specifically limit the specific fusion method.

[0047] Step 104: Perform target detection based on the target features to obtain target object information.

[0048] After obtaining the target features, target detection can be performed based on the target features, thereby extracting the target object information from the target features.

[0049] Specifically, the sliding window method can be used to detect the object in the target feature to obtain the target object information. This method is to divide the target feature into blocks, each block is detected using a sliding window, and the target object information is identified from the target feature by comparing the features in the sliding window with the preset object feature information. The target feature can also be processed using a neural network model such as a bounding box to obtain the target object information. The embodiment of the present invention does not specifically limit the method for obtaining the target object information.

[0050] The target object information may include, but is not limited to, object size (length, width, height), object spatial position (e.g., x, y, z coordinates of the object in the spatial coordinate system), orientation angle (e.g., the orientation direction of the head of other vehicles), and category (e.g., pedestrians, bicycles, tricycles, etc.). The specific information type included in the target object information can be flexibly set by the technician, and is not specifically limited in the embodiments of the present invention.

[0051] In summary, an embodiment of the present invention provides a target detection method, including: obtaining a first feature corresponding to a current environment image, and second features corresponding to a plurality of historical environment images; aligning the content of the second feature with the content of the first feature to obtain a correction feature corresponding to the second feature; fusing the first feature with each correction feature to obtain a target feature; performing target detection based on the target feature to obtain target object information. In an embodiment of the present invention, the second feature of the historical environment image can be aligned with the first feature of the current environment image, and the correction feature obtained after the alignment can be fused with the first feature to obtain the target feature. Since the currently obscured obstacle may not have been obscured at the historical moment, the target feature may contain feature information of the currently obscured obstacle in the historical environment image, so that the target object information obtained by performing target detection on the target feature can contain information of the currently obscured obstacle, which can improve the accuracy of target detection of the obscured obstacle.

[0052] Reference Figure 2 , Figure 2 A flowchart of another target detection method provided by an embodiment of the present invention.

[0053] Step 201, obtaining second features corresponding to a plurality of historical environment images respectively.

[0054] In the embodiment of the present invention, environmental images can be continuously acquired based on a preset time interval, and features of the acquired environmental images can be extracted to obtain corresponding features and saved. For each acquired environmental image, the environmental image can be used as the current environmental image, and the latest multiple environmental images acquired before the environmental image can be used as historical environmental images for subsequent feature fusion and target detection processes.

[0055] Therefore, before obtaining the first feature corresponding to the current environment image, the second feature corresponding to the historical environment image has actually been obtained. The specific method of generating the second feature is similar to the following steps 202 to 203, and the embodiment of the present invention will not be repeated.

[0056] Step 202: extract features of the current environment image based on each convolution step in the convolution step set to obtain primary features corresponding to each convolution step.

[0057] In an embodiment of the present invention, after the current environment image is acquired, feature extraction may be performed on the current environment image based on a feature pyramid network (FPN) to obtain a first feature consisting of multi-scale features of the current environment image.

[0058] The feature pyramid network can extract image features at different scales, which helps to better capture the multi-scale features of the image. By adopting different receptive fields in different layers, it can better capture small targets, and by connecting features between different layers, it can better utilize existing network features and reduce the amount of computation.

[0059] Specifically, a convolution step size set consisting of multiple preset convolution step sizes can be set, and a feature pyramid network that can contain a feature extraction layer corresponding to each preset convolution step size can be constructed. Based on the feature pyramid network, from the minimum convolution step size to the maximum convolution step size in the convolution step size set, the primary features corresponding to each convolution step size in the convolution step size set are obtained in sequence.

[0060] Optionally, step 202 may include:

[0061] Sub-step 2021, based on the minimum convolution step in the convolution step set, feature extraction is performed on the current environment image to obtain primary features corresponding to the minimum convolution step.

[0062] In an embodiment of the present invention, feature extraction can be first performed on the current environment image based on the minimum convolution step in the convolution step set to obtain primary features corresponding to the minimum convolution step, and then primary features corresponding to a larger convolution step are obtained based on the primary features corresponding to the minimum convolution step, so as to gradually express the feature information extracted from the current environment image as high-level semantic features.

[0063] Reference Figure 3 , Figure 3 A schematic diagram of a feature pyramid network provided by an embodiment of the present invention is shown. Figure 3 As shown, the feature pyramid network can include four first extraction layers with different convolution steps: first extraction layer 1, first extraction layer 2, first extraction layer 3 and first extraction layer 4. The structure of the first extraction layer can be a bottleneck network structure. Different convolution steps (block strides) are set for the first extraction layers. For example, the convolution step length of the first extraction layer 1 is 4, the convolution step length of the first extraction layer 2 is 8, the convolution step length of the first extraction layer 3 is 16, and the convolution step length of the first extraction layer 4 is 32. When extracting the primary features of the current environment image, the current environment image can be first input into the first extraction layer 1 to obtain the primary features output by the first extraction layer 1. The primary features output by the first extraction layer 1 are the primary features corresponding to the minimum convolution step length.

[0064] Sub-step 2022, in the feature pyramid network, performing step-by-step feature extraction on the primary features corresponding to the minimum convolution step size in a first direction, and obtaining the primary features corresponding to each convolution step size in the convolution step size set in turn; wherein the first direction points from a smaller convolution step size in the convolution step size set to a larger convolution step size.

[0065] After obtaining the primary features corresponding to the minimum convolution step size, the primary features corresponding to the minimum convolution step size can be extracted step by step based on a larger convolution step size, so as to obtain the primary features corresponding to each convolution step size in turn, wherein the direction of the step-by-step feature extraction is a first direction, and the first direction points from a smaller convolution step size to a larger convolution step size.

[0066] like Figure 3As shown, after obtaining the primary features corresponding to the convolution step length of the first extraction layer 1, the primary features corresponding to the convolution step length of the first extraction layer 1 can be input into the first feature extraction layer 2 to obtain the primary features corresponding to the convolution step length of the first feature extraction layer 2, and then the primary features corresponding to the convolution step length of the first feature extraction layer 2 can be input into the first feature extraction layer 3 to obtain the primary features corresponding to the convolution step length of the first feature extraction layer 3, and so on, until the primary features corresponding to all convolution step lengths are obtained.

[0067] Step 203: The primary feature corresponding to the convolution step size is merged with the primary features corresponding to other convolution step sizes to obtain first sub-features corresponding to each convolution step size.

[0068] In an embodiment of the present invention, for each primary feature corresponding to a convolution step, the primary feature can be fused with primary features corresponding to other convolution steps to obtain a first sub-feature corresponding to each convolution step, so that the first sub-feature corresponding to each convolution step can contain more feature information, thereby improving the accuracy of subsequent target recognition.

[0069] It should be noted that each fusion process can fuse at least two primary features and at most all primary features.

[0070] Optionally, step 203 may include:

[0071] Sub-step 2031, in the feature pyramid network, feature fusion is performed on the primary features based on the second direction, and intermediate features corresponding to each convolution step in the convolution step set are obtained in sequence.

[0072] In the feature pyramid network, the primary features can be fused in a second direction opposite to the first direction to obtain intermediate features corresponding to each convolution step size in sequence. For example, if the first direction is bottom-up, that is, from a smaller convolution step size to a larger convolution step size, then the second direction is top-down, that is, from a larger convolution step size to a smaller convolution step size.

[0073] like Figure 3As shown, based on the feature pyramid network, each first extraction layer can be connected to the corresponding second extraction layer, for example, the first extraction layer 1 is connected to the second extraction layer 1, the first extraction layer 2 is connected to the second extraction layer 2, the first extraction layer 3 is connected to the second extraction layer 3, and the first extraction layer 4 is connected to the second extraction layer 4. The structure of the second extraction layer can be a bottleneck network structure, and each second extraction layer is connected to a higher-level feature extraction layer (a feature extraction layer with a larger convolution step) for feature fusion. The second extraction layer 3 receives the primary features output by the first extraction layer 3 and the intermediate features output by the second extraction layer 4, and outputs the intermediate features corresponding to the second extraction layer 3 after fusion. The second extraction layer 2 receives the primary features output by the first extraction layer 2 and the intermediate features output by the second extraction layer 3, and outputs the intermediate features corresponding to the second extraction layer 2 after fusion, and so on.

[0074] Sub-step 2032: In the feature pyramid network, feature fusion is performed on the intermediate features based on the first direction to sequentially obtain first sub-features corresponding to each convolution step in the convolution step set.

[0075] In an embodiment of the present invention, feature fusion can also be performed on intermediate features in a first direction from bottom to top through a feature pyramid network, and first character features corresponding to each convolution step can be obtained in sequence. Specifically, in the feature pyramid network, a third extraction layer for outputting the first sub-feature can be set, the number of third extraction layers is the same as the above-mentioned second extraction layer, and the number of third extraction layers has a one-to-one correspondence with the above-mentioned second extraction layer, and the convolution step of each third extraction layer is the same as the convolution step of the corresponding second extraction layer. The third extraction layer can output the first sub-feature corresponding to the third extraction layer based on the intermediate features output by the corresponding second extraction layer and the first sub-feature output by the upper third extraction layer (corresponding to the third extraction layer with a smaller convolution step).

[0076] like Figure 3As shown, in addition to including several second extraction layers, a third extraction layer may also be included, and each first extraction layer may be connected to a corresponding third extraction layer, for example, the second extraction layer 1 is connected to the third extraction layer 1, the second extraction layer 2 is connected to the third extraction layer 2, the second extraction layer 3 is connected to the third extraction layer 3, and the second extraction layer 4 is connected to the third extraction layer 4. The structure of the third extraction layer may be a bottleneck network structure, and each third extraction layer is connected to a higher-level feature extraction layer (a feature extraction layer with a larger convolution step) for feature fusion. The third extraction layer 1 receives the intermediate features output by the second extraction layer 1 and outputs the first sub-feature of the third extraction layer 1; the third extraction layer 2 receives the intermediate features output by the second extraction layer 2 and the first sub-feature output by the third extraction layer 1, and outputs the first sub-feature corresponding to the third extraction layer 2 after fusion; the third extraction layer 3 receives the intermediate features output by the second extraction layer 3 and the first sub-feature output by the third extraction layer 2, and outputs the first sub-feature corresponding to the third extraction layer 3 after fusion, and so on.

[0077] In an embodiment of the present invention, multi-scale features of the current environment image can be extracted through a feature pyramid network. Taking into account the various sizes of obstacles in the current environment image, a feature fusion structure in the first direction and the second direction is included in the feature pyramid. Feature fusion is performed in two directions, top-down and bottom-up, to obtain multiple first sub-features of different scales, which helps to improve the accuracy of subsequent obstacle detection.

[0078] Step 204: determine the offset between the second sub-feature and the first sub-feature corresponding to the same convolution step size.

[0079] In an embodiment of the present invention, the second sub-feature of each historical environment image includes the second sub-feature corresponding to each convolution step. The specific determination method is similar to the above-mentioned method for determining the first sub-feature, and the embodiment of the present invention will not be repeated.

[0080] For the second sub-feature and the first sub-feature with the same convolution step size, the offset between the two can be calculated, and the second sub-feature can be adjusted according to the offset to obtain a corrected sub-feature aligned with the first sub-feature.

[0081] Specifically, the offset between the first sub-feature and the second sub-feature can be determined based on the average distance between multiple similar feature elements (such as feature points) in the first sub-feature and the second sub-feature. Alternatively, the first sub-feature and the second sub-feature corresponding to the same convolution step size can be fused to obtain fused features corresponding to each second sub-feature, and the offset between the first sub-feature and the second sub-feature corresponding to the same convolution step size can be determined based on the fused features.

[0082] In an embodiment of the present invention, the offset between the second sub-feature and the first sub-feature can be determined in the temporal fusion enhancement module. In the process of extracting features from the environmental image, the first sub-feature corresponding to each convolution step can be input into a temporal fusion enhancement module, and the temporal fusion enhancement module fuses the first sub-feature with the second sub-feature. Figure 3 As shown, each third extraction layer can be connected to a temporal fusion enhancement module, and the temporal fusion enhancement module can obtain the first sub-feature output by the third extraction layer and retain the second sub-feature output by the third extraction layer for each historical environmental image.

[0083] Reference Figure 4 , Figure 4 FIG. 4 shows a schematic diagram of a processing flow of a timing fusion enhancement module provided by an embodiment of the present invention. Figure 4 As shown, the temporal fusion enhancement module performs feature fusion on each second sub-feature with the first sub-feature to obtain fusion features corresponding to each second sub-feature, and determines the offset between the second sub-feature and the first sub-feature corresponding to the fusion feature based on the distance between the same feature elements in a fusion feature.

[0084] Among them, the first sub-feature and the second sub-feature can be fused by feature splicing, neural network fusion, etc. For example, for the feature splicing method, the feature vector corresponding to the first sub-feature can be fused with the feature vectors of each second sub-feature through matrix multiplication to obtain a fused feature. The first sub-feature and the second sub-feature can also be fused by the CONCATENATE function to obtain a fused feature corresponding to each second sub-feature. The embodiment of the present invention does not specifically limit the specific fusion method.

[0085] Step 205: adjusting the second sub-feature based on the offset to obtain a corrected sub-feature corresponding to the second sub-feature.

[0086] In the embodiment of the present invention, after obtaining the offset corresponding to each second sub-feature, the corresponding second sub-feature can be adjusted based on the offset to obtain the correction sub-feature corresponding to each second sub-feature. Figure 4 As shown, the first sub-feature is fused with the second sub-feature 1 to obtain fused feature 1, and offset 1 is calculated based on fused feature 1, the first sub-feature is fused with the second sub-feature 2 to obtain fused feature 2, and offset 2 is calculated based on fused feature 2, the second sub-feature 1 is adjusted by using offset 1 to obtain corrected sub-feature 1, and the second sub-feature 2 is adjusted by using offset 2 to obtain corrected sub-feature 2.

[0087] Step 206: All correction sub-features corresponding to the same convolution step length are merged with the first character feature to obtain target sub-features corresponding to each convolution step length.

[0088] After obtaining the corrected sub-features corresponding to each second sub-feature, all corrected sub-features can be fused with the first sub-features to obtain the target sub-feature of the convolution step corresponding to the enhancement module that needs to be fused at this time.

[0089] like Figure 4 As shown, if Figure 4 This is a schematic diagram of the processing flow of the temporal fusion enhancement module corresponding to the convolution step size of 4. The correction sub-feature 1, the correction sub-feature 2 and the first sub-feature can be fused to obtain the target sub-feature corresponding to the convolution step size of 4.

[0090] When processing the current environment image, for each convolution step in the above convolution step set, a target sub-feature corresponding to each convolution step can be generated, thereby obtaining the target sub-feature corresponding to each convolution step.

[0091] In the embodiment of the present invention, the multi-frame scene images collected at different times contain rich object motion feature information, but due to the movement of the vehicle and surrounding objects, the image features at the same position in the multi-frame scene images collected at different times are not aligned, and simply fusing them will cause feature confusion. Therefore, the above method can be used to align the object features in the multi-frame scene images collected at different times, and generate target sub-features that can represent occluded objects of different sizes, thereby improving the accuracy of subsequent feature recognition based on the target sub-features and helping to identify occluded obstacles.

[0092] Step 207: Perform target detection based on the target features to obtain target object information.

[0093] This step can refer to the above-mentioned step 104, and will not be described in detail in this embodiment of the present invention.

[0094] Optionally, step 207 may include:

[0095] Sub-step 2071, performing target detection based on target sub-features corresponding to the convolution step in the first convolution step set to obtain a class of target object information; wherein the first convolution step set is a subset of the convolution step set.

[0096] In an embodiment of the present invention, taking into account the different scales of different types of objects in the real world (pedestrians, bicycles, tricycles) and (cars, vans, large trucks), objects can be classified and different types of objects can be detected using different target sub-features to improve the accuracy of target detection.

[0097] Specifically, objects can be divided into a first type (VRU type) and a second type (vehicle type) according to their size, wherein the first type may include smaller obstacles such as pedestrians, bicycles, and tricycles, and the second type may include larger obstacles such as cars, vans, and large trucks.

[0098] It should be noted that technicians can classify objects according to actual needs, and the classification of objects is not limited to the first type and the second type mentioned above.

[0099] For the first type of object, a corresponding first convolution step size set may be set, and target detection may be performed on the target sub-features corresponding to the convolution step sizes in the first convolution step size set to obtain a class of target object information of the first type of object. Wherein, when the convolution step size set includes 4, 8, 16, and 32 convolution step sizes, the first convolution step size set may include 4, 8, and 16 convolution step sizes.

[0100] It should be noted that the number and types of convolution step sizes included in the above-mentioned convolution step size set, as well as the number and types of convolution step sizes included in the above-mentioned first convolution step size set, can be flexibly set by technical personnel, and the embodiments of the present invention do not make specific limitations.

[0101] Sub-step 2072, performing target detection based on target sub-features corresponding to the convolution step in the second convolution step set to obtain a class of target object information; wherein the second convolution step set is a subset of the convolution step set.

[0102] For the second type of object, a corresponding second convolution step size set may be set, and target detection is performed on the target sub-features corresponding to the convolution step sizes in the second convolution step size set to obtain a class of target object information of the second type of object. Wherein, when the convolution step size set includes 4, 8, 16, and 32 convolution step sizes, the second convolution step size set may include 8, 16, and 32 convolution step sizes.

[0103] It should be noted that the number and types of convolution step sizes included in the above-mentioned convolution step size set, as well as the number and types of convolution step sizes included in the above-mentioned second convolution step size set, can be flexibly set by technical personnel, and the embodiments of the present invention do not make specific limitations.

[0104] In an embodiment of the present invention, different types of objects can be distinguished based on their sizes, and target detection can be performed using appropriate target sub-feature combinations for different types. This not only improves the accuracy of target detection, but also reduces the number of target sub-features required for target detection of a certain type of object. There is no need to use all target sub-features for target detection, thereby improving the efficiency of target detection.

[0105] In summary, an embodiment of the present invention provides another target detection method, including: obtaining a first feature corresponding to a current environment image, and second features corresponding to a plurality of historical environment images; aligning the content of the second feature with the content of the first feature to obtain a correction feature corresponding to the second feature; fusing the first feature with each correction feature to obtain a target feature; performing target detection based on the target feature to obtain target object information. In an embodiment of the present invention, the second feature of the historical environment image can be aligned with the first feature of the current environment image, and the correction feature obtained after the alignment can be fused with the first feature to obtain the target feature. Since the currently obscured obstacle may not have been obscured at the historical moment, the target feature may contain feature information of the currently obscured obstacle in the historical environment image, so that the target object information obtained by performing target detection on the target feature can contain information of the currently obscured obstacle, which can improve the accuracy of target detection of the obscured obstacle.

[0106] Based on the above embodiments, an embodiment of the present invention further provides a target detection device.

[0107] refer to Figure 5 , Figure 5 A structural block diagram of a target detection device provided by an embodiment of the present invention:

[0108] An acquisition module 601 is used to acquire a first feature corresponding to a current environment image and second features corresponding to a plurality of historical environment images respectively;

[0109] An alignment module 602, configured to align the content of the second feature with the content of the first feature to obtain a correction feature corresponding to the second feature;

[0110] A fusion module 603, used for fusing the first feature with each of the correction features to obtain a target feature;

[0111] The detection module 604 is used to perform target detection based on the target features to obtain target object information.

[0112] Optionally, the first feature includes a plurality of first sub-features, and the acquisition module includes:

[0113] A primary feature submodule, used for extracting features of the current environment image based on each convolution step in the convolution step set, and obtaining primary features corresponding to each convolution step;

[0114] The primary feature fusion submodule is used to fuse the primary features corresponding to the convolution step length with the primary features corresponding to other convolution step lengths to obtain the first sub-features corresponding to each convolution step length.

[0115] Optionally, the primary feature submodule includes:

[0116] A first convolution submodule, configured to extract features of the current environment image based on a minimum convolution step in the convolution step set, and obtain primary features corresponding to the minimum convolution step;

[0117] The second convolution submodule is used to extract the primary features corresponding to the minimum convolution step size step by step in the feature pyramid network according to the first direction, and obtain the primary features corresponding to each convolution step size in the convolution step size set in turn; wherein the first direction points from a smaller convolution step size in the convolution step size set to a larger convolution step size.

[0118] Optionally, the primary feature fusion submodule includes:

[0119] An intermediate feature submodule, used for performing feature fusion on the primary features based on the second direction in the feature pyramid network, and sequentially obtaining intermediate features corresponding to each convolution step in the convolution step set;

[0120] The intermediate feature fusion submodule is used to perform feature fusion on the intermediate features based on the first direction in the feature pyramid network, and sequentially obtain the first sub-features corresponding to each convolution step in the convolution step set.

[0121] Optionally, the second feature includes a plurality of second sub-features extracted based on a plurality of convolution step sizes, the correction feature corresponding to the second feature includes a plurality of correction sub-features respectively corresponding to the second sub-features, and the alignment module includes:

[0122] An offset quantum module, used to determine the offset between the second sub-feature and the first sub-feature corresponding to the same convolution step size;

[0123] An alignment submodule is used to adjust the second sub-feature based on the offset to obtain a correction sub-feature corresponding to the second sub-feature.

[0124] Optionally, the target feature includes a plurality of target sub-features corresponding to a plurality of convolution step sizes respectively, and the fusion module includes:

[0125] The fusion submodule is used to fuse all the correction sub-features corresponding to the same convolution step size with the first word feature to obtain the target sub-features corresponding to each convolution step size.

[0126] Optionally, the target object information includes first-class target object information and second-class target object information, and the detection module includes:

[0127] A first detection submodule, configured to perform target detection based on target sub-features corresponding to convolution steps in a first convolution step set, to obtain information of a class of target objects; wherein the first convolution step set is a subset of the convolution step set;

[0128] The second detection submodule is used to perform target detection based on the target sub-features corresponding to the convolution step in the second convolution step set to obtain a class of target object information; wherein the second convolution step set is a subset of the convolution step set.

[0129] In summary, an embodiment of the present invention provides a target detection device, including: an acquisition module, used to acquire a first feature corresponding to a current environment image, and second features corresponding to a plurality of historical environment images; an alignment module, used to align the content of the second feature with the content of the first feature to obtain a correction feature corresponding to the second feature; a fusion module, used to fuse the first feature with each correction feature to obtain a target feature; and a detection module, used to perform target detection based on the target feature to obtain target object information. In an embodiment of the present invention, the second feature of the historical environment image can be aligned with the first feature of the current environment image, and the correction feature obtained after the alignment can be fused with the first feature to obtain the target feature. Since the currently obscured obstacle may not have been obscured at the historical moment, the target feature may contain the feature information of the currently obscured obstacle in the historical environment image, so that the target object information obtained by performing target detection on the target feature can contain the information of the currently obscured obstacle, which can improve the accuracy of target detection of the obscured obstacle.

[0130] An embodiment of the present invention further provides a vehicle controller, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the above-mentioned target detection method.

[0131] An embodiment of the present invention further provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the above-mentioned target detection method is implemented.

[0132] An embodiment of the present invention further provides a vehicle, comprising the above-mentioned vehicle controller.

[0133] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned device embodiments and will not be repeated here.

[0134] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0135] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A target detection method, characterized in that: The method comprises: Obtaining a first feature corresponding to the current environment image and second features corresponding to a plurality of historical environment images respectively; Aligning the content of the second feature with the content of the first feature to obtain a correction feature corresponding to the second feature; Fusion the first feature with each of the correction features to obtain a target feature; Target detection is performed based on the target features to obtain target object information.

2. The method according to claim 1, characterized in that The first feature includes a plurality of first sub-features, and the first feature corresponding to the current environment image is obtained, including: Performing feature extraction on the current environment image based on each convolution step in the convolution step set to obtain primary features corresponding to each convolution step; The primary feature corresponding to the convolution step length is fused with the primary features corresponding to other convolution step lengths to obtain the first sub-features corresponding to each convolution step length.

3. The method according to claim 2, characterized in that The step of extracting features of the current environment image based on each convolution step in the convolution step set to obtain primary features corresponding to each convolution step includes: Based on the minimum convolution step size in the convolution step size set, feature extraction is performed on the current environment image to obtain primary features corresponding to the minimum convolution step size; In the feature pyramid network, the primary features corresponding to the minimum convolution step size are extracted step by step according to a first direction, and the primary features corresponding to each convolution step size in the convolution step size set are obtained in turn; wherein the first direction points from a smaller convolution step size in the convolution step size set to a larger convolution step size.

4. The method according to claim 2, characterized in that: The step of fusing the primary feature corresponding to the convolution step with the primary features corresponding to other convolution step sizes to obtain first sub-features corresponding to the respective convolution step sizes includes: In the feature pyramid network, the primary features are subjected to feature fusion based on the second direction, and intermediate features corresponding to each convolution step in the convolution step set are obtained in sequence; In the feature pyramid network, feature fusion is performed on the intermediate features based on the first direction to obtain first sub-features corresponding to each convolution step in the convolution step set in sequence.

5. The method according to claim 2, characterized in that: The second feature includes multiple second sub-features extracted based on multiple convolution step sizes, the correction feature corresponding to the second feature includes multiple correction sub-features respectively corresponding to the second sub-features, and the content of the second feature is aligned with the content of the first feature to obtain the correction feature corresponding to the second feature, including: Determine the offset between the second sub-feature and the first sub-feature corresponding to the same convolution step size; The second sub-feature is adjusted based on the offset to obtain a corrected sub-feature corresponding to the second sub-feature.

6. The method according to claim 5, characterized in that The target feature includes a plurality of target sub-features corresponding to a plurality of convolution step sizes respectively, and the first feature is fused with each of the correction features to obtain the target feature, including: All the corrected sub-features corresponding to the same convolution step size are fused with the first character feature to obtain the target sub-features corresponding to each convolution step size.

7. The method according to claim 6, characterized in that The target object information includes first-class target object information and second-class target object information, and the target detection is performed based on the target features to obtain the target object information, including: Performing target detection based on target sub-features corresponding to the convolution step lengths in the first convolution step length set to obtain a class of target object information; wherein the first convolution step length set is a subset of the convolution step length set; Target detection is performed based on target sub-features corresponding to the convolution step in the second convolution step set to obtain a class of target object information; wherein the second convolution step set is a subset of the convolution step set.

8. A target detection device, characterized in that: The device comprises: An acquisition module, used to acquire a first feature corresponding to a current environment image, and second features corresponding to a plurality of historical environment images respectively; an alignment module, used to align the content of the second feature with the content of the first feature to obtain a correction feature corresponding to the second feature; A fusion module, used for fusing the first feature with each of the correction features to obtain a target feature; The detection module is used to perform target detection based on the target features to obtain target object information.

9. A vehicle controller, characterized in that: The vehicle controller includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the target detection method according to any one of claims 1 to 7 is implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the target detection method according to any one of claims 1 to 7 is implemented.

Images