Perceptual method and apparatus for automated driving

Abstract

Provided is a perception method for automatic driving, comprising: obtaining a first detection result for a first image by using a first neural network based on the first image, the first image being obtained for a target area; obtaining at least one second detection result for the first image using at least one second neural network on the basis of the first image, at least one additional input being generated on the basis of the first image, each of the at least one additional input representing at least a change in the first image, and using the at least one second neural network to obtain the at least one second detection result for the first image based on the at least one additional input; and fusing the first detection result and the at least one second detection result to obtain a final perception result for the first image. This improves the accuracy and robustness of the perception, thereby improving the safety of automatic driving.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation, and in particular to the field of autonomous driving. Background Art

[0002] The accuracy and robustness of environmental perception are very important as the basis for vehicle positioning, making decisions about the vehicle's driving path, and controlling the vehicle to execute the decision in the field of autonomous driving. Accurate and robust environmental perception can improve the safety of autonomous driving, otherwise it may lead to immeasurable traffic accidents.

[0003] With the development of computer vision, neural networks have been widely used in environmental perception applications. However, due to the limited training data and the imbalanced distribution of training data (long tail problem), coupled with the limited parameters of current neural networks and the high dimensionality of real-world data, there is still room for improvement even when neural networks are applied to environmental perception.

[0004] Currently, in order to increase the safety of autonomous driving, redundant or parallel neural networks are usually added. However, the redundant or parallel neural networks currently added are usually similar to the original neural networks, which may cause them to fail in similar situations (common cause failure). In other words, their failures are highly correlated, so satisfactory results cannot be achieved. Summary of the invention

[0005] It is desired to provide a perception method and device for autonomous driving, which can improve the accuracy and robustness of environmental perception, thereby providing a guarantee for safe autonomous driving.

[0006] According to one aspect, a perception method for autonomous driving is provided, comprising: obtaining a first detection result for a first image using a first neural network based on a first image, wherein the first image is acquired for a target area; obtaining at least one second detection result for the first image using at least one second neural network based on the first image, wherein obtaining the at least one second detection result comprises: generating at least one additional input based on the first image, each of the at least one additional input at least representing a change in the first image, and obtaining the at least one second detection result for the first image using the at least one second neural network based on the at least one additional input; and fusing the first detection result and the at least one second detection result to obtain a final perception result for the first image.

[0007] According to another aspect, a training method for a neural network is provided, the neural network being used for perception in autonomous driving and comprising a first neural network, at least one second neural network and a fusion module, the fusion module being used to fuse the output of the first neural network with the output of the at least one second neural network to obtain the output of the neural network, the training method comprising: pre-training the first neural network based on a plurality of first pre-training images and an object annotation result for each first pre-training image to obtain a pre-trained first neural network, wherein the first neural network is used to obtain a corresponding first pre-training detection result based on each first pre-training image in the plurality of first pre-training images; and pre-training each second neural network in the at least one second neural network based on a plurality of second pre-training images and a foreground object annotation result for each second pre-training image to obtain a pre-trained at least one second neural network, wherein at least one additional pre-training input is generated based on each second pre-training image and a corresponding second pre-training detection result is obtained based on each additional pre-training input in the at least one additional pre-training input and the corresponding second neural network, the at least one additional pre-training input at least representing a change in the second pre-training image, and the first pre-training image and the second pre-training image may be the same or different pre-training images.

[0008] According to another aspect, a perception device for autonomous driving is provided, comprising: a memory, and a processing unit, which is used to: obtain a first detection result for a first image based on a first image using a first neural network, wherein the first image is acquired for a target area; obtain at least one second detection result for the first image based on the first image using at least one second neural network, wherein obtaining the at least one second detection result comprises: generating at least one additional input based on the first image, each of the at least one additional input at least representing a change in the first image, and obtaining the at least one second detection result for the first image based on the at least one additional input using the at least one second neural network; and fusing the first detection result and the at least one second detection result to obtain a final perception result for the first image.

[0009] According to another aspect, an autonomous driving system is provided, comprising: a camera for acquiring a first image of a target area; and a perception device for autonomous driving according to various embodiments of the present invention.

[0010] According to another aspect, a computer-readable medium is provided, storing a computer program, which, when executed by a processor, enables the processor to execute the perception method for autonomous driving according to various embodiments of the present invention or the training method for a neural network according to various embodiments of the present invention.

[0011] According to various embodiments of various aspects of the present invention, on the basis of the original neural network (first neural network) focusing on the detection and classification of objects in the input image, at least one parallel neural network (at least one second neural network) focusing on changes occurring in the input image is added, and the input image is detected from different levels. On the one hand, the sensitivity to the background in the input image is retained, and the sensitivity to the changing objects in the input image is provided. On the other hand, the detection result based on the second neural network can introduce prior knowledge (for example, prior knowledge unrelated to classification). As a result, not only common cause failures are reduced, the accuracy, reliability and robustness of environmental perception are improved, thereby improving the safety of autonomous driving, but also the interpretability of the perception results is increased.

[0012] In some embodiments, unlike the original first neural network that focuses on the detection and classification of objects in the input image, the at least one parallel neural network focuses on the detection task of the changed objects in the input image without involving classification, that is, the at least one parallel neural network detects the existence of the changed objects in the image. Therefore, even if an object or a category that has not appeared in the training data set, such as an obstacle or a construction area, appears in the input image in actual applications, such a neural network can still detect the existence of the object and prompt the existence of an unknown object; this can further reduce the common cause failure between different neural networks, and further improve the accuracy, reliability and robustness of environmental perception. Furthermore, the neural network structure according to each embodiment of the present invention can be realized by simply inserting the above-mentioned parallel neural network into the original neural network, which is also convenient in use. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Embodiments are illustrated by way of example only and not by way of limitation in the figures of the accompanying drawings in which like reference numerals refer to similar elements.

[0014] Figure 1 A perception method for autonomous driving according to an embodiment of the present invention is shown;

[0015] Figure 2 A method for training a neural network according to an embodiment of the present invention is shown;

[0016] Figure 3 A perception device for autonomous driving according to an embodiment of the present invention is shown.

[0017] Various aspects and features of various embodiments of the present invention are described with reference to the above drawings. The above drawings are only illustrative and not restrictive. In addition, not all parts of the device of the embodiment of the present invention are marked with reference numerals in the above drawings. Only relevant parts are marked in some drawings, which does not limit the parts to those shown in the drawings of the specification. DETAILED DESCRIPTION

[0018] Currently, the accuracy and robustness of environmental perception are increased by providing a parallel neural network on the basis of the original neural network. However, on the one hand, the parallel neural network is often trained with data similar to that used to train the original neural network. In the field of autonomous driving, for example, images collected by image sensors (such as cameras) set on the roadside or on the vehicle are usually used to train the original neural network and the parallel neural network. Although these images themselves may be different, their feature distributions are likely to be similar; on the other hand, the original neural network and the parallel neural network usually have similar structures; based on the above two aspects, the provided parallel neural network is prone to common cause failure with the original neural network, resulting in false positive or false negative perception results, for example, the perception result includes an object that does not originally exist in the input image (false positive) or the perception result does not include an object that actually exists in the input image (false negative). As can be seen from the above, even if a parallel neural network is provided, the current method is still difficult to achieve satisfactory environmental perception. For example, when objects or categories that have not appeared in the training image, such as obstacles or construction areas, appear on the road, whether or not the parallel neural network is provided, it may lead to failure of environmental perception.

[0019] The inventors of the present application recognize that the feature distribution and structure of the training data of the current parallel neural network are similar to those of the original neural network, which may easily lead to common cause failures. Therefore, the following embodiments of the present invention are designed to detect input images from different levels, reduce the occurrence of common cause failures, improve the accuracy, reliability and robustness of environmental perception, and thus provide a basis for safe autonomous driving.

[0020] According to various embodiments of the present invention, at least one parallel neural network is added on the basis of the original neural network, and the at least one parallel neural network focuses on the detection of changes occurring in the input image, which is different from the original neural network which focuses more on the detection and classification of all objects in the input image. By combining the original neural network which focuses on the detection and classification of objects in the input image with the parallel neural network which focuses on the detection of changes occurring in the input image, the original neural network and the corresponding parallel neural network focus on different aspects of the input image, while retaining the sensitivity to the background in the input image, providing sensitivity to the changing objects in the input image, which not only reduces common cause failures, but also improves the accuracy, reliability and robustness of environmental perception, thereby improving the safety of autonomous driving, but also increases the interpretability of the perception results.

[0021] In some embodiments, the at least one parallel neural network is different from the original neural network in that it does not involve classification, i.e., the at least one parallel neural network only includes detecting the changed objects in the input image (i.e., detecting the existence of the changed objects) but does not include classifying the changed objects in the input image (e.g., detecting the changed objects of unknown type). Thus, even if objects or categories that do not appear in the training data set, such as obstacles or construction areas, appear in the input image in actual applications, such a neural network can still provide accurate, reliable and robust perception results, and in particular can detect the existence of an object, even if it is not certain what the object is, it can still perceive the existence of an object; thereby, it can further reduce common cause failures, improve perception accuracy, reliability and robustness in different environments (e.g., changes in the background of the captured image caused by lighting, shadows, etc.), and thus improve the safety of autonomous driving.

[0022] In addition, according to various embodiments of the present invention, at least one parallel neural network (hereinafter referred to as at least one second neural network) that focuses on changes occurring in an input image is combined with an original neural network (hereinafter referred to as the first neural network). The parallel neural network can be combined with the original neural network as a plug-in framework, which facilitates transformation and upgrading based on the original neural network.

[0023] In the field of autonomous driving, the change of the present invention may refer to that relative to the original input image or the reference image, certain areas of the current input have obvious differences that have an impact on the driving control decision, and the degree of significance of this change may be consistent with a person's subjective judgment.

[0024] Figure 1 FIG. 1 shows a perception method 100 for autonomous driving according to an embodiment of the present invention. Figure 1As shown, according to the method 100, in step 110, the first image (i.e., input image I) is input into the first neural network, thereby obtaining a first detection result for the first image based on the first image using the first neural network. The first neural network focuses on the detection and classification of each target in the first image, and is particularly used for perceiving the image. The first neural network may include a first encoder and a first decoder, the first encoder is used to determine a first feature map based on the first image, and the first decoder is used to obtain a first detection result based on the first feature map. The first detection result at least includes a detection result for detecting the object in the first image, and may further include a classification result for classifying the object in the first image. The first neural network may be obtained by training based on the training image and the annotation results of the objects in the training image. The first image is obtained for the target area, for example, collected by a camera. In one example, the first image is an image showing traffic conditions collected by a camera set on the roadside. In another example, the first image is an image showing the surrounding environment of the vehicle collected by a camera set on the vehicle. The first image may be an image in a sequence of images collected by the camera, or a frame of an image in a video image collected by the camera.

[0025] Next, at least one second detection result for the first image is obtained based on the first image using at least one second neural network. The second neural network is a neural network for detecting changes in the first image, which can be trained based on the labeled change objects in the training image. During the training process of the second neural network, it is expected to use data augmentation to improve the generalization of the second neural network for different scenes, which will be further described below. In one embodiment, the second neural network may include a second encoder and a second decoder, the second encoder is at least used to determine a second feature map based on an additional input representing the change in the first image, and the second decoder is used to obtain a second detection result based on the second feature map. In one embodiment, the second neural network is used to detect the change object in the target area in the first image, but does not involve the classification of the change object. Due to the different input data compared to the first neural network, the feature weights that the second neural network focuses on are different from those of the first neural network, especially focusing on the features with larger changes in the first image and whether the features with larger changes have foreground objects.

[0026] Specifically, at least one additional input is generated based on the first image, each of the at least one additional input at least represents a change in the first image, and at least one second detection result for the first image is obtained based on the at least one additional input using the corresponding at least one second neural network. Figure 1In the case where one second neural network is shown in FIG. 1 , it is also expected that there are multiple second neural networks, and it is also expected that multiple second neural networks respectively obtain corresponding second detection results based on multiple additional inputs that are different from each other. Figure 1 The embodiment of the present invention is described using the case of a second neural network.

[0027] In one embodiment, in step 120, an additional input is generated based on the first image, and the additional input at least represents a change in the first image, and the change may include at least one of the following: a change in the first image compared to a reference image, a change between various regions in the first image, and a change between the first image and a second image. In some embodiments, the additional input may be an image, but this is not restrictive, and it may also be a feature map. The reference image includes prior knowledge of the target area, which may be a pre-acquired image of the target area or an image obtained by processing an image sequence or video stream from a camera. The first image and the second image are images captured adjacent to each other in time for the target area, for example, the first image and the second image are adjacent images in an image sequence captured by a camera, or the first image and the second image are adjacent frames in a video image captured by a camera, and the second image may be one or more images.

[0028] Different methods can be used to determine changes in the first image and thereby generate the additional input.

[0029] In one example, the additional input includes a first additional input representing a change between the first image and the reference image. It is possible to stack and align multiple images in an image sequence from a camera or multiple image frames from a video stream (hereinafter referred to as multiple images) and average them to generate a mean image. For example, the first additional input can be generated by merging the mean image with the first image as a reference image, wherein the first image can be one of the multiple image frames. It is expected that by stacking multiple images or multiple image frames, for example, averaging over time, an image with foreground objects in multiple image frames of an image sequence or video stream removed is obtained as a reference image. Although it is expected to remove all foreground objects as much as possible to obtain a reference image including only background objects, this is not restrictive and some foreground objects may also be retained, such as vehicles stationary on the road.

[0030] The algorithm for fusing the reference image and the first image includes but is not limited to matrix subtraction, Hadamard product and attention mechanism. In fact, any image processing algorithm that can detect the change between the first image and the reference image can be applied to the embodiments of the present invention. In addition, the reference image is not limited to the mean image, and other types of images containing prior knowledge can also be used as reference images. For example, when the camera used to capture the first image is a vehicle-mounted camera, the map of the target area in the vehicle navigation system can be used as a reference image. In addition, the reference image can also be obtained through an optimization algorithm.

[0031] The above fusion is different from the usual concatenation of different information into the neural network. Concatenation refers to the processing of tensors on the data, for example, concatenating two tensors H*W*N to obtain H*W*2N, thereby retaining the complete information of the two input tensors. Unlike concatenation, the above fusion can detect the changes between the first image and the reference image. Using it for parallel neural networks can increase the accuracy and robustness of perception.

[0032] In another example, the additional input includes a second additional input representing changes between regions in the first image. Saliency detection can be performed on the first image to detect changes between regions in the first image, thereby generating the second additional input. For example, the most "salient" region in the first image, such as the most eye-catching region (e.g., a foreground object), can be detected to generate the additional input. Any type of saliency detection algorithm can be used in embodiments of the present invention to detect changes between regions in the first image.

[0033] In another example, the additional input includes a third additional input representing a change between the first image and the second image captured adjacent in time. The first image and the second image can be processed based on an optical flow method to generate a third additional input representing a change in motion of corresponding pixels in the first image and the second image. Any algorithm based on an optical flow method can be used in an embodiment of the present invention to extract the change in motion of corresponding pixels in the first image and the second image, including algorithms based on deep neural networks, such as RAFT (Recurrent All-Pairs Field Transforms).

[0034] Although the above method is described to determine the change in the first image, this is not restrictive, and any algorithm that can determine the change in the first image can be used in the embodiments of the present invention.

[0035] After the above additional input is generated, the additional input is input into the second neural network in step 130 to obtain a second detection result for the first image. That is, the additional input input into the second neural network is the additional input determined in advance according to the first image in step 120. Since the additional input input into the second neural network represents the change in the first image, the second detection result is a detection result obtained for the changed object appearing in the first image, which makes the second detection result a detection result for the change in the first image in a targeted manner.

[0036] Since the second neural network only focuses on the detection of changes in the input image, which is different from the detection and classification of all objects in the input image in the first neural network, the detection and classification performed by the first neural network can be called high-level, while the detection performed by the second neural network can be called low-level. Compared with the first neural network, the second neural network adjusts the weights of the features that are focused on, learned, and inferred. The low-level detection performed by the second neural network based on the various types of additional inputs as described above (for example, only performing the detection task of whether there is a foreground object) can provide supplementary input for the high-level detection performed by the first neural network (for example, object detection and object classification), thereby increasing the accuracy, reliability, and robustness of perception. In addition, additional inputs representing different types of changes are used as intermediate state variables, combined with the perception results of the first neural network for reasoning, so as to detect additional dangerous objects, so as to improve the safety and confidence level of perception.

[0037] In step 140, the first detection result and the second detection result are fused to obtain a final perception result for the first image. By fusing the first detection result focusing on the detection and classification of the target in the first image and the second detection result focusing on the change in the first image, the perception method according to the present invention can focus on the target area in a differentiated manner, which avoids common cause failure and increases the accuracy, reliability and robustness of perception.

[0038] In one embodiment, the second neural network only focuses on the detection of changed objects and does not involve the classification of changed objects. Therefore, even if the input image includes objects or categories that have never appeared in the training image data, the perception method of the present invention can avoid common cause failures and achieve accurate, reliable and robust perception. The object will not be arbitrarily identified as other types of targets (for example, a construction area that has not appeared in the training image will be identified as a road background) because the neural network has not "seen" the object.

[0039] The first detection result and the second detection result can be fused in a variety of ways. In particular, the neural network according to the present invention may include a fusion module in addition to the first and second neural networks described above, which fuses the output of the first neural network and the output of the second neural network to obtain the output of the neural network according to the present invention, that is, the final perception result of the input image. Specifically, the first detection result from the first neural network and the second detection result from the second neural network can be fused as described in step 140 through the fusion module. In one embodiment, the detection result of detecting the object in the first image included in the first detection result (only involving the detection of whether there is a foreground object) and the detection result of the changed object from the second neural network (only involving the detection of whether there is a foreground object) can be fused through the fusion module. It can be expected that in this case, the first detection result from the first neural network can also include the classification result of the object, and the detection result of the changed object from the second neural network does not include the classification result of the object. After the fusion of the first detection result and the second detection result, the classification result included in the first detection result can be retained.

[0040] The fusion of the first detection result and the second detection result may be an object-level fusion or a pixel-level fusion.

[0041] In one embodiment, Bayesian fusion is used to fuse the first detection result and the second detection result. In this embodiment, the fusion of the first detection result and the second detection result can be a fusion of pixels and pixels, a fusion of objects and objects, or a fusion of objects and pixels.

[0042] For example, the first detection result can show the detection result of the object at the pixel level, and the second detection result can also show the detection result of the changed object at the pixel level. For example, the second detection result can show whether there is a changed object (for example, the pixels corresponding to the unknown type of changed object are marked as gray), or the pixel value of each pixel in the second detection result can be set according to the probability of the pixel being occupied by the changed object. The individual pixels in the first detection result can be set according to similar rules. Thus, the fusion of the first detection result and the second detection result can be a fusion at the pixel level (pixel and pixel).

[0043] For example, the first detection result may show the detection result of the object at the object level, and the second detection result may also show the detection result of the changed object at the object level. In the image perception process, the detected object is usually encircled with a bounding box. The detection result of the bounding box of the object can be represented by the coordinate position in the image coordinate system. Thus, the fusion of the first detection result and the second detection result may be the fusion of the bounding box of the object, that is, the fusion of the object level (object and object).

[0044] It is also contemplated that the first detection result shows the detection result of the object at the object level, while the second detection result shows the detection result of the changed object at the pixel level. Thus, the fusion of the first detection result and the second detection result can be a fusion of objects and pixels.

[0045] In another embodiment, a multilayer perceptron (MLP) is used to fuse the first detection result and the second detection result. In this embodiment, the fusion of the first detection result and the second detection result can be the fusion of pixels and pixels or the fusion of objects and objects as described above.

[0046] Regardless of the fusion method used and the fusion level, by fusing the first detection result and the second detection result, the neural network according to the present invention detects the input image from different levels, which not only retains the sensitivity to the changing background in the input image, but also provides sensitivity to the changing foreground objects in the input image, which reduces common cause failures.

[0047] On the basis of using the above-mentioned fusion method to achieve the fusion purpose, different logics can also be used to design the fusion algorithm according to specific downstream tasks to achieve different effects. In one embodiment, the fusion of the first detection result and the second detection result includes adding the changed object detected in the second detection result to the first detection result. For example, because the confidence of the detection result of the first neural network for a certain changed area is lower than a threshold, the changed area fails to appear in the first detection result (false negative, FN). If the probability of each pixel being occupied in the changed area detected in the second detection result is very high, then after a fusion step (such as Bayesian fusion), the object detection result of the changed area in the first detection result can be adjusted or re-acquired based on the second detection result, so that it outputs a detection result that an object exists in the changed area. Specifically, by only adding the changed objects detected in the second detection result to the first detection result while keeping the first detection result unchanged, (1) if an object is detected in a certain area in the first detection result, and no changed object is detected in the area in the second detection result, then there is no detected changed object in the second detection result that needs to be added to the first detection result. At this time, there is no need to add the changed object to the first detection result, and the final perception result depends on the first detection result, which includes the object in the area; (2) if an object is detected in a certain area in the first detection result, and a changed object is detected in the area in the second detection result, then the changed object in the second detection result is added to the first detection result. After being included in a detection result, it is further superimposed so that the final perception result also includes the object in the area; (3) If no object is detected in a certain area in the first detection result, and no changed object is detected in the area in the second detection result, then there is no detected changed object in the second detection result that needs to be added to the first detection result. At this time, there is no changed object that needs to be added, so the final result is still that there is no object in the area, (4) If no object is detected in a certain area in the first detection result, and an object is detected in the area in the second detection result, then the object is added to the first detection result, and the final result depends on the second detection result and includes the object in the area. Thus, for one or more areas in the image, as long as any one of the first neural network and the second neural network detects the existence of an object, the object detected in the area is output. In another embodiment, for one or more areas in the image, only objects that are not included in the second detection result are removed from the first detection result.For example, for a certain image area, the first neural network may detect that there is an object in the area because the confidence of the detection result of the area is higher than a threshold (false positive, FP), while the second neural network detects that the probability of the area being occupied is low. In this case, the detection result of the first neural network for the area can be removed from the first detection result through the fusion step. This means that when the second neural network does not detect an object at an image position, even if the first neural network detects an object at that position, the detected object will not be output. By removing only the objects not included in the second detection result from the first detection result, for a certain image area, (1) if the object is detected in the first detection result but not in the second detection result, the object detected in the first detection result is not included in the second detection result and needs to be removed from the first detection result, and the final perception result obtained will no longer include the detection result of the object; (2) if both the first detection result and the second detection result detect the object, there is no object included in the first detection result but not in the second detection result, so there is no need to remove any object from the first detection result, and the final perception result obtained will depend on the first detection result and include the object; (3) if the object is not detected in the first detection result but detected in the second detection result, there is also no object included in the first detection result but not in the second detection result, and there is no object that needs to be removed in the first detection result, and the final perception result obtained will depend on the first detection result and not include the object. Therefore, the detected object is output only when both the first and second neural networks detect the existence of the object. In summary, the detection results of the first neural network and the second neural network can be fused according to the logic of "AND" or "OR" according to the specific downstream tasks.

[0048] Although the perception method 100 is described with reference to the first neural network and the second neural network, this does not mean that the first neural network and the second neural network must have different network structures. They may also have the same network structure, for example, they are obtained through different training data. Although the first neural network and the second neural network may have the same network structure, their weights may be different after training because they focus on different input images.

[0049] Of course, it is more expected that the first neural network and the second neural network have different network structures. For example, the first neural network is used for detection and classification of objects in the first image, while the second neural network is only used for detection of changed objects in the first image without involving classification. This enables the second neural network to be designed to have a structure that is more suitable for paying attention to changes in the input image, thereby achieving a more differentiated focus on the input image compared to the first neural network, avoiding common cause failures, and further improving the accuracy, reliability and robustness of perception. It is also expected that the first and second neural networks are obtained through different training data based on different network structures. Although the first neural network and the second neural network are described separately above, it can be understood that the first and second neural networks can also share part of the structure. For example, the first encoder of the first neural network and the second encoder of the second neural network can be the same encoder.

[0050] Although the above embodiment is described with reference to first generating an additional input and then inputting the additional input into the second neural network, this is not restrictive, and it is also conceivable that the process of generating the additional input based on the first image is part of the second neural network, thereby obtaining a second detection result for the first image based on the first image using the second neural network. In one embodiment, the second neural network includes a second encoder and a second decoder as described above, and the perception method according to the present invention includes inputting the first image into the second encoder to generate an additional input representing the change in the first image based on the first image and determining a second feature map based on the additional input, including extracting the second feature map based on the additional input, and being able to generate the additional input based on the first image according to at least one of the various examples described above, and then inputting the second feature map determined based on the additional input into the second decoder to obtain a second detection result. In one embodiment, the second encoder and the second decoder can be implemented by different sub-neural networks, respectively. In a specific embodiment, the additional input to be determined includes a first additional input, and the first additional input represents the change between the first image and the reference image. In this case, the first image and the reference image can be input into an encoder shared by the first neural network and the second neural network to respectively output a first feature map extracted based on the first image, and a second feature map output based on the first image and the reference image for characterizing the first additional input; the first feature map is input into a first decoder of the first neural network to obtain a first detection result; and the second feature map is input into a second decoder of the second neural network to obtain a second detection result.

[0051] In addition, although obtaining additional inputs is described above with reference to different examples, on the one hand, this is not restrictive, and other methods can also be used to obtain additional inputs representing changes in the first image; on the other hand, these examples can be used in combination, for example, multiple additional inputs representing different types of changes in the first image can be generated, and then the second detection result is detected using the second neural network based on the multiple additional inputs. In one example, the multiple additional inputs can be combined and the combined multiple additional inputs can be input into the second neural network to determine the second detection result.

[0052] In another example, without limitation, Figure 1 In the case of only one second neural network shown in FIG. 1 , there may be multiple second neural networks. That is, according to an embodiment of the present invention, there may be at least one second neural network. The at least one second neural network obtains at least one second detection result for the first image based on at least one additional input. The structure of each second neural network may be the same as that in the previous reference. Figure 1 The second neural network described is similar to that described in reference Figure 1 The second neural network described may represent any one of the at least one second neural network. In one embodiment, each second neural network only involves detection of a changed object and does not involve classification. In one embodiment, each second neural network includes a second encoder and a second decoder as described above, and optionally, the second encoder of any one or more second neural networks may be a shared encoder with the first encoder of the first neural network. In another embodiment, each second neural network may be designed to detect different types of changes. Each second neural network obtains a corresponding second detection result based on different types of changes as described above, for example, one second neural network outputs a second detection result based on an additional input representing a change between a first image and a reference image, and another second neural network outputs a second detection result based on an additional input representing a change between regions in the first image. Multiple second neural networks for different types of changes can be used to generate multiple second detection results based on multiple additional inputs representing different types of changes, and then the multiple second detection results are fused with the first detection result to obtain a final perception result. The fusion algorithm between multiple second detection results and the first detection result is also similar to the algorithm for fusion of one second detection result and the first detection result.

[0053] References Figure 1 The perception method 100 shown describes the application of the neural network structure of the present invention (i.e., a neural network structure including a first neural network focusing on detection and classification of targets and a second neural network focusing on changes in input images). It can be understood that before the neural network according to the present invention is used for perception, it is first necessary to train the neural network.

[0054] Figure 2 A training method 200 of a neural network according to an embodiment of the present invention is shown, the neural network is used for perception in autonomous driving and has a neural network structure according to various embodiments of the present invention, namely, a first neural network focusing on the detection and classification of targets and at least one second neural network focusing on the detection of changing objects in an input image. The neural network may also include a fusion module, which is used to fuse the output of the first neural network (the above-mentioned first detection result) and the output of at least one second neural network (the above-mentioned second detection result) as the output of the entire neural network based on a parameterized model. The parameterized model includes a multilayer perceptron or a gated recurrent unit. In the fusion module, the output from the first neural network and the output from at least one second neural network can first be merged into one output to input the parameterized model through a non-parameterized operation (non-parameterized method). The non-parameterized operation includes at least one of the following: concatenation, Hadamard product, and elementwise addition.

[0055] according to Figure 2 The neural network training method 200 shown includes a pre-training process and a training process. In the pre-training process, the first neural network and the at least one second neural network are first trained separately. The pre-training of the first neural network and the at least one second neural network can use different loss functions from each other, and can use different or the same pre-training images. On the basis of the pre-training process, in the further training (joint training) of the image, the first training image with the object annotation result can be input into the pre-trained first neural network and the at least one second neural network to at least perform the training of the parameterized model used in the fusion module, thereby obtaining the final trained neural network.

[0056] Specifically, in the pre-training process of the image, the first neural network and the at least one second neural network are first pre-trained respectively to obtain at least part of the weights therein. At step 210, a plurality of first pre-training images for pre-training the first neural network and a plurality of second pre-training images for pre-training at least one second neural network are obtained, the first pre-training images having object annotation results, and the second pre-training images having foreground object annotation results. The object annotation results of the first pre-training images may be annotation results involving detection of the position, shape, size and classification of the object, while the foreground object annotation results only involve detection of the position and size of the object, and do not involve classification of the object. The first pre-training image and the second pre-training image may be the same or different pre-training images.

[0057] After obtaining the first pre-training image and the second pre-training image, the first neural network and each of the at least one second neural network can be pre-trained based on them in steps 220 and 230, respectively, wherein the object labeling result of the first pre-training image and the foreground object labeling result of the second pre-training image are used as the respective output true values ​​for each of the first neural network and the at least one second neural network, and the first neural network and the at least one second neural network are pre-trained to obtain the pre-trained first neural network and the at least one second neural network respectively. In the training process of the second neural network, it is necessary to generate at least one additional pre-training input based on the corresponding second pre-training image to train the at least one second neural network respectively, especially in the training process, based on each of the at least one additional pre-training input and the corresponding second neural network, a corresponding second pre-training detection result is obtained as the output of the second neural network, and the output corresponds to the foreground object labeling result of the second pre-training image as the output true value. With reference to Figure 1 Similar to the description, each of the at least one additional pre-trained input represents at least one of the following changes in the corresponding second pre-trained image: changes between the corresponding second pre-trained image and a corresponding reference image, the corresponding reference image including prior knowledge of the target area targeted by the corresponding second pre-trained image; changes between the respective areas in the corresponding second pre-trained image; and changes between the corresponding second pre-trained image and a second image, the corresponding second pre-trained image and the second image being images captured temporally adjacent to the target area.

[0058] The loss functions used to train the first neural network and at least one second neural network may be different from each other. It is also currently contemplated that the loss functions are the same. Specifically, the loss function of the first neural network can be designed (depending on the task to be performed, the loss function will be different) to compare the result of the first neural network with the object annotation result. If the two are the same or the difference between the two is within an acceptable range, it means that the first neural network has been trained; if the two are not the same or the difference between the two is not within an acceptable range, then return to adjust the parameters of the first neural network. This cycle is repeated to pre-train the first neural network for each first pre-training image. The first neural network according to the present invention is pre-trained by using multiple of the above-mentioned first pre-training images.

[0059] In addition, the loss function of each second neural network can be designed (depending on the task to be performed, the loss function will be different) to compare the detection result of the second neural network with the foreground object labeling result. If the two are the same or the difference between the two is within an acceptable range, it means that the second neural network has been trained; if the two are not the same or the difference between the two is not within an acceptable range, then the parameters of the second neural network are adjusted. This cycle is repeated to train the second neural network for each second pre-trained image. The second neural network according to the present invention is pre-trained by using multiple second pre-trained images.

[0060] The first neural network includes a first encoder and a first decoder, and each second neural network includes a second encoder and a second decoder. During the pre-training process, the first encoder of the first neural network is used to extract a first feature map based on the first pre-training image, and the first decoder is used to obtain a first pre-training detection result as an output of the first neural network based on the first feature map; the second encoder of each second neural network is used to generate a corresponding first additional pre-training input based on the second pre-training image and determine a corresponding second feature map based on the corresponding first additional pre-training input, the corresponding first additional training input at least represents a change in the second pre-training image, and the second decoder is used to obtain a corresponding second pre-training detection result based on the corresponding second feature map.

[0061] For example, at least one additional pre-training input includes a first additional pre-training input representing a change between a corresponding second pre-training image and a corresponding second reference image. The second pre-training image and the corresponding second reference image are acquired for the same target area, and the second reference image includes prior knowledge about the same target area. The first additional pre-training input is used to pre-train a specific second neural network in the at least one second neural network.

[0062] Pre-training the specific second neural network may include: taking the corresponding second pre-training image and the second reference image as inputs of the specific second neural network, and taking the foreground object labeling result of the corresponding second pre-training image as the output true value of the specific second neural network to pre-train it, wherein the second encoder of the specific second neural network is used to generate a first additional pre-training input based on the input corresponding second pre-training image and the second reference image, and determine a second feature map based on the first additional pre-training input, and the second decoder of the specific second neural network is used to obtain a specific second detection result based on the second feature map as the output of the specific second neural network.

[0063] Although reference is made to performing an operation of obtaining a first additional pre-trained input based on a second pre-trained image in a second encoder of the second neural network, this is not restrictive, and the operation may also be performed before the second pre-trained image is input into the encoder of the second neural network. Specifically, pre-training a specific second neural network may include first generating a first additional pre-trained input based on a corresponding second pre-trained image and a second reference image, and then pre-training the specific second neural network using the first additional pre-trained input as an input of the specific second neural network, and using a foreground object labeling result of the corresponding second pre-trained image as an output true value of the specific second neural network. Specifically, the first additional pre-trained input may be generated based on the corresponding second pre-trained image and the second reference image by an image difference algorithm (e.g., matrix subtraction, etc.).

[0064] After obtaining the pre-trained first neural network and at least one second neural network in steps 220 and 230, respectively, a neural network including the pre-trained first neural network and the pre-trained at least one second neural network is obtained in step 240, and the neural network also includes a fusion module based on a parameterized model. A further training process for the entire neural network can then be performed.

[0065] Specifically, in step 250, a first training image S is obtained, which is obtained for the target area and has an object annotation result. The first training image may be different from the first and second pre-training images previously used for pre-training. A relatively small training image set may be used to perform pre-training while a relatively large training image set may be used to perform the training process.

[0066] After the first training image is acquired, the pre-trained first neural network and at least one pre-trained second neural network obtained in step 240 are further trained based on the acquired first training image, wherein the object labeling result for the first training image is used as the output truth value of the entire neural network to at least train the parameterized model (e.g., a model including a multi-layer perceptron or a gated recurrent unit) used by the fusion module in the entire neural network, which includes adjusting the parameters of the parameterized model (e.g., the weights corresponding to each neuron in the multi-layer perceptron). For the second neural network, it is necessary to obtain at least one additional training input representing the changes in the first training image based on the first training image to be used for training each second neural network in the at least one second neural network.

[0067] In one scenario, the parameters (such as weights) of the first neural network and each second neural network obtained in the pre-training process remain unchanged, and only the parameters (such as weights) of the parameter model used in the fusion module are obtained through the training process. That is, during the training process, the pre-trained first neural network and each second neural network are not retrained, and only the parameter model used in the fusion module is trained.

[0068] In another scenario, based on the parameters of the first neural network and each second neural network obtained during the pre-training process, during the training process, not only the parameters of the parameterized model in the fusion module (such as weights) are determined, but also the parameters of the pre-trained first neural network and / or each second neural network (such as the weights corresponding to each neuron therein) are further adjusted. That is, during the training process, not only the parameterized model used in the fusion module is trained, but also the pre-trained first neural network and / or second neural network is further trained.

[0069] In the above training process, the first encoder of the first neural network is used to extract the first feature map based on the first training image, and the first decoder is used to obtain the first training detection result as the output of the first neural network based on the first feature map for the first training image. At least one additional training input for training the second neural network can be obtained by the second encoder of each second neural network, and of course, it can also be obtained before inputting into the second neural network.

[0070] For example, the further training includes: generating a first additional training input representing a change between the first training image and the corresponding first reference image based on the first training image and the corresponding first reference image, wherein the first training image and the corresponding first reference image are obtained for the same target area, and the first reference image includes prior knowledge about the same target area. Using the first additional training input as the input of a pretrained specific second neural network in the at least one pretrained second neural network, the second encoder of the pretrained specific second neural network determines a second feature map based on the first additional training input, and the second decoder of the pretrained specific second neural network obtains a specific second detection result as the output of the pretrained specific second neural network based on the second feature map for the first additional training input; or using the first training image and the corresponding first reference image as the input of a pretrained specific second neural network in the at least one pretrained second neural network, the second encoder of the pretrained specific second neural network generates the first additional training input, determines a second feature map based on the first additional training input, and the second decoder of the pretrained specific second neural network obtains a specific second detection result as the output of the pretrained specific second neural network based on the second feature map for the first additional training input. In the fusion module, the output of the first neural network and the output of at least one second neural network can be combined into one output to input the parameterized model, so as to fuse their detection results to obtain the true output value of the neural network, that is, the above-mentioned labeling result of the first training image.

[0071] The first and second training detection results are intermediate results generated by the first and second neural networks respectively during the model training process. It is necessary to train 1) a model with parameters, or 2) a model with parameters and a pre-trained first and / or second neural network (adjust the model parameters of the first and / or second neural network) so that the final perception result obtained after the first and second training detection results as their output are fused can correspond to the annotation result of the first training image (that is, the annotation result of the first training image is used as the output true value of the entire neural network). The parameters of the first neural network, each second neural network and the model with parameters (such as the weights of each neuron) are determined, and the final trained entire neural network can be obtained in step 260. In one embodiment, in the above training process, the loss function of the entire neural network can be designed (depending on the task to be performed, the loss function will be different) to compare the fused result with the annotation result. If the two are the same or the difference between the two is within an acceptable range, it means that the neural network including the first neural network, the second neural network and the model with parameters has been trained; if the two are not the same or the difference between the two is not within an acceptable range, then return to adjust the parameters of the first neural network, the parameters of the second neural network, and / or the relevant parameters of the fusion of the outputs of the first neural network and the second neural network (such as the parameters of the fusion algorithm in the fusion module). This cycle is repeated to train the neural network for each training image. By using a plurality of the first training images to train the neural network according to the present invention, various parameters in the neural network, such as weights, can be obtained, thereby generating a neural network that can be used for perception applications.

[0072] Although the reference Figure 2 A method for training a neural network is described, according to which the training of a first and at least one second neural network is coupled or strongly associated. However, this is not restrictive and it is also conceivable to train the first and each second neural network separately and then fuse the trained first and second neural networks. In this case, the fusion of the outputs of the first and second neural networks may be based on a non-parametric model, such as a Bayesian fusion. The following description is made with reference only to the case of one second neural network, but it will be appreciated that this also applies to the case of multiple second neural networks.

[0073] Specifically, a training method for a neural network is provided, wherein the neural network is used for perception in autonomous driving and includes a first neural network and a second neural network, and the output of the first neural network and the output of the second neural network are fused to obtain the output of the neural network, and the training method includes: obtaining a first training image, wherein the first training image is obtained for a target area and has a first object annotation result; inputting the first training image into the first neural network and using the first annotation result as an output truth value to train the first neural network to obtain a trained first neural network; obtaining a second training image, wherein the second training image has an annotated foreground object; inputting the second training image into the second neural network and using the annotated foreground object as an output truth value to train the second neural network to obtain a trained second neural network, and the details of the respective training processes of the first neural network and the second neural network can refer to the above-mentioned process of pre-training the first neural network and the second neural network, and they can adopt the same method. After obtaining the trained first neural network and the trained second neural network, the output of the trained first neural network and the output of the trained second neural network can be fused via a non-parametric model (e.g., Bayesian fusion) to obtain the trained entire neural network. The first training image and the second training image can be the same or different training images.

[0074] The separate training situation can be applied as follows Figure 2 The pre-training steps 220-230 shown are implemented, and the first training image and the second training image correspond to the first pre-training image and the second pre-training image in the above steps respectively.

[0075] Although the data for training the first neural network and the second neural network are described separately with reference to the first pre-trained image and the second pre-trained image (pre-training process) and the first training image (joint training) when the training of the first and at least one second neural network is coupled or strongly associated, or the first training image and the second training image when the first and at least one second neural network are trained separately, this is not restrictive. The respective images may also be the same image. The above-mentioned training images can be obtained in a variety of ways.

[0076] In one embodiment, data augmentation is expected to be used during the training of the second neural network to improve the generalization of the second neural network for different scenarios. Thus, while the first neural network is trained using the first training image and the annotation results of the objects therein, the annotated foreground objects in the first training image are synthesized with different background objects to obtain augmented training data (e.g., the foreground objects in the first training image are synthesized with background objects from other images to obtain augmented training data), and then the second neural network is trained based on the augmented training data and the annotation results of the foreground objects contained therein to obtain a trained second neural network.

[0077] Specifically, when the training of the first and at least one second neural network is coupled or strongly associated (or in the above-mentioned joint training process): a first training image can be formed by synthesizing a plurality of first reference images containing only the background with the annotated foreground object, so that each first training image is synthesized by the annotated foreground object and the first reference image including the corresponding background. Then, a plurality of first reference images corresponding to each of the plurality of first training images are transformed (such as rotated) to obtain a transformed first reference image corresponding to each first training image, and the first training image and its corresponding transformed first reference image are used to obtain the first additional training input. Specifically, the transformed plurality of first reference images and the corresponding plurality of first training images can be used as inputs of the second neural network, or additional training inputs can be obtained based on the transformed plurality of first reference images and the corresponding plurality of first training images and then input into the second neural network, and each first training image has a corresponding first reference image. The synthesized plurality of first training images are simultaneously used as inputs of the first neural network and the second neural network, that is, it is required that in the joint training stage, the first neural network and the second neural network have the same data source.

[0078] In the case where the first and at least one second neural network are trained separately (or in the above-mentioned pre-training process), the first training image (or first pre-training image) and the second training image (or second pre-training image) used to train the first and at least one second neural network may be different, but the second training image (or second pre-training image) and the second reference image used to pre-train the second neural network may also be obtained by referring to the same method as described in the previous paragraph. Therefore, whether the first and second neural networks are trained separately or jointly, the training data may be obtained by referring to the above paragraph. Since the background contained in the reference image is first used to synthesize the training image, and then the reference image is transformed, additional training input is obtained based on the transformed reference image and the training image, so that the background of each training image used to train a specific second neural network has a certain difference or change with the corresponding transformed reference image. In the training stage, the second neural network can learn based on this difference / change, which improves the attention weight of the second neural network to the difference features of the training image compared with the corresponding transformed reference image, so that it is easy to detect whether there is a foreground object in the changed area of ​​the image, while the training of the first neural network focuses more on the overall object detection and classification in the image, so that the first neural network and the second neural network obtained by training have better complementarity or double verification advantages (avoiding common cause failure) due to the different weights of the features of attention, and the accuracy of downstream tasks can be greatly improved by fusing the detection results of the first neural network and the second neural network. However, this does not exclude that there may be a small number of training images with no difference and no change in the background of the corresponding reference image.

[0079] Figure 3 A perception device 10 for autonomous driving according to an embodiment of the present invention is shown, the device comprising a memory 11 and a processor 12, the processor 12 can be used to execute the various steps of the method described according to the perception method 100. In particular, the processor 12 is used to obtain a first detection result for the first image using a first neural network based on a first image, wherein the first image is obtained for a target area; obtain at least one second detection result for the first image using at least one second neural network based on the first image, wherein obtaining the at least one second detection result comprises: generating at least one additional input based on the first image, each of the at least one additional input at least representing a change in the first image, and obtaining the at least one second detection result for the first image using the at least one second neural network based on the at least one additional input; and fusing the first detection result and the at least one second detection result to obtain a final perception result for the first image.

[0080] The device for training the neural network may have a similar composition to the above-mentioned perception device 10, that is, including a memory and a processor, or the device for training the neural network may be the same device as the perception device. The processor is particularly used to execute the reference above Figure 2 The individual training methods described.

[0081] Embodiments of the present invention also relate to an autonomous driving system (not shown), which includes a camera for acquiring a first image of a target area and a perception device for autonomous driving according to various embodiments of the present invention. It is also conceivable that it includes a device for training a neural network.

[0082] The inventors of the present invention have verified and compared the perception method according to the present invention with the existing methods in roadside perception. It has been confirmed that the method according to the present invention can identify objects that have not appeared in the training data set (such as cartons that unexpectedly appeared on the road) together with foreground objects of known categories (such as pedestrians, cars, bicycles, etc.) as high-dimensional features, instead of mixing the unseen objects (such as the cartons mentioned above) into the background features and making them undetectable as in the existing methods.

[0083] The functions of the perception method and device for autonomous driving and the training method and device for neural network of the present invention can be implemented by software or corresponding hardware, or by means of a processor to implement the functions of the above-mentioned units. For example, the processor can read the computer programs in the memory and run these computer programs to implement the functions of the above-mentioned units. In one embodiment, the above-mentioned device of the present invention can also be implemented by a memory and a processor.

[0084] It can be understood that the methods of various embodiments of the present disclosure can be implemented by computer programs / software. These software can be loaded into the working memory of the processor and used to execute the methods according to various embodiments of the present disclosure when running.

[0085] It is understood that the method and apparatus according to the present invention can be performed by any device with processing capabilities, and can also be performed at a remote location. The exemplary embodiments of the present disclosure cover both: creating / using the computer program / software of the present disclosure from the beginning, and converting an existing program / software to use the computer program / software of the present disclosure by means of an update.

[0086] According to another embodiment of the present disclosure, a computer program product is provided, such as a machine (e.g., computer) readable medium, such as a CD-ROM, including a computer program code, which, when executed, causes a computer or a processor to perform a method according to each embodiment of the present disclosure. The machine readable medium is, for example, an optical storage medium or a solid-state medium supplied with or as part of other hardware.

[0087] According to another embodiment of the present disclosure, a system for boxing is provided, the system comprising a three-dimensional camera disposed above a boxing container and the device according to various embodiments of the present disclosure.

[0088] The above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0089] The present disclosure is described above with reference to specific embodiments. It should be understood by those skilled in the art that the technical solution of the present disclosure can be implemented in various ways without departing from the spirit and basic features of the present disclosure. The specific embodiments are merely illustrative and not restrictive. In addition, these embodiments can be arbitrarily combined to achieve the purpose of the present disclosure. The scope of protection of the present disclosure is defined by the attached claims.

[0090] The word "comprising" in the specification and claims does not exclude the existence of other elements or steps. The expressions "first", "second" and the order of the steps shown in the figures do not limit their order or quantity. The functions of the various elements described in the specification or recorded in the claims can also be split or combined and implemented by corresponding multiple elements or a single element.

Claims

1. A perception method for autonomous driving, comprising: Obtaining a first detection result for the first image using a first neural network based on the first image, wherein the first image is obtained for a target area; Obtaining at least one second detection result for the first image using at least one second neural network based on the first image, wherein obtaining the at least one second detection result comprises: generating at least one additional input based on the first image, each of the at least one additional input representing at least a change in the first image, and obtaining the at least one second detection result for the first image using the at least one second neural network based on the at least one additional input; and The first detection result and the at least one second detection result are fused to obtain a final perception result for the first image.

2. The sensing method according to claim 1, wherein: The first neural network is used to detect and classify objects in the target area; Each of the at least one second neural network is used to detect a changed object in the target area but does not involve classification of the changed object.

3. The sensing method according to claim 1, wherein: Obtaining the at least one second detection result includes: generating the at least one additional input based on the first image, when the at least one additional input comprises a plurality of additional inputs, the plurality of additional inputs are different from each other; Each additional input of the at least one additional input is input into a corresponding second neural network of the at least one second neural network to obtain a corresponding second detection result.

4. The sensing method according to claim 1, wherein: Each of the at least one second neural network comprises a second encoder and a second decoder, and the perception method further comprises: inputting the first image into the second encoder to generate a corresponding additional input based on the first image and determining a second feature map based on the corresponding additional input; and The second feature map is input into the second decoder to obtain a corresponding second detection result.

5. The sensing method according to claim 4, wherein: The first neural network includes a first encoder and a first decoder, the first encoder is used to determine a first feature map based on the first image, and the first decoder is used to obtain the first detection result based on the first feature map, wherein the first encoder of the first neural network and the second encoder of any one of the at least one second neural network are the same encoder or different encoders.

6. The sensing method according to any one of claims 1 to 3, wherein: Each of the at least one additional input represents at least one of the following changes in the first image: a change between the first image and a corresponding reference image, the reference image including prior knowledge of the target region; changes between regions in the first image; and The change between the first image and the second image, wherein the first image and the second image are images captured adjacent to each other in time with respect to the target area.

7. The sensing method according to claim 6, wherein: The at least one additional input comprises a first additional input, the first additional input representing a change between the first image and the corresponding reference image, the perceptual method comprising: Generate the first additional input based on the first image and the corresponding reference image, and input the first additional input into a corresponding second neural network in the at least one second neural network to obtain a corresponding second detection result; or The first image and the corresponding reference image are input into an encoder to respectively output a first feature map extracted based on the first image, and a second feature map output based on the first image and the corresponding reference image for characterizing the first additional input; the first feature map is input into a first decoder to obtain the first detection result; the second feature map is input into a second decoder to obtain a corresponding second detection result; wherein the first neural network and the corresponding second neural network include a common encoder, the first neural network also includes the first decoder, and the corresponding second neural network also includes the second decoder.

8. The sensing method according to claim 6, wherein: The at least one additional input comprises a first additional input, the first additional input representing a change between the first image and the corresponding reference image, the perception method further comprising: averaging a plurality of images acquired for the target area to generate the corresponding reference image; and The first additional input is generated based on the first image and the corresponding reference image.

9. The sensing method according to claim 8, wherein: The first additional input is generated based on the first image and the corresponding reference image by using at least one of the following algorithms: Matrix subtraction; Hadamard; and Attention mechanism.

10. The sensing method according to claim 6, wherein: The at least one additional input includes a second additional input, the second additional input representing changes between regions in the first image, the perception method further comprising: Saliency detection is performed on the first image to generate the second additional input.

11. The sensing method according to claim 6, wherein: The at least one additional input includes a third additional input, the third additional input representing a change between the first image and the second image, the perception method further comprising: The first image and the second image are processed based on an optical flow method to generate the third additional input representing motion changes of corresponding pixels in the first image and the second image.

12. The sensing method according to any one of claims 1 to 3, wherein: The network structures of the first neural network and any one of the at least one second neural network are different and / or are obtained through different training data.

13. The sensing method according to claim 12, wherein: The first neural network is trained based on a training image and a result of labeling an object in the training image; and The second neural network is obtained through training based on augmented training data, wherein the augmented training data is obtained by synthesizing a foreground object annotation result in the training image with different background objects.

14. The sensing method according to claim 2, wherein: The first detection result at least includes a detection result of detecting the object in the target area, and each of the at least one second detection result includes a detection result of detecting the changed object in the target area.

15. The sensing method according to claim 14, wherein: The first detection result also includes a classification result of classifying the object in the target area.

16. The sensing method according to claim 14, wherein: The fusing of the first detection result and the at least one second detection result comprises: For a specific area in the first image, adding the changed object detected in each of the at least one second detection result to the first detection result to obtain the final perception result; and / or For a specific area in the first image, objects included in the first detection result but not included in the at least one second detection result are removed from the first detection result to obtain the final perception result.

17. A method for training a neural network, the neural network being used for perception in autonomous driving and comprising a first neural network, at least one second neural network and a fusion module, the fusion module being used to fuse an output of the first neural network with an output of the at least one second neural network to obtain an output of the neural network, the training method comprising: Pre-training the first neural network based on a plurality of first pre-training images and an object annotation result of each first pre-training image to obtain a pre-trained first neural network, wherein the first neural network is used to obtain a corresponding first pre-training detection result based on each first pre-training image in the plurality of first pre-training images; Based on multiple second pre-trained images and the foreground object labeling results of each second pre-trained image, each second neural network in the at least one second neural network is pre-trained to obtain at least one pre-trained second neural network, wherein at least one additional pre-trained input is generated based on each second pre-trained image and a corresponding second pre-trained detection result is obtained based on each additional pre-trained input in the at least one additional pre-trained input and the corresponding second neural network, the at least one additional pre-trained input at least represents a change in the second pre-trained image, and the first pre-trained image and the second pre-trained image may be the same or different pre-trained images.

18. The training method according to claim 17, wherein: The fusion module is used to fuse the output of the first neural network and the output of the at least one second neural network based on the parameter model, and the training method also includes: Based on a plurality of first training images and an object annotation result of each of the plurality of first training images, one of the following is trained to obtain the trained neural network: i) the parameterized model on which the fusion module is based; or ii) the parameterized model, and the first neural network and / or the at least one second neural network included in the neural network; The output of the pre-trained first neural network and the output of the at least one pre-trained second neural network are used as inputs of the fusion module, and at least the parameterized model is trained based on the object labeling result of the first training image as the output true value.

19. The training method according to claim 18, wherein: Each of the at least one additional pre-trained input represents at least one of the following changes in the corresponding second pre-trained image: a change between the corresponding second pre-trained image and a corresponding reference image, the corresponding reference image including prior knowledge of a target area targeted by the corresponding second pre-trained image; Changes between regions in the corresponding second pre-training image; and The change between the corresponding second pre-training image and the second image, the corresponding second pre-training image and the second image are images captured adjacent to each other in time for the target area.

20. The training method according to claim 19, wherein: The at least one additional pre-training input includes a first additional pre-training input, the first additional pre-training input represents a change between a corresponding second pre-training image and a corresponding second reference image and is used to train a specific second neural network in the at least one second neural network, each second neural network in the at least one second neural network includes a second encoder and a second decoder, and pre-training the specific second neural network includes: generating the first additional pre-training input based on the corresponding second pre-training image and the second reference image, using the first additional pre-training input as the input of the specific second neural network, and using the foreground object labeling result of the corresponding second pre-training image as the output true value of the specific second neural network to pre-train the specific second neural network; or, The specific second neural network is pre-trained by taking the corresponding second pre-trained image and the second reference image as inputs of the specific second neural network, and taking the foreground object labeling result of the corresponding second pre-trained image as the output true value of the specific second neural network, wherein the second encoder of the specific second neural network is used to generate the first additional pre-trained input based on the input corresponding second pre-trained image and the second reference image, and determine a second feature map based on the first additional pre-trained input, and the second decoder of the specific second neural network is used to obtain a specific second detection result based on the second feature map as the output of the specific second neural network.

21. The training method according to claim 20, wherein: The method of obtaining the trained neural network based on a plurality of first training images and an object labeling result of each of the plurality of first training images comprises: Generate a first additional training input based on each of the first training images and the corresponding first reference image, use the first additional training input as the input of a pretrained specific second neural network in the at least one pretrained second neural network, determine a second feature map based on the first additional training input by the second encoder of the pretrained specific second neural network, and obtain a specific second detection result as the output of the pretrained specific second neural network based on the second feature map for the first additional training input by the second decoder of the pretrained specific second neural network; or Using each of the first training images and the corresponding first reference image as inputs of a pre-trained specific second neural network among the at least one pre-trained second neural network, generating the first additional training input by the second encoder of the pre-trained specific second neural network, determining a second feature map based on the first additional training input, and obtaining a specific second detection result as an output of the pre-trained specific second neural network by the second decoder of the pre-trained specific second neural network based on the second feature map for the first additional training input; The first additional training input represents the change between each of the first training images and the corresponding first reference image, each of the first training images and the corresponding first reference image are acquired for the same target area, and the first reference image includes prior knowledge about the same target area.

22. The training method according to claim 21, wherein: The plurality of first training images and the corresponding first reference images are obtained by the following steps: Combining a plurality of first reference images containing only backgrounds with the annotated foreground objects to form the plurality of first training images, so that each first training image is a combination of the annotated foreground object and the first reference image including the corresponding background; and A plurality of first reference images corresponding to each of the plurality of first training images are transformed to obtain transformed first reference images corresponding to each of the first training images.

23. The training method according to claim 19, wherein: The first neural network includes a first encoder and a first decoder, the first encoder is used to extract a first feature map based on an input image, and the first decoder is used to obtain a first detection result based on the first feature map as an output of the first neural network; The first encoder and the second encoder are the same encoder or different encoders.

24. The training method according to claim 18, wherein: The parameterized model includes a multi-layer perceptron or a gated recurrent unit.

25. A perception device for autonomous driving, comprising: Memory, and A processing unit for: Obtaining a first detection result for the first image using a first neural network based on the first image, wherein the first image is obtained for a target area; Obtaining at least one second detection result for the first image using at least one second neural network based on the first image, wherein obtaining the at least one second detection result comprises: generating at least one additional input based on the first image, each of the at least one additional input representing at least a change in the first image, and obtaining the at least one second detection result for the first image using the at least one second neural network based on the at least one additional input; and The first detection result and the at least one second detection result are fused to obtain a final perception result for the first image.

26. An automatic driving system comprising: a camera for acquiring a first image of a target area; and A perception device for autonomous driving according to claim 25.

27. A computer-readable medium storing a computer program, which, when executed by a processor, causes the processor to perform the perception method for autonomous driving according to any one of claims 1 to 16 or the training method for a neural network according to any one of claims 17 to 24.

Images