Neural network, training method, environment sensing system and method, and program product

Abstract

The present application relates to a neural network system, including at least: a multi-modal input neural network, configured to perform first encoding on multiple pieces of acquired environment sensing data corresponding to multiple input modalities to obtain multiple pieces of feature data for the multiple pieces of environment sensing data and perform first fusion on the multiple pieces of feature data to obtain fused multi-modal feature data; an encoder neural network, configured to at least perform second encoding on the multi-modal feature data to obtain multi-scale feature data, wherein the encoder neural network includes, for the multi-modal feature data, multiple stages respectively including at least two parallel branches for different resolutions; a decoder neural network, configured to decode the multi-scale feature data to obtain decoder output data; and a multi-modal output neural network, configured to obtain an environment sensing result when the decoder output data is used. The present application further relates to a training method, an environment sensing system and method, and a computer program product.

Description

NEURAL NETWORK, TRAINING METHOD, ENVIRONMENT SENSING SYSTEM AND METHOD, AND PROGRAM PRODUCTTECHNICAL FIELD

[0001] The present application relates to a neural network system. The present application further relates to a training method for a neural network system, an environment sensing system, an environment sensing method, and a computer program product.BACKGROUND

[0002] With increasing development of automation technology and artificial intelligence, autonomous systems are increasingly used in various industries. For example, the autonomous systems are widely used in medical assistance, home services, intelligent transportation, Mars exploration, and the like. It is very important herein to achieve accuracy and efficiency of scene recognition / object detection. Generally, data redundancy is achieved by fusing data from different information sources to improve accuracy. However, such data fusion may result in high computational complexity and memory usage. In addition, in object detection / environment sensing, there are items of small scales and items of large scales, and the items usually present different appearances at different scales, which requires that an object detection system / environment sensing system be able to adapt to detection requirements of items of different scales.SUMMARY

[0003] The present application is based on a task of proposing an improved technical solution, particularly a multi-modal end-to-end deep learning modeling solution, so that data from different information sources can be fused / associated, and multi-scale feature fusion can be performed on the data and can be performed in a computationally efficient manner, thereby improving accuracy, computational efficiency, generalization, and the like of object detection / environment sensing.

[0004] The task is achieved by a neural network system, a training method for a neural network system, an environment sensing system, an environment sensing method, and a computer program product. Advantageous embodiments of the present application are also described in detail hereinafter.

[0005] According to a first aspect of the present application, a neural network system is provided. The neural network system at least comprises :

[0006] a multi-modal input neural network, configured to perform first encoding on multiple pieces of acquired environment sensing data corresponding to multiple input modalities to obtain multiple pieces of feature data for the multiple pieces of environment sensing data, and perform first fusion on the multiple pieces of feature data to obtain fused multi-modal feature data;

[0007] an encoder neural network, configured to at least perform second encoding on the multi-modal feature data to obtain multi-scale feature data, wherein the encoder neural network comprises multiple stages for the multimodal feature data, and the multiple stages respectively comprise at least two parallel branches for different resolutions;

[0008] a decoder neural network, configured to decode the multi-scale feature data to obtain decoder output data; and

[0009] a multi-modal output neural network, configured to obtain an environment sensing result when the decoder output data is used.

[0010] Within the scope of the present application, an “input modality” can be understood as input data, and therefore, different input modalities mean different input data. Exemplarily, these input modalities and / or these pieces of input data differ at least in at least one of the following: a sensing principle, a sensing device, a sensing region (field of view), and sensing time. Preferably, these input modalities and / or these pieces of input data are for a same scene. In other words, sensing regions corresponding to these input modalities and / or these pieces of input data at least partially overlap each other.

[0011] In the scope of the present application, “multi-scale feature data” can be understood as comprising both low-level detail information and / or texture information and high-level semantic information. Within the scope of thepresent application, “at least two parallel branches for different resolutions” can be understood as at least two parallel branches differing in resolution. Exemplarily, a first branch has a first resolution, a second branch has a second resolution lower than the first resolution, a third branch has a third resolution lower than the first resolution and the second resolution, and so on.

[0012] According to the present application, based on a neural network, especially based on a deep learning model architecture, multi-modal fusion and multi-scale fusion of sensing data are implemented, data association is performed on the multiple input modalities, and a low-level texture feature and a high-level semantic feature are fused, so that accuracy and generalization are improved.

[0013] According to a preferred embodiment arrangement of the present application, encoder input data of each of the multiple stages of the encoder neural network is from the multi-modal feature data or encoder output data of a previous stage preceding the stage. Each individual input modality herein has its own encoding layer and, after respective (first) encoding, enters the encoder neural network in a fused manner, that is, in a form of fused multimodal feature data. Advantageously, encoder input data (of an initial stage or a first stage) of the encoder neural network is a multi-modal fusion result.

[0014] According to a preferred embodiment arrangement of the present application, the encoder neural network is configured to perform unimodal second encoding respectively on at least one piece of the multiple pieces of feature data to obtain multi-scale feature data; the encoder neural network comprises further multiple stages respectively for the at least one piece of the multiple pieces of feature data, and the further multiple stages respectively comprise at least two parallel branches for different resolutions; encoder input data of each of the multiple stages of the encoder neural network comprises the multi-modal feature data and / or encoder output data of a previous stage preceding each of the multiple stages and / or further encoder output data of a previous stage preceding each of the further multiple stages. A network architecture of the encoder neural network herein is decomposed at a fine granularity. Each individual input modality herein has its own unimodalencoding network of the encoder neural network. In other words, each individual input modality, after the respective (first) encoding, additionally enters its own unimodal encoding network of the encoder neural network in an individual manner, that is, in a form of feature data. Corresponding encoder output data of these unimodal encoding networks additionally also enters the encoder neural network separately. Advantageously, the encoder input data of the encoder neural network comprises a unimodal result in addition to the multi-modal fusion result, so that relevance of data and interactivity of information are improved.

[0015] According to a preferred embodiment arrangement of the present application, the decoder neural network is configured to decode further encoder output data of a last stage of the further multiple stages of the encoder neural network to obtain the decoder output data. Advantageously, the relevance of the data and the interactivity of the information are further increased.

[0016] According to a preferred embodiment arrangement of the present application, each of the multiple stages and / or each of the further multiple stages of the encoder neural network at least comprises: at least one of the following operations: a branch generation operation, used for generating another parallel branch, wherein the another parallel branch is for a lower resolution; and a branch fusion operation, used for fusing encoder output data of at least two parallel branches of each of the multiple stages and / or further encoder output data of at least two parallel branches of each of the further multiple stages. Advantageously, high resolution information can always be maintained.

[0017] According to a preferred embodiment arrangement of the present application, the encoder neural network comprises a first attention enhancement module, and the first attention enhancement module is used to fuse at least two parallel branches of at least one of the multiple stages and / or fuse at least two parallel branches of at least one of the further multiple stages. Advantageously, weights are dynamically adjusted according to importance of branch data to be fused.

[0018] According to a preferred embodiment arrangement of the present application, the multi-modal output neural network comprises a second attention enhancement module, and the second attention enhancement module is related to at least one, preferably at least two, of the multiple input modalities. Advantageously, weights are dynamically adjusted according to importance of input data to be fused. Exemplarily, it is possible to dynamically adapt to different quantities of input modalities (for example, one input modality, two input modalities, or more input modalities) or different cases of input modalities.

[0019] According to a preferred embodiment arrangement of the present application, the second attention enhancement module is related to detection regions of the at least two of the multiple input modalities. Exemplarily, the detection regions of the at least two input modalities are independent of each other or overlap with each other. Exemplarily, in a case of mutual overlapping, the size (or degree of overlap) of an overlapping region, the position of the overlapping region, and the like can additionally also be determined. Advantageously, weights are dynamically adjusted according to importance and / or relevance of input data to be fused.

[0020] According to a preferred embodiment arrangement of the present application, the multiple input modalities at least comprise: at least one, preferably at least two, of the following input modalities: a radar input; an ultrasonic input; a visual input; a laser radar input; and an infrared input. It can be understood that the input modalities listed herein are merely exemplary, rather than limiting.

[0021] According to a preferred embodiment arrangement of the present application, in training of the neural network system, a loss function comprises an auxiliary loss and / or an HEGM (hierarchical elastic graph matching) loss. Advantageously, a learning process / training process of the neural network system is optimized. Alternatively or additionally, according to a preferred embodiment arrangement of the present application, in the training of the neural network system, the loss function can be calculated individually or correlatively, and / or the loss function is defined to be less than a predeterminedthreshold. Advantageously, the learning process / training process of the neural network system is carried out, particularly in an association manner.

[0022] According to a second aspect of the present application, a training method for the neural network system according to the present application is provided.

[0023] According to a third aspect of the present application, an environment sensing system is provided, the environment sensing system comprises the neural network system according to the present application.

[0024] According to a fourth aspect of the present application, an environment sensing method is provided, the environment sensing method is used to perform environment sensing when the neural network system according to the present application is applied.

[0025] According to a fifth aspect of the present application, a computer program product, for example, a computer-readable program medium, is provided. The computer program product comprises or stores computer program instructions that, when executed by a processor, cause the training method according to the present application and / or the environment sensing method according to the present application to be at least partially executed.

[0026] Features and details described in connection with the neural network system according to the present application also apply to the training method according to the present application, the environment sensing system according to the present application, the environment sensing method according to the present application, and the computer program product according to the present application, and vice versa.BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present application is described in more detail below with reference to the accompanying drawings. The accompanying drawings show:

[0028] FIG. 1 is a schematic diagram of a schematic network architecture of a neural network system according to the present application;

[0029] FIG. 2 is a schematic block diagram of a training method according to the present application;

[0030] FIG. 3 shows an exemplary embodiment of an encoder neural network ENC2 of a neural network system according to the present application;

[0031] FIG. 4 shows an exemplary embodiment of a neural network system according to the present application and an exemplary embodiment of a training method according to the present application, where FIG. 4(a) exemplarily shows a training method for an individual input modality, and FIG. 4(b) exemplarily shows a training method for multiple input modalities;

[0032] FIG. 5 shows another exemplary embodiment of a neural network system according to the present application and another exemplary embodiment of a training method according to the present application, where FIG. 5(a) exemplarily shows a training method for an individual input modality, FIG. 5(b) exemplarily shows a training method for multiple input modalities, and FIG. 5(c) and FIG. 5(d) exemplarily show possible embodiments of the training method for multiple input modalities after dimension reduction;

[0033] FIG. 6 is a schematic block diagram of an environment sensing system according to the present application;

[0034] FIG. 7 is a schematic flowchart of an environment sensing method according to the present application; and

[0035] FIG. 8 is a schematic diagram of a computer program product according to the present application.DETAILED DESCRIPTION

[0036] To make the technical tasks, the technical solutions, and the technical effects of the present application clearer, the present application is further described below in detail with reference to the accompanying drawings and exemplary embodiments. It should be understood that the exemplary embodiments described herein are only used for explanation, but not for limitation.

[0037] In some examples, the terms first, second, and the like are used for description. However, it should be understood that these terms are only used to distinguish one element from another unless stated otherwise.

[0038] FIG. 1 is a schematic diagram of a schematic network architecture of a neural network system 1 according to the present application.

[0039] The neural network system 1 includes or is connected to a multimodal input neural network ENC1 . The multi-modal input neural network ENC1 is configured to perform first encoding on multiple pieces of acquired environment sensing data IN_1 , IN_2, and IN_3 corresponding to multiple input modalities to obtain multiple pieces of feature data for the multiple pieces of environment sensing data IN_1 , IN_2, and IN_3. The feature data herein has a high resolution. The multi-modal input neural network ENC1 is further configured to perform first fusion on the multiple pieces of feature data to obtain fused multi-modal feature data OLIT1.

[0040] Exemplarily, the multiple input modalities or the multiple pieces of environment sensing data IN_1 , IN_2, and IN_3 include at least: at least one, preferably at least two, of the following input modalities: a radar input; an ultrasonic input; a visual input; a laser radar input; and an infrared input. It should be noted that types and / or quantities of input modalities listed herein are merely exemplary, rather than limiting. Hereinafter, an example in which the radar input is used as a first input modality, the ultrasonic input is used as a second input modality, and the visual input is used as a third input modality is used for description.

[0041] In one embodiment, the multi-modal input neural network ENC1 outputs the fused multi-modal feature data OLIT1. Additionally, in one embodiment, the multi-modal input neural network ENC1 further outputs unimodal feature data respectively corresponding to the multiple pieces of environment sensing data IN_1 , IN_2, and IN_3.

[0042] The neural network system 1 includes an encoder neural network ENC2, that is, a multi-modal encoder neural network. The encoder neural network ENC2 is configured to at least perform second encoding on the multimodal feature data OLIT1 to obtain multi-scale feature data OLIT2. The encoder neural network ENC2 includes multiple stages S1 , S2, S3, and S4 for the multi-modal feature data OLIT1 , and the multiple stages S1 , S2, S3, and S4 respectively include at least two parallel branches for different resolutions.The at least two parallel branches herein differ in resolution. In other words, the encoder neural network ENC2 includes both low-level detail information and / or texture information and high-level semantic information. Hereinafter, the encoder neural network ENC2 and the multiple stages S1 , S2, S3, and S4 thereof are described in detail by using HRNet (High-Resolution Net). Details are not described here.

[0043] The neural network system 1 includes a decoder neural network DEC. The decoder neural network DEC is configured to decode the multi-scale feature data OLIT2 to obtain decoder output data.

[0044] The neural network system 1 includes a multi-modal output neural network HEAD. The multi-modal output neural network HEAD is configured to obtain an environment sensing result when the decoder output data is used. Within the scope of the present application, the multi-modal output neural network HEAD can be understood as a segmentation task, especially a pixellevel segmentation task, for example a segmentation head, for classification prediction, especially pixel-level classification prediction. Such classification prediction can be applied to medical assistance, home services, intelligent transportation, Mars exploration, and the like, for example, for autonomous navigation, obstacle avoidance, and simultaneous localization and mapping (SLAM). Exemplarily, environment sensing and mapping can be performed in robots and autonomous systems (for example, self-driving vehicles), for example, an evidential occupancy grid map that represents confidence or an evidence level of a spatial occupancy state of each grid cell in a map in a probability distribution manner.

[0045] Exemplarily, the multi-modal output neural network HEAD includes a second attention enhancement module. The second attention enhancement module is related to at least one, preferably at least two, of the multiple input modalities IN_1 , IN_2, and IN_3. Preferably, the second attention enhancement module is related to relevance or detection regions of the at least two of the multiple input modalities IN_1 , IN_2, and IN_3. For example, an overlapping region of the detection regions of the multiple pieces of environment sensing data IN_1 , IN_2, and IN_3 and a position thereof in theentire detection region are determined, and a weight is dynamically adjusted based thereon. In this way, spatially associated detection regions and objects or surroundings located therein, for example, objects or surroundings detected simultaneously by two or more sensing devices, can be processed in an improved and more precise manner.

[0046] FIG. 2 is a schematic block diagram of a training method 10 according to the present application.

[0047] In step S101 , multiple pieces of environment sensing data corresponding to multiple input modalities, such as a first input modality IN_1 , a second input modality IN_2, and a third input modality IN_3, are acquired. Additionally, in step S101 , one piece of or multiple pieces of corresponding label data are further acquired.

[0048] In step S103, the multiple pieces of environment sensing data are input into a neural network system 1 to be trained. Exemplarily, feature data and / or encoder output data and / or decoder output data and / or an environment sensing result (for example, a classification prediction result, for example, an occupancy state) are obtained herein to be used as intermediate output data or output data.

[0049] In step S105, the intermediate output data or the output data obtained in step S103 is compared with the label data. In this case, for example, a loss function is calculated, to quantify the difference between the classification prediction result and an actual result of the neural network system 1 or of a deep learning model 1. Exemplarily, when the loss function is less than a predetermined threshold, it is considered that the training method 10 can be stopped.

[0050] Exemplarily, in addition to a main loss function, an auxiliary loss function is introduced. This is very advantageous for a convolutional neural network (CNN), especially a deep convolutional neural network. In this way, the loss can be transferred to a shallow layer, and a learning process of the neural network system 1 or the deep learning model 1 can be optimized.

[0051] Exemplarily, an HEGM loss is also introduced, and in particular, an HEGM loss is introduced in a learning process / training process of a multimodal encoder neural network ENC2.

[0052] Exemplarily, a loss function for an individual modality (for example, F1_LOSS_1 ) and a loss function for multiple modalities (for example, F2_LOSS) can be calculated separately or correlatively. In other words, training of a corresponding neural network (ENC1_1 ) or a corresponding unimodal encoding network (ENC2") of a neural network stops when the loss function (for example, F1_LOSS_1 and F2_LOSS") for an individual input modality is less than a predetermined threshold thereof, or training of a corresponding unimodal encoding network (ENC2") of a neural network stops when the loss function (for example, F1_LOSS_1 and F2_LOSS") for an individual input modality is less than a predetermined threshold thereof, and the loss function (F2_LOSS) for the multiple input modalities is also less than a predetermined threshold thereof.

[0053] Hereinafter, the training method 10 is further described in detail in connection with exemplary embodiments of the neural network system 1 .

[0054] FIG. 3 shows an exemplary embodiment of an encoder neural network ENC2 of a neural network system 1 according to the present application.

[0055] The encoder neural network ENC2, that is, a multi-modal encoder neural network ENC2, is configured to perform second encoding on the multimodal feature data OUT 1 to obtain multi-scale feature data OUT2. Additionally, the encoder neural network ENC2, that is, the multi-modal encoder neural network ENC2, is further configured to perform second encoding on unimodal feature data respectively corresponding to the multiple pieces of environment sensing data IN_1 , IN_2, and IN_3 to obtain multi-scale feature data OUT2.

[0056] The encoder neural network ENC2 herein exemplarily includes four stages S1 , S2, S3, and S4. A box herein represents a channel map or a feature map, an arrow extending in the horizontal direction represents a convolution operation or a convolution unit, an arrow extending in the diagonally downward direction represents down-sampling, and an arrow extending in the diagonallyupward direction represents up-sampling. The convolution depth herein is from shallow to deep, parallel branches are referred to as a first branch, a second branch, a third branch, and a fourth branch from top to bottom, respectively, and the resolution is from high to low (or semantic information is from weak to strong) in sequence.

[0057] The first stage S1 (at the top in FIG. 3) includes a basic structure K1 and a transition structure K2, as shown in dashed-line boxes. There is a convolution operation in the basic structure K1 , for example, a 3x3 convolution kernel is used. There is also a branch generation operation in the transition structure K2. The branch generation operation generates another parallel branch by, for example, a two-time down-sampling operation. The another parallel branch is for a lower resolution, has a larger receptive field, and obtains stronger semantic information.

[0058] The second stage S2 (at the top of FIG. 3) and the third stage S3 (at the bottom of FIG. 3) are similar to the first stage S1 , and the main difference is that there is also a branch fusion operation in the transition structure K2. The branch fusion operation is used to fuse encoder output data of different parallel branches, for example, by using a down-sampling operation and / or an up- sampling operation. Using the transition structure K2 of the third stage S3 as an example, for a branch fusion operation of the first branch, an input thereof is a first branch of a previous layer, two times up-sampling of a second branch of the previous layer, and four times up-sampling of a third branch of the previous layer. For a branch fusion operation of the second branch, an input thereof is 3x3 convolution down-sampling of the first branch of the previous layer, the second branch of the previous layer, and two times up-sampling of the third branch of the previous layer. For a branch fusion operation of the third branch, an input thereof is two consecutive 3x3 convolution down-samplings of the first branch of the previous layer, 3x3 convolution down-sampling of the second branch of the previous layer, and the third branch of the previous layer. In this way, information exchange of different branches is achieved. In addition, for a branch generation operation of the fourth branch, one possible example is that the fourth branch is generated only by using 3x3 convolution down-sampling of the third branch of the previous layer, and another possible example is that the fourth branch is additionally generated in a fused manner by using branches of a lower layer, such as the first branch and / or the second branch (for example, using consecutive multiple times of 3x3 convolution down-sampling).

[0059] The fourth stage S4 (at the bottom of FIG. 3) is similar to the second stage S2 and the third stage S3, and the main difference is that there is no branch generation operation in the transition structure K2. However, it is not excluded to perform the branch generation operation if necessary, as in the first stage S1 to the third stage S3.

[0060] Exemplarily, an attention mechanism is introduced into the encoder neural network ENC2. For example, a first attention enhancement module is introduced into the encoder neural network ENC2, and the first attention enhancement module is used to fuse at least two parallel branches, for example, is used in a branch fusion operation.

[0061] Exemplarily, atrous convolution can be used in the convolution operation and / or the down-sampling in the encoder neural network ENC2 or the neural network system 1 , so that a receptive field can be enlarged without increasing the amount of computation. In addition, different atrous rates are set, so that feature responses at different scales can be obtained.

[0062] In this way, high resolution can be always maintained and high- resolution low-level texture information and high-level semantic information can be obtained / maintained simultaneously.

[0063] Finally, depending on specific application, only feature maps with the highest resolution can be used, as shown by the encoder output data of the first branch of the fourth stage S4. Alternatively or additionally, the feature maps of all branches can also be associated, for example, for subsequent semantic segmentation and especially for keypoint detection. Alternatively or additionally, a feature pyramid can further be formed on the basis of associating the feature maps of branches, for example, for object detection.

[0064] It should be noted that the network architecture of the encoder neural network ENC listed herein is merely exemplary, rather than limiting. It can beunderstood that the network architecture of the encoder neural network ENC can at least satisfy that high-resolution information and high semantic information are always maintained simultaneously, and / or at least satisfy that a feature response of a small scale and a feature response of a large scale are always obtained simultaneously. It can be understood that the number of the multiple stages, the number of layers of the basic structure K1 , the size of the convolution kernel, the rate of the up-sampling or down-sampling, and the like may alternatively be other values.

[0065] FIG. 4 shows an exemplary embodiment of a neural network system 1 according to the present application and an exemplary embodiment of a training method 10 according to the present application, wherein FIG. 4(a) exemplarily shows a training method for an individual input modality I N_1 , and FIG. 4(b) exemplarily shows a training method for multiple input modalities IN_1 , IN_2, and IN_3.

[0066] In FIG. 4(a), an acquired first input modality, that is, a radar input IN_1 , is input into a first sub-network ENC1_1 of a multi-modal input neural network ENC1 to be trained to obtain first feature data. The first feature data is decoded by the DEC1_1 to obtain a first classification prediction result, for example, a first occupancy state. The first classification prediction result is compared with first label data to obtain a value of a first loss function F1_LOSS_1. For example, an auxiliary loss is introduced. The first subnetwork ENC1_1 is trained until the value of the first loss function F1_LOSS_1 satisfies a predetermined condition. The same or similar situation applies to an acquired second input modality, that is, an ultrasonic input IN_2, and an acquired third input modality, that is, a visual input IN_3. Details are not described herein.

[0067] In FIG. 4(b), the first feature data, the second feature data, and the third feature data corresponding to the first input modality (radar input) IN_1 , the second input modality (ultrasonic input) IN_2, and the third input modality (visual input) IN_3, that is, the fused multi-modal feature data OLIT1 , are collectively input into the encoder neural network ENC2 in a fusion manner, and in particular, are used as the first encoder input data to be collectively inputinto the first stage S1 (with reference to FIG. 3) of the encoder neural network ENC2 to obtain first encoder output data. Subsequently, the first encoder output data is used as second encoder input data to be input into the second stage S2 (with reference to FIG. 3) of the encoder neural network ENC2 to obtain second encoder output data. Subsequently, the second encoder output data is used as third encoder input data to be input into the third stage S3 (with reference to FIG. 3) of the encoder neural network ENC2 to obtain third encoder output data. Finally, the third encoder output data is used as fourth encoder input data to be input into the fourth stage S4 (with reference to FIG. 3) of the encoder neural network ENC2 to obtain fourth encoder output data, that is, multi-scale feature data OLIT2. The multi-scale feature data OLIT2 is decoded by a decoder neural network DEC to obtain decoder output data. The decoder output data is subjected to, for example, semantic segmentation by using a multi-modal output neural network HEAD to obtain a classification prediction result, for example, especially an occupancy state in a form of a probability. The classification prediction result is compared with corresponding label data to obtain a value of a loss function F2_LOSS. For example, an HEGM loss is introduced. The encoder neural network ENC2 is trained until the value of the loss function F2_LOSS satisfies a predetermined condition.

[0068] FIG. 5 shows another exemplary embodiment of a neural network system 1 according to the present application and another exemplary embodiment of a training method 10 according to the present application, where FIG. 5(a) exemplarily shows a training method for an individual input modality, FIG. 5(b) exemplarily shows a training method for multiple input modalities, and FIG. 5(c) and FIG. 5(d) exemplarily show possible embodiments of the training method for multiple input modalities after dimension reduction.

[0069] FIG. 5(a) is similar to FIG. 4(a). Details are not described herein again.

[0070] In comparison with FIG. 4(b), in FIG. 5(b), the encoder neural network ENC2 is additionally configured to perform unimodal second encoding ENC2 on at least one piece, preferably all feature data, of the multiple piecesof feature data to obtain multi-scale feature data. As shown in FIG. 5(b), in addition to a multi-modal encoding network ENC2", which is similar to that in FIG. 4(b) and includes four stages S1 , S2, S3, and S4, the encoder neural network ENC2 further includes a unimodal encoding network ENC2' for each individual input modality. The unimodal encoding network also includes four stages S1 , S2, S3, and S4, as shown in FIG. 5(b) by using vertical dashed lines. Correspondingly, the decoder neural network DEC further includes, in addition to the multi-modal decoding network DEC", a unimodal encoding network DEC' for each individual input modality.

[0071] In FIG. 5(b), the first feature data, the second feature data, and the third feature data corresponding to the first input modality (radar input) IN_1 , the second input modality (ultrasonic input) IN_2, and the third input modality (visual input) IN_3 are separately input into the corresponding unimodal encoding network ENC2' of the encoder neural network ENC2, and in particular, are used as the first encoder input data to be input into the first stage S1 of the unimodal encoding network ENC2' of the encoder neural network ENC2 to obtain the first encoder output data. Using the first input modality (radar input) as an example (with reference to FIG. 5(c)), the first feature data corresponding to the first input modality IN_1 (on which first encoding is performed via ENC1_1 ) is used as the first encoder input data to be separately input into a first stage ENC2'_S1 of the unimodal encoding network ENC2' of the encoder neural network ENC2 to obtain the first encoder output data. Subsequently, the first encoder output data is used as second encoder input data to be input into a second stage ENC2"_S2 of the unimodal encoding network ENC2' of the encoder neural network ENC2 to obtain second encoder output data. Subsequently, the second encoder output data is used as third encoder input data to be input into a third stage ENC2'_S3 of the unimodal encoding network ENC2' of the encoder neural network ENC2 to obtain third encoder output data. Finally, the third encoder output data is used as fourth encoder input data to be input into a fourth stage ENC2'_S4 of the unimodal encoding network ENC2' of the encoder neural network ENC2 to obtain fourth encoder output data. The fourth encoder output data is decoded by theunimodal decoding network DEC'_1 of the decoder neural network DEC to obtain decoder output data. The decoder output data is compared with corresponding label data to obtain the value of the loss function F2_LOSS'_1 . The unimodal encoding network ENC21of the encoder neural network ENC2 is trained until the value of the loss function F2_LOSS'_1 satisfies a predetermined condition. The same or similar situation applies to an acquired second input modality, that is, an ultrasonic input IN_2, and an acquired third input modality, that is, a visual input IN_3. Details are not described herein.

[0072] Meanwhile, in FIG. 5(b), the first feature data, the second feature data, and the third feature data corresponding to the first input modality (radar input) IN_1 , the second input modality (ultrasonic input) IN_2, and the third input modality (visual input) IN_3 are collectively input into the encoder neural network ENC2 in a fusion manner, namely as the fused multi-modal feature data OUT1 , and in particular, are used as the first encoder input data to be collectively input into a first stage S1 of the multi-modal encoding network ENC2" of the encoder neural network ENC2 to obtain first encoder output data. Subsequently, the first encoder output data is used as second encoder input data to be input into a second stage S2 of the multi-modal encoding network ENC2" of the encoder neural network ENC2 to obtain second encoder output data. Subsequently, the second encoder output data is used as third encoder input data to be input into a third stage S3 of the multi-modal encoding network ENC2" of the encoder neural network ENC2 to obtain third encoder output data. Finally, the third encoder output data is used as fourth encoder input data to be input into a fourth stage S4 of the multi-modal encoding network ENC2" of the encoder neural network ENC2 to obtain fourth encoder output data. Preferably, encoder output data of the individual stages S1 , S2, and S3 of the unimodal encoding network ENC2' is further used herein, in a fused manner, as encoder input data of next stages S2, S3, and S4 to be input into the multimodal encoding network ENC2" of the encoder neural network ENC2. This is shown in FIG. 5(b) by using three elbow-shaped arrows each at the output of the individual stages S1 , S2, and S3.

[0073] The fourth encoder output data of the multi-modal encoding network ENC2" is decoded by the multi-modal decoding network DEC" of the decoder neural network DEC to obtain decoder output data. Preferably, as shown in FIG. 5(b) by using three elbow arrows at an output of the last stage S4, the fourth encoder output data of the unimodal encoding network ENC2' is additionally decoded in a fused manner by the multi-modal decoding network DEC" of the decoder neural network DEC to obtain decoder output data. The decoder output data is subjected to, for example, semantic segmentation to obtain a classification prediction result (for example, an occupancy state), especially a classification prediction result in a form of a probability. The classification prediction result is compared with corresponding label data to obtain a value of a loss function F2_LOSS". The encoder neural network ENC2 is trained until the value of the loss function F2_LOSS" satisfies a predetermined condition.

[0074] Preferably, in an example, the neural network system 1 includes a second attention enhancement module. The second attention enhancement module is related to at least one, preferably at least two, of the multiple input modalities IN_1 , IN_2, and IN_3. For example, when there are only two input modalities, a weight can be dynamically adjusted based on the two input modalities. For example, when there is only one input modality, for example, a radar input, a weight can be dynamically adjusted based on the one input modality, and the neural network system 1 “performs dimension reduction” or “degrades” as shown in FIG. 5(c) or FIG. 5(d). Advantageously, this allows dynamic adaptation to different numbers of input modalities.

[0075] FIG. 6 is a schematic block diagram of an environment sensing system 2 according to the present application.

[0076] The environment sensing system 2 includes a detection module 201 . The detection module is configured to perform object detection / environment sensing, especially for the same scene. Exemplarily, the detection module 201 includes, but is not limited to, a radar sensor, an ultrasonic sensor, a visual sensor (for example, a video sensor or a camera), a laser radar, an infraredsensor, and the like. It can be understood that the input modalities listed herein are merely exemplary, rather than limiting.

[0077] The environment sensing system 2 further includes a processing module 203. The processing module includes the neural network system 1 according to the present application. Exemplarily, one piece of or multiple pieces of input data acquired by the detection module 201 are processed by the processing module 203 to generate an established or updated evidential occupancy grid map.

[0078] The environment sensing system 2 further includes an execution module 205. The execution module uses result data received from the processing module 203 for subsequent processing. Exemplarily, an autonomous system, for example, a vehicle, can perform navigation, obstacle avoidance, mapping, positioning, and the like based on an environment sensing result. Exemplarily, a robot can perform autonomous navigation, simultaneous localization and mapping, and the like based on the environment sensing result.

[0079] FIG. 7 is a schematic flowchart of an environment sensing method 20 according to the present application.

[0080] In step S202, multiple pieces of environment sensing data IN_1 , IN_2, and IN_3 corresponding to multiple input modalities are acquired.

[0081] In step S204, the multiple pieces of environment sensing data IN_1 , IN_2, and IN_3 are input into the processing module 203 of the environment sensing system 2. In this way, data from different information sources can be fused, and objects at different scales can be detected.

[0082] In step S206, a corresponding instruction is generated according to a detection result obtained in step S204, to subsequently implement corresponding processing measures.

[0083] FIG. 8 is a schematic diagram of a computer program product 30 according to the present application.

[0084] The computer program product 30 includes computer program instructions that, when executed by a processor or controller, cause the training method 10 according to the present application and / or the environmentsensing method 20 according to the present application to be at least partially executed. For example, the processor may be a central processing unit (CPU), or may be another general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or the like. The general-purpose processor may be a microprocessor, any conventional processor, or the like.

[0085] The computer program product 30 is stored on a computer-readable program medium, or in other words, the computer-readable program medium stores the computer program instructions of the computer-readable program medium. When the computer program instructions are executed by the processor or the controller, the computer program instructions can at least partially perform the training method 10 according to the present application and / or the environment sensing method 20 according to the present application.

[0086] In the present application, a multi-modal end-to-end deep learning modeling solution is proposed on a deep learning model architecture. Based on pixel-level classification and accuracy regression tasks, a multi-scale fusion technical solution is used on structure modeling to fuse high-level semantic features and low-level texture features, so that semantic information at different scales is effectively captured, and a detection and segmentation capability of a semantic segmentation network for items of different sizes is improved. In particular, the present application has the following advantages:

[0087] enhancing context information: By fusing features at multi-scales, a receptive field is expanded, and a larger range of context information is obtained, so that a more accurate semantic segmentation result / classification result is provided;

[0088] preserving details and textures: By fusing features at multi-scales, when detail and texture information is retained, higher-level semantic information is also obtained, which helps to more accurately distinguish regions with similar texture but belonging to different semantic categories;

[0089] adapting to objects with different scales: For a case in which the objects / targets in an image have different scales, an item of a large scale andan item of a small scale can be better detected by fusing the multi-scale features, so that detection and segmentation requirements of objects of different scales can be better adapted, thereby improving robustness and generalizability of a model; and

[0090] alleviating a problem of scale invariance: Items in an image may present different appearances at different scales, and a semantic segmentation network can better handle scale changes by fusing multi-scale features, so that impact of scale invariance on segmentation performance is alleviated.

[0091] Although particular embodiments of the present application have been described in detail herein, the particular embodiments are given merely for the purpose of explanation, and should not be considered to limit the scope of the present application. Various replacement solutions and modification solutions may be provided without departing from the core and scope of the present application.

Claims

CLAIMS1 . A neural network system, at least comprising: a multi-modal input neural network, configured to perform first encoding on multiple pieces of acquired environment sensing data corresponding to multiple input modalities to obtain multiple pieces of feature data for the multiple pieces of environment sensing data, and perform first fusion on the multiple pieces of feature data to obtain fused multi-modal feature data; an encoder neural network, configured to at least perform second encoding on the multi-modal feature data to obtain multi-scale feature data, wherein the encoder neural network comprises multiple stages for the multimodal feature data, and the multiple stages respectively comprise at least two parallel branches for different resolutions; a decoder neural network, configured to decode the multi-scale feature data to obtain decoder output data; and a multi-modal output neural network, configured to obtain an environment sensing result when the decoder output data is used.

2. The neural network system according to claim 1 , wherein encoder input data of each of the multiple stages of the encoder neural network is from the multi-modal feature data or encoder output data of a previous stage preceding the stage.

3. The neural network system according to claim 1 , wherein the encoder neural network is configured to perform unimodal second encoding respectively on at least one piece of the multiple pieces of feature data to obtain multi-scale feature data, wherein the encoder neural network comprises further multiple stages respectively for the at least one piece of the multiple pieces of feature data, and the further multiple stages respectively comprise at least two parallel branches for different resolutions, and wherein encoder input data of each of the multiple stages of the encoder neural network comprises the multi-modal feature data and / or encoder output data of a previous stagepreceding each of the multiple stages and / or further encoder output data of a previous stage preceding each of the further multiple stages.

4. The neural network system according to claim 3, wherein the decoder neural network is configured to decode further encoder output data of a last stage of the further multiple stages of the encoder neural network to obtain the decoder output data.

5. The neural network system according to any one of the preceding claims, wherein each of the multiple stages and / or each of the further multiple stages of the encoder neural network at least comprises: at least one of the following operations: a branch generation operation, used for generating another parallel branch, wherein the another parallel branch is for a lower resolution; and a branch fusion operation, used for fusing encoder output data of at least two parallel branches of each of the multiple stages and / or further encoder output data of at least two parallel branches of each of the further multiple stages.

6. The neural network system according to any one of the preceding claims, wherein the encoder neural network comprises a first attention enhancement module, and the first attention enhancement module is used to fuse at least two parallel branches of at least one of the multiple stages and / or fuse at least two parallel branches of at least one of the further multiple stages.

7. The neural network system according to any one of the preceding claims, wherein the multi-modal output neural network comprises a second attention enhancement module, and the second attention enhancement module is related to at least one, preferably at least two, of the multiple input modalities.

8. The neural network system according to claim 7, wherein the second attention enhancement module is related to detection regions of the at least two of the multiple input modalities.

9. The neural network system according to any one of the preceding claims, wherein the multiple input modalities at least comprise: at least one, preferably at least two, of the following input modalities: a radar input; an ultrasonic input; a visual input; a laser radar input; and an infrared input.

10. The neural network system according to any one of the preceding claims, wherein in training of the neural network system, a loss function comprises an auxiliary loss and / or an HEGM loss, and / or the loss function can be calculated individually or correlatively, and / or the loss function is defined to be less than a predetermined threshold.

11. A training method for the neural network system according to any one of claims 1 to 10.

12. An environment sensing system, comprising the neural network system according to any one of claims 1 to 10.

13. An environment sensing method, used to perform environment sensing when the neural network system according to any one of claims 1 to 10 is applied.

14. A computer program product, for example, a computer-readable program medium, characterized in that the computer program product comprises or stores computer program instructions that, when executed by a processor, cause the training method according to claim 11 and / or the environment sensing method according to claim 13 to be at least partially executed.

Images