Semantic segmentation method, apparatus and system, and engineering machinery equipment

Abstract

The present disclosure relates to the technical field of engineering machinery automation. Provided are a semantic segmentation method, apparatus and system, and engineering machinery equipment. The method comprises: processing an original image of an operation area of engineering machinery equipment, so as to obtain at least one image feature map group, wherein each image feature map group comprises a plurality of image feature maps having different scales; processing point cloud data of the operation area, so as to obtain a radar feature map group; for each image feature map group, executing fusion with the radar feature map group, so as to obtain a fused feature map; and obtaining a semantic segmentation result on the basis of the fused feature map.

Description

Semantic segmentation method, device, system and engineering machinery equipment

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the application with CN application number 202311658025.6 and application date December 5, 2023, and claims its priority. The disclosed content of the CN application is hereby introduced as a whole into this application. Technical Field

[0003] The present disclosure relates to the technical field of engineering machinery automation, and in particular to a semantic segmentation method, device, system and engineering machinery equipment. Background Art

[0004] Various types of engineering machinery and equipment are widely used in construction in various scenarios such as mining, water conservancy projects, transportation, and power projects.

[0005] As the current main development direction, unmanned engineering machinery and equipment has various advantages. It can alleviate the labor shortage problem caused by the aging population and avoid operators working in dangerous environments (for example, places with landslide risks, places with toxic and harmful gases, etc.).

[0006] Summary of the Invention

[0007] According to one aspect of an embodiment of the present disclosure, a semantic segmentation method is provided, comprising: processing an original image of a working area of ​​engineering machinery to obtain at least one image feature map group, each image feature map group including multiple image feature maps of different scales; processing point cloud data of the working area to obtain a radar feature map group; for each image feature map group, performing a fusion operation with the radar feature map group to obtain a fused feature map; and obtaining a semantic segmentation result based on the fused feature map.

[0008] In some embodiments, each image feature map group corresponds to M subgroups of image feature maps, and the radar feature map group includes M radar feature maps of different scales, M≥2; wherein, for each image feature map group, performing a fusion operation with the radar feature map group to obtain a fused feature map includes: performing a first fusion operation on the image feature map in the 1st subgroup to obtain a first fused feature map of the 1st subgroup; performing a second fusion operation on the first fused feature map of the i-th subgroup and the i-th radar feature map to obtain a second fused feature map of the i-th subgroup, where i is each integer from 1 to M; and performing the first fusion operation on the second fused feature map of the j-th subgroup and the image feature map in the j+1-th subgroup to obtain a first fused feature map of the j+1-th subgroup, where j is each integer from 1 to M-1, wherein the fused feature map is the second fused feature map of the M-th subgroup.

[0009] In some embodiments, performing a second fusion operation on the first fused feature map and the i-th radar feature map of the i-th subgroup to obtain the second fused feature map of the i-th subgroup includes: performing one or more consecutive groups of operations on the i-th radar feature map to obtain a first intermediate feature map, each group of operations includes one or more consecutive groups of sub-operations, and the output feature map of each group of operations is obtained by adding the output feature map of the last group of sub-operations in the group of operations to the input feature map of the group of operations, wherein each group of sub-operations consists of a convolution operation and an activation operation after the convolution operation; and adding the first fused feature map of the i-th subgroup to the first intermediate feature map, and performing chain residual pooling on the feature map obtained by the addition to obtain the second fused feature map of the i-th subgroup.

[0010] In some embodiments, the first fusion operation includes: performing convolution operations on the multiple feature maps to be processed respectively to obtain multiple second intermediate feature maps, wherein the feature map with the largest scale among the multiple second intermediate feature maps is the first feature map, wherein the multiple feature maps to be processed are the image feature maps in the first subgroup, or the second fused feature map of the j-th subgroup and the image feature map in the j+1-th subgroup; upsampling the other feature maps in the multiple second intermediate feature maps except the first feature map so that the scale of the other feature maps is the same as the scale of the first feature map; and adding the upsampled other feature maps to the first feature map.

[0011] In some embodiments, the number of the multiple image feature maps in each image feature map group is M, the M image feature maps in each image feature map group are arranged in order from small to large scale, and the M radar feature maps in the radar feature map group are arranged in order from small to large scale, wherein: the j-th subgroup is the j-th image feature map and the j+1-th image feature map, and the M-th subgroup is the M-th image feature map.

[0012] In some embodiments, processing an original image of an operating area of ​​engineering machinery equipment to obtain at least one image feature map group includes: preprocessing the original image using at least one preprocessing method to obtain at least one image; and downsampling each of a plurality of images to be processed to obtain an image feature map group corresponding to each image to be processed, wherein the plurality of images to be processed include the original image and the at least one image.

[0013] In some embodiments, the working area includes bulk materials, and the at least one preprocessing method includes at least one of preprocessing based on a dust removal algorithm and preprocessing based on a homomorphic filtering algorithm.

[0014] In some embodiments, processing the point cloud data of the work area to obtain a radar feature map group includes: processing the point cloud data using a projection method to obtain a first radar image; processing the first radar image using a bilateral filter to obtain a second radar image; and performing a pooling operation on the second radar image to obtain the radar feature map group.

[0015] In some embodiments, the operation area includes bulk materials, and the method further includes: determining characteristic parameters of the bulk materials according to the semantic segmentation result, so as to control the operation of the engineering machinery equipment according to the characteristic parameters.

[0016] According to another aspect of an embodiment of the present disclosure, a semantic segmentation apparatus is provided, comprising: a module configured to execute the semantic segmentation method described in any one of the above embodiments.

[0017] According to another aspect of an embodiment of the present disclosure, a semantic segmentation device is provided, comprising: a memory; and a processor coupled to the memory, configured to execute the semantic segmentation method described in any one of the above embodiments based on instructions stored in the memory.

[0018] According to another aspect of the embodiments of the present disclosure, a semantic segmentation system is provided, which is used to be installed on engineering machinery and includes: the semantic segmentation device described in any one of the above embodiments; an image acquisition unit, configured to send the acquired original image to the semantic segmentation device; and a point cloud acquisition unit, configured to send the acquired point cloud data to the semantic segmentation device.

[0019] According to another aspect of the embodiments of the present disclosure, there is provided an engineering machinery equipment, comprising: the semantic segmentation system described in any one of the above embodiments.

[0020] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, comprising computer program instructions, wherein when the computer program instructions are executed by a processor, the semantic segmentation method described in any one of the above embodiments is implemented.

[0021] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the semantic segmentation method described in any one of the above embodiments is implemented.

[0022] According to another aspect of the embodiments of the present disclosure, a computer program is provided, wherein when the computer program is executed by a processor, the method of any one of the above embodiments is implemented.

[0023] In the disclosed embodiments, a raw image of the working area of ​​the construction machinery is processed to obtain at least one image feature map group, and the point cloud data of the working area is processed to obtain a radar feature map group. Each image feature map group is then fused with the radar feature map group to obtain a fused feature map, and semantic segmentation results are obtained based on the fused feature maps. This improves the accuracy of semantic segmentation, thereby facilitating improved accuracy in automated control of the construction machinery.

[0024] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] FIG1 is a flow chart of a semantic segmentation method according to some embodiments of the present disclosure;

[0027] FIG2 is a schematic diagram of an imaging model of an image acquisition unit according to some embodiments of the present disclosure;

[0028] FIG3 is a schematic diagram of a flow chart of a second fusion operation according to some embodiments of the present disclosure;

[0029] FIG4 is a schematic diagram of a fusion operation performed on an image feature map group and a radar feature map group according to some embodiments of the present disclosure;

[0030] FIG5 is a schematic diagram showing the comparison before and after preprocessing based on the homomorphic filtering algorithm;

[0031] FIG6 is a schematic diagram of a homomorphic filter according to some embodiments of the present disclosure;

[0032] FIG7 is a schematic structural diagram of a semantic segmentation apparatus according to some embodiments of the present disclosure;

[0033] FIG8 is a schematic structural diagram of a semantic segmentation apparatus according to other embodiments of the present disclosure;

[0034] FIG9 is a structural diagram of a semantic segmentation apparatus according to yet other embodiments of the present disclosure;

[0035] FIG10 is a schematic diagram of the structure of a multi-path refinement network in a decoder according to some embodiments of the present disclosure;

[0036] FIG11 is a schematic structural diagram of a semantic segmentation system according to some embodiments of the present disclosure;

[0037] FIG. 12 is a schematic diagram of engineering machinery equipment according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0039] Unless specifically stated otherwise, the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0040] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0041] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0042] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0043] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0044] In related technologies, the accuracy of automated control of unmanned engineering machinery and equipment is low.

[0045] Analysis revealed that this is due to low semantic segmentation accuracy, which makes it difficult to accurately perceive environmental information for unmanned operations (for example, it is difficult to accurately locate materials in the work area), resulting in low automation control accuracy.

[0046] In view of this, the embodiments of the present disclosure propose the following solutions.

[0047] FIG1 is a flowchart of a semantic segmentation method according to some embodiments of the present disclosure.

[0048] As shown in FIG1 , the semantic segmentation method includes steps 102 to 108 .

[0049] In step 102 , an original image of a working area of ​​the engineering machinery is processed to obtain at least one image feature map group.

[0050] Here, each image feature map group includes multiple image feature maps of different scales. In some embodiments, the original image can be downsampled to obtain multiple image feature maps of different scales as one image feature map group.

[0051] In some embodiments, the engineering machinery equipment is an excavator, a bulldozer, etc.

[0052] In some embodiments, the original image of the working area is collected by an image collection unit installed on the engineering machinery equipment. The image collection unit may be, for example, a camera including an image sensor.

[0053] In step 104 , the point cloud data of the operation area is processed to obtain a radar feature map group.

[0054] The radar signature map group may include one or more radar signature maps. For example, the radar signature map group may include multiple radar signature maps of different scales.

[0055] In some embodiments, the point cloud data of the working area is collected by a point cloud collection unit installed on the engineering machinery. The point cloud collection unit may include, for example, a point cloud sensor.

[0056] It can be understood that step 102 and step 104 can be performed successively or simultaneously.

[0057] In step 106 , a fusion operation is performed on each image feature map group with the radar feature map group to obtain a fused feature map.

[0058] The fusion operation will be further explained below with reference to some embodiments.

[0059] In step 108, a semantic segmentation result is obtained based on the fused feature map.

[0060] In some embodiments, an image feature map group is obtained by performing step 102. In this case, a semantic segmentation result is obtained based on a fused feature map obtained by fusing the image feature map group with the radar feature map group. For example, an activation operation can be performed on the fused feature map to obtain a semantic segmentation result.

[0061] In other embodiments, multiple image feature map groups are obtained by executing step 102. In this case, a semantic segmentation result is obtained based on multiple fused feature maps obtained by performing a fusion operation on each of the multiple image feature map groups and the radar feature map group. For example, the multiple fused feature maps can be added together, and an activation operation can be performed on the added feature maps to obtain a semantic segmentation result.

[0062] Since each image feature map group includes multiple image feature maps of different scales, the fusion operation of each image feature map group and the radar feature map group involves at least two fusions, that is, the first fusion involves the fusion between image feature maps of different scales, and the second fusion is the fusion between the image feature map and the radar feature map.

[0063] By performing a fusion operation involving at least two fusions on the image feature map group and the radar feature map group to obtain a fused feature map, a semantic segmentation result is obtained based on the fused feature map. In this way, the accuracy of semantic segmentation can be improved.

[0064] In the above embodiment, the original image of the working area of ​​the construction machinery is processed to obtain at least one image feature map group, and the point cloud data of the working area is processed to obtain a radar feature map group. Each image feature map group is then fused with the radar feature map group to obtain a fused feature map, and semantic segmentation results are obtained based on the fused feature maps. This improves the accuracy of semantic segmentation, thereby facilitating improved accuracy in automated control of the construction machinery.

[0065] In addition, the point cloud data of the working area is first processed to obtain a radar feature map group, and then the fusion operation of the radar feature map group and the image feature map group is performed. This can reduce the computing power requirement while retaining the point cloud target features, thereby alleviating the processing pressure of the execution device of the semantic segmentation method and reducing the possibility of failure of the execution device of the semantic segmentation method.

[0066] In some embodiments, the work area includes bulk materials, such as coal, sand, and soil. In this case, after obtaining semantic segmentation results, characteristic parameters of the bulk materials can be determined based on the semantic segmentation results to control the operation of the construction machinery and equipment based on these characteristic parameters. This can improve the accuracy of automated control of the construction machinery and equipment. The characteristic parameters of the bulk materials may include, but are not limited to, the location, texture, appearance, and properties of the bulk materials.

[0067] Taking the location of bulk materials as an example, the 2D coordinates of the bulk materials in the image coordinate system can be determined based on the semantic segmentation results. Based on the conversion relationship between the 2D coordinates in the image coordinate system and the 3D coordinates in the 3D coordinate system of the construction machinery, the 3D coordinates of the bulk materials in the 3D coordinate system of the construction machinery can be determined, thereby obtaining the location of the bulk materials. This is illustrated below with reference to Figure 2.

[0068] FIG2 is a schematic diagram of an imaging model of an image acquisition unit according to some embodiments of the present disclosure.

[0069] Figure 2 shows three coordinate systems, namely the three-dimensional coordinate system of the engineering machinery equipment (X w Axis, Y w Axis, Z w Axis and origin O w ), the three-dimensional coordinate system of the image acquisition unit (denoted by X c Axis, Y c Axis, Z c Axis and origin O c ), and the image coordinate system (consisting of the u-axis, v-axis and the origin O i express).

[0070] In Figure 2, point P on the material in the working area w Corresponding point P in the image planei . Connect point P w , click P i and the origin O c The straight line is point P w The projection line on the image plane is from point P w and the origin O c The intersection of the projection line and the image plane is point P w The imaging point on the image.

[0071] Click P w The three-dimensional coordinates (x w ,y w ,z w ) and point P i The two-dimensional coordinates in the image coordinate system (u i ,v i The conversion relationship between ) and can be shown as the following formula (1):

[0072] In formula (1), Z is the scale factor, f is the focal length of the image acquisition unit, dX represents the physical length of a pixel on the photosensitive plate in the X-axis direction, dY represents the physical length of a pixel on the photosensitive plate in the Y-axis direction, (u0, v0) represents the coordinates of the center of the photosensitive plate in the pixel coordinate system, R is the rotation matrix between the three-dimensional coordinate system of the engineering machinery equipment and the three-dimensional coordinate system of the image acquisition unit, t is the translation vector from the three-dimensional coordinate system of the engineering machinery equipment to the three-dimensional coordinate system of the image acquisition unit, K is the intrinsic parameter matrix of the image acquisition unit, and T is the transformation matrix between the three-dimensional coordinate system of the engineering machinery equipment and the three-dimensional coordinate system of the image acquisition unit.

[0073] In the above formula (1), 1 / Z and point P w By multiplying the three-dimensional coordinates in the three-dimensional coordinate system of the engineering machinery equipment, we can get point P w Normalized coordinates in the three-dimensional coordinate system of the image acquisition unit. The transformation matrix composed of R and t represents a rigid body transformation. The transformation between the three-dimensional coordinate system of the image acquisition unit and the image coordinate system is a perspective transformation, and the transformation between the image coordinate system and the pixel coordinate system is an affine transformation.

[0074] Some embodiments of the fusion operation of the image feature map group and the radar feature map group are described below.

[0075] In some embodiments, each image feature map group corresponds to M subgroups of image feature maps, and the radar feature map group includes M radar feature maps of different scales. Here, M ≥ 2. That is, each image feature map group includes multiple subgroups of image feature maps, and the radar feature map group includes multiple radar feature maps of different scales.

[0076] It is understood that each subgroup may include one or more image feature maps, and the image feature maps in different subgroups are different. For example, if an image feature map group includes three image feature maps, these three image feature maps can be composed of three subgroups. In this case, the first subgroup may include the first image feature map and the second image feature map, the second subgroup may include the second image feature map and the third image feature map, and the third subgroup may include the third image feature map.

[0077] In these embodiments, a fusion operation may be performed on the image feature map group and the radar feature map group in the following manner to obtain a fused feature map.

[0078] First, a first fusion operation is performed on the image feature maps in the first subgroup to obtain a first fused feature map of the first subgroup.

[0079] Then, a second fusion operation is performed on the first fused feature map of the i-th subgroup and the i-th radar feature map to obtain a second fused feature map of the i-th subgroup.

[0080] Here, i is an integer from 1 to M. In other words, each of the M subgroups performs a second fusion operation with a radar signature map corresponding to the M radar signature maps.

[0081] Next, a first fusion operation is performed on the second fused feature map of the jth subgroup and the image feature map in the j+1th subgroup to obtain a first fused feature map of the j+1th subgroup. Here, j is any integer from 1 to M-1.

[0082] In this case, the fused feature map obtained by performing the fusion operation on the image feature map group and the radar feature map group is the second fused feature map of the Mth subgroup, that is, the feature map obtained by performing the second fusion operation on the last radar feature map and the last subgroup.

[0083] For ease of understanding, the following description is made by taking as an example a case where the image feature map group includes two subgroups of image feature maps and the radar feature map group includes two radar feature maps.

[0084] First, a first fusion operation is performed on multiple image feature maps in the first subgroup to obtain a first fused feature map of the first subgroup.

[0085] Then, a second fusion operation is performed on the first fused feature map of the first subgroup and the first radar feature map to obtain a second fused feature map of the first subgroup.

[0086] Next, a first fusion operation is performed on the second fused feature map of the first subgroup and the image feature map in the second subgroup to obtain a first fused feature map of the second subgroup.

[0087] Finally, a second fusion operation is performed on the first fused feature map of the second subgroup and the second radar feature map to obtain a second fused feature map of the second subgroup. The second fused feature map of the second subgroup is the fused feature map obtained by performing the fusion operation on the image feature map group and the radar feature map group.

[0088] In the above embodiment, in a fusion operation involving at least two fusions performed on a group of image feature maps and a group of radar feature maps, the second fused feature map of the previous subgroup, obtained by performing a second fusion operation on the first fused feature map of the previous subgroup and the previous radar feature map, is used as a feature map to participate in the first fusion operation performed on the image feature map of the subsequent subgroup. In other words, the first fusion operation performed on the image feature map of the subsequent subgroup involves the result obtained by fusing the previous subgroup with the corresponding radar feature map. In this way, the accuracy of semantic segmentation can be further improved, thereby further improving the accuracy of automated control of engineering machinery and equipment.

[0089] As some implementations, the first fusion operation is performed as follows.

[0090] First, convolution operations are performed on the multiple feature maps to be processed respectively to obtain multiple intermediate feature maps (hereinafter referred to as second intermediate feature maps).

[0091] Here, the feature map with the largest scale among the multiple second intermediate feature maps is the first feature map, that is, the feature map obtained by performing the convolution operation on the feature map with the largest scale among the multiple feature maps to be processed is the first feature map.

[0092] It can be understood that in the first fusion operation performed on the image feature map in the 1st subgroup, the multiple feature maps to be processed are the image feature maps in the 1st subgroup; in the first fusion operation performed on the second fused feature map of the jth subgroup and the image feature map in the j+1th subgroup, the multiple feature maps to be processed are the second fused feature map of the jth subgroup and the image feature map in the j+1th subgroup.

[0093] Then, upsampling is performed on the other feature maps except the first feature map in the plurality of second intermediate feature maps so that the scales of the other feature maps are the same as the scale of the first feature map.

[0094] Finally, the other upsampled feature maps are added to the first feature map to obtain the corresponding first fused feature map.

[0095] In this implementation, in the first fusion operation, convolution operations are performed on each of the multiple feature maps to be processed. Then, some of the multiple second intermediate feature maps obtained by convolution are upsampled to obtain feature maps of the same scale, which are then summed. This approach can further improve the accuracy of semantic segmentation, thereby further improving the accuracy of automated control of construction machinery.

[0096] As some implementations, a second fusion operation is performed on the first fused feature map and the i-th radar feature map of the i-th subgroup in the following manner.

[0097] First, one or more consecutive groups of operations are performed on the i-th radar feature map to obtain an intermediate feature map (hereinafter referred to as the first intermediate feature map).

[0098] Here, each group of operations includes one or more consecutive groups of sub-operations, and the output feature map of each group of operations is obtained by adding the output feature map of the last group of sub-operations in the group of operations to the input feature map of the group of operations. Each group of sub-operations consists of a convolution operation and an activation operation after the convolution operation.

[0099] It can be understood that the first intermediate feature map is the output feature map of the last group of operations in this group or multiple groups of consecutive operations.

[0100] Then, the first fused feature map of the i-th subgroup is added to the first intermediate feature map, and the feature map obtained by the addition is subjected to chained residual pooling (CRP) processing to obtain the second fused feature map of the i-th subgroup.

[0101] For ease of understanding, the following description is made in conjunction with Figure 3. Figure 3 is a schematic flow chart of the second fusion operation according to some embodiments of the present disclosure.

[0102] As shown in Figure 3, two consecutive groups of operations are performed on the i-th radar feature map. Each group of operations includes two groups of sub-operations, and each sub-operation consists of a convolution operation and an activation operation after the convolution operation.

[0103] Here, the output feature map of each group of operations is obtained by adding the output feature map of the last group of sub-operations in the group of operations to the input feature map of the group of operations.

[0104] Taking the first group of operations as an example, the output feature map of the first group of operations can be obtained by adding the input feature map of the first group of operations (i.e., the i-th radar feature map) and the feature map obtained by performing convolution operation, activation operation, convolution operation and activation operation on the i-th radar feature map in sequence (i.e., the output feature map of the last group of sub-operations).

[0105] The input feature map of the second set of operations is the output feature map of the first set of operations. The operation method is similar to that of the first set of operations and will not be described in detail here.

[0106] It should be understood that FIG3 merely schematically illustrates a situation where two consecutive groups of operations are performed on the i-th radar characteristic diagram, and each group of operations includes two groups of sub-operations, and the embodiments of the present disclosure are not limited thereto.

[0107] In the second fusion operation of the above implementation, one or more groups of operations are performed on the radar signature map. Each group of operations uses a direct connection structure to add the input feature map and the output feature map of the last group of sub-operations to obtain the output feature map of the group. This can effectively alleviate problems such as gradient divergence and gradient vanishing. Furthermore, the sub-operations in each group of operations consist of a convolution operation and an activation operation after the convolution operation, which is more suitable for processing radar signature maps. This approach can further improve the accuracy of semantic segmentation, thereby further improving the accuracy of automated control of construction machinery and equipment.

[0108] In some embodiments, the number of multiple image feature maps in the image feature map group is M, and the M image feature maps are arranged in order from small to large scale, and the M radar feature maps in the radar feature map group are also arranged in order from small to large scale.

[0109] In this case, the j-th subgroup is the j-th image feature map and the j+1-th image feature map, and the M-th subgroup is the M-th image feature map.

[0110] This approach allows image feature maps and radar feature maps to be fused in ascending scale order. Furthermore, each image feature map only participates in the first fusion operation of two subgroups at most. This not only further improves semantic segmentation accuracy but also reduces the computational complexity of the fusion operation for image and radar feature map groups, thereby increasing semantic segmentation efficiency. This approach further enhances both the accuracy and real-time performance of automated control of construction machinery and equipment.

[0111] To facilitate understanding, the following description is made in conjunction with Figure 4. Figure 4 is a schematic diagram of a fusion operation performed on an image feature map group and a radar feature map group according to some embodiments of the present disclosure.

[0112] As shown in FIG4 , the image feature map group includes three image feature maps, which are, in order from small to large scale, a 1 / 16 image feature map with a scale of 1 / 16 of the original image, a 1 / 8 image feature map with a scale of 1 / 8 of the original image, and a 1 / 4 image feature map with a scale of 1 / 4 of the original image.

[0113] In this case, the first subgroup is a 1 / 16 image feature map and a 1 / 8 image feature map, the second subgroup is a 1 / 8 image feature map and a 1 / 4 image feature map, and the third subgroup is a 1 / 4 image feature map.

[0114] As shown in FIG4 , the radar feature map group also includes three radar feature maps. The three radar feature maps are, in order from small to large scale, a 1 / 16 radar feature map with a scale of 1 / 16 of the radar image obtained based on point cloud data, a 1 / 8 radar feature map with a scale of 1 / 8 of the radar image, and a 1 / 4 radar feature map with a scale of 1 / 4 of the radar image.

[0115] In this case, the fusion operation of the image feature map group and the radar feature map group can be performed by three Image Point Cloud Fusion Units (IPFUs). The IPFU can be a sensor-level fusion structure.

[0116] Specifically, the image feature map and the i-th radar feature map of the i-th subgroup can be input into the i-th IPFU, and the output of the j-th IPFU is input into the j+1-th IPFU. The i-th IPFU can perform the first fusion operation and the second fusion operation in the manner described above to obtain the second fused feature map of the i-th subgroup. The output of the last IPFU is the fused feature map.

[0117] It is understood that FIG4 merely schematically illustrates the case where M=3, and the embodiments of the present disclosure are not limited thereto. For example, in some embodiments, the image feature map group may further include a 1 / 32 image feature map having a scale of 1 / 32 of the original image.

[0118] So far, some embodiments of the fusion operation of the image feature map group and the radar feature map group have been described.

[0119] Next, some embodiments of processing an original image to obtain at least one image feature map group are described.

[0120] In some embodiments, at step 102, the original image is first preprocessed using at least one preprocessing method to obtain at least one image, and then each of the multiple images to be processed is downsampled to obtain an image feature map group corresponding to each image to be processed. Here, the multiple images to be processed include the original image and the at least one image obtained by preprocessing.

[0121] In other words, in these embodiments, the original image is downsampled to obtain an image feature map group corresponding to the original image, and at least one image obtained after preprocessing the original image is downsampled to obtain at least one image feature map group corresponding to the at least one image. Each image feature map group is then fused with the radar feature map group to obtain a fused feature map, and semantic segmentation results are obtained based on the fused feature map. This can increase sample diversity, facilitate data expansion and feature enhancement, and thus further improve the accuracy of semantic segmentation, which in turn helps further improve the accuracy of automated control of construction machinery.

[0122] In some implementations, at least one preprocessing method may include at least one of a geometric transformation and a color saturation transformation. The geometric transformation may include, but is not limited to, adjusting the original image's orientation (e.g., randomly flipping the original image horizontally with a certain probability, such as 0.5), size, quality, or the number of pixel values ​​in at least one of the length and width directions (e.g., adjusting the number to 1 / 4 or 1 / 3 of the original pixel values). The color saturation transformation may include, but is not limited to, subtracting the average of the three RGB channels.

[0123] In other implementations, the work area includes bulk materials. In this case, the at least one preprocessing method includes at least one of preprocessing based on a dust removal algorithm and preprocessing based on a homomorphic filtering algorithm. For example, the at least one preprocessing method may include preprocessing based on a dust removal algorithm and preprocessing based on a homomorphic filtering algorithm.

[0124] Since bulk materials have characteristics such as irregular shape, large area, and easy occlusion between materials, the difficulty of semantic segmentation increases and the accuracy is further reduced when the working area includes bulk materials.

[0125] In the above implementation, when the work area includes bulk materials, preprocessing the original image using at least one of a dust removal algorithm and a homomorphic filtering algorithm can be performed. This can alleviate the problem of reduced semantic segmentation accuracy caused by bulk materials. This can further improve the accuracy of semantic segmentation, thereby further enhancing the accuracy of automated control of construction machinery.

[0126] As further implementations, at least one preprocessing method may include geometric transformation, color saturation transformation, preprocessing based on a dust removal algorithm, and preprocessing based on a homomorphic filtering algorithm. Using these preprocessing methods to enhance the original image can further improve the accuracy of semantic segmentation, thereby further improving the accuracy of automated control of construction machinery.

[0127] The following is a brief description of some examples of preprocessing based on the dust removal algorithm.

[0128] Raw images of work areas are typically captured outdoors, which makes them susceptible to degradation due to atmospheric turbidity (e.g., particles, water droplets, etc.). Dust, fog, and smoke are caused by atmospheric absorption and scattering. The radiation received by the image acquisition unit from the scene point decays along the line of sight, causing degraded raw images to lose contrast and color, which is detrimental to target recognition, feature detection, photometric analysis, and more.

[0129] In these scenarios, the model of the original image (also called the foggy image) can be described by the following formula (2): I(x)=J(x)t(x)+A(1-t(x)).

[0130] In formula (2), I(x) represents the observed intensity, that is, the intensity of the original image; J(x) represents the scene radiance, that is, the true intensity of the scene; A(1-t(x)) represents the global atmospheric light; and t(x) represents the transmittance of the medium.

[0131] In formula (2), J(x)t(x) can be called direct attenuation, and A(1-t(x)) can be called atmospheric light (i.e., airlight). Direct attenuation describes the scene radiance and its attenuation in the medium. Atmospheric light describes the changes in scene color caused by previously scattered light.

[0132] When the atmosphere is uniform, t(x) can be expressed as the following formula (3): t(x) = e -βd(x) , where β is the scattering coefficient of the atmosphere and d is the scene depth. Formula (3) shows that the scene radiance decays exponentially with the scene depth.

[0133] From formulas (2) and (3), we can see that the original image of the work area consists of two parts: one is the residual value after the true brightness is attenuated by the atmosphere (the true part is not scattered), and the other is the light scattered by the atmospheric light (the atmospheric part is scattered in the air). In addition, the farther the distance, the more the original scene's radiant brightness will attenuate.

[0134] It can be seen that in order to better remove dust and fog in the original image, it is necessary to estimate the atmospheric light and then estimate the transmittance, so as to restore a fog-free image without dust and fog.

[0135] The purpose of preprocessing the original image based on the dust removal algorithm is to restore the fog-free image as much as possible without dust and fog. The specific operations of this preprocessing are basically as follows.

[0136] First, assume that atmospheric light is constant and that the transmittance of a small local area (i.e., patch) is constant. Based on this assumption, perform the minimum operation on both sides of the previous formula (2) and perform it independently on the three channels to obtain the following formula (4):

[0137] Then, the three minimum values ​​obtained from the three channels are minimized to obtain the following formula (5):

[0138] According to the dark channel prior, the dark channel value of the fog-free image tends to zero, that is, the dark channel value J dark (x) can be expressed as the following formula (6):

[0139] Because A c It is always positive, so we can get the following formula (7):

[0140] Substituting the above formula (7) into formula (5), we can obtain the following formula (8):

[0141] Thus, the expression formula of transmittance is obtained.

[0142] In fact, that is dark channel.

[0143] Then, a dehazing operation can be performed to compute a haze-free image based on the hazy image.

[0144] Specifically, the obtained transmittance can be used to restore the scene radiance according to formula (4). When the transmittance t(x) is close to zero, J(x)t(x) is also close to zero. In this case, directly restoring the scene radiance J(x) is prone to noise. Therefore, the lower limit of t(x) can be limited to t0, which means that a small amount of fog is retained in dense fog areas. In this way, the final scene radiance J(x) can be restored and expressed by the following formula (9):

[0145] In some embodiments, t0 may be set to 0.1.

[0146] In some embodiments, the dehazed image can be exposed at a higher exposure. Because scene radiance is typically not as bright as atmospheric light, the dehazed image appears darker. This increased exposure can further improve the accuracy of semantic segmentation, thereby further enhancing the accuracy of automated control of construction machinery.

[0147] By performing preprocessing based on the dust removal algorithm on the original image, the problem of original image degradation in outdoor scenes due to turbid media in the atmosphere can be alleviated, thereby improving the accuracy of semantic segmentation.

[0148] Next, some examples of preprocessing based on the homomorphic filtering algorithm are described.

[0149] The original image of the working area often has a large dynamic range but unclear details in the dark areas. It is desired to enhance the details in the dark areas without losing the details in the bright areas.

[0150] The image f(x,y) can be modeled as the product of illumination intensity (i.e., illumination, represented by i(x,y)) and reflection intensity (i.e., reflection, represented by r(x,y)), from which we can obtain formula (10): f(x,y)=i(x,y)r(x,y).

[0151] Generally speaking, the lighting in natural images is uniform and gradual, so i(x,y) should be a low-frequency component; while the reflection of light by different objects is sudden, so r(x,y) is a high-frequency component.

[0152] We can take the logarithm of both sides of formula (10) to obtain formula (11): lnf(x,y)=lni(x,y)+lnr(x,y).

[0153] Performing Fourier transform on formula (11) yields formula (12): FET(lnf(x,y))=FET(lni(x,y))+FET(lnr(x,y)).

[0154] Figure 5 is a schematic diagram showing the comparison before and after preprocessing based on the homomorphic filtering algorithm. Referring to Figure 5 , the low-frequency component i can be suppressed to reduce the dynamic range, while the high-frequency component r can be enhanced to enhance the contrast.

[0155] In some embodiments, a homomorphic filter may be used to perform pre-processing based on a homomorphic filtering algorithm. Figure 6 is a schematic diagram of a homomorphic filter according to some embodiments of the present disclosure.

[0156] In Figure 6, H(u,v) represents the homomorphic filter function, and D(u,v) represents the distance between the frequency (u,v) and the filter center (u0,v0). Using the homomorphic filter shown in Figure 6 to filter an image produces a new image with low-frequency suppression and high-frequency enhancement.

[0157] By performing preprocessing based on the homomorphic filtering algorithm on the original image, the pixel values ​​of the lower brightness areas in the original image in conditions such as dark outdoor weather or weak indoor lighting can be effectively enhanced, thereby improving the accuracy of semantic segmentation.

[0158] So far, some examples of preprocessing based on the dust removal algorithm and some examples of preprocessing based on the homomorphic filtering algorithm have been described respectively.

[0159] Next, some embodiments of processing point cloud data of an operation area to obtain a radar signature map group are described.

[0160] In some embodiments, the point cloud data may be processed in the following manner to obtain a radar signature map group.

[0161] First, the point cloud data may be processed using a projection method to obtain a first radar image.

[0162] As some implementations, the point cloud data may be processed using a pyramid projection method to obtain the first radar image.

[0163] As some implementation methods, calibration parameters between the image acquisition unit that acquires the original image and the point cloud acquisition unit that acquires the point cloud data can be obtained first. The calibration parameters may include but are not limited to the transformation matrix and the internal parameters of the image acquisition unit (such as a camera).

[0164] In this case, the point cloud data can be projected according to the following formula (13) to obtain the first radar image:

[0165] p img =K·T·p lidar ,

[0166] In formula (13), the 3D point in the point cloud data is represented as p lidar =(x,y,z,1), where x, y, z are position parameters and 1 is the intensity value. The projection result of the three-dimensional point in the first radar image (i.e., RGB image plane) is expressed as p img =(u,v,1). K is the internal parameter of the image acquisition unit, T is the transformation matrix, R is the rotation matrix, and t is the translation vector.

[0167] A frame of point cloud data typically consists of over 100,000 3D points. However, only a portion of these 3D points can be projected onto the RGB image plane to form the first radar image. In this case, the projected 3D points are relatively sparse in the first radar image.

[0168] Based on this, after obtaining the first radar image, the first radar image can be processed using a bilateral filter to obtain the second radar image, so that the sparse first radar image can be interpolated into a dense second radar image.

[0169] Finally, a pooling operation may be performed on the second radar image to obtain a radar signature map group.

[0170] In the above embodiment, after processing the point cloud data using a projection method to obtain a first radar image, the sparse first radar image is processed using a bilateral filter to obtain a dense second radar image. Pooling is then performed on the second radar image to obtain a radar feature map group. This approach can further improve the accuracy of semantic segmentation, thereby further enhancing the accuracy of automated control of construction machinery.

[0171] The embodiment of the present disclosure also provides a semantic segmentation device.

[0172] In some embodiments, the semantic segmentation apparatus includes a memory and a processor coupled to the memory, wherein the processor is configured to execute the semantic segmentation method of any one of the above embodiments based on instructions stored in the memory.

[0173] In some other embodiments, the semantic segmentation apparatus includes a module configured to execute the semantic segmentation method of any one of the above embodiments.

[0174] FIG7 is a schematic structural diagram of a semantic segmentation apparatus according to some embodiments of the present disclosure.

[0175] As shown in FIG7 , a semantic segmentation apparatus 700 includes a memory 701 and a processor 702 coupled to the memory 701 . The processor 702 is configured to execute the semantic segmentation method of any one of the above embodiments based on instructions stored in the memory 701 .

[0176] The memory 701 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.

[0177] In some embodiments, the semantic segmentation apparatus 700 may further include an input / output interface 703, a network interface 704, a storage interface 705, and the like. These interfaces, such as the input / output interface 703, the network interface 704, and the storage interface 705, as well as the memory 701 and the processor 702, may be connected via a bus 706, for example. The input / output interface 703 provides a connection interface for input / output devices such as a display, mouse, keyboard, and touch screen. The network interface 604 provides a connection interface for various networked devices. The storage interface 705 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0178] FIG8 is a schematic structural diagram of a semantic segmentation apparatus according to some other embodiments of the present disclosure.

[0179] As shown in FIG8 , the semantic segmentation apparatus 800 includes a processing module 801 and a fusion module 802 .

[0180] The processing module 801 is configured to process the original image of the working area of ​​the construction machinery to obtain at least one image feature map group, and process the point cloud data of the working area to obtain a radar feature map group. Here, each image feature map group includes multiple image feature maps of different scales.

[0181] The fusion module 802 is configured to perform a fusion operation with the radar feature map group for each image feature map group to obtain a fused feature map, and obtain a semantic segmentation result based on the fused feature map.

[0182] In some embodiments, each image feature map group corresponds to M subgroups of the image feature map, and the radar feature map group includes M radar feature maps of different scales, where M≥2.

[0183] In this case, the fusion module 802 is further configured to: perform a first fusion operation on the image feature map in the 1st subgroup to obtain a first fused feature map of the 1st subgroup; perform a second fusion operation on the first fused feature map of the i-th subgroup and the i-th radar feature map to obtain a second fused feature map of the i-th subgroup, where i is each integer from 1 to M; and perform a first fusion operation on the second fused feature map of the j-th subgroup and the image feature map in the j+1-th subgroup to obtain a first fused feature map of the j+1-th subgroup, where j is each integer from 1 to M-1, wherein the fused feature map is the second fused feature map of the M-th subgroup.

[0184] In some embodiments, the fusion module 802 is further configured to: perform one or more consecutive groups of operations on the i-th radar feature map to obtain a first intermediate feature map, each group of operations includes one or more consecutive groups of sub-operations, and the output feature map of each group of operations is obtained by adding the output feature map of the last group of sub-operations in the group of operations to the input feature map of the group of operations, wherein each group of sub-operations consists of a convolution operation and an activation operation after the convolution operation; and add the first fused feature map of the i-th subgroup to the first intermediate feature map, and perform chain residual pooling on the feature map obtained by the addition to obtain the second fused feature map of the i-th subgroup.

[0185] In some embodiments, the first fusion operation includes: performing convolution operations on the multiple feature maps to be processed respectively to obtain multiple second intermediate feature maps, the feature map with the largest scale among the multiple second intermediate feature maps is the first feature map, wherein the multiple feature maps to be processed are the image feature maps in the 1st subgroup, or the second fused feature map of the jth subgroup and the image feature map in the j+1th subgroup; upsampling the other feature maps in the multiple second intermediate feature maps except the first feature map so that the scales of the other feature maps are the same as the scale of the first feature map; and adding the upsampled other feature maps to the first feature map.

[0186] In some embodiments, the number of multiple image feature maps in each image feature map group is M, the M image feature maps in each image feature map group are arranged in order from small to large scale, and the M radar feature maps in the radar feature map group are arranged in order from small to large scale, wherein: the j-th subgroup is the j-th image feature map and the j+1-th image feature map, and the M-th subgroup is the M-th image feature map.

[0187] In some embodiments, the processing module 801 is further configured to: preprocess the original image using at least one preprocessing method to obtain at least one image; and downsample each of a plurality of images to be processed to obtain an image feature map group corresponding to each image to be processed, wherein the plurality of images to be processed include the original image and at least one image.

[0188] In some embodiments, the working area includes bulk materials, and the at least one preprocessing method includes at least one of preprocessing based on a dust removal algorithm and preprocessing based on a homomorphic filtering algorithm.

[0189] In some embodiments, the processing module 801 is further configured to: process the point cloud data using a projection method to obtain a first radar image; process the first radar image using a bilateral filter to obtain a second radar image; and perform a pooling operation on the second radar image to obtain a radar feature map group.

[0190] In some embodiments, the operation area includes bulk materials, and the semantic segmentation apparatus 800 further includes a control module 803. The control module 803 is configured to determine characteristic parameters of the bulk materials according to the semantic segmentation results, so as to control the operation of the engineering machinery equipment according to the characteristic parameters.

[0191] It should be understood that the semantic segmentation apparatus 800 may further include various other modules to execute the semantic segmentation method of any of the above embodiments.

[0192] FIG9 is a schematic structural diagram of a semantic segmentation apparatus according to yet other embodiments of the present disclosure.

[0193] As shown in FIG9 , the semantic segmentation apparatus 900 includes an image data processing module 901 , an image encoder 902 , a point cloud data processing module 903 and a decoder 904 .

[0194] The image data processing module 901 is configured to preprocess the original image using at least one preprocessing method to obtain at least one image.

[0195] The image encoder 902 is configured to downsample each of the plurality of images to be processed to obtain an image feature map group corresponding to each image to be processed. Here, the plurality of images to be processed include the original image and at least one image from the image data processing module 901.

[0196] The image encoder 902 may, but is not limited to, adopt a residual network (Residual Network, ResNet), such as a ResNet50 structure.

[0197] The point cloud data processing module 903 is configured to process the point cloud data of the operation area to obtain a radar feature map group.

[0198] The decoder 904 is configured to perform a fusion operation on each image feature map group with the radar feature map group to obtain a fused feature map, and obtain a semantic segmentation result based on the fused feature map. During the fusion operation, the decoder 904 also upsamples the feature map.

[0199] It should be understood that the semantic segmentation apparatus 900 may further include various other modules to execute the semantic segmentation method of any of the above embodiments.

[0200] In some embodiments, the decoder 904 includes a semantic segmentation network based on deep learning (hereinafter referred to as deep learning network).

[0201] As some implementations, the deep learning network is a multi-path refinement network, and the multi-path refinement network is configured to perform a fusion operation of the image feature map group and the radar feature map group in the manner shown in Figures 3 and 4, and obtain a semantic segmentation result based on the fused feature map.

[0202] Some examples of multi-path refinement networks are given below. Figure 10 is a schematic diagram of the structure of a multi-path refinement network in a decoder according to some embodiments of the present disclosure.

[0203] As shown in Figure 10, the multipath refinement network may include an IPFU. The IPFU may be configured to perform a first fusion operation and a second fusion operation in accordance with the aforementioned embodiments. The IPFU may include a CRP module, which may be configured to perform CRP processing in accordance with the aforementioned embodiments. For related details, please refer to the aforementioned description and will not be repeated here.

[0204] In Figure 10, the 3*3 convolution layer is configured to perform convolution operations based on 3 (rows) × 3 (columns) convolution kernels, the 1*1 convolution layer is configured to perform convolution operations based on 1 (row) × 1 (row) convolution kernels, and the 5*5 pooling layer is configured to perform pooling operations based on 5 (rows) × 5 (columns) pooling kernels.

[0205] However, it should be understood that FIG10 is merely illustrative and not restrictive, and the embodiments of the present disclosure are not limited to performing operations based on convolution kernels and pooling kernels of these sizes.

[0206] 10 is configured to perform an activation operation, for example, an activation operation based on a Rectified Linear Unit (ReLU). The upsampling layer is configured to perform upsampling.

[0207] It is understood that FIG10 is merely a schematic illustration of an IPFU of a multi-path refinement network, but the present disclosure is not limited thereto. For example, referring to FIG4 , a multi-path refinement network may include multiple IPFUs.

[0208] The semantic segmentation device 900 of the above embodiment performs semantic segmentation based on an end-to-end deep learning network and follows the encoder-decoder design pattern, which can not only improve the accuracy of semantic segmentation, but also improve the efficiency of semantic segmentation, thereby helping to improve the accuracy and real-time performance of automated control of engineering machinery equipment.

[0209] The present disclosure also provides a semantic segmentation system for installation on engineering machinery. FIG11 is a schematic diagram of the structure of a semantic segmentation system according to some embodiments of the present disclosure.

[0210] As shown in FIG11 , the semantic segmentation system includes a semantic segmentation device 1110 , an image acquisition unit 1120 and a point cloud acquisition unit 1130 .

[0211] The semantic segmentation device 1110 may be the semantic segmentation device of any of the above embodiments (eg, the semantic segmentation device 700 / 800 / 900).

[0212] The image acquisition unit 1120 may be configured to acquire original images of the working area of ​​the engineering machinery equipment and send the acquired original images to the semantic segmentation device 1110 .

[0213] The point cloud acquisition unit 1130 may be configured to acquire point cloud data of an operation area of ​​the engineering machinery and send the acquired point cloud data to the semantic segmentation device 1110 .

[0214] In some embodiments, referring to FIG. 11 , a semantic segmentation device 1110 may be deployed in a controller 1140 of an engineering machinery device.

[0215] Before the construction machinery equipment starts working, the controller 1140 can drive the image acquisition unit 1120 and the point cloud acquisition unit 1130 to respectively acquire original images and point cloud data of the working area.

[0216] Then, the semantic segmentation device 1110 can execute the semantic segmentation method of any one of the above embodiments to obtain a semantic segmentation result, and can output the characteristic parameters of the material (such as bulk material) in the working area determined according to the semantic segmentation result, thereby providing sufficient perception basis for the automated control of engineering machinery equipment.

[0217] Afterwards, the controller 1140 can control the operation of the engineering machinery equipment according to the characteristic parameters of the material output by the semantic segmentation device 1110.

[0218] The present disclosure also provides an engineering machinery, including the semantic segmentation system of any one of the above embodiments. The engineering machinery may include but is not limited to excavators, bulldozers, etc.

[0219] Figure 12 is a schematic diagram of engineering machinery according to some embodiments of the present disclosure. Figure 12 schematically shows an excavator.

[0220] As shown in Figure 12, the excavator cab is equipped with an image acquisition unit 1120 and a point cloud acquisition unit 1130. These units can rotate with the cab. Image acquisition unit 1120 can capture raw images of the excavator's operating area, while point cloud acquisition unit 1130 can capture point cloud data of the operating area.

[0221] The excavator further includes a controller 1140 , which includes a semantic segmentation device 1110 . The semantic segmentation device 1110 can execute the semantic segmentation method of any of the above embodiments based on the original image and point cloud data from the image acquisition unit 1120 and the point cloud acquisition unit 1130 .

[0222] FIG12 also schematically shows the established three-dimensional coordinate system of the excavator (by X w Axis, Y w Axis, Z w Axis and origin O w ), the three-dimensional coordinate system of the image acquisition unit 1120 (denoted by X c Axis, Y c Axis and Z c Axis) and the three-dimensional coordinate system of the point cloud acquisition unit 1130 (denoted by X p Axis, Y p Axis, Z p axis).

[0223] In some embodiments, in the three-dimensional coordinate system of the excavator, the origin O w Y is the base center point of the excavator's rotation center on the ground. w The axis is in the direction in front of the excavator cab, Z wThe axis is perpendicular to the ground and points upwards, X w Axis perpendicular to Y w Axis and Z w axis and in the direction of the right side of the cab.

[0224] The transformation between the three-dimensional coordinate system of the excavator and the three-dimensional coordinate system of the image acquisition unit can be referred to the above description of the transformation between the three-dimensional coordinate system of the engineering machinery equipment and the three-dimensional coordinate system of the image acquisition unit, so it is not repeated here.

[0225] Here, a brief description is given of the transformation between the three-dimensional coordinate system of the point cloud acquisition unit 1130 and the three-dimensional coordinate system of the excavator (ie, the engineering machinery equipment).

[0226] Specifically, the three-dimensional coordinates (X l ,Y l ,Z l ) and the three-dimensional coordinates (X t ,Y t ,Z t The transformation matrix between ) is shown in the following formula (14): Among them, R is the rotation matrix of the three-dimensional coordinate system of the point cloud acquisition unit 1130 and the three-dimensional coordinate system of the excavator, and T is the translation vector of the three-dimensional coordinate system of the point cloud acquisition unit 1130 and the three-dimensional coordinate system of the excavator.

[0227] In some embodiments, the point cloud acquisition unit 1130 is installed directly above the cab, and X p Axis, Y p Axis, Z p The directions of the axes are respectively w Axis, Y w Axis, Z w In this way, the transformation between the three-dimensional coordinate system of the excavator and the three-dimensional coordinate system of the point cloud acquisition unit 1130 can be simplified, thereby reducing the processing pressure of the semantic segmentation device 1110 and reducing the possibility of failure of the semantic segmentation device 1110.

[0228] An embodiment of the present disclosure further provides a computer-readable storage medium, comprising computer program instructions, which, when executed by a processor, implement the semantic segmentation method of any one of the above embodiments.

[0229] The embodiments of the present disclosure further provide a computer program product, including a computer program, which implements the semantic segmentation method of any one of the above embodiments when executed by a processor.

[0230] The embodiments of the present disclosure further provide a computer program, which implements the method of any one of the above embodiments when executed by a processor.

[0231] Thus far, various embodiments of the present disclosure have been described in detail. To avoid obscuring the concept of the present disclosure, some details known in the art have not been described. Based on the above description, those skilled in the art can fully understand how to implement the technical solutions disclosed herein.

[0232] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. References to the same or similar parts between the various embodiments are sufficient. For device, system, and equipment embodiments, since they generally correspond to the method embodiments, their descriptions are relatively simple. For relevant parts, references to the descriptions of the method embodiments are sufficient.

[0233] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0234] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0235] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0236] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0237] Although some specific embodiments of the present disclosure have been described in detail through examples, those skilled in the art will understand that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. Those skilled in the art will understand that the above embodiments may be modified or some technical features may be replaced with equivalents without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A semantic segmentation method, comprising: Processing an original image of an operation area of ​​the engineering machinery equipment to obtain at least one image feature map group, each image feature map group including a plurality of image feature maps of different scales; Processing the point cloud data of the operation area to obtain a radar feature map group; For each image feature map group, a fusion operation with the radar feature map group is performed to obtain a fused feature map; as well as A semantic segmentation result is obtained based on the fused feature map.

2. The method according to claim 1, wherein: Each image feature map group corresponds to M subgroups of the image feature map, and the radar feature map group includes M radar feature maps of different scales, M≥2; Wherein, for each image feature map group, performing a fusion operation with the radar feature map group to obtain a fused feature map includes: Performing a first fusion operation on the image feature map in the first subgroup to obtain a first fused feature map of the first subgroup; Performing a second fusion operation on the first fused feature map of the i-th subgroup and the i-th radar feature map to obtain a second fused feature map of the i-th subgroup, where i is each integer from 1 to M; and The first fusion operation is performed on the second fused feature map of the j-th subgroup and the image feature map in the j+1-th subgroup to obtain the first fused feature map of the j+1-th subgroup, where j is each integer from 1 to M-1. Among them, the fused feature map is the second fused feature map of the Mth subgroup.

3. The method according to claim 2, wherein: Performing a second fusion operation on the first fused feature map of the i-th subgroup and the i-th radar feature map to obtain a second fused feature map of the i-th subgroup includes: Performing one or more consecutive groups of operations on the i-th radar feature map to obtain a first intermediate feature map, each group of operations includes one or more consecutive groups of sub-operations, and the output feature map of each group of operations is obtained by adding the output feature map of the last group of sub-operations in the group of operations to the input feature map of the group of operations, wherein each group of sub-operations consists of a convolution operation and an activation operation after the convolution operation; and The first fused feature map of the i-th subgroup is added to the first intermediate feature map, and the feature map obtained by the addition is subjected to chain residual pooling processing to obtain a second fused feature map of the i-th subgroup.

4. The method according to claim 2 or 3, wherein: The first fusion operation includes: Perform convolution operations on the multiple feature maps to be processed respectively to obtain multiple second intermediate feature maps, wherein the feature map with the largest scale among the multiple second intermediate feature maps is the first feature map, wherein the multiple feature maps to be processed are the image feature maps in the first subgroup, or the second fused feature map of the j-th subgroup and the image feature map in the j+1-th subgroup; Upsampling the other feature maps in the plurality of second intermediate feature maps except the first feature map so that scales of the other feature maps are the same as that of the first feature map; and The upsampled other feature maps are added to the first feature map.

5. The method according to claim 2 or 3, wherein: The number of the multiple image feature maps in each image feature map group is M, the M image feature maps in each image feature map group are arranged in order of scale from small to large, and the M radar feature maps in the radar feature map group are arranged in order of scale from small to large, wherein: The j-th subgroup is the j-th image feature map and the j+1-th image feature map, and the M-th subgroup is the M-th image feature map.

6. The method according to any one of claims 1 to 5, wherein: Processing the original image of the working area of ​​the engineering machinery equipment to obtain at least one image feature map group includes: Preprocessing the original image using at least one preprocessing method to obtain at least one image; and Down-sampling each to-be-processed image among a plurality of to-be-processed images to obtain an image feature map group corresponding to each to-be-processed image, wherein the plurality of to-be-processed images include the original image and the at least one image.

7. The method according to claim 6, wherein: The working area includes bulk materials, and the at least one preprocessing method includes at least one of preprocessing based on a dust removal algorithm and preprocessing based on a homomorphic filtering algorithm.

8. The method according to any one of claims 1 to 7, wherein: Processing the point cloud data of the operation area to obtain a radar feature map group includes: Processing the point cloud data using a projection method to obtain a first radar image; Processing the first radar image using a bilateral filter to obtain a second radar image; and A pooling operation is performed on the second radar image to obtain the radar feature map group.

9. The method according to any one of claims 1 to 8, wherein: The working area includes bulk materials, and the method further includes: The characteristic parameters of the bulk material are determined according to the semantic segmentation result, so as to control the operation of the engineering machinery equipment according to the characteristic parameters.

10. A semantic segmentation device, comprising: A module configured to execute the semantic segmentation method described in any one of claims 1-9.

11. A semantic segmentation device, comprising: Memory; as well as A processor coupled to the memory is configured to execute the semantic segmentation method according to any one of claims 1 to 9 based on instructions stored in the memory.

12. A semantic segmentation system, for installation on engineering machinery equipment, comprising: The semantic segmentation device according to claim 10 or 11; An image acquisition unit, configured to send the acquired original image to the semantic segmentation device; and The point cloud acquisition unit is configured to send the acquired point cloud data to the semantic segmentation device.

13. An engineering machinery equipment, comprising: The semantic segmentation system of claim 12.

14. A computer-readable storage medium comprising computer program instructions, wherein: When the computer program instructions are executed by a processor, the semantic segmentation method according to any one of claims 1 to 9 is implemented.

15. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

16. A computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images