Semantic segmentation method, apparatus and system, and excavator

Abstract

The present disclosure relates to the field of intelligentization and automation of construction machinery. Provided are a semantic segmentation method, apparatus and system, and an excavator. The semantic segmentation method comprises: acquiring image information and radar point cloud information of an operation environment; determining a first feature representation of the image information; on the basis of labeling information of a target in the image information and the radar point cloud information, determining a second feature representation of the labeling information of the target; and fusing the first feature representation of the image information and the second feature representation of the labeling information of the target, so as to segment the target in the image information. In the present disclosure, multi-dimensional sensing information such as image information and radar information of an operation environment is comprehensively detected, and a feature representation of the image information and a feature representation of a labeled target in the image information and the radar information are fused for semantic segmentation, such that the target is more accurately detected from an image on the basis of a prompt regarding the labeled target, thereby enabling an excavator to comprehensively perceive an operation surface environment and providing the foundation for unmanned excavation operations.

Description

Semantic segmentation method, device, system and excavator

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the application with CN application number 202410220047.2 and application date February 27, 2024, and claims its priority. The disclosed content of the CN application is hereby introduced as a whole into this application. Technical Field

[0003] The present disclosure relates to the field of intelligent and automated engineering machinery, and in particular to a semantic segmentation method, device, system, and excavator. Background Art

[0004] Excavators are versatile construction machines widely used in mining, water conservancy projects, transportation, and power projects. Unmanned excavators can replace operators in operations and construction in environments with landslide hazards and toxic and hazardous gases, while also alleviating the labor shortage caused by an aging population. Therefore, the development of unmanned excavators has significant industry demand and practical significance. Perceiving the excavator's working surface environment is fundamental to achieving unmanned excavation operations, and accurate perception of this environment has become an urgent need.

[0005] Summary of the Invention

[0006] Some embodiments of the present disclosure provide a semantic segmentation method, including:

[0007] Obtain image information and radar point cloud information of the working environment;

[0008] determining a first feature representation of the image information;

[0009] determining a second feature representation of the target's marking information based on the image information and the target's marking information in the radar point cloud information;

[0010] The first feature representation of the image information and the second feature representation of the marking information of the target are fused to segment the target in the image information.

[0011] In some embodiments, obtaining image information and radar point cloud information of the operating environment includes:

[0012] Acquiring image information of the working environment using a camera device; and

[0013] A laser radar is used to obtain first radar point cloud information of the operating environment, and / or a four-dimensional millimeter wave radar is used to obtain second radar point cloud information of the operating environment.

[0014] In some embodiments, the method further includes determining target tag information using the following method:

[0015] According to the installation positions and angles of the camera device, laser radar, and four-dimensional millimeter-wave radar on the excavator, the conversion matrices between the excavator coordinate system and the image coordinate system of the camera device, the laser radar coordinate system, and the four-dimensional millimeter-wave radar coordinate system are determined;

[0016] determining, according to the conversion matrix, second marking information of the first marking information of the target in the first radar point cloud information in the image coordinate system, and / or determining fourth marking information of the third marking information of the target in the second radar point cloud information in the image coordinate system;

[0017] The original fifth marking information of the target in the image coordinate system and the second marking information and / or the fourth marking information in the image coordinate system in the image information are fused to determine the marking information of the target.

[0018] In some embodiments, fusing the original fifth marking information of the target in the image information in the image coordinate system with the second marking information and / or the fourth marking information in the image coordinate system to determine the marking information of the target includes:

[0019] Perform weighted averaging on the fifth label information and the second label information and / or the fourth label information in the image coordinate system, and use the obtained average label information as the target label information,

[0020] Among them, the weights of the fifth marking information, the second marking information and the fourth marking information are configured to be: the same, or determined according to the image clarity of the camera device, laser radar, and four-dimensional millimeter wave radar under the current operating conditions.

[0021] In some embodiments, fusing the original fifth marking information of the target in the image information in the image coordinate system with the second marking information and / or the fourth marking information in the image coordinate system to determine the marking information of the target includes:

[0022] According to the image clarity of the camera device, lidar, and four-dimensional millimeter-wave radar under the current operating conditions, the marking information corresponding to the highest image clarity is selected from the fifth marking information and the second marking information and / or fourth marking information in the image coordinate system as the marking information of the target.

[0023] In some embodiments, determining the first feature representation of the image information includes:

[0024] Convert the image into fixed-size image blocks and perform feature encoding on the image blocks;

[0025] Reduce the dimension of the feature-encoded data to a fixed-size first feature representation.

[0026] In some embodiments, determining a second feature representation of the target's marking information includes:

[0027] Each marking point represented by the marking information of the target is encoded to obtain a second feature representation of the marking information of the target.

[0028] In some embodiments, further comprising:

[0029] determining mask information of the target based on the image information;

[0030] determining a third feature representation of the mask information of the target based on the mask information of the target;

[0031] The first feature representation of the image information is fused with the second feature representation of the target's marking information and / or the third feature representation of the target's mask information to segment the target in the image information.

[0032] In some embodiments, determining mask information of the target based on the image information includes:

[0033] Performing semantic segmentation based on the image information to obtain an image region of the target;

[0034] Based on the image region of the object, mask information of the object is generated.

[0035] In some embodiments, determining a third feature representation of the mask information of the target includes:

[0036] The mask information of the target is convolved and encoded to obtain a third feature representation of the mask information of the target.

[0037] Some embodiments of the present disclosure provide a semantic segmentation device, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute a semantic segmentation method based on instructions stored in the memory.

[0038] Some embodiments of the present disclosure provide a computer-readable storage medium having a computer program stored thereon, which implements the steps of a semantic segmentation method when executed by a processor.

[0039] Some embodiments of the present disclosure provide a computer program product, including a computer program, which implements the steps of a semantic segmentation method when executed by a processor.

[0040] Some embodiments of the present disclosure provide a semantic segmentation system, including:

[0041] A camera device for collecting image information of the working environment;

[0042] Radar, including: a laser radar for collecting first radar point cloud information of the operating environment, and / or a four-dimensional millimeter wave radar for collecting second radar point cloud information of the operating environment;

[0043] The semantic segmentation device is configured to perform a semantic segmentation method.

[0044] Some embodiments of the present disclosure provide an excavator, comprising: a semantic segmentation system. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The following briefly introduces the drawings required for describing the embodiments or related technologies. The present disclosure can be more clearly understood based on the following detailed description with reference to the drawings.

[0046] Obviously, the drawings described below are only some embodiments of the present disclosure. A person skilled in the art can derive other drawings based on these drawings without inventive effort.

[0047] FIG1 shows a schematic diagram of the deployment of multiple sensors on an excavator and a schematic diagram of the corresponding coordinate system according to some embodiments of the present disclosure.

[0048] FIG2 is a schematic diagram showing an imaging model of a camera device in an excavator construction area according to some embodiments of the present disclosure.

[0049] FIG3 shows a flow chart of an interactive semantic segmentation method according to some embodiments of the present disclosure.

[0050] FIG4 shows a schematic diagram of an interactive semantic segmentation model according to some embodiments of the present disclosure.

[0051] FIG5 shows a schematic diagram of an interactive semantic segmentation method according to some embodiments of the present disclosure.

[0052] FIG6 shows a schematic structural diagram of a semantic segmentation apparatus according to some embodiments of the present disclosure.

[0053] FIG7 shows a schematic diagram of a semantic segmentation system according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0054] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0055] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, and do not represent any specific technical meanings, nor do they indicate a necessary logical order between them.

[0056] It should also be understood that in the embodiments of the present disclosure, “a plurality of” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0057] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0058] In addition, the term "and / or" in this disclosure is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the related objects are in an "or" relationship.

[0059] It should also be understood that the description of the various embodiments in this disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced with each other. For the sake of brevity, they will not be described one by one.

[0060] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0061] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0062] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0063] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0064] In addition, in order to avoid obscuring the present disclosure with unnecessary details, only the processing steps and / or device structures that are closely related to at least the solution according to the present disclosure are shown in the drawings, while other details that are not closely related to the present disclosure are omitted. It should also be noted that similar reference numerals and letters in the drawings indicate similar items, and therefore once an item is defined in one drawing, it does not need to be discussed again for subsequent drawings.

[0065] The disclosed embodiments comprehensively detect multi-dimensional sensor information such as images and radars of the working environment, fuse the feature representation of the image information and the feature representation of the marked targets in the image information and radar information to perform semantic segmentation, thereby more accurately detecting targets from the image based on the prompts of the marked targets, enabling the excavator to fully perceive the working surface environment, and providing a basis for unmanned excavation operations.

[0066] The semantic segmentation perception system for intelligent mining equipment disclosed in this disclosure includes a sensor hardware matching solution and a software algorithm architecture design solution, multi-sensor joint calibration, and interactive semantic segmentation technology. The implementation methods of each part will be elaborated in detail later. Using multiple sensors such as laser radar, four-dimensional (4D) millimeter wave radar, and camera devices, a multi-sensor joint calibration solution is designed to achieve spatiotemporal synchronization of multiple types of sensors. Using a semantic segmentation model that integrates prompt words, interactive semantic segmentation of laser point cloud data, 4D millimeter wave radar point cloud data, and images is achieved, thereby achieving interactive semantic segmentation of environmental surface shape, size, and texture. Taking full advantage of the respective advantages of laser radar, 4D millimeter wave radar, and camera devices, the perception system is provided with accurate semantic segmentation boundaries in different working scenarios, providing effective support for comprehensive environmental perception. While obtaining the three-dimensional coordinates of the target, the target's texture, appearance, true height, and other features can also be obtained, thereby inferring the target type, providing an important basis for the excavator's decision-making and planning. Based on the interactive semantic segmentation method, dynamic information of the segmented area can be provided in real time and accurately, providing sufficient perception basis for the excavator's automated and intelligent planning and control.

[0067] The hardware matching solution for the sensor is described below.

[0068] The first step in establishing the perception system architecture for intelligent excavators is to select sensors and hardware configurations based on the unmanned excavator's operating scenarios. Sensor hardware configurations for unmanned excavators primarily fall into three categories: lidar, cameras, and 4D millimeter-wave radar.

[0069] Camera devices, such as cameras, are low-cost and can identify different objects. They have advantages in object height and width measurement accuracy, lane line recognition, and pedestrian recognition accuracy. They are indispensable sensors for realizing functions such as obstacle recognition and road area detection. However, their effective range and ranging accuracy are not as good as 4D millimeter-wave radars, and they are easily affected by factors such as lighting and weather.

[0070] LiDAR offers the advantages of high measurement accuracy and a wide scanning range, but its range is limited and susceptible to lighting. LiDAR's azimuth and elevation resolution can reach 0.1 by 0.1 degrees.

[0071] 4D millimeter-wave radar is an upgraded version of traditional millimeter-wave radar, offering enhanced detection capabilities, higher resolution, and higher accuracy. 4D refers to the four dimensions of speed, distance, horizontal angle, and vertical height. Compared to traditional three-dimensional (3D) millimeter-wave radar, 4D millimeter-wave radar adds "altitude" detection, integrating the fourth dimension into traditional millimeter-wave radar. This provides the following advantages: It captures richer information dimensions, including the ability to measure elevation angles with an angular resolution of approximately 1 degree; it also offers a longer detection range, reaching over 300 meters; it also features a denser target point cloud, enabling point cloud imaging-level output, enabling data-driven image recognition; and it overcomes the effects of rain, fog, and dust, improving detection reliability. Currently, the azimuth and elevation angle resolution of 4D millimeter-wave radars can reach approximately 1 degree by 1 degree. 4D millimeter-wave radars are limited by the Doppler effect, meaning they still struggle to identify laterally moving objects, close vehicles, and pedestrians. Overall, 4D millimeter-wave radar can provide more reliable data support for the planning and control of intelligent mining equipment.

[0072] Sensors such as cameras, lidar, and millimeter-wave radar each have their own advantages and disadvantages. This disclosure utilizes a multi-sensor configuration to leverage their strengths and provide a more comprehensive understanding of the environment. The lidar and cameras operate in good weather, while the 4D millimeter-wave radar and cameras operate in adverse environments such as rain, fog, and dust, enabling the unmanned excavator to operate normally in a variety of conditions.

[0073] During implementation, we've designed a variety of sensor matching solutions based on an evaluation of the advantages and disadvantages of each of the aforementioned sensor types. Each sensor configuration solution has its optimal application scenario, and a detailed user manual has been compiled to help customers select the optimal sensor configuration solution based on their specific application scenario. These sensor configuration solutions include information on sensor type, number of sensors to be installed, and sensor installation locations, enabling rapid and accurate selection of multiple sensors.

[0074] The multi-sensor joint calibration scheme is described below.

[0075] FIG1 shows a schematic diagram of the deployment of multiple sensors on an excavator and a schematic diagram of the corresponding coordinate system according to some embodiments of the present disclosure.

[0076] Figure 1 shows a schematic diagram of the deployment of a camera 11, a four-dimensional millimeter-wave radar 12, and a laser radar 13 on an excavator 14. The camera 11, such as a camera or a video camera, is mounted directly in front of the excavator, facing downward, and rotates with the rotation of the excavator cab. The four-dimensional millimeter-wave radar 12 is mounted directly in front of the excavator, facing downward, and rotates with the rotation of the excavator cab. The laser radar 13 is mounted directly above the excavator.

[0077] In Figure 1, the excavator coordinate system is represented by Ow, Xw, Yw, and Zw; the imaging device (e.g., camera) coordinate system is represented by Oc, Xc, Yc, and Zc; the lidar coordinate system is represented by Ot, Xt, Yt, and Zt; and the four-dimensional millimeter-wave radar coordinate system is represented by Om, Xm, Ym, and Zm. These coordinate systems are all three-dimensional.

[0078] In the excavator coordinate system, the center point of the base of the excavator located at the center of rotation on the horizontal ground plane is the origin Ow, the Y-axis Yw is along the front of the excavator cab, the Z-axis Zw is perpendicular to the horizontal ground and facing upward, and the X-axis Xw is perpendicular to the plane of the Yw-axis Zw and along the front of the cab to the right.

[0079] The directions of the XYZ axes of the laser radar coordinate system and the excavator coordinate system are consistent, but their origins are different. The origin Ot of the laser radar coordinate system is at the center of the laser radar.

[0080] In the coordinate system of a camera device (such as a camera or a camera), the center position of the camera device is the origin Oc, the positive shooting direction of the camera device is the Y-axis Yc, the Z-axis Zc is perpendicular to the Yc-axis and facing upward, and the X-axis Xc is perpendicular to the Yc-axis Zc-axis and along the front of the cab to the right.

[0081] In the 4D millimeter-wave radar coordinate system, the radar's center is the origin Om. The radar's forward-facing direction is the Y-axis Ym, perpendicular to Ym and pointing upward is the Z-axis Zm, and perpendicular to the YmZm axis and pointing right along the front of the cab is the X-axis Xm. Therefore, if the radar and camera have the same forward-facing direction, the X, Y, and Z axes of the radar and camera coordinate systems will be aligned, with only the origin being different.

[0082] In order to accurately establish the correspondence between the excavator coordinate system and each pixel point in the image, it is necessary to establish an imaging model of the camera device in the excavator construction area, as shown in Figure 2.

[0083] FIG2 shows the coordinate system of the excavator represented by Ow, Xw, Yw, and Zw, the coordinate system of the camera (such as a camera or video camera) represented by Oc, Xc, Yc, and Zc, and the image coordinate system of the camera represented by the origin Oi and the mutually perpendicular u-axis and v-axis. The image coordinate system is two-dimensional. Point Pw is a point on the material in the excavator construction area. Pi is the point corresponding to point Pw in the image plane. The straight line connecting point Pw, point Pi, and point Oc is the projection line of point Pw on the excavator construction plane on the image plane. The projection line is obtained by connecting point Pw and point Oc. The intersection point Pi of the projection line and the image plane is the imaging point of point Pw on the image plane.

[0084] Among them, the three-dimensional coordinates of point Pw in the excavator coordinate system (x w ,y w ,z w ) to the two-dimensional coordinates (u i ,v i ) is shown in the following formula:

[0085] Wherein, Z is the scale factor; f is the focal length of the camera; dX and dY represent the physical lengths of a pixel on the photosensitive plate in the X-axis and Y-axis directions, respectively; (u0, v0) represent the coordinates of the center of the camera photosensitive plate in the pixel coordinate system, respectively; R is the rotation matrix from the three-dimensional excavator coordinate system to the three-dimensional camera device (such as a camera or a webcam) coordinate system; t is the translation vector from the three-dimensional excavator coordinate system to the three-dimensional camera device (such as a camera or a webcam) coordinate system; K is the intrinsic parameter matrix of the camera device (such as a camera or a webcam); T is the transformation matrix from the three-dimensional excavator coordinate system to the three-dimensional camera device (such as a camera or a webcam) coordinate system.

[0086] In the above formula, multiplying 1 / Z by the 3D excavator coordinates of point Pw yields the normalized coordinates of Pw in the camera coordinate system, located on the plane in front of the camera at Z = 1. The transformation matrix composed of R and t represents the rigid body transformation. The transformation from the camera coordinate system to the image coordinate system is a perspective transformation, and the transformation from the image coordinate system to the pixel coordinate system is an affine transformation.

[0087] The transformation matrix between the lidar coordinate system and the excavator coordinate system is as follows:

[0088] formula:

[0089] in,

[0090] Rt is the rotation matrix from the lidar coordinate system to the excavator coordinate system;

[0091] Tt is the translation vector from the laser radar coordinate system to the excavator coordinate system;

[0092] Xw, Yw, and Zw are the coordinate point positions of the excavator coordinate system;

[0093] Xt, Yt, and Zt are the coordinate points in the laser radar coordinate system.

[0094] The transformation matrix between the four-dimensional millimeter-wave radar coordinate system and the excavator coordinate system is as follows:

[0095] formula:

[0096] in,

[0097] Rm is the rotation matrix from the four-dimensional millimeter wave radar coordinate system to the excavator coordinate system;

[0098] Tm is the translation vector from the four-dimensional millimeter-wave radar coordinate system to the excavator coordinate system;

[0099] Xw, Yw, and Zw are the coordinate point positions of the excavator coordinate system;

[0100] Xm, Ym, and Zm are the coordinate points in the four-dimensional millimeter-wave radar coordinate system.

[0101] Based on a multi-sensor joint calibration scheme, the data from the camera, LiDAR, and 4D millimeter-wave radar are converted to the excavator coordinate system, achieving joint and unified calibration of the multi-sensor coordinate system. This allows for conversion between the image coordinate system, the LiDAR coordinate system, and the 4D millimeter-wave radar coordinate system based on the excavator coordinate system.

[0102] The software algorithm architecture design scheme is described below.

[0103] FIG3 shows a flow chart of an interactive semantic segmentation method according to some embodiments of the present disclosure.

[0104] As shown in FIG3 , the interactive semantic segmentation method of this embodiment includes the following steps.

[0105] In step 31a, the camera device collects image information of the working environment.

[0106] In some embodiments, the camera device collects image information of the working environment based on the drive of a control component such as a processor.

[0107] In step 31b, the laser radar collects first radar point cloud information of the working environment.

[0108] In some embodiments, the laser radar collects first radar point cloud information of the working environment based on the drive of control components such as a processor.

[0109] In step 31c, the four-dimensional millimeter wave radar collects second radar point cloud information of the operating environment.

[0110] In some embodiments, the four-dimensional millimeter wave radar collects second radar point cloud information of the operating environment based on the drive of control components such as a processor.

[0111] The execution order of steps 31a, 31b, and 31c is not specific. Depending on the working conditions, only part of steps 31a, 31b, and 31c may need to be executed, and correspondingly, only part of steps 32a, 32b, and 32c need to be executed.

[0112] In step 32a, the image information is processed. Such processing includes, but is not limited to, geometric transformation of the image, color saturation transformation of the image, image enhancement using a dust removal algorithm in the event of inclement weather, and homomorphic filtering to enhance pixel values ​​in low-brightness areas of the image.

[0113] In step 32b, the first radar point cloud information is processed, such as but not limited to: noise reduction and filtering of the point cloud.

[0114] In step 32c, the second radar point cloud information is processed, including but not limited to calibration, coordinate conversion, noise removal, and static-dynamic separation.

[0115] The execution order of steps 32a, 32b, and 32c is not specific.

[0116] In step 33, based on the collected or processed information of the above-mentioned multiple sensors, an interactive semantic segmentation method is executed to output descriptive information of the target, such as position, height, texture, precise edge, etc., to achieve interactive semantic segmentation of point cloud data and images, and to achieve interactive semantic segmentation of environmental surface shape and size and environmental surface texture.

[0117] The interactive semantic segmentation method can be implemented using an interactive semantic segmentation model. Figure 4 shows a schematic diagram of an interactive semantic segmentation model according to some embodiments of the present disclosure.

[0118] As shown in Figure 4, the interactive semantic segmentation model consists of the following parts.

[0119] The image encoder 41 is used to encode the features of the image and can be implemented using an attention module. For example, the image encoder can have the same architecture as the visual transformer and be pre-trained on a self-collected labeled dataset of various excavator operation scenes. The image encoder takes an input image of any size, unifies the size of the data format, converts it into a fixed-size image patch, and performs feature encoding on the image patch so that the image output by the image encoder has the same dimensionality. The input image is converted into a fixed-size image patch, for example: first, the size is proportionally changed to the maximum side, and then the mean is subtracted and divided by the variance, and finally, padding is performed with a padding value of 0.

[0120] The feature embedding layer 42 converts (reduces) the feature-encoded data into a fixed-size feature representation (feature vector, also called the first feature representation) for easier processing and computation (e.g., distance calculation). The main purpose of embedding is to reduce the dimensionality of (sparse) features. The dimensionality reduction method can be compared to a fully connected layer (without an activation function), which reduces the dimensionality by calculating the weight matrix of the embedding layer.

[0121] The semantic segmentation model 43 can be obtained by pre-training and is used to implement the semantic segmentation function of the image. It can be implemented using an existing model that can implement semantic segmentation. The semantic segmentation model is usually based on the architecture of a deep convolutional neural network (CNN), and obtains the category prediction of each pixel by performing convolution operations, pooling operations, and upsampling operations on the input image. The semantic segmentation model achieves semantic segmentation by minimizing the classification loss at the pixel level during the training process. Semantic segmentation models include, for example, FCN (Fully Convolutional Networks) models, U-Net models, DeepLab models, SegNet models, GMMSeg models, etc., but are not limited to the examples given.

[0122] Image mask 44 refers to masking an image. Masking refers to using a selected image, graphic, or object to block (completely or partially) the processed image to control the image processing area or processing process. Masking is used to block certain areas of the image so that they do not participate in processing or calculation of processing parameters, or only the blocked areas are processed or statistically analyzed. Specifically in the present disclosure, image masking refers to masking the target area after the original image is processed through some mature pre-trained semantic segmentation model to obtain the target area.

[0123] The convolution layer 45 performs convolution processing on the input data. Specifically, it performs convolution processing on the mask information of the target.

[0124] The hint encoder 46 is used to encode the input marking information or mask information to obtain the corresponding feature representation. For marking point encoding, the marking point is converted from a point to a vector; for marking box encoding, the marking box is converted from the two corner points of the box to a vector; for mask encoding, it is only necessary to downsample the mask to ensure that it is consistent with the output of the image encoder. Among them, each marking point represented by the marking information of the target is encoded to obtain the second feature representation of the marking information of the target; the mask information of the target is convolved and encoded to obtain the third feature representation of the mask information of the target. The hint encoder supports both sparse hints (points, boxes, text) and dense hints (masks). Sparse hints are projected and connected with the image embedding, while dense hints are embedded using convolution and summed element-by-element with the image implantation.

[0125] The following method is used to determine the target's tag information:

[0126] (1) Based on the installation positions and angles of the camera, lidar, and 4D millimeter-wave radar on the excavator, determine the transformation matrices between the excavator coordinate system and the camera image coordinate system, lidar coordinate system, and 4D millimeter-wave radar coordinate system, respectively. The specific forms of each transformation matrix are described above.

[0127] (2) Determine, based on the transformation matrix, the second marking information of the first marking information of the target in the first radar point cloud information (i.e., the radar point cloud information of the working environment obtained by using a laser radar) in the image coordinate system, and / or determine the fourth marking information of the third marking information of the target in the second radar point cloud information (i.e., the radar point cloud information of the working environment obtained by using a four-dimensional millimeter-wave radar) in the image coordinate system.

[0128] The first label information is, for example, any label information that can represent the location of the target in the first radar point cloud information. The first label information is converted from the lidar coordinate system to the image coordinate system to obtain the second label information. The third label information is, for example, any label information that can represent the location of the target in the second radar point cloud information. The third label information is converted from the four-dimensional millimeter-wave radar coordinate system to the image coordinate system to obtain the fourth label information.

[0129] Among them, according to the transformation matrix between the lidar coordinate system and the excavator coordinate system, and the transformation matrix between the excavator coordinate system and the image coordinate system, the transformation matrix between the lidar coordinate system and the image coordinate system can be determined, and based on the transformation matrix, the second marking information of the first marking information of the target in the first radar point cloud information in the image coordinate system can be determined.

[0130] Similarly, based on the transformation matrix between the four-dimensional millimeter-wave radar coordinate system and the excavator coordinate system, and the transformation matrix between the excavator coordinate system and the image coordinate system, the transformation matrix between the four-dimensional millimeter-wave radar coordinate system and the image coordinate system can be determined. Based on the transformation matrix, the fourth marking information of the third marking information of the target in the second radar point cloud information in the image coordinate system can be determined.

[0131] (3) Fusing the original fifth marking information of the target in the image information in the image coordinate system and the second marking information and / or the fourth marking information in the image coordinate system to determine the marking information of the target, for example, including the following exemplary fusion method.

[0132] That is, based on the original fifth marking information of the target in the image information in the image coordinate system, the second marking information and / or the fourth marking information in the image coordinate system are fused to determine the marking information of the target.

[0133] (3-1) Performing a weighted average of the fifth label information and the second label information and / or the fourth label information in the image coordinate system, and using the obtained average label information as the label information of the target, wherein the weights for weighting the fifth label information, the second label information, and the fourth label information are configured to be: the same, or determined based on the image clarity of the camera device, the laser radar, and the four-dimensional millimeter-wave radar under the current operating conditions. The fifth label information is determined based on the image clarity of the camera device under the current operating conditions, the second label information is determined based on the image clarity of the laser radar under the current operating conditions, and the fourth label information is determined based on the image clarity of the four-dimensional millimeter-wave radar under the current operating conditions.

[0134] That is, the method of determining the target's marking information includes any of the following:

[0135] Performing a weighted average on the fifth label information and the second label information, and using the obtained average label information as the target label information;

[0136] Performing a weighted average on the fifth label information and the fourth label information, and using the obtained average label information as the target label information;

[0137] A weighted average is performed on the fifth label information, the second label information, and the fourth label information, and the obtained average label information is used as the target label information.

[0138] (3-2) Based on the image clarity of the camera device, laser radar, and four-dimensional millimeter-wave radar under the current operating conditions, the marking information corresponding to the highest image clarity is selected from the fifth marking information and the second marking information and / or the fourth marking information in the image coordinate system as the marking information of the target.

[0139] That is, according to the image clarity of the camera device, laser radar, and four-dimensional millimeter-wave radar under the current operating conditions, the mark information corresponding to the highest image clarity is selected as the mark information of the target from the fifth mark information and the second mark information in the image coordinate system, or from the fifth mark information and the fourth mark information in the image coordinate system, or from the fifth mark information, the second mark information, and the fourth mark information.

[0140] Interpreter 47 upsamples the image embedding using two transposed convolutional layers and fuses the upscaled image embedding with the output data from hint encoder 46 to obtain more accurate object prediction results. Fusion methods include, but are not limited to, fusing the first feature representation of the image information with a second feature representation of the target's marking information and / or a third feature representation of the target's mask information to detect the target in the image information. Both the second and third feature representations can indicate the target's location information. Based on this target's location information, the target is detected in the image information according to the first feature representation of the image information.

[0141] That is, the fusion method includes any of the following:

[0142] fusing the first feature representation of the image information with the second feature representation of the target's marking information to segment the target in the image information;

[0143] fusing the first feature representation of the image information with the third feature representation of the mask information of the target to segment the target in the image information;

[0144] The first feature representation of the image information is fused with the second feature representation of the target's marking information and the third feature representation of the target's mask information to segment the target in the image information.

[0145] The following describes the interactive semantic segmentation method in conjunction with the interactive semantic segmentation model. Figure 5 shows a schematic diagram of the interactive semantic segmentation method according to some embodiments of the present disclosure.

[0146] In step 51 , image information and radar point cloud information (including first radar point cloud information and / or second radar point cloud information) of the working environment are acquired.

[0147] The image information of the working environment is acquired by a camera device. The first radar point cloud information of the working environment is acquired by a laser radar. The second radar point cloud information of the working environment is acquired by a four-dimensional millimeter wave radar.

[0148] At step 52, a first feature representation of the image information is determined.

[0149] The image is converted into an image block of a fixed size and feature encoding is performed on the image block. This step can be implemented using the image encoder 41 ; the feature encoding data is reduced in dimension to a first feature representation of a fixed size. This step can be implemented using the feature embedding layer 42 .

[0150] In step 53 , a second feature representation of the target's marking information is determined based on the image information and the target's marking information in the radar point cloud information.

[0151] Each marking point represented by the marking information of the target is encoded to obtain a second feature representation of the marking information of the target. This step can be implemented using a prompt encoder.

[0152] The following method is used to determine the target's tag information:

[0153] (1) Based on the installation positions and angles of the camera, lidar, and 4D millimeter-wave radar on the excavator, determine the transformation matrices between the excavator coordinate system and the camera image coordinate system, lidar coordinate system, and 4D millimeter-wave radar coordinate system, respectively. The specific forms of each transformation matrix are described above.

[0154] (2) Determine, based on the transformation matrix, the second marking information of the first marking information of the target in the first radar point cloud information (i.e., the radar point cloud information of the working environment obtained by using a laser radar) in the image coordinate system, and / or determine the fourth marking information of the third marking information of the target in the second radar point cloud information (i.e., the radar point cloud information of the working environment obtained by using a four-dimensional millimeter-wave radar) in the image coordinate system.

[0155] Among them, according to the transformation matrix between the lidar coordinate system and the excavator coordinate system, and the transformation matrix between the excavator coordinate system and the image coordinate system, the transformation matrix between the lidar coordinate system and the image coordinate system can be determined, and based on the transformation matrix, the second marking information of the first marking information of the target in the first radar point cloud information in the image coordinate system can be determined.

[0156] Similarly, based on the transformation matrix between the four-dimensional millimeter-wave radar coordinate system and the excavator coordinate system, and the transformation matrix between the excavator coordinate system and the image coordinate system, the transformation matrix between the four-dimensional millimeter-wave radar coordinate system and the image coordinate system can be determined. Based on the transformation matrix, the fourth marking information of the third marking information of the target in the second radar point cloud information in the image coordinate system can be determined.

[0157] (3) Fusing the original fifth marking information of the target in the image information in the image coordinate system and the second marking information and / or the fourth marking information in the image coordinate system to determine the marking information of the target, for example, including the following exemplary fusion method.

[0158] (3-1) Performing a weighted average on the fifth marking information and the second marking information and / or the fourth marking information in the image coordinate system, and using the obtained average marking information as the marking information of the target, wherein the weights of the fifth marking information, the second marking information, and the fourth marking information are configured to be: the same, or determined according to the image clarity of the camera device, the laser radar, and the four-dimensional millimeter wave radar under the current operating conditions.

[0159] (3-2) Based on the image clarity of the camera device, laser radar, and four-dimensional millimeter-wave radar under the current operating conditions, the marking information corresponding to the highest image clarity is selected from the fifth marking information and the second marking information and / or the fourth marking information in the image coordinate system as the marking information of the target.

[0160] In step 54 , a third feature representation of the mask information of the object is determined based on the mask information of the object.

[0161] Based on the image information, the target and its mask information are determined. The target is segmented using a semantic segmentation model 43, and the target mask information is determined using an image mask 44. The target mask information is convolved using a convolutional layer 45 and then encoded using a hint encoder 46 to obtain a third feature representation of the target mask information.

[0162] There is no particular order in which steps 52, 53 and 54 are executed. Either one or both of step 53 and step 54 can be executed as needed.

[0163] In step 55 , the first feature representation of the image information is fused with the second feature representation of the target's marking information and / or the third feature representation of the target's mask information to segment the target in the image information.

[0164] Both the second feature representation and the third feature representation can indicate the location information of the target. Based on the location information of the target and according to the first feature representation of the image information, the target in the image information is detected.

[0165] In addition to obtaining the target's 3D coordinates, the system also captures features such as its location, texture, appearance, true height, and precise edges. This allows the target's type to be inferred, providing crucial information for excavator planning. Based on interactive semantic segmentation, dynamic information about the segmented area can be accurately and in real time, providing a robust sensory basis for automated and intelligent excavator planning and control.

[0166] FIG6 is a schematic diagram illustrating the structure of a semantic segmentation apparatus according to some embodiments of the present disclosure. As shown in FIG6 , the semantic segmentation apparatus 600 according to this embodiment includes a memory 610 and a processor 620 coupled to the memory 610 . The processor 620 is configured to execute the semantic segmentation method according to any of the embodiments based on instructions stored in the memory 610 .

[0167] The apparatus 600 may further include an input / output interface 630 , a network interface 640 , a storage interface 650 , etc. These interfaces 630 , 640 , 650 , the memory 610 , and the processor 620 may be connected via a bus 660 , for example.

[0168] The memory 610 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs.

[0169] The processor 620 may be implemented using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, or discrete hardware components such as discrete gates or transistors.

[0170] Among them, the input and output interface 630 provides a connection interface for input and output devices such as a display, mouse, keyboard, and touch screen. The network interface 640 provides a connection interface for various networked devices. The storage interface 650 provides a connection interface for external storage devices such as SD cards and USB flash drives. The bus 660 can use any of a variety of bus structures. For example, bus structures include but are not limited to the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, and the Peripheral Component Interconnect (PCI) bus.

[0171] The embodiments of the present disclosure provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the semantic segmentation method in any of the embodiments are implemented.

[0172] The embodiments of the present disclosure provide a computer program product, including a computer program, which, when executed by a processor, implements the steps of the semantic segmentation method in any of the embodiments.

[0173] The present disclosure provides a semantic segmentation system that can be used in an excavator 14. As shown in FIG7 , the semantic segmentation system 700 includes:

[0174] A camera device 11 is used to collect image information of the working environment;

[0175] Radar, for collecting radar point cloud information of the working environment, the radar includes, for example: a laser radar 13, for collecting first radar point cloud information of the working environment, and / or a four-dimensional millimeter wave radar 12, for collecting second radar point cloud information of the working environment;

[0176] The semantic segmentation device 600 is configured to collect information from multiple sensors and execute the semantic segmentation method in any of the embodiments.

[0177] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more non-transitory computer-readable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer program code.

[0178] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0179] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0180] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0181] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A semantic segmentation method, comprising: Obtain image information and radar point cloud information of the working environment; determining a first feature representation of the image information; determining a second feature representation of the target's marking information based on the image information and the target's marking information in the radar point cloud information; The first feature representation of the image information and the second feature representation of the marking information of the target are fused to segment the target in the image information.

2. The semantic segmentation method according to claim 1, wherein: Obtaining image information and radar point cloud information of the working environment includes: Acquiring image information of the working environment using a camera device; and A laser radar is used to obtain first radar point cloud information of the operating environment, and / or a four-dimensional millimeter wave radar is used to obtain second radar point cloud information of the operating environment.

3. The semantic segmentation method according to claim 2, further comprising determining target label information by: According to the installation positions and angles of the camera device, laser radar, and four-dimensional millimeter-wave radar on the excavator, the conversion matrices between the excavator coordinate system and the image coordinate system of the camera device, the laser radar coordinate system, and the four-dimensional millimeter-wave radar coordinate system are determined; determining, according to the conversion matrix, second marking information of the first marking information of the target in the first radar point cloud information in the image coordinate system, and / or determining fourth marking information of the third marking information of the target in the second radar point cloud information in the image coordinate system; The second marking information and / or the fourth marking information in the image coordinate system are fused based on the original fifth marking information of the target in the image information to determine the marking information of the target.

4. The semantic segmentation method according to claim 3, wherein: Based on the original fifth marking information of the target in the image information in the image coordinate system, the second marking information and / or the fourth marking information in the image coordinate system are fused to determine the marking information of the target, which includes any one of the following: Performing a weighted average on the fifth label information and the second label information in the image coordinate system, and using the obtained average label information as the label information of the target; Perform weighted averaging on the fifth marking information and the fourth marking information in the image coordinate system, and use the obtained average The value tag information is used as the tag information of the target; Perform a weighted average on the fifth label information, the second label information, and the fourth label information, and use the obtained average label information as the target label information. Among them, the weights of the fifth marking information, the second marking information and the fourth marking information are configured to be: the same, or determined according to the image clarity of the camera device, laser radar, and four-dimensional millimeter wave radar under the current operating conditions.

5. The semantic segmentation method according to claim 3, wherein: The step of fusing the second marking information and / or the fourth marking information in the image coordinate system based on the original fifth marking information of the target in the image information in the image coordinate system to determine the marking information of the target includes: According to the image clarity of the camera device, laser radar, and four-dimensional millimeter-wave radar under the current operating conditions, the mark information corresponding to the highest image clarity is selected as the mark information of the target from the fifth mark information and the second mark information in the image coordinate system, or from the fifth mark information and the fourth mark information in the image coordinate system, or from the fifth mark information, the second mark information, and the fourth mark information.

6. The semantic segmentation method according to claim 1, wherein: Determining a first feature representation of the image information includes: Convert the image into fixed-size image blocks and perform feature encoding on the image blocks; Reduce the dimension of the feature-encoded data to a fixed-size first feature representation.

7. The semantic segmentation method according to claim 1, wherein: Determining a second feature representation of the target's label information includes: Each marking point represented by the marking information of the target is encoded to obtain a second feature representation of the marking information of the target.

8. The semantic segmentation method according to claim 1, further comprising: determining mask information of the target based on the image information; determining a third feature representation of the mask information of the target based on the mask information of the target; The fusion includes: fusing the first feature representation of the image information with the second feature representation of the target's marking information and the third feature representation of the target's mask information to segment the target in the image information.

9. The semantic segmentation method according to claim 8, wherein: Determining mask information of the target based on the image information includes: Performing semantic segmentation based on the image information to obtain an image region of the target; Based on the image region of the object, mask information of the object is generated.

10. The semantic segmentation method according to claim 8, wherein: Determining a third feature representation of the mask information of the target includes: The mask information of the target is convolved and encoded to obtain a third feature representation of the mask information of the target.

11. A semantic segmentation device, comprising: Memory; and a processor coupled to the memory, wherein the processor is configured to execute the semantic segmentation method according to any one of claims 1 to 10 based on instructions stored in the memory.

12. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the semantic segmentation method according to any one of claims 1 to 10 are implemented.

13. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the semantic segmentation method according to any one of claims 1 to 10.

14. A semantic segmentation system comprising: A camera device for collecting image information of the working environment; Radar, including: a laser radar for collecting first radar point cloud information of the operating environment, and / or a four-dimensional millimeter wave radar for collecting second radar point cloud information of the operating environment; A semantic segmentation device, configured to execute the semantic segmentation method according to any one of claims 1 to 10.

15. An excavator comprising: The semantic segmentation system of claim 14.

Images