Intelligent operation method and system of chain bucket ship unloader based on point cloud data processing

Abstract

The present application provides an intelligent operation method and system for a chain bucket ship unloader based on point cloud data processing. For a point cloud rendering image, a visual transformer algorithm is called to perform obstacle detection to obtain a first support set, and an obstacle detection algorithm is called to perform obstacle detection to obtain a second support set. The first support set and the second support set are combined to obtain the target obstacle in the point cloud rendering image. The detection capability based on the obstacle detection algorithm in the obstacle recognition process alleviates the impact of the obstacle type prediction capability in the obstacle detection processing of the visual transformer algorithm, and increases the detection accuracy. In addition, compared with using point cloud coding tensor sets of various scales to detect the first spatial object and the second spatial object, the present application uses different point cloud coding tensor sets for separate detection, which can effectively save the amount of calculation in the detection process, save costs, and improve efficiency.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an intelligent operation method and system for a chain bucket ship unloader based on point cloud data processing. Background Art

[0002] In modern logistics and port operations, chain bucket ship unloaders play a vital role. With the continuous growth of port throughput and the increasingly complex operating environment, the intelligent operation of chain bucket ship unloaders has become a key requirement for improving operating efficiency and ensuring operating safety. In the operation scenario of chain bucket ship unloaders, accurately detecting obstacles and determining their types are important links to ensure safe operation. Traditional single algorithms often have many problems when performing obstacle detection and type prediction.

[0003] Some algorithms rely too much on their own single logical structure when performing obstacle detection. For example, if some algorithms take into account both obstacle detection and type prediction functions during the detection process, they may be interfered with by their type prediction ability when detecting obstacles due to the limitations of the algorithm's internal structure. Specifically, when an algorithm needs to determine whether an object exists (obstacle detection) and also make a preliminary judgment on what type the object may be (obstacle type prediction), the two functions may affect each other. For example, when detecting an object with a vague shape, the algorithm may have a preconceived tendency to predict its type, which may lead to a deviation in the judgment of whether the object is an obstacle, thereby reducing the accuracy of the detection. Similarly, in terms of obstacle type prediction, a single algorithm may affect the accuracy of type prediction due to the limitations of its detection ability because its own structure focuses on one aspect of detection or prediction. For example, if an algorithm can only obtain relatively rough object feature information when detecting an object, it may lack sufficient accurate basis when predicting the type, resulting in type prediction errors. In addition, point cloud data is an important data source for describing the surrounding environment of the chain bucket unloader operation scene. However, in the prior art, the processing method of point cloud data is insufficient in efficiency. Summary of the invention

[0004] In view of this, the present application provides an intelligent operation method and system for a chain bucket ship unloader based on point cloud data processing. The technical solution of the present application is implemented as follows: On the one hand, the present application provides an intelligent operation method for a chain bucket ship unloader based on point cloud data processing, the method comprising: acquiring point cloud data of the area where the chain bucket ship unloader is located; performing point cloud rendering on the point cloud data to obtain a point cloud rendered image; combining the point cloud rendered image, calling the visual transformer algorithm to perform obstacle detection on the point cloud rendered image to obtain a first support set relative to a preset obstacle set, calling the obstacle detection algorithm to perform obstacle detection on the point cloud rendered image to obtain a second support set relative to the preset obstacle set, the preset obstacle set including multiple candidate obstacles; combining the first support set and the second support set to obtain the target obstacle in the point cloud rendered image. obstacle; combining the point cloud rendering image and the target obstacle, calling the visual transformer algorithm to predict the obstacle type of the target obstacle, and obtaining a third support set relative to the preset obstacle type set, calling the obstacle type prediction algorithm to predict the obstacle type of the target obstacle, and obtaining a fourth support set relative to the preset obstacle type set, wherein the preset obstacle type set includes multiple reference obstacle types; combining the third support set and the fourth support set to obtain the obstacle type of the target obstacle; combining the target obstacle and the obstacle type of the target obstacle to obtain the obstacle annotation result of the point cloud rendering image, so as to perform the operation path planning of the chain bucket ship unloader based on the obstacle annotation result.

[0005] On the other hand, the present application provides a computer system, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements the steps in the above method when executing the program.

[0006] Beneficial effects of the present application: The intelligent operation method of the chain bucket ship unloader based on point cloud data processing of the present application, for the point cloud rendering image, calls the visual transformer algorithm to perform obstacle detection to obtain a first support set, calls the obstacle detection algorithm to perform obstacle detection to obtain a second support set, and combines the first support set and the second support set to obtain the target obstacle in the point cloud rendering image. In the obstacle recognition process, the detection capability of the obstacle detection algorithm is used to alleviate the influence of the obstacle type prediction capability in the obstacle detection processing of the visual transformer algorithm, thereby increasing the detection accuracy. In addition, the visual transformer algorithm is called through the point cloud rendering image and the target obstacle to predict the obstacle type of the target obstacle to obtain a third support set, and the obstacle type prediction algorithm is called to predict the obstacle type to obtain a fourth support set. At the same time, the obstacle type is obtained through the third support set and the fourth support set. In this way, when predicting the obstacle type, the prediction capability of the obstacle type prediction algorithm can be combined to alleviate the influence of the detection capability in the obstacle type prediction processing of the visual transformer algorithm, thereby increasing the accuracy of the obstacle type prediction. In this way, the obstacle detection and obstacle type prediction are decoupled, and different algorithms are alternately used for detection or obstacle type prediction in different links to further improve the reliability of the obstacle labeling results.

[0007] In addition, when acquiring a point cloud rendering image, feature encoding is performed based on the point cloud data to obtain point cloud coding tensor sets of multiple scales. At the same time, the target object is detected according to the point cloud coding tensor set of the smallest scale, and the second spatial object is detected through point cloud coding tensor sets of other scales. Compared with using point cloud coding tensor sets of various scales to detect the first spatial object and the second spatial object, the present application adopts a solution of separate detection using different point cloud coding tensor sets, which can effectively save the amount of computation in the detection process, save costs, and improve efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A schematic diagram of the implementation flow of an intelligent operation method of a chain bucket ship unloader based on point cloud data processing provided in an embodiment of the present application;

[0009] Figure 2 A hardware entity diagram of a computer system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0010] The embodiment of the present application provides an intelligent operation method of a chain bucket ship unloader based on point cloud data processing, which can be executed by a processor of a computer system, wherein the computer system can refer to a device with data processing capabilities such as a server, a laptop, a tablet computer, and a desktop computer. Figure 1A schematic diagram of the implementation flow of an intelligent operation method of a chain bucket ship unloader based on point cloud data processing provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method comprises the following steps:

[0011] Step S100: acquiring point cloud data of the area where the chain bucket ship unloader is located;

[0012] Step S200: performing point cloud rendering on the point cloud data to obtain a point cloud rendering image;

[0013] Step S300: combining the point cloud rendering image, calling the visual transformer algorithm to perform obstacle detection on the point cloud rendering image, obtaining a first support set relative to a preset obstacle set, calling the obstacle detection algorithm to perform obstacle detection on the point cloud rendering image, obtaining a second support set relative to the preset obstacle set, wherein the preset obstacle set includes a plurality of candidate obstacles;

[0014] Step S400: combining the first support set and the second support set to obtain a target obstacle in the point cloud rendering image;

[0015] Step S500: combining the point cloud rendering image and the target obstacle, calling the visual transformer algorithm to predict the obstacle type of the target obstacle, obtaining a third support set relative to a preset obstacle type set, calling the obstacle type prediction algorithm to predict the obstacle type of the target obstacle, obtaining a fourth support set relative to the preset obstacle type set, wherein the preset obstacle type set includes a plurality of reference obstacle types;

[0016] Step S600: combining the third support set and the fourth support set to obtain the obstacle type of the target obstacle;

[0017] Step S700: Obtain obstacle annotation results of the point cloud rendering image in combination with the target obstacle and the obstacle type of the target obstacle, so as to plan the operation path of the chain bucket ship unloader based on the obstacle annotation results.

[0018] In step S100, point cloud data is a set of vectors in a three-dimensional coordinate system. In the operation scene of the chain bucket ship unloader, these vectors can represent discrete points on the surfaces of various objects in the surrounding environment of the chain bucket ship unloader. For example, the dock where the ship unloader is located may have stacked goods, other equipment, buildings, and staff, etc. The surface feature points of these objects constitute point cloud data in the form of three-dimensional coordinates. The computer system obtains this data through specific sensor devices, which may include laser radars, etc. The laser radar determines the distance information of surrounding objects by emitting laser beams and receiving reflected light, thereby constructing point cloud data. These data contain rich spatial information and are the basis for subsequent various processing and analysis.

[0019] In step S200, point cloud rendering is the process of converting point cloud data into an image form for subsequent processing and analysis. The computer system will use a series of algorithms and techniques to achieve this conversion. Each point in the point cloud data contains three-dimensional coordinate information. During the rendering process, these points will be mapped to a two-dimensional plane according to certain rules, and the display color or grayscale value in the image will be determined according to other attributes of the points (such as reflection intensity, etc.). For example, assuming that there is a point set representing a cargo pile in the point cloud data, when rendering, the points close to the top of the cargo pile may have a higher reflection intensity due to being close to the sensor, and may be displayed as a brighter color or a higher grayscale value in the rendered image; while the points at the bottom of the cargo pile close to the ground may be partially blocked or relatively far from the sensor, and the reflection intensity is low, and may be displayed as a darker color or a lower grayscale value in the rendered image. In this way, the entire point cloud data is converted into a point cloud rendering image that can intuitively represent the surrounding environment of the chain bucket unloader.

[0020] In step S300, the preset obstacle set is a set of obstacle types that may appear in the operation area of ​​the chain bucket unloader that is preset by the computer system. For example, this set may include other ships, dock facilities (such as the support structure of the crane, etc.), personnel, cargo piles (if they are in an inappropriate position, they are considered obstacles), etc. The visual transformer algorithm (ViT) is a visual processing algorithm based on the transformer structure. The computer system inputs the point cloud rendering image into the ViT algorithm, and the ViT algorithm analyzes and processes the image. It divides the image into multiple small blocks (similar to patches of the image), and then encodes these small blocks, and captures the relationship between different parts of the image through the self-attention mechanism of the transformer, so as to determine whether there are obstacles in the preset obstacle set in the image, and calculate a support for each candidate obstacle. This support can be understood as the probability or confidence that the candidate obstacle appears in the image, and these support constitute the first support set. For example, if there is a candidate obstacle "personnel" in the preset obstacle set, after processing, the ViT algorithm may give a support value of 0.6 for the candidate obstacle "personnel" in the current point cloud rendering image, which means that the algorithm believes that there is a 60% probability that there is an obstacle such as a person in the image.

[0021] The computer system also inputs the point cloud rendering image into the selected obstacle detection algorithm (such as YOLO, Faster-R-CNN, CRNN, etc.). These algorithms have their own detection mechanisms. Taking the YOLO algorithm as an example, it divides the input image into a grid, and each grid cell is responsible for detecting whether a certain target exists and the location of the target.

[0022] When detecting obstacles in the preset obstacle set, it will calculate a support for each candidate obstacle. For example, for the candidate obstacle "ship", the YOLO algorithm may calculate its support in the image as 0.7 based on the ship features in the image (such as outline, color, etc.), which means that the algorithm believes that there is a 70% chance that there is a ship obstacle in the image. These supports calculated for each candidate obstacle constitute the second support set.

[0023] In step S400, after obtaining the first support set and the second support set, the computer system will conduct a comprehensive analysis on the two sets. For example, for a candidate obstacle in the preset obstacle set, assume that its support in the first support set is 0.4 and its support in the second support set is 0.5. The computer system can use a simple weighted average or other fusion strategy to determine whether the candidate obstacle is ultimately determined as a target obstacle. If a weighted average is used, and the weight of the visual transformer algorithm (ViT) is set to 0.4 and the weight of the obstacle detection algorithm is set to 0.6, then the comprehensive support of the candidate obstacle is 0.4*0.4+0.6*0.5=0.46. If a threshold is set to 0.45, since 0.46 is greater than 0.45, then the candidate obstacle may be determined as a target obstacle. By performing such processing on all candidate obstacles in the preset obstacle set, the computer system can determine the target obstacle in the point cloud rendering image.

[0024] In another embodiment, a matching score between the first support set and the second support set is calculated. This matching score can be calculated based on a variety of methods, such as calculating the sum of the absolute values ​​of the differences in the supports of the corresponding candidate obstacles in the two sets, and then performing normalization to obtain the matching score. Assuming that the supports of the three candidate obstacles in the first support set are [0.3, 0.5, 0.2], and the supports of the corresponding candidate obstacles in the second support set are [0.4, 0.4, 0.3], then the sum of the absolute values ​​of the differences is |0.3-0.4|+|0.5-0.4|+|0.2-0.3|=0.3, and after normalization (assuming that it is divided by the maximum possible sum of differences, such as 3), a matching score of 0.1 is obtained. Then, the first difference of the second support set is determined based on this matching score. Assuming that a certain mapping relationship (such as a pre-set functional relationship) is used here, the first difference is 0.05 according to the matching score of 0.1.

[0025] Next, the computer system calculates the difference between the second support set and the first difference to obtain the first optimization vector relative to the first support set. For example, the second support set is [0.4, 0.4, 0.3], and the first difference is 0.05, then the first optimization vector is [0.4-0.05, 0.4-0.05, 0.3-0.05] = [0.35, 0.35, 0.25]. Afterwards, the computer system optimizes the first support set using the first optimization vector to obtain the first target support set. Assuming that the first support set is [0.3, 0.5, 0.2] and the first optimization vector is [0.35, 0.35, 0.25], then the first target support set is [0.3+0.35, 0.5+0.35, 0.2+0.25] = [0.65, 0.85, 0.45].

[0026] Finally, the computer system determines the target obstacle according to the first target support set. For example, if a threshold is set to 0.5, then the candidate obstacles with support greater than 0.5 in the first target support set are determined as target obstacles, and the candidate obstacles corresponding to "0.65" and "0.85" are determined as target obstacles.

[0027] In step S500, the preset obstacle type set is a set of all possible obstacle types predefined by the computer system, including, for example, types such as "ship", "personnel", "cargo", and "terminal facilities". The computer system inputs the point cloud rendering image portion containing the target obstacle into the visual transformer algorithm (ViT) for the determined target obstacle. The ViT algorithm analyzes the characteristics of the target obstacle, determines which type it belongs to in the preset obstacle type set, and calculates a support for each type. For example, for an object determined to be a target obstacle, the ViT algorithm may determine that its support for the "personnel" type is 0.3, its support for the "cargo" type is 0.2, its support for the "terminal facility" type is 0.1, and so on, and these support constitute the third support set.

[0028] The computer system also inputs the portion of the point cloud rendering image containing the target obstacle into the selected obstacle type prediction algorithm. This algorithm predicts the type of the target obstacle based on its own model structure and algorithm logic, and calculates the support for each type. For example, an obstacle type prediction algorithm may determine that the target obstacle belongs to the "personnel" type with a support of 0.4, the "cargo" type with a support of 0.1, the "terminal facility" type with a support of 0.05, etc. These support constitute the fourth support set.

[0029] In step S600, as an implementation method, after obtaining the third support set and the fourth support set, the computer system will perform comprehensive processing on them. For example, for the obstacle type "personnel", the support in the third support set is 0.3, and the support in the fourth support set is 0.4. Weighted average or other fusion strategies can be used to determine the final type judgment. If the weight of the visual transformer algorithm (ViT) is set to 0.3 and the weight of the obstacle type prediction algorithm is set to 0.7, then the comprehensive support of the "personnel" type is 0.3*0.3+0.7*0.4=0.37. By performing such calculations on all types in the preset obstacle type set, the computer system will select the type with the highest comprehensive support as the obstacle type of the target obstacle.

[0030] In another embodiment, the computer system calculates the matching score between the third support set and the fourth support set in a manner similar to step S400, and determines the difference of the fourth support set (similar to the first difference concept in step S400). At the same time, other related support sets (if any) may also be considered. For example, in an extended embodiment, there may be a fifth support set and a sixth support set, etc. Then, the third support set is optimized according to these differences and other support sets to obtain an optimized support set. For example, by adjusting the value of the support, it is similar to the optimization vector calculation and application process in step S400. Finally, the obstacle type of the target obstacle is determined according to the optimized support set, that is, the obstacle type with the highest support is selected as the final result.

[0031] In step S700, after the computer system determines the target obstacles and their respective obstacle types, it can obtain the obstacle annotation results of the point cloud rendering image. For example, if there are two target obstacles in the point cloud rendering image, one is determined to be of the "personnel" type and the other is determined to be of the "cargo" type, then the obstacle annotation results will clearly indicate what types of obstacles the two objects in the image are.

[0032] During the operation of the chain bucket ship unloader, the operation path planning needs to avoid these obstacles. For example, if there is an obstacle marked as "cargo" type in front of the ship unloader, the computer system will avoid the collision between the chain bucket of the ship unloader and the cargo when planning the operation path. If there are personnel in the operating area (obstacles marked as "personnel" type), the operation path also needs to be adjusted to ensure the safety of the personnel. In this way, the obstacle marking results provide an important decision-making basis for the intelligent operation of the chain bucket ship unloader, enabling the ship unloader to operate safely and efficiently in a complex operating environment.

[0033] In one embodiment, the method further comprises:

[0034] Step S101: combining the point cloud rendering image, calling the obstacle type prediction algorithm to perform obstacle detection on the point cloud rendering image, and obtaining a fifth support set relative to the preset obstacle set;

[0035] Step S102: combining the point cloud rendering image and the target obstacle, calling an obstacle detection algorithm to predict the obstacle type of the target obstacle, and obtaining a sixth support set relative to the preset obstacle type set;

[0036] In this case, step S400, combining the first support set and the second support set to obtain the target obstacle in the point cloud rendering image, includes: step S400A: combining the first support set, the second support set and the fifth support set to obtain the target obstacle;

[0037] Step S600, combining the third support set and the fourth support set to obtain the obstacle type of the target obstacle, includes: Step S600A: combining the third support set, the fourth support set and the sixth support set to obtain the obstacle type corresponding to the target obstacle.

[0038] In step S101 of the embodiment of the present application, the preset obstacle set is still a set of obstacle types that may appear in the operation area of ​​the chain bucket unloader that is preset by the computer system. The obstacle type prediction algorithm is used here for obstacle detection, which is different from the conventional one that is only used for type prediction, but from the perspective of the algorithm function, it can still give a probability measure of each candidate obstacle existing in the image based on the image features, that is, the support. For example, assume that the preset obstacle set includes "ship", "personnel", "cargo pile", "dock building structure", etc. The computer system inputs the point cloud rendering image into the obstacle type prediction algorithm, and the algorithm analyzes the image. For the candidate obstacle "ship", the algorithm may calculate its support as 0.4 based on certain features in the image (such as the shape, color distribution, etc., and the degree of matching with the ship features); for "personnel", the support may be calculated as 0.3 due to factors such as some suspected human outlines in the image. By analogy, a support will be obtained for each candidate obstacle in the preset obstacle set, and these support degrees together constitute the fifth support set. This method can use the obstacle type prediction algorithm to evaluate the possibility of the existence of obstacles in the image from another perspective, providing more information for subsequent more accurate determination of the target obstacle.

[0039] Next is step S102, where the preset obstacle type set is a set of all possible obstacle types previously defined. After the target obstacle has been determined, the computer system uses an obstacle detection algorithm to predict the type of the target obstacle. The obstacle detection algorithm is used to predict the type of the target obstacle. For example, the target obstacle is a specific object in the image, and the computer system inputs the image portion containing the target obstacle into the obstacle detection algorithm. Assume that the preset obstacle type set includes types such as "movable device", "fixed facility", and "bulk cargo". For this target obstacle, the obstacle detection algorithm may judge that it belongs to the "movable device" type based on its shape, texture and other features, and the support degree is 0.5, the support degree of the "fixed facility" type is 0.2, and the support degree of the "bulk cargo" type is 0.1, etc. These supports calculated for each preset obstacle type constitute the sixth support set. This step provides information from another perspective of the obstacle detection algorithm for the type judgment of the target obstacle, which helps to improve the accuracy of type prediction.

[0040] In step S400A, the computer system needs to integrate the information of the three support sets to determine the target obstacle. To illustrate with a specific example, assume that the preset obstacle set has three candidate obstacles: A, B, and C. In the first support set, the support of candidate obstacle A is 0.3, B is 0.4, and C is 0.2; in the second support set, the support of A is 0.4, B is 0.3, and C is 0.3; in the fifth support set, the support of A is 0.2, B is 0.5, and C is 0.1. The computer system can adopt a comprehensive calculation method, for example, first perform a weighted summation of the support of each candidate obstacle in the three support sets. Assume that the weights of the first support set, the second support set, and the fifth support set are set to 0.3, 0.5, and 0.2, respectively. For candidate obstacle A, the comprehensive support is calculated as: 0.3*0.3+0.5*0.4+0.2*0.2=0.33; for B: 0.3*0.4+0.5*0.3+0.2*0.5=0.37; for C: 0.3*0.2+0.5*0.3+0.2*0.1=0.23. Then, the computer system sets a threshold, assuming it is 0.3. Since B's comprehensive support of 0.37 is greater than the threshold of 0.3, B is determined to be the target obstacle. This method of combining multiple support sets can more comprehensively consider the results of different algorithms for target obstacle detection, avoid the limitations that may exist in a single algorithm, and thus improve the accuracy of target obstacle determination.

[0041] In step S600A, the computer system integrates three support sets to determine the type of the target obstacle. For example, assume that the preset obstacle type set includes three types: "large equipment", "small equipment", and "materials". For a certain target obstacle, in the third support set, the support of the "large equipment" type is 0.3, "small equipment" is 0.2, and "materials" is 0.1; in the fourth support set, the support of "large equipment" is 0.4, "small equipment" is 0.3, and "materials" is 0.1; in the sixth support set, the support of "large equipment" is 0.2, "small equipment" is 0.5, and "materials" is 0.05. The computer system can also use a weighted summation method to integrate the three support sets. Assume that the weights of the third support set, the fourth support set, and the sixth support set are set to 0.3, 0.5, and 0.2, respectively. For the "large equipment" type, the comprehensive support is calculated as: 0.3*0.3+0.5*0.4+0.2*0.2=0.33; for "small equipment": 0.3*0.2+0.5*0.3+0.2*0.5=0.31; for "material": 0.3*0.1+0.5*0.1+0.2*0.05=0.09. Then, the computer system selects the type with the highest comprehensive support. Here, the comprehensive support of "large equipment" is the highest, so this target obstacle is determined to be a "large equipment" type. By integrating multiple support sets, the advantages of different algorithms in obstacle type prediction can be fully utilized to improve the accuracy of type determination, thereby providing a more reliable basis for operations such as the operation path planning of the chain bucket unloader, ensuring that obstacles and their types can be accurately identified in complex operating environments, and ensuring the safety and efficiency of operations.

[0042] As another implementation of step S400, that is, combining the first support set and the second support set to obtain the target obstacle in the point cloud rendering image, includes:

[0043] Step S410: Determine a first difference of the second support set by combining the matching scores between the first support set and the second support set;

[0044] Step S420: combining the difference between the second support set and the first difference to obtain a first optimization vector relative to the first support set;

[0045] Step S430: Optimizing the first support set in combination with the first optimization vector to obtain a first target support set;

[0046] Step S440: Combine the first target support set to obtain the target obstacle.

[0047] In step S410 of the embodiment of the present application, the first support set and the second support set are sets of metrics about the possibility of existence of each candidate obstacle obtained by the computer system after detecting the candidate obstacles in the preset obstacle set through different algorithms (visual transformer algorithm and obstacle detection algorithm). The matching score is used to measure the similarity or consistency between the two sets. There are many ways to calculate the matching score. For example, for each candidate obstacle in the preset obstacle set, the absolute value of the difference in support between the first support set and the second support set is calculated, and then the absolute values ​​of these differences are summed up, and then the matching score is obtained after normalization.

[0048] Assume that there are three candidate obstacles in the preset obstacle set: obstacle A, obstacle B, and obstacle C. The support of these three candidate obstacles in the first support set is [0.3, 0.5, 0.2], and the corresponding support in the second support set is [0.4, 0.4, 0.3]. First, the absolute value of the difference in support of each candidate obstacle is calculated. For obstacle A, it is |0.3-0.4|=0.1, for obstacle B, it is |0.5-0.4|=0.1, and for obstacle C, it is |0.2-0.3|=0.1. Then the absolute values ​​of these differences are summed to get 0.3. Assuming that the normalization is divided by the sum of the maximum possible absolute values ​​of the difference in support of all candidate obstacles in the two sets (here it is assumed to be 1, because the value range of each support is 0 to 1), then the matching score is 0.3.

[0049] After obtaining the matching score, the first difference of the second support set is determined according to the matching score. The determination of the first difference can be based on a pre-set functional relationship or mapping rule. For example, it may be set that when the matching score is between 0 and 0.2, the first difference is 0.05; when the matching score is between 0.2 and 0.4, the first difference is 0.1; when the matching score is between 0.4 and 0.6, the first difference is 0.15, and so on. In the above example, the matching score of 0.3 is between 0.2 and 0.4, so the first difference is determined to be 0.1.

[0050] Next is step S420. In the previous example, the second support set is [0.4, 0.4, 0.3], and the first difference is 0.1. Calculate the difference between the second support set and the first difference, and calculate for each element separately to obtain [0.4-0.1, 0.4-0.1, 0.3-0.1] = [0.3, 0.3, 0.2]. This result is the first optimization vector relative to the first support set. The significance of this first optimization vector is that it adjusts the second support set according to the relationship between the first support set and the second support set, so that it can be better integrated or compared with the first support set later.

[0051] Then comes step S430. The first support set was previously [0.3, 0.5, 0.2], and the first optimization vector is [0.3, 0.3, 0.2]. There are many ways to optimize the first support set. Here, we assume that a simple method of adding corresponding elements is used. Then the optimized first target support set is [0.3+0.3, 0.5+0.3, 0.2+0.2]=[0.6, 0.8, 0.4]. This optimization method integrates the information in the first optimization vector into the first support set to obtain a new support set. This new set combines the original information of the first support set and the information adjusted based on the relationship with the second support set.

[0052] Finally, step S440 is performed, where the computer system determines the target obstacle based on the first target support set. This usually involves setting a threshold to determine which candidate obstacles are target obstacles. For example, the threshold is set to 0.5. In the first target support set [0.6, 0.8, 0.4], candidate obstacles with a support greater than 0.5 are determined to be target obstacles. Here, the support of obstacle A (0.6) and the support of obstacle B (0.8) are both greater than 0.5, so obstacles A and B are determined to be target obstacles. Through such a series of steps, the computer system starts from the first support set and the second support set, and finally determines the target obstacle after matching scores, difference calculations, optimization vector generation, and optimization of the first support set. This method helps to improve the accuracy of target obstacle determination because it fully considers the relationship between the support sets obtained by two different algorithms and makes reasonable adjustments and integrations thereof.

[0053] In one implementation, step S400A, combining the first support set, the second support set, and the fifth support set to obtain a target obstacle, includes:

[0054] Step S400A1: Determine a first difference of the second support set by combining the matching scores between the first support set and the second support set;

[0055] Step S400A2: Determine a second difference of the fifth support set by combining the matching scores between the first support set and the fifth support set;

[0056] Step S400A3: Determine a first optimization vector for the first support set by combining the difference between the second support set and the first difference;

[0057] Step S400A4: Optimizing the first support set in combination with the first optimization vector;

[0058] Step S400A5: obtaining a second target support set based on the difference between the optimized first support set and the second difference;

[0059] Step S400A6: Combine the second target support set to obtain the target obstacle.

[0060] In step S400A1 of the embodiment of the present application, the first support set and the second support set are respectively sets of metrics about the possibility of existence of each candidate obstacle obtained by the computer system after detecting the candidate obstacles in the preset obstacle set through different algorithms (such as the visual transformer algorithm and the obstacle detection algorithm). The matching score is used to measure the similarity or consistency between the two sets.

[0061] Assume that there are four candidate obstacles in the preset obstacle set: obstacle 1, obstacle 2, obstacle 3, and obstacle 4. The support of these four candidate obstacles in the first support set is [0.2, 0.3, 0.4, 0.1], and the corresponding support in the second support set is [0.3, 0.25, 0.35, 0.15].

[0062] One way to calculate the matching score is to calculate the absolute value of the support difference between each candidate obstacle in the first support set and the second support set, then sum the absolute values ​​of these differences, and then obtain the matching score after normalization.

[0063] For obstacle 1, the absolute value of the support difference is |0.2-0.3|=0.1; for obstacle 2, it is |0.3-0.25|=0.05; for obstacle 3, it is |0.35-0.4|=0.05; for obstacle 4, it is |0.15-0.1|=0.05. The absolute values ​​of these differences are summed to get 0.25.

[0064] Assuming that normalization is to divide this sum by the maximum possible sum of the absolute values ​​of the support differences of all candidate obstacles in the two sets (here assumed to be 1, because the value range of each support is 0 to 1), then the resulting matching score is 0.25.

[0065] After obtaining the matching score, the first difference of the second support set is determined according to the matching score. This determination process can be based on a pre-set functional relationship or mapping rule. For example, when the matching score is between 0 and 0.2, the first difference is set to 0.03; when the matching score is between 0.2 and 0.4, the first difference is set to 0.05; when the matching score is between 0.4 and 0.6, the first difference is set to 0.08, and so on. In the above example, the matching score of 0.25 is between 0.2 and 0.4, so the first difference is determined to be 0.05.

[0066] Next is step S400A2, the fifth support set is the support set obtained by the computer system through the obstacle type prediction algorithm to detect the preset obstacle set. The matching score between the first support set and the fifth support set is calculated in the same way as the matching score between the first support set and the second support set. Assume that the support for obstacle 1, obstacle 2, obstacle 3 and obstacle 4 in the fifth support set are [0.15, 0.35, 0.2, 0.3] respectively. For obstacle 1, the support in the first support set is 0.2, and the support in the fifth support set is 0.15. The absolute value of the support difference is |0.2-0.15|=0.05; for obstacle 2, it is |0.3-0.35|=0.05; for obstacle 3, it is |0.4-0.2|=0.2; for obstacle 4, it is |0.1-0.3|=0.2. The absolute values ​​of these differences are summed to get 0.5. Assume that after normalization (also divided by 1, here it is assumed that the sum of the maximum possible differences) is obtained, the matching score is 0.5. The second difference of the fifth support set is determined based on this matching score. Assume that when the matching score is between 0.4 and 0.6, the second difference is 0.08 (according to the pre-set mapping rule), so the second difference here is 0.08.

[0067] Then, step S400A is performed, the second support set is [0.3, 0.25, 0.35, 0.15], the first difference is 0.05, and the difference between the second support set and the first difference is calculated for each candidate obstacle.

[0068] For obstacle 1, the difference is 0.3-0.05=0.25; for obstacle 2, it is 0.25-0.05=0.2; for obstacle 3, it is 0.35-0.05=0.3; for obstacle 4, it is 0.15-0.05=0.1. The vector [0.25,0.2,0.3,0.1] composed of these differences is the first optimization vector for the first support set. The significance of this first optimization vector is that it is a kind of guiding information for adjusting the first support set according to the relationship between the second support set and the first difference.

[0069] Next is step S400A4, the first support set is [0.2, 0.3, 0.4, 0.1], and the first optimization vector is [0.25, 0.2, 0.3, 0.1]. There are many ways to optimize the first support set, and here it is assumed that a simple method of adding corresponding elements is used.

[0070] The optimized first support set is [0.2+0.25, 0.3+0.2, 0.4+0.3, 0.1+0.1]=[0.45, 0.5, 0.7, 0.2]. This optimization method integrates the information in the first optimized vector into the first support set to obtain a new support set. This new set combines the original information of the first support set and the information adjusted based on the relationship with the second support set.

[0071] Then, step S400A5 is performed, the optimized first support set is [0.45, 0.5, 0.7, 0.2], and the second difference is 0.08. The difference between the optimized first support set and the second difference is calculated for each candidate obstacle.

[0072] For obstacle 1, the difference is 0.45-0.08=0.37; for obstacle 2, it is 0.5-0.08=0.42; for obstacle 3, it is 0.7-0.08=0.62; for obstacle 4, it is 0.2-0.08=0.12. The set of these differences [0.37, 0.42, 0.62, 0.12] is the second target support set. This second target support set further integrates the adjustment information related to the optimized first support set and the fifth support set.

[0073] Finally, in step S400A6, the computer system determines the target obstacle according to the second target support set. This usually involves setting a threshold to determine which candidate obstacles are target obstacles. For example, the threshold is set to 0.4.

[0074] In the second target support set [0.37, 0.42, 0.62, 0.12], candidate obstacles with support greater than 0.4 are determined to be target obstacles. Here, the support of obstacle 2 0.42 and the support of obstacle 3 0.62 are both greater than 0.4, so obstacles 2 and obstacle 3 are determined to be target obstacles.

[0075] In one embodiment, the process of obtaining the first support set and the second support set includes the following steps:

[0076] Step S201: combining the point cloud rendering image, calling the visual transformer algorithm to perform obstacle detection on the point cloud rendering image, obtaining a first temporary support set relative to the preset obstacle set, calling the obstacle detection algorithm to perform obstacle detection on the point cloud rendering image, and obtaining a second temporary support set relative to the preset obstacle set; wherein the first temporary support set includes the first temporary support of each candidate obstacle, and the second temporary support includes the second temporary support of each candidate obstacle;

[0077] Step S202: combining the first temporary support set and the second temporary support set, selecting target candidate obstacles whose first temporary support and second temporary support are not less than the support threshold from each candidate obstacle;

[0078] Step S203: filtering out the support of candidate obstacles that are not target candidate obstacles in the first temporary support set and the second temporary support set, respectively, to obtain a first support set and a second support set.

[0079] In step S201 of the embodiment of the present application, in combination with the point cloud rendering image, the visual transformer algorithm is called to perform obstacle detection on the point cloud rendering image to obtain a first temporary support set relative to a preset obstacle set, and the obstacle detection algorithm is called to perform obstacle detection on the point cloud rendering image to obtain a second temporary support set relative to the preset obstacle set.

[0080] The preset obstacle set is a set of obstacle types that may appear in the operation area of ​​the chain bucket ship unloader, which is preset by the computer system, including, for example, ships, personnel, cargo piles, dock facilities, etc.

[0081] The Visual Transformer (ViT) algorithm is a visual processing algorithm. The computer system inputs the point cloud rendering image into the Visual Transformer algorithm, which analyzes and processes the image to detect obstacles in the preset obstacle set and calculates a support for each candidate obstacle. These supports constitute the first temporary support set. Assume that there are three candidate obstacles in the preset obstacle set: ships, people, and cargo piles. After the computer system inputs the point cloud rendering image into the Visual Transformer algorithm, for the candidate obstacle of the ship, the algorithm calculates its support as 0.3 based on certain features in the image (such as shape, color, texture, etc. that may be related to the ship), indicating that the algorithm believes that there is a 30% probability that there is an obstacle such as a ship in the image; for the person, the support is calculated as 0.4 based on the features such as the outline and color distribution of the person in the image; for the cargo pile, the support is calculated as 0.2 based on the features such as the shape and height of the cargo pile. In this way, the first temporary support set [0.3, 0.4, 0.2] is obtained.

[0082] At the same time, the computer system inputs the point cloud rendering image into the selected obstacle detection algorithm (such as YOLO, etc.). This algorithm will also detect obstacles in the preset obstacle set and calculate the support for each candidate obstacle to obtain a second temporary support set.

[0083] Assume that for the same three candidate obstacles, the obstacle detection algorithm calculates a support of 0.25 for the ship, because it determines that the possibility of the ship in the image is 25% based on its own detection mechanism (for example, by dividing the image into grids and analyzing the features in each grid, etc.); the support calculated for the personnel is 0.35; and the support calculated for the cargo pile is 0.15. In this way, the second temporary support set [0.25, 0.35, 0.15] is obtained.

[0084] The supports in the first temporary support set and the second temporary support set are only preliminary judgment results based on their respective algorithms, and further processing is needed to improve accuracy and reliability.

[0085] Next is step S202: combining the first temporary support set and the second temporary support set, screening target candidate obstacles whose first temporary support and second temporary support are not less than a support threshold from among the candidate obstacles.

[0086] The support threshold is a standard value pre-set by the computer system and is used to screen candidate objects with a high probability of being obstacles. For example, the support threshold is set to 0.2.

[0087] For the first temporary support set [0.3, 0.4, 0.2] and the second temporary support set [0.25, 0.35, 0.15] obtained above, the computer system will check each candidate obstacle one by one.

[0088] For the candidate obstacle of the ship, the first temporary support of 0.3 is greater than 0.2, and the second temporary support of 0.25 is also greater than 0.2, so the ship is a target candidate obstacle; for the personnel, the first temporary support of 0.4 is greater than 0.2, and the second temporary support of 0.35 is greater than 0.2, so the personnel is also a target candidate obstacle; and for the cargo pile, although the first temporary support of 0.2 is greater than 0.2, the second temporary support of 0.15 is less than 0.2, so the cargo pile is not a target candidate obstacle. In this way, the computer system screens out those candidate obstacles that have a high probability of being obstacles in the preliminary detection results of both algorithms, so that some objects that may be misjudged as obstacles in a single algorithm but have low support in another algorithm can be excluded, thereby improving the accuracy of subsequent determination of target obstacles.

[0089] Finally, step S203 is performed: the support of candidate obstacles that are not target candidate obstacles in the first temporary support set and the second temporary support set are filtered out respectively to obtain the first support set and the second support set.

[0090] In the previous example, the cargo pile is not a candidate obstacle. For the first temporary support set [0.3, 0.4, 0.2], after filtering out the support 0.2 corresponding to the cargo pile, we get the first support set [0.3, 0.4]; for the second temporary support set [0.25, 0.35, 0.15], after filtering out the support 0.15 corresponding to the cargo pile, we get the second support set [0.25, 0.35].

[0091] The first support set and the second support set obtained in this way are the results of preliminary screening and optimization. They only contain the support information of candidate obstacles that have a high probability of being obstacles in the preliminary detection of the two algorithms. This processing method helps to reduce the amount of data for subsequent processing, while improving the accuracy and reliability of the support set, laying the foundation for further accurate determination of target obstacles.

[0092] In one implementation, the point cloud rendering image includes a plurality of target obstacles, and the obstacle annotation result includes a plurality of target obstacles and obstacle types of the target obstacles; the obstacle annotation result is obtained by taking t processing stages, each processing stage is obstacle detection processing or obstacle type prediction processing, and the processing result of each processing stage is the target obstacle or obstacle type; the sth processing stage includes the following processes:

[0093] Step S10: Based on the processing result of the s-1th processing stage, determining the result classification of the sth processing result, the result classification is obstacle detection or obstacle type prediction, and the result classification corresponding to the first processing stage is obstacle detection;

[0094] Step S20: combining the point cloud rendering image and the processing results of the previous s-1 rounds of processing stage, calling the visual transformer algorithm to obtain a public support set, calling the obstacle detection algorithm to obtain an obstacle detection support set, calling the obstacle type prediction algorithm to obtain an obstacle type prediction support set; the public support set is the first support set or the third support set, the obstacle detection support set is the second support set or the sixth support set, and the obstacle type prediction support set is the fourth support set or the fifth support set;

[0095] Step S30: combining the public support set, the obstacle detection support set and the obstacle type prediction support set to obtain the processing result of the sth processing stage.

[0096] In step S10 of the embodiment of the present application, based on the processing results of the s-1th processing stage, the result classification of the sth processing result is determined, and the result classification is obstacle detection or obstacle type prediction, and the result classification corresponding to the first processing stage is obstacle detection. During the entire processing process, the computer system will perform multiple processing in sequence. Each processing is associated with the result of the previous processing. For the first processing stage, since there is no reference to the previous processing result, its result classification is pre-set to obstacle detection. This means that when the computer system processes for the first time, the main task is to detect whether there are obstacles in the image.

[0097] For example, in the operation scenario of a bucket chain unloader, when the computer system first processes the point cloud rendering image, the focus is on finding various objects that may exist in the image, which may or may not be obstacles. For example, if the image detects the outline of some objects that are shaped like ships, people, or cargo piles, this is the task of obstacle detection in the first processing stage.

[0098] As the processing progresses, when entering the subsequent processing rounds (s>1), the computer system will determine the classification of the results of the current round (s round) based on the processing results of the previous round (s-1 round). If the previous processing result is the detection of some objects suspected of being obstacles, then the classification of the processing results of the current round may be to predict the types of these suspected obstacles; conversely, if the previous round predicted the types of some objects and found that there are some objects that have not been accurately classified or need further confirmation, then the classification of the processing results of the current round may return to obstacle detection to re-evaluate whether these objects are really obstacles or whether there are other obstacles that have not been detected.

[0099] Next comes step S20. In each processing, the computer system uses the results of the previous processing stage and the information of the point cloud rendering image itself to call different algorithms to obtain different support sets.

[0100] To illustrate with a specific example, suppose that in a certain round of processing (s>1), the previous round of processing results detected three suspected obstacle objects A, B, and C. The computer system will combine the point cloud rendering image part containing these three objects and the previous round of detection results to perform the following operations.

[0101] When the visual transformer algorithm is called, it will analyze the current content to be processed (which may be obstacle detection or type prediction, depending on the current result classification) based on image features and previous detection information. If obstacle detection is currently being performed, it will calculate a support for each candidate obstacle in the preset obstacle set, and these supports constitute a public support set (similar to the concept of the first support set). For example, for candidate obstacles such as ships, personnel, and cargo piles in the preset obstacle set, the supports are calculated to be [0.3, 0.4, 0.2] respectively, and this set is the public support set. If obstacle type prediction is currently being performed, then this public support set is similar to the third support set, which is a support set calculated for each type in the preset obstacle type set.

[0102] At the same time, the computer system will call the obstacle detection algorithm. If the current result classification is obstacle detection, this algorithm will calculate the support for the candidate obstacles in the preset obstacle set to form an obstacle detection support set (similar to the second support set). For example, for the same candidate obstacles such as ships, personnel, and cargo piles, the calculated support is [0.25, 0.35, 0.15]. If the current result classification is obstacle type prediction, this obstacle detection algorithm will calculate the support for the types in the preset obstacle type set in a special way (based on its own algorithm structure and logic, used for type prediction functions), forming a result similar to the sixth support set.

[0103] In addition, the computer system will also call the obstacle type prediction algorithm. If the current result classification is obstacle type prediction, it will calculate the support for each type in the preset obstacle type set to form an obstacle type prediction support set (similar to the fourth support set). For example, for movable equipment, fixed facilities, bulk cargo and other types in the preset obstacle type set, the calculated support is [0.3, 0.2, 0.1]. If the current result classification is obstacle detection, this algorithm will calculate the support for the candidate obstacles in the preset obstacle set in a special way (for obstacle detection functions), forming a result similar to the fifth support set.

[0104] Finally, in step S30, after obtaining the three different support sets, the computer system needs to perform comprehensive processing on them to obtain the processing result of the current answer (s answer).

[0105] Continuing with the above example, if the current result classification is obstacle detection, the public support set is [0.3, 0.4, 0.2], the obstacle detection support set is [0.25, 0.35, 0.15], and it is assumed that there is also a special support set for detection obtained from the obstacle type prediction algorithm (similar to the fifth support set) of [0.2, 0.3, 0.1]. The computer system can use a variety of methods to combine these support sets. One method is weighted summation. Assume that the weights of the public support set, obstacle detection support set, and special fifth support set are set to 0.4, 0.4, and 0.2, respectively. For candidate obstacle ships, the comprehensive support is calculated as: 0.4*0.3+0.4*0.25+0.2*0.2=0.26; for personnel: 0.4*0.4+0.4*0.35+0.2*0.3=0.34; for cargo piles: 0.4*0.2+0.4*0.15+0.2*0.1=0.14. Then, the computer system can make a judgment based on a set threshold (for example, 0.2). Since the comprehensive support of personnel 0.34 is greater than 0.2, personnel are determined as obstacles in the current back processing.

[0106] If the current result classification is obstacle type prediction. The public support set (similar to the third support set) is [0.3, 0.2, 0.1], the obstacle type prediction support set is [0.3, 0.2, 0.1], and it is assumed that there is also a special support set for type prediction obtained from the obstacle detection algorithm (similar to the sixth support set) of [0.25, 0.15, 0.1]. The weighted summation method is also adopted, assuming that the weights are 0.3, 0.4 and 0.3 respectively. For movable equipment in the preset obstacle type, the comprehensive support is calculated as: 0.3*0.3+0.4*0.3+0.3*0.25=0.295; for fixed facilities: 0.3*0.2+0.4*0.2+0.3*0.15=0.185; for bulk cargo: 0.3*0.1+0.4*0.1+0.3*0.1=0.1. Then, the computer system will select the type with the highest comprehensive support. Here, the comprehensive support of the movable device is the highest, so for an object detected as an obstacle, its type is determined to be a movable device.

[0107] In one embodiment, the obstacle detection algorithm is optimized by combining the matching scores between the first estimated obstacles and the real obstacle labels of each first template image; the obstacle type prediction algorithm is optimized by combining the matching scores between the second estimated obstacle types and the real obstacle type labels of each second template image; the visual transformer algorithm is optimized by combining the matching scores between the third obstacle labeling results and the real obstacle labeling results of each third template image, the third obstacle labeling results include the third estimated obstacles and the third obstacle type, and the real obstacle labeling results include the real obstacle label and the real obstacle type label of the third template image.

[0108] Specifically, in one implementation, the optimization process of the obstacle detection algorithm includes:

[0109] Step S11: obtaining a first image template set, the first image template set comprising a plurality of first template images containing real obstacle annotation results, the real obstacle annotation result of each first template image comprising a real obstacle label of the first template image;

[0110] Step S12: Based on each first template image, calling the obstacle detection algorithm to be optimized to perform obstacle detection on each first template image, and obtaining a first estimated obstacle for each first template image;

[0111] Step S13: combining each first template image and the first estimated obstacle, calling the obstacle detection algorithm to be optimized to predict the obstacle type of the first estimated obstacle, obtaining the first estimated obstacle type of the first estimated obstacle, and combining the first estimated obstacle and the first estimated obstacle type of each first template image to obtain the obstacle labeling result of each first template image;

[0112] Step S14: determining a first optimization cost value based on the matching score between the first estimated obstacle and the real obstacle label of each first template image, and optimizing the algorithm parameters of the obstacle detection algorithm to be optimized in combination with the first optimization cost value.

[0113] In step S11 of the embodiment of the present application, a first image template set is obtained, where the first image template set includes multiple first template images containing real obstacle annotation results, and the real obstacle annotation result of each first template image includes the real obstacle label of the first template image.

[0114] In the operating environment of the chain bucket unloader, in order to optimize the obstacle detection algorithm, the computer system needs a set of images with accurate annotation information as a reference, which is the first image template set. These template images cover various possible scenes and obstacle situations. For example, in the dock scene, the images in the first image template set may include images taken from different angles of ships docking, cargo stacking, people walking, etc. Each first template image comes with a real obstacle annotation result, where the real obstacle label clearly indicates what the obstacle is in the image. For example, if there is a ship, two people and a pile of cargo in an image, then the real obstacle label of this image will accurately mark the three obstacle categories of ships, people and cargo. Such a first image template set provides accurate benchmark data for subsequent evaluation and optimization of obstacle detection algorithms.

[0115] Next is step S12, where the computer system uses the obstacle detection algorithm to be optimized to detect each template image in the first image template set. The algorithm analyzes the image according to its own algorithm logic and model structure (which can refer to the existing feasible neural network structure), tries to find obstacles in the image, and uses the detected result as the first estimated obstacle.

[0116] Then comes step S13. After obtaining the first estimated obstacles, the computer system calls the obstacle detection algorithm again (because some obstacle detection algorithms have certain type prediction capabilities, or can be used for type prediction through certain adjustments) to perform type prediction on these first estimated obstacles.

[0117] Finally, in step S14, in order to quantify the performance of the current obstacle detection algorithm, the computer system needs to calculate the matching score between the first estimated obstacle and the real obstacle label of each first template image. There are many ways to calculate the matching score, for example, an indicator such as the intersection over union (IoU) between the first estimated obstacle predicted and the real obstacle label can be used.

[0118] After such matching score calculation is performed on all template images in the first image template set, these scores are combined (for example, by averaging) to obtain an overall matching score. This overall matching score can be converted into a first optimization cost value. For example, a mapping relationship can be set. When the matching score is low, the first optimization cost value is large, indicating that the algorithm performance is poor and more optimization adjustments are required. Then, the computer system optimizes the algorithm parameters of the obstacle detection algorithm that needs to be optimized according to the first optimization cost value. Different obstacle detection algorithms have different parameters, such as the weight and threshold of the neural network structure. If the first optimization cost value is large, the computer system may use a larger adjustment range to update the algorithm parameters. For example, if the algorithm is based on a neural network, the back propagation algorithm may be used in combination with the first optimization cost value to adjust the weight of the neural network, so that the algorithm can detect obstacles more accurately in subsequent detections, reduce the difference between the real obstacle labels, and thus improve the accuracy and reliability of the obstacle detection algorithm in the chain bucket unloader operation scenario.

[0119] Accordingly, in one embodiment, the optimization process of the obstacle type prediction algorithm includes:

[0120] Step S21: obtaining a second image template set, where the second image template set includes a plurality of second template images with real obstacle labeling results, where the real obstacle labeling result of each second template image includes a real obstacle type label of each real obstacle label of the second template image;

[0121] Step S22: Based on each second template image, calling the obstacle type prediction algorithm to be optimized to perform obstacle detection on each second template image to obtain a second estimated obstacle for each second template image;

[0122] Step S23: combining each second template image and the second estimated obstacle, calling the obstacle type prediction algorithm to be optimized to predict the obstacle type of the second estimated obstacle, obtaining the second estimated obstacle type of the second estimated obstacle, and combining the second estimated obstacle and the second estimated obstacle type of each second template image to obtain the obstacle labeling result of each second template image;

[0123] Step S24: Determine a second optimization cost value based on the matching score between the second estimated obstacle type of each second template image and the real obstacle type label, and optimize the algorithm parameters of the obstacle type prediction algorithm to be optimized in combination with the second optimization cost value.

[0124] In step S21 of the embodiment of the present application, in the operating scenario of the chain bucket unloader, in order to optimize the obstacle type prediction algorithm, the computer system needs a set of images containing accurate annotation information as a reference, which is the second image template set. These template images cover various scenes that may appear and the obstacles therein. For example, in the operating environment of the dock, the images in the second image template set may contain scenes where different objects such as ships, personnel, cargo piles, and dock facilities exist at the same time. Each second template image carries a real obstacle annotation result, in which the real obstacle type label clearly identifies the type of each obstacle in the image. For example, in an image, there is a ship, two personnel and a pile of cargo. Then, for the ship obstacle, its real obstacle type label is "ship"; for the personnel, the type label is "personnel"; for the cargo pile, the type label is "cargo". Such a second image template set provides accurate reference data for subsequent evaluation and optimization of the obstacle type prediction algorithm.

[0125] Next is step S22, where the computer system calls the obstacle type prediction algorithm that needs to be optimized for each template image in the second image template set. This algorithm analyzes the image according to its own logic and model structure to detect obstacles in the image, and these detected obstacles are marked as second estimated obstacles. For example, for a template image containing a ship, a person, and a cargo pile, the obstacle type prediction algorithm may detect obstacles by performing operations such as feature extraction and pattern recognition on the image. However, due to the limitations that the algorithm may have, it may have some problems during the detection process. For example, it may misjudge part of the ship as other objects, or fail to fully detect all the personnel. Ultimately, the second estimated obstacle output by the algorithm may be a partially inaccurately identified ship, a person (one person is omitted), and a suspected other object (a part of the cargo pile is mistakenly identified as other categories). This shows that the current obstacle type prediction algorithm has inaccuracies in the obstacle detection link on this template image and needs to be further optimized.

[0126] Then comes step S23. After obtaining the second estimated obstacles, the computer system calls the obstacle type prediction algorithm again to predict the type of these second estimated obstacles. Continuing with the example mentioned above, for the partially inaccurately identified ship (second estimated obstacle) detected, the algorithm may predict its type as a ship (although there are some inaccuracies in the detection, the type prediction may be correct); for a detected person (second estimated obstacle), its type is predicted as a person; for other suspected objects (misidentified cargo pile parts), it may be predicted as a wrong type. Then, the second estimated obstacles and the corresponding second estimated obstacle types in each second template image are combined to obtain the obstacle labeling result of each second template image. This obstacle labeling result is obtained based on the obstacle type prediction algorithm that currently needs to be optimized, and there are differences with the real obstacle labeling results of each second template image mentioned above, and these differences are the key basis for the optimization algorithm.

[0127] Finally, in step S24, in order to quantify the performance of the current obstacle type prediction algorithm, the computer system needs to calculate the matching score between the second estimated obstacle type and the real obstacle type label of each second template image. The matching score can be calculated in a variety of ways, for example, the proportion of the number of correctly predicted types to the total number of types can be calculated. Assume that in a template image containing a ship, a person, and a cargo pile, the real obstacle type labels are "ship", "person", and "cargo" respectively. The second estimated obstacle types predicted by the algorithm are "ship", "person", and "wrong type". In this example, there are two correctly predicted types (ship and person), and the total number of types is three, so the matching score obtained by preliminary calculation is 2 / 3≈0.67. After performing such matching score calculations on all template images in the second image template set, these scores are combined (for example, by averaging) to obtain an overall matching score. This overall matching score can be converted into a second optimization cost value. For example, a mapping relationship can be set. When the matching score is low, the second optimization cost value is large, indicating that the algorithm performance is poor and more optimization adjustments are required. Then, the computer system optimizes the algorithm parameters of the obstacle type prediction algorithm that needs to be optimized according to the second optimization cost value. Different obstacle type prediction algorithms have different parameters, such as weights and thresholds in neural networks or split thresholds in decision trees. If the second optimization cost value is large, a larger adjustment range is used to update the algorithm parameters. For example, if the algorithm is based on a neural network, the back propagation algorithm can be combined with the second optimization cost value to adjust the weight of the neural network, so that the algorithm can more accurately predict the obstacle type in subsequent type predictions, reduce the difference between the actual obstacle type label, and thus improve the accuracy and reliability of the obstacle type prediction algorithm in the chain bucket unloader operation scenario.

[0128] Accordingly, in one embodiment, the optimization process of the visual transformer algorithm includes:

[0129] Step S31: obtaining a third image template set, the third image template set comprising a plurality of third template images having real obstacle labeling results, the real obstacle labeling result of each third template image comprising a real obstacle label of the third template image and a real obstacle type label of each real obstacle label;

[0130] Step S32: Based on each third template image, calling the visual transformer algorithm to be optimized to perform obstacle detection on each third template image to obtain a third estimated obstacle for each third template image;

[0131] Step S33: combining each third template image and the third estimated obstacle, calling the visual transformer algorithm to be optimized to predict the obstacle type of the third estimated obstacle, obtaining the third estimated obstacle type of the third estimated obstacle, and combining the third estimated obstacle and the third estimated obstacle type of each third template image to obtain the obstacle labeling result of each third template image;

[0132] Step S34: Determine a third optimization cost value based on the matching score between the third obstacle annotation result of each third template image and the real obstacle annotation result, and optimize the algorithm parameters of the visual transformer algorithm to be optimized in combination with the third optimization cost value.

[0133] In step S31 of the embodiment of the present application, in the operation scene of the chain bucket unloader, the computer system needs a set of standard image template sets as references in order to optimize the visual transformer algorithm. This is the third image template set. These template images contain various situations that may occur in the operation environment, such as the docking of ships, the stacking of goods, and the activities of personnel in the dock scene.

[0134] Each third template image has an accurate real obstacle labeling result. The real obstacle label clearly indicates which obstacles exist in the image. For example, it may contain different obstacle identifications such as ships, personnel, cargo piles, etc. The real obstacle type label of each real obstacle label further refines the type information of these obstacles. For example, ships may be marked as specific types such as "cargo ships" or "tankers", personnel may be marked as types such as "dock workers" or "crew members", and cargo piles may be marked as types such as "bulk cargo piles" or "container piles". Such a detailed and accurate third image template set provides a reliable reference for the optimization of the visual transformer algorithm.

[0135] Next is step S32, the computer system calls the visual transformer algorithm to be optimized for processing each template image in the third image template set. The visual transformer algorithm analyzes the image according to its own structure and algorithm principle, tries to find obstacles in the image, and marks the detected result as the third estimated obstacle.

[0136] Then comes step S33. After obtaining the third estimated obstacles, the computer system again calls the visual transformer algorithm to perform type prediction on these third estimated obstacles.

[0137] Finally, in step S34, in order to measure the performance of the visual transformer algorithm, the computer system needs to calculate the matching score between the third obstacle labeling result of each third template image and the real obstacle labeling result. There are many ways to calculate this matching score, for example, it can be calculated by comprehensively considering the accuracy of obstacle detection and the accuracy of type prediction.

[0138] For the calculation of obstacle detection accuracy, the intersection over union (IoU) method can be used (refer to the previous description, which will not be repeated here).

[0139] After such a matching score calculation is performed on all template images in the third image template set, these scores are combined (for example, averaged) to obtain an overall matching score. This overall matching score can be converted into a third optimization cost value. For example, a mapping relationship can be set. When the matching score is low, the third optimization cost value is large, indicating that the algorithm performance is poor and more optimization adjustments are required. Then, the computer system optimizes the algorithm parameters of the visual transformer algorithm that needs to be optimized according to this third optimization cost value. The visual transformer algorithm contains a variety of parameters, such as the number of heads in the transformer structure, the embedding dimension, etc. If the third optimization cost value is large, the computer system may use a larger adjustment range to update these parameters. For example, if it is an optimization method based on gradient descent, the update amount of the parameters will be calculated according to the third optimization cost value, thereby adjusting the relevant parameters in the algorithm, so that the visual transformer algorithm can obtain more accurate results in subsequent obstacle detection and type prediction, reduce the difference between the actual obstacle labeling results, and thereby improve the accuracy and reliability of the visual transformer algorithm in the chain bucket unloader operation scenario.

[0140] In one embodiment, step S200, performing point cloud rendering on the point cloud data to obtain a point cloud rendering image, includes:

[0141] Step S210: obtaining point cloud data to be rendered;

[0142] Step S220: performing feature encoding based on the point cloud data to obtain point cloud encoding tensor sets of multiple scales;

[0143] Step S230: taking a point cloud coding tensor set with the smallest scale among the multiple point cloud coding tensor sets as a first point cloud coding tensor set, obtaining a first rendering result of a first spatial object based on the first point cloud coding tensor set, and obtaining a second rendering result of a second spatial object based on point cloud coding tensor sets other than the first point cloud coding tensor set; the first spatial object includes a target object;

[0144] Step S240: if the first rendering result and the second rendering result meet the preset rendering result, the point cloud data, the first rendering result and the second rendering result are used as components of the point cloud rendering data to obtain the point cloud rendering data.

[0145] In step S210 of the embodiment of the present application, the point cloud data is a set of three-dimensional data collected by specific sensors (such as laser radar, etc.) in the area where the chain bucket unloader is located. These data represent the surface information of objects in the surrounding environment in the form of discrete points. For example, in the dock scene, the point cloud data contains the surface points of the chain bucket unloader's own structure, the outline points of the surrounding ships, the surface points of the cargo pile, and the points of possible personnel and other facilities. The computer system obtains these original point cloud data from the sensor, and these data are the basis for subsequent point cloud rendering. Each point in the point cloud data has three-dimensional coordinates (x, y, z), and may also contain other attribute information, such as reflection intensity, etc., but in this step, the main focus is on its three-dimensional coordinate information. These coordinates determine the position of the point in three-dimensional space, thereby depicting the spatial structure of the entire scene.

[0146] Next is step S220, feature encoding is to convert the information in the point cloud data into a representation form that is more suitable for subsequent processing (such as rendering). The computer system encodes the information of each point through a specific algorithm. To put it in a simple way, it is like converting and combining the three-dimensional coordinates of each point and other related information (if any) according to certain rules. For example, assuming that in a dock scene, for the part of the point cloud data representing the surface of the ship, the computer system may construct features based on the spatial relationship between points (such as the distance and direction of adjacent points) during the feature encoding process. For a certain point, the distribution of its surrounding points may be encoded as a feature vector. This encoding is performed at multiple scales, and different scales can be understood as different resolutions or observation levels.

[0147] At a larger scale, the point cloud data in a larger range is encoded as a whole to obtain a more macroscopic feature representation. This feature representation exists in the form of a tensor (a multidimensional array), forming a larger-scale point cloud encoding tensor. At a smaller scale, more local point cloud data will be focused on, and more detailed features will be encoded, also in the form of tensors, thus obtaining a set of point cloud encoding tensors of multiple scales. The point cloud encoding tensor of each scale contains the feature information of the point cloud data at the corresponding scale, which can be used by the subsequent rendering process.

[0148] Then, in step S230, the first point cloud coding tensor set (i.e., the minimum scale point cloud coding tensor set) contains the most detailed point cloud data feature information. The computer system uses this minimum scale tensor set to render the first spatial object, i.e., the target object. The target object may be an object that needs to be focused on in the operation scene of the chain bucket ship unloader. For example, when performing an operation safety analysis, a person or other small equipment close to the chain bucket ship unloader may be defined as a target object.

[0149] Taking a person as the target object, for example, the feature information about the person in the minimum-scale point cloud encoding tensor set can accurately depict the person's appearance, posture and other details. The computer system renders according to these feature information to obtain the first rendering result of the person (the first spatial object). This rendering result may be presented in the form of an image, in which the shape and position of the person in the image correspond to the person in the actual scene, and because it is based on the minimum-scale tensor set, the rendering result will be more detailed and can reflect the specific characteristics of the person, such as the outline of the clothes worn, the movement of the limbs, etc.

[0150] At the same time, for the second spatial object, the computer system uses other point cloud coded tensor sets in addition to the first point cloud coded tensor set for rendering. The second spatial object can be other objects in the scene, such as distant ships or large cargo piles. These objects do not require the same high level of detail when rendering as the target object. For example, for distant ships, it is sufficient to use a larger scale point cloud coded tensor set for rendering. The larger scale tensor set contains more macroscopic feature information. The second rendering result of the ship (second spatial object) rendered based on this information may roughly show the overall shape, approximate position and other information of the ship, but will not show many details like the first rendering result. This method of rendering using point cloud coded tensor sets of different scales according to different objects can improve rendering efficiency and reduce unnecessary calculations while ensuring the rendering quality of important target objects.

[0151] Finally, step S240 is performed, where the preset rendering result is some standards or requirements about the rendering effect preset by the computer system, which may include the clarity of the image, the accuracy of the object shape, whether the relative position relationship between different objects is correct, and other aspects.

[0152] For example, for the first rendering result (such as the rendering result of a person), the preset rendering result may require that the outline of the person is clearly identifiable, the body movements can be accurately reflected, and the position of the person in the entire scene is consistent with the position in the actual point cloud data. For the second rendering result (such as the rendering result of a ship), the overall shape of the ship is required to be correct, and the relative position relationship with other objects (such as dock facilities, surrounding cargo piles, etc.) is reasonable. When the first rendering result and the second rendering result both meet the requirements of these preset rendering results, the computer system will combine the original point cloud data, the first rendering result and the second rendering result to form point cloud rendering data. This point cloud rendering data not only contains the original point cloud data (which may be useful for subsequent re-analysis or adjustment of the rendering result), but also contains the rendering results of the first spatial object (target object) and the second spatial object after rendering processing. These rendering results exist in the form of images or other visualizations, so that various objects in the chain bucket unloader operation scene are presented in a more intuitive way, providing a more easily processed data basis for subsequent obstacle detection, operation path planning and other operations.

[0153] In one implementation, step S220, feature encoding is performed based on the point cloud data to obtain a set of point cloud encoding tensors of multiple scales, including:

[0154] Step S221: performing multiple rounds of feature encoding based on the point cloud data to obtain a first encoding tensor set of multiple scales;

[0155] Step S222: Perform feature encoding based on a first encoding tensor set of multiple scales to obtain a point cloud encoding tensor set of multiple scales; wherein each round of feature encoding includes: Step S2211: Perform feature encoding based on the first processed data to obtain a first transition encoding tensor set; wherein the first processed data is data of a set percentage of data slices in the input data corresponding to the current round of feature encoding; the input data corresponding to the first round of feature encoding is point cloud data; Step S2212: Channel-mix the first transition encoding tensor set and the second transition encoding tensor set to obtain a third transition encoding tensor set; the second transition encoding tensor set is data other than the first processed data in the input data; Step S2213: Perform feature compression on the third transition encoding tensor set to obtain an initial encoding tensor set; wherein the initial encoding tensor set is the input data corresponding to the next round of feature compression; the first encoding tensor set of multiple scales includes an initial encoding tensor set obtained by multiple rounds of feature encoding in multiple rounds of feature encoding; the spatial positions of some data slices in the first processed data based on the feature encoding in the previous and next rounds are different in the corresponding input data.

[0156] In step S221 of the embodiment of the present application, the point cloud data includes discrete point information on the surfaces of various objects in the area where the chain bucket ship unloader is located, and this information exists in the form of three-dimensional coordinates, etc. In order to better process this data for subsequent operations (such as rendering, etc.), the computer system needs to encode its features.

[0157] The feature encoding process is carried out in multiple rounds, and each round will obtain the encoding results at a specific scale, and these results together constitute the first encoding tensor set of multiple scales. For example, in the operation scene of the chain bucket unloader, it is assumed that the point cloud data contains information about objects such as ships, cargo piles, and personnel. In the first round of feature encoding, the computer system will process these point cloud data in a holistic manner. Assuming that the point cloud data can be regarded as a huge point set, the first round of feature encoding may perform preliminary feature extraction on this point set. For example, for the point cloud data of the ship part, it may be encoded according to the approximate spatial distribution characteristics between points. If the point cloud data of the ship presents a specific geometric shape (such as a long strip with a certain curvature), the computer system may encode this shape feature as a part of a vector or tensor. For the cargo pile, due to its relatively irregular shape, its height change, surface area and other features may be encoded. For the personnel part, the general body shape characteristics of the personnel may be encoded (such as the overall shape characteristics corresponding to the posture information such as whether standing or bending).

[0158] As the number of rounds increases, each round of feature encoding will further refine or process these features from different angles, thereby obtaining a set of first encoding tensors of different scales. Different scales mean different levels of abstraction or detail. At a larger scale, the overall characteristics of the object may be more reflected, while at a smaller scale, more attention is paid to local detail features. For example, in later rounds, the features of local structures such as the bow and stern of a ship may be further encoded, which are more detailed features compared to the overall shape features in the first round of encoding, and the encoding results of these different scales are stored in the first encoding tensor set in the form of tensors.

[0159] Next is the sub-step S2211 in each round of feature coding. In each round of feature coding, the computer system selects a portion of data as the first processed data for feature coding. The first processed data is a subset of the input data corresponding to the current round of feature coding, specifically a data slice with a set percentage (the data slice here can be analogous to the concept of channels in a matrix).

[0160] Take the point cloud data in the bucket chain unloader operation scene as an example. Assume that in a certain round of feature encoding, the input data is an intermediate result after the previous round of processing (if it is the first round, it is the original point cloud data). If the percentage is set to 30%, the computer system will select this 30% data slice as the first processing data according to a certain rule (for example, according to the spatial order of the points or random selection).

[0161] For this part of data, the computer system performs feature encoding. For example, for the selected part of the point cloud data (which may contain part of the point cloud data of objects such as ships, cargo piles, and personnel), the computer system will analyze the relationship between these points. If the point cloud data of the bow of the ship is included in this 30% data slice, the computer system will encode according to the spatial relationship of these points (such as the distance and angle between adjacent points).

[0162] Then, in sub-step S2212, after obtaining the first set of transitional coding tensors, the computer system also needs to consider the portion of the input data other than the first processed data, which is defined as the second set of transitional coding tensors. Continuing with the above example, if the first processed data is 30% of the input data, then the second set of transitional coding tensors is the remaining 70% of the data.

[0163] The computer system performs channel mixing on the first set of transition coding tensors and the second set of transition coding tensors. Channel mixing is an operation that fuses feature information from different sources. For example, the first set of transition coding tensors contains information such as the shape features of the bow point cloud data mentioned earlier, while the second set of transition coding tensors may contain feature information of other parts of the ship's hull, cargo piles, personnel, etc. During the channel mixing process, the computer system may use some specific algorithms to ensure the effective fusion of different features. For example, different weights are assigned according to the importance or relevance of different features, and then the corresponding channels are weighted summed and other operations are performed. Through this channel mixing operation, a third set of transition coding tensors is obtained, which fuses the feature information of different parts of the input data, including both local features in the first processed data and other features in the second set of transition coding tensors.

[0164] Next is sub-step S2213. After obtaining the third transition coding tensor set, the computer system needs to perform feature compression operation on it. Feature compression is to reduce the amount of data while retaining key feature information. For example, the third transition coding tensor set may contain a large amount of feature information, which may be redundant. The computer system will use some methods, such as dimensionality reduction algorithm, to process this tensor set.

[0165] Assuming that the third transition coding tensor set represents the complex feature information of objects such as ships, cargo piles and personnel, some similar features may be merged or some less important features may be removed through feature compression. For example, for the ship part, if there are multiple feature channels representing the color of the ship in the third transition coding tensor set (these channels may have certain correlations), the computer system may merge these channels or select the most representative channel among them, thereby reducing the amount of data.

[0166] The initial set of encoded tensors obtained after feature compression will be used as the input data for the next round of feature encoding. Moreover, in the previous and next rounds of feature encoding, the spatial positions of the data slices in the corresponding input data based on the partial data slices in the first processed data are different. This means that the part of the point cloud data that each round of feature encoding focuses on is different in space. For example, the first round of feature encoding may focus on partial data slices in the front area of ​​the point cloud data (defined in a certain spatial order) as the first processed data, then the second round of feature encoding may focus on partial data slices in the middle or back area of ​​the point cloud data as the first processed data, so that features can be extracted from different spatial regions, thereby obtaining first encoded tensor sets of different scales.

[0167] Finally, in step S222, after obtaining the first coded tensor set of multiple scales, the computer system needs to further perform feature encoding on it to obtain a point cloud coded tensor set of multiple scales that is ultimately used for rendering and other operations.

[0168] For example, the first coded tensor set of multiple scales contains feature information of different abstraction levels and details. For the first coded tensor of larger scale (tensor reflecting more overall features of the object), the computer system may further encode it to better meet the subsequent rendering requirements. In this process, adjustments may be made according to the requirements for different object features during rendering.

[0169] Assume that for the rendering of a ship, its features need to be accurately displayed at different distances. For the overall feature part of the ship in the larger-scale first coding tensor, the computer system may further encode its appearance features under different viewing angles (such as the changing features of the overall outline of the ship in different directions when viewed from a distance). For the smaller-scale first coding tensor (a tensor containing more local detail features), for example, for the detail feature tensor of the bow of the ship, the computer system may further encode the display features of the detailed structure of the bow under different lighting conditions, etc.

[0170] By performing feature encoding on the first coded tensor set of multiple scales, the computer system obtains a point cloud coded tensor set of multiple scales. Each tensor in these point cloud coded tensor sets contains optimized coding information for point cloud data features at different scales, providing a suitable data basis for subsequent rendering according to different objects (such as rendering the target object according to the point cloud coded tensor set of the smallest scale in step S230, and rendering the second space object according to the point cloud coded tensor set of other scales), thereby ensuring the rendering effect while improving the rendering efficiency, and better adapting to various needs in the intelligent operation scene of the chain bucket ship unloader.

[0171] In one implementation, step S222, performing feature encoding based on the first coded tensor sets of multiple scales to obtain point cloud coded tensor sets of multiple scales, includes:

[0172] Step S2221: performing feature expansion based on the first coded tensor sets of multiple scales to obtain second coded tensor sets of multiple scales; wherein the second coded tensor set of the largest scale is the first coded tensor set of the largest scale; the second coded tensor sets of other scales are obtained by performing feature expansion on adjacent large-scale second coded tensor sets and merging first coded tensor sets of the same scale;

[0173] Step S2222: Feature compression is performed based on the second coded tensor sets of multiple scales to obtain point cloud coded tensor sets of multiple scales; wherein the point cloud coded tensor set of the smallest scale is the second coded tensor set of the smallest scale; point cloud coded tensor sets of other scales are obtained based on feature compression of adjacent small-scale point cloud coded tensor sets and merging second coded tensor sets of the same scale.

[0174] In step S2221 of the embodiment of the present application, the first encoding tensor set of multiple scales is obtained through a series of feature encoding steps, and each tensor contains the feature information of the point cloud data at the corresponding scale. In order to obtain a richer feature representation, the computer system needs to perform feature expansion on these first encoding tensor sets to facilitate subsequent operations (such as rendering, etc.). Taking the point cloud data in the chain bucket unloader operation scene as an example, it is assumed that there are three scales of first encoding tensor sets, corresponding to large, medium and small scales respectively. The first encoding tensor set of the largest scale contains relatively macroscopic feature information of the entire operation scene. For example, it may describe the overall distribution of large objects (such as ships, large cargo piles, etc.) in the scene, the approximate shape outline and the relative position relationship between them. When the feature is expanded, for the first encoding tensor set of the largest scale, since it is already the largest scale, the second encoding tensor set of the largest scale is directly the first encoding tensor set of the largest scale. This means that at this scale, its feature information does not undergo additional expansion operations in this step, but will serve as the basis for subsequent operations on other scales. For the generation of the second coded tensor set at the mesoscale, the computer system will perform feature expansion based on the adjacent large-scale (i.e., maximum-scale) second coded tensor set and merge the first coded tensor set at the same scale (mesoscale). Assuming that the large-scale second coded tensor set describes the overall outline and positional relationship of the ship, the computer system may incorporate some local feature information of the ship when performing feature expansion. For example, for a ship, only its overall shape is known to be a rectangle at a large scale, but during the feature expansion process, feature information of local structures such as the bow and stern of the ship is obtained from the first coded tensor set at the mesoscale, and this information is added to the result after expansion based on the large-scale second coded tensor set. At the same time, other relevant features that may exist in the first coded tensor set at the mesoscale will also be merged, such as some special textures on the surface of the ship (if reflected in the first coded tensor set) or the approximate location of certain equipment on the ship.

[0175] Similarly, for the small-scale second coding tensor set, the computer system will perform feature expansion based on the adjacent middle-scale second coding tensor set and merge the small-scale first coding tensor set. For example, the small-scale first coding tensor set may contain detailed appearance features of a person (such as body movements, details of clothing, etc.) and the shape features of a single item in a pile of goods. The computer system merges the feature information in these small-scale first coding tensor sets with the results of the expansion based on the middle-scale second coding tensor set. Assuming that the middle-scale second coding tensor set already contains the approximate position and posture of the person in the scene, by merging the detailed appearance features in the small-scale first coding tensor set, a more complete small-scale second coding tensor set is obtained, which not only contains the overall posture information of the person, but also contains more detailed appearance features.

[0176] In this way, the computer system obtains a second set of coded tensors at multiple scales. The second set of coded tensors at each scale has richer feature information than the corresponding first set of coded tensors. This information includes both the part expanded from the adjacent large scale and other features in the first set of coded tensors at the same scale.

[0177] Next is step S2222. After obtaining the second coded tensor set of multiple scales, the computer system needs to perform feature compression to obtain the final point cloud coded tensor set of multiple scales. The purpose of feature compression is to reduce the amount of data while retaining key feature information, making the data easier to process and store, and also more suitable for subsequent operations (such as rendering, etc.).

[0178] Taking the second coded tensor set of multiple scales obtained previously as an example, the second coded tensor set of the smallest scale contains the most detailed feature information, such as the detailed appearance features of personnel, the precise shape of a single item in a pile of goods, and the spatial relationship in a small area. For this second coded tensor set of the smallest scale, since it is already the smallest scale, the point cloud coded tensor set of the smallest scale is directly the second coded tensor set of the smallest scale. This is because at this scale, there is no need to perform operations based on adjacent small scales. It will directly serve as the final point cloud coded tensor set of the smallest scale to provide data support for subsequent rendering of the target object (such as the target object in step S230).

[0179] For the generation of point cloud coding tensor sets of other scales (non-minimum scale), the computer system will perform feature compression based on adjacent small-scale point cloud coding tensor sets and merge the second coding tensor sets of the same scale. For example, for the generation of the sub-minimum scale point cloud coding tensor set, the computer system will perform feature compression on the point cloud coding tensor set based on the minimum scale (that is, the second coding tensor set of the minimum scale). Assuming that the minimum scale point cloud coding tensor set contains very detailed information such as the precise position of each joint of the person's limbs and various texture details on the clothing, during the feature compression process, the computer system may remove some details that have little impact on the overall situation, such as some minor texture changes on the clothing or minor deviations in the joint position. At the same time, the computer system will merge other relevant features in the second coding tensor set of the sub-minimum scale. For example, the second coding tensor set of the sub-minimum scale may contain features such as the activity trajectory of the person in a small range and the relationship with the surrounding small objects (such as nearby small tools). These features are merged with the minimum scale point cloud coding tensor set after feature compression to obtain the sub-minimum scale point cloud coding tensor set.

[0180] Similarly, for a larger scale point cloud coding tensor set, the computer system will first perform feature compression on the adjacent small scale point cloud coding tensor set in the same way, and then merge the second coding tensor set of the same scale. For example, for a medium scale point cloud coding tensor set, the computer system will perform feature compression on the point cloud coding tensor set based on the sub-small scale, remove some relatively minor features, and then merge other features in the second medium scale coding tensor set, such as the overall structural features of ships, cargo piles, etc. at the medium scale, their mutual relationships, and their relationships with other larger objects in the scene (such as large equipment), so as to obtain a medium scale point cloud coding tensor set.

[0181] In one implementation, step S2221, performing feature expansion based on a first set of coded tensors at multiple scales to obtain a second set of coded tensors at multiple scales, includes:

[0182] Step S22211: feature expansion is performed on the first set of coded tensors to be processed to obtain an expanded set of coded tensors; wherein the first set of coded tensors to be processed based on which the first round of feature expansion is performed is the first set of coded tensors obtained by the last round of feature compression;

[0183] Step S22212: merging the extended coded tensor set with the first coded tensor set of the same scale to obtain a second coded tensor set;

[0184] Step S22213: Determine the second set of encoded tensors as the first set of encoded tensors to be processed corresponding to the next round of feature expansion.

[0185] In step S22211 of the embodiment of the present application, in this process, the first set of coded tensors to be processed is the input object of the feature expansion operation. Taking the point cloud data processing in the chain bucket unloader operation scene as an example, it is assumed that we have completed the previous multiple rounds of feature encoding to obtain the first set of coded tensors of multiple scales. For each scale, there is a corresponding first coded tensor, which contains point cloud data feature information at different levels. When performing feature expansion, the first set of coded tensors to be processed in the first round is the first set of coded tensors obtained by the last round of feature compression. For example, in the previous feature encoding process, for point cloud data processing of a certain scale (assuming a medium scale), a first coded tensor is obtained after multiple rounds of feature encoding and the last round of feature compression. This tensor may contain feature information of objects around the chain bucket unloader (such as cargo piles, part of the ship structure, etc.) at this scale. These feature information may be simplified and refined, such as the general shape of the cargo pile, the relative position relationship between the cargo pile and the surrounding objects, and some surface features of the cargo pile at this scale.

[0186] The computer system performs feature expansion on this first set of coded tensors to be processed. Feature expansion is an operation that increases the richness of features. For example, in the case of a cargo pile, the original first coded tensor may only describe the overall shape of the cargo pile as a rectangular parallelepiped. During the feature expansion process, the computer system may add feature information such as the general direction of the surface texture of the cargo pile and the distribution trend of the cargo on the top of the cargo pile based on more detailed information in the point cloud data. For the ship structure, if only part of the contour features of the ship structure are recorded in the first coded tensor, the features of certain key connection parts of the ship structure, some local curvature changes on the surface of the ship structure, and other information may be supplemented during feature expansion. Through such operations, the computer system obtains an extended set of coded tensors, which contains richer feature information than the first set of coded tensors to be processed.

[0187] Next is step S22212, the expanded coding tensor set contains rich feature information after feature expansion, but it may not be a complete feature representation at this scale. The first coding tensor set of the same scale may also contain some other feature information that is not involved in the feature expansion process.

[0188] Continuing with the example of the medium scale in the chain bucket unloader operation scene, it is assumed that the extended coding tensor set adds features such as the surface texture direction of the cargo pile and the distribution trend of the top cargo in the description of the cargo pile, but the first coding tensor set of the same scale may contain some characteristic information of the environment around the cargo pile, such as the distance relationship between small equipment near the cargo pile (such as handling tools) and the cargo pile, the impact of light and shadow around the cargo pile on the cargo pile at this scale, and other features.

[0189] The computer system merges the extended coded tensor set with the first coded tensor set of the same scale. During the merging process, it is necessary to ensure that the feature information from different sources can be reasonably combined. For example, for the features of the cargo pile, they may be integrated according to the type and importance of the features. If it is to describe the shape, texture and other features of the cargo pile itself, the more detailed texture direction information in the extended coded tensor set may be merged with the cargo pile shape information in the first coded tensor set to form a more comprehensive description of the cargo pile features; for the relationship between the cargo pile and the surrounding environment, the distance relationship between the cargo pile and the small equipment, the influence of light and shadow and other features in the first coded tensor set will be considered together with the distribution trend of the cargo on the top of the cargo pile in the extended coded tensor set, so as to obtain a complete feature representation of the cargo pile and its surrounding environment at this scale. Through such a merging operation, a second coded tensor set is obtained, which contains more comprehensive and richer feature information at this scale, and is a more complete description of the point cloud data features at this scale.

[0190] Then, step S22213 is performed, and the second set of coded tensors is obtained through the previous feature expansion and merging operations, and contains more comprehensive feature information at the current scale. In the whole process of performing feature expansion based on the first set of coded tensors at multiple scales to obtain the second set of coded tensors at multiple scales, this is an iterative step.

[0191] For example, in the chain bucket unloader operation scenario, if we are processing feature extensions at multiple scales (such as small, medium, and large scales), for the medium-scale processing, after obtaining the second set of coded tensors, this set will be used as the first set of coded tensors to be processed for the next round of feature extension (if there is a next round). This means that the next round of feature extension will be further operated based on this already richer and more comprehensive feature set. If the next round of feature extension is to perform feature extension on a more complex relationship with other objects (such as ships, personnel, etc.) at this scale, then this second set of coded tensors provides the basis for the next round of operations. It may contain detailed features of the cargo pile itself and preliminary relationship features with surrounding objects. The next round of feature extension can further mine and add more complex feature information such as the relationship between the cargo pile and the walking path of the personnel, and the potential connection between the cargo pile and the docking position of the ship. Through such an iterative process, a richer and more accurate feature representation at each scale is gradually constructed, and finally a second set of coded tensors at multiple scales is obtained, providing a suitable data basis for subsequent operations (such as the feature compression operation in step S2222).

[0192] In one implementation, in step S2222, multiple rounds of feature compression are performed based on multiple second coded tensor sets to obtain point cloud coded tensor sets of multiple scales, including:

[0193] Step S22221: feature compression is performed on the second set of coded tensors to be processed to obtain a compressed set of coded tensors; wherein the second set of coded tensors obtained by the last round of feature expansion is the second set of coded tensors to be processed by the first round of feature compression;

[0194] Step S22222: merging the compressed coded tensor set with the second coded tensor set of the same scale to obtain a point cloud coded tensor set;

[0195] Step S22223: Determine the point cloud coded tensor set as the second set of coded tensors to be processed corresponding to the next round of feature compression.

[0196] In step S22221 of the embodiment of the present application, in the entire processing flow, the second set of coded tensors to be processed is the input object for feature compression operation. Taking the point cloud data in the chain bucket unloader operation scene as an example, after the previous steps (such as feature expansion and other operations), the second set of coded tensors of multiple scales is obtained. For each scale, the second set of coded tensors contains rich feature information.

[0197] Taking one of the scales (e.g., the medium scale) as an example, the second set of encoded tensors obtained by the last round of feature expansion contains a variety of feature information of objects around the chain bucket unloader (such as cargo piles, ship structures, personnel, etc.) at this scale. Assume that for a cargo pile, this set of tensors not only contains the overall shape, surface texture, and distribution of cargo, but also includes detailed relationship information between the cargo pile and surrounding objects (such as nearby small equipment, other cargo piles, and the path of personnel walking, etc.); for a ship structure, it may contain detailed structural features of each part of the ship, special markings on the surface of the ship, and the relationship between the ship and the terminal facilities, etc.; for personnel, it contains the personnel's posture, movements, and interaction information between the personnel and surrounding objects, etc.

[0198] However, these rich feature information may be redundant or not necessary for subsequent operations (such as rendering). Therefore, the computer system needs to perform feature compression on this second set of coded tensors to be processed. Feature compression is an operation that reduces the amount of data while retaining key feature information. For example, for the characteristics of a cargo pile, there may be some information that describes the surface texture of the cargo pile in too much detail, which is redundant to a certain extent for the overall representation of the cargo pile. The computer system can use some algorithms (such as principal component analysis, etc.) to perform feature compression. Through these algorithms, the multiple descriptive dimensions of the surface texture of the cargo pile are simplified into several key dimensions, retaining the information that best represents the texture features, while removing those dimensional information that has little impact on the overall. For the relationship information between the cargo pile and the surrounding objects, it may be screened according to the distance and importance. For example, only the relationship information with objects that are close and have a greater impact on the operation (such as small equipment being unloaded) is retained, and the relationship information with objects that are far away and have little relevance is removed. Similar processing is also performed on the characteristics of ship structures and personnel to obtain a set of compressed coded tensors. This compressed coded tensor set retains key feature information, but the amount of data is reduced compared to the original second set of coded tensors to be processed.

[0199] Next is step S22222. Although the compressed coded tensor set has been feature compressed and retains key feature information, it may not be complete in some aspects. The second coded tensor set of the same scale may contain some information that is partially discarded during the feature compression process but is still valuable in some cases.

[0200] Continuing with the example of the medium scale, assume that the features of the cargo pile are simplified in the compressed coding tensor set, mainly retaining the overall shape of the cargo pile, key texture features, and relationship information with important objects (such as unloading equipment). In addition to this information, the second coding tensor set of the same scale may also contain some appearance features of the cargo pile under specific lighting conditions (such as the impact of shadows on the shape of the cargo pile) and the state changes of the cargo pile over a period of time (such as a small displacement of the cargo, etc.).

[0201] The computer system merges the compressed coded tensor set with the second coded tensor set of the same scale. During the merging process, the feature information in the two tensor sets needs to be comprehensively considered. For example, for the shape features of the cargo pile, if the shape information in the compressed coded tensor set is obtained based on the simplified algorithm, and there is some shape supplementary information obtained based on the original data in the second coded tensor set of the same scale (such as visual shape changes caused by lighting), the computer system needs to fuse this information. Weighted processing may be performed according to the reliability and importance of the features. If the simplified shape information is considered to be more basic and reliable, it may be given a higher weight, while the shape change information caused by lighting is given a lower weight, and then the weighted sum of the two is obtained to obtain the merged cargo pile shape features. Similar processing is performed for other features (such as texture, relationship with surrounding objects, etc.). Through such a merging operation, a point cloud coded tensor set is obtained. This point cloud coded tensor set combines the compressed key feature information and the supplementary feature information in the original second coded tensor set, and is a more comprehensive feature representation at this scale that is more suitable for subsequent operations (such as rendering).

[0202] Then comes step S22223, the point cloud coded tensor set is obtained through the previous feature compression and merging operations, which is a comprehensive feature representation at the current scale. This is an iterative step in the entire process of performing feature compression based on the second coded tensor set at multiple scales to obtain the point cloud coded tensor set at multiple scales.

[0203] For example, in the operation scenario of the chain bucket ship unloader, if we are processing feature compression at multiple scales (such as small, medium, and large scales), for the medium-scale processing, after obtaining the point cloud coding tensor set, this set will be used as the second set of coding tensors to be processed for the next round of feature compression (if there is a next round). This means that the next round of feature compression will be further operated based on this set that has integrated multiple feature information. If the next round of feature compression is to further optimize the feature representation at this scale to better adapt to different operating requirements (such as under different observation distances or accuracy requirements), then this point cloud coding tensor set provides the basis for the next round of operations. It may contain optimized feature information of objects such as cargo piles, ships, and personnel at the current scale. The next round of feature compression can further remove some relatively minor features or more accurately represent key features on this basis, so as to gradually obtain a point cloud coding tensor set at this scale that is more in line with the requirements. Through such an iterative process, point cloud coding tensor sets of multiple scales are finally obtained, providing a suitable data basis for subsequent operations in the intelligent operation of the chain bucket ship unloader (such as rendering based on point cloud coding tensor sets of different scales).

[0204] In one implementation, step S22222, merging the compressed coded tensor set with the second coded tensor set of the same scale to obtain a point cloud coded tensor set, includes:

[0205] Step S222221: merging the compressed coded tensor set with the second coded tensor set of the same scale to obtain a first merged coded tensor set;

[0206] Step S222222: determining data slice weights corresponding to respective data slices in the first merged coded tensor set based on the first merged coded tensor set, and merging respective data slices of the second coded tensor set according to respective data slice weights to obtain a second merged coded tensor set;

[0207] Step S222223: Determine the set of spatial weight coding tensors corresponding to the second merged feature, and determine the set of point cloud coding tensors based on the set of spatial weight coding tensors and the second merged feature.

[0208] In step S222221 of the embodiment of the present application, in the operation scene of the chain bucket ship unloader, after a series of processing, the point cloud data obtains a compressed coding tensor set and a second coding tensor set of the same scale. Both tensor sets contain feature information about objects in the operation scene, but the information focuses on different aspects.

[0209] Taking the description of the cargo pile and the surrounding environment at a medium scale as an example, the compressed coded tensor set is obtained through feature compression operation, which retains the key feature information of the cargo pile. Assume that in this compressed coded tensor set, the information about the cargo pile includes its simplified overall shape (such as the basic size of the rectangular shape), the height information of the main part of the cargo pile, and the approximate distance relationship between the cargo pile and the adjacent important unloading equipment (such as the unloading area corresponding to the reach of the crane arm). This information is filtered and simplified to highlight the most critical parts for subsequent operations (such as rendering or job decision-making).

[0210] The second set of coded tensors of the same scale contains richer but possibly partially redundant information. For a cargo pile, in addition to the information in the compressed coded tensor set above, it may also contain a more detailed description of the surface texture of the cargo pile (such as the general situation of different texture patterns formed on the surface by different cargoes), a history of small changes in the shape of the cargo pile due to loading and unloading operations at different time periods, and the relative position relationship between the cargo pile and other surrounding auxiliary equipment (such as cargo fixing devices, temporary lighting equipment, etc.).

[0211] The computer system merges the two tensor sets. During the merging process, the corresponding elements or features in the two tensor sets need to be combined. For example, for the shape information of the cargo pile, the simplified shape information in the compressed coded tensor set is combined with the history of small shape changes in the second coded tensor set. This may mean integrating the basic size of the cargo pile with the shape change information to form a more comprehensive shape description, which includes both the overall static shape and the dynamic change information within a certain time range. For the relationship information between the cargo pile and the surrounding objects, the approximate distance relationship with the important unloading equipment in the compressed coded tensor set and the relative position relationship with other auxiliary equipment in the second coded tensor set are merged to obtain a description of the relationship between the cargo pile and the surrounding environment containing more relationship information. In this way, the compressed coded tensor set is merged with all relevant features in the second coded tensor set of the same scale to obtain a first merged coded tensor set. This first merged coded tensor set combines the information of the two original tensor sets, makes up for the information missing that may exist in the compressed coded tensor set to a certain extent, and also integrates more detailed information in the second coded tensor set.

[0212] Next is step S222222, each data slice in the first merged coding tensor set represents certain feature information. However, the importance of these feature information to the final construction of the point cloud coding tensor set is different. The computer system needs to determine the weight of each data slice to reflect its relative importance in the entire feature representation. Continuing with the first merged coding tensor set related to the cargo pile at a medium scale as an example, the data slices therein may correspond to different aspects of the characteristics of the cargo pile. Assuming that one of the data slices corresponds to the height information of the cargo pile, this information is of great significance for judging the loading capacity of the cargo pile and its matching with the unloading equipment (such as the lifting height limit of the crane); another data slice may correspond to partial information on the surface texture of the cargo pile, which is relatively less important for operational decisions (such as unloading order, stacking stability, etc.).

[0213] The computer system will determine the weight of each data slice according to pre-set rules or data-based analysis. For data slices with higher importance, such as the slice corresponding to the cargo pile height information, a higher weight, such as 0.8, may be assigned; while for the slice corresponding to the surface texture information of the cargo pile, a lower weight, such as 0.2, may be assigned. These weights can be determined based on the understanding of the chain bucket unloader operation process, the analysis of historical data, or through some machine learning algorithms (such as feature importance evaluation algorithms based on sample data).

[0214] After determining the weights, the computer system merges the data slices of the second set of coded tensors according to these weights. Since the second set of coded tensors contains rich original feature information, merging guided by weights can more specifically retain important information and reasonably integrate different features. For example, for the height information slice of the cargo pile (in the second set of coded tensors), since its corresponding weight in the first merged set of coded tensors is higher, more attention will be given in the merging process. The computer system may merge in a weighted summation manner. Assuming that there are multiple slices related to the height of the cargo pile in the second set of coded tensors (due to the previous different processing, there may be multiple slices representing height information), these slices are weighted summed according to their respective weights (may be based on the weights adjusted according to the importance in the first merged set of coded tensors) to obtain a comprehensive representation of the height information. Other data slices (such as the surface texture of the cargo pile, etc.) are also processed in a similar manner, and all data slices are weighted merged to obtain the second merged set of coded tensors. This second merged coding tensor set is the result of selectively integrating the second coding tensor set based on the weights of each data slice in the first merged coding tensor set. It further optimizes the feature representation and highlights important feature information.

[0215] Then comes step S222223. The second merged coding tensor set has optimized and integrated the features, but in order to more accurately construct a point cloud coding tensor set suitable for the intelligent operation requirements of the chain bucket unloader, the spatial information of the features needs to be considered. The computer system needs to determine the spatial weight coding tensor set corresponding to the second merged feature.

[0216] Taking cargo piles as an example, assuming that there are multiple cargo piles at the dock, among which the cargo piles close to the unloading area of ​​the chain bucket unloader have higher spatial importance during the operation process, while the cargo piles far away from the unloading area have relatively lower spatial importance. The computer system constructs a set of spatial weight coding tensors based on these spatial relationships. For the spatial features corresponding to the cargo piles close to the unloading area (such as the coordinate position in the entire dock space, the distance from the surrounding spatial boundary, etc.), a higher weight, such as 0.7, will be assigned in the set of spatial weight coding tensors; while the spatial features corresponding to the cargo piles far away from the unloading area may be assigned a lower weight, such as 0.3. After determining the set of spatial weight coding tensors, the computer system determines the set of point cloud coding tensors based on this set and the second merged feature. This process may involve some form of fusion operation of the weight information in the set of spatial weight coding tensors with the feature information in the second merged coding tensor set. For example, for the feature information of the cargo pile (in the second merged coding tensor set), it is weightedly combined with the corresponding spatial weight (in the set of spatial weight coding tensors). If a feature of the cargo pile (such as a shape feature) is represented as a vector in the second combined coding tensor set, each element of this vector is multiplied by the corresponding spatial weight, and then summed or other combined operations are performed to obtain the fused cargo pile shape feature representation. Other features in the second combined coding tensor set and features of other objects in the dock scene (such as ships, equipment, etc.) are processed in a similar manner to finally obtain a point cloud coding tensor set. This point cloud coding tensor set not only contains optimized and integrated object feature information, but also incorporates spatial weight information, which can better adapt to various needs in the intelligent operation scene of the chain bucket unloader, such as accurate obstacle detection and reasonable operation path planning.

[0217] Figure 2 A hardware entity diagram of a computer system provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the hardware entity of the computer system 1000 includes: a processor 1001 and a memory 1002, wherein the memory 1002 stores a computer program that can be run on the processor 1001, and the processor 1001 implements the steps in the method of any of the above embodiments when executing the program.

Claims

1. An intelligent operation method of a chain bucket ship unloader based on point cloud data processing, characterized in that: The method comprises: acquiring point cloud data of an area where a chain bucket ship unloader is located; performing point cloud rendering on the point cloud data to obtain a point cloud rendering image; combining the point cloud rendering image, calling a visual transformer algorithm to perform obstacle detection on the point cloud rendering image to obtain a first support set relative to a preset obstacle set, calling an obstacle detection algorithm to perform obstacle detection on the point cloud rendering image to obtain a second support set relative to the preset obstacle set, wherein the preset obstacle set includes a plurality of candidate obstacles; combining the first support set and the second support set to obtain a target obstacle in the point cloud rendering image; combining the point cloud rendering image and the target obstacle, calling the visual transformer algorithm to perform obstacle type prediction on the target obstacle to obtain a third support set relative to the preset obstacle type set, calling an obstacle type prediction algorithm to perform obstacle type prediction on the target obstacle to obtain a fourth support set relative to the preset obstacle type set, wherein the preset obstacle type set includes a plurality of reference obstacle types; combining the third support set and the fourth support set to obtain the target obstacle in the point cloud rendering image; the obstacle type of the target obstacle; combining the target obstacle and the obstacle type of the target obstacle, obtaining the obstacle annotation result of the point cloud rendering image, so as to perform the operation path planning of the chain bucket ship unloader based on the obstacle annotation result; combining the point cloud rendering image, calling the obstacle type prediction algorithm to perform obstacle detection on the point cloud rendering image, and obtaining a fifth support set relative to the preset obstacle set; combining the point cloud rendering image and the target obstacle, calling the obstacle detection algorithm to predict the obstacle type of the target obstacle, and obtaining a sixth support set relative to the preset obstacle type set; the combining the first support set and the second support set to obtain the target obstacle in the point cloud rendering image includes: combining the first support set, the second support set and the fifth support set to obtain the target obstacle; the combining the third support set and the fourth support set to obtain the obstacle type of the target obstacle includes: combining the third support set, the fourth support set and the sixth support set to obtain the obstacle type corresponding to the target obstacle.

2. The method according to claim 1, characterized in that The combining of the first support set and the second support set to obtain the target obstacle in the point cloud rendering image includes: determining a first difference of the second support set in combination with the matching score between the first support set and the second support set; obtaining a first optimization vector relative to the first support set in combination with the difference between the second support set and the first difference; optimizing the first support set in combination with the first optimization vector to obtain a first target support set; and obtaining the target obstacle in combination with the first target support set.

3. The method according to claim 1, characterized in that The method of combining the first support set, the second support set and the fifth support set to obtain the target obstacle includes: determining a first difference of the second support set in combination with a matching score between the first support set and the second support set; determining a second difference of the fifth support set in combination with a matching score between the first support set and the fifth support set; determining a first optimization vector for the first support set in combination with a difference between the second support set and the first difference; optimizing the first support set in combination with the first optimization vector; obtaining a second target support set based on the difference between the optimized first support set and the second difference; and obtaining the target obstacle in combination with the second target support set.

4. The method according to any one of claims 1 to 3, characterized in that: The process of acquiring the first support set and the second support set includes: combining the point cloud rendering image, calling the visual transformer algorithm to perform obstacle detection on the point cloud rendering image, and obtaining a first temporary support set relative to the preset obstacle set; calling the obstacle detection algorithm to perform obstacle detection on the point cloud rendering image, and obtaining a second temporary support set relative to the preset obstacle set; wherein the first temporary support set includes the first temporary support of each candidate obstacle, and the second temporary support includes the second temporary support of each candidate obstacle; combining the first temporary support set and the second temporary support set, screening target candidate obstacles whose first temporary support and second temporary support are not less than a support threshold from each candidate obstacle; filtering out the support of candidate obstacles that are not target candidate obstacles in the first temporary support set and the second temporary support set, respectively, to obtain the first support set and the second support set.

5. The method according to claim 1, characterized in that The point cloud rendering image includes a plurality of target obstacles, and the obstacle annotation result includes a plurality of target obstacles and obstacle types of the target obstacles; the obstacle annotation result is obtained by taking t processing stages, each processing stage is obstacle detection processing or obstacle type prediction processing, and the processing result of each processing stage is a target obstacle or an obstacle type; The sth processing stage The method comprises the following processes: determining the result classification of the s-th processing result based on the processing result of the s-1th processing stage, wherein the result classification is obstacle detection or obstacle type prediction, and the result classification corresponding to the first processing stage is obstacle detection; combining the point cloud rendering image and the processing result of the previous s-1th processing stage, calling the visual transformer algorithm to obtain a public support set, calling the obstacle detection algorithm to obtain an obstacle detection support set, and calling the obstacle type prediction algorithm to obtain an obstacle type prediction support set; the public support set is the first support set or the third support set, the obstacle detection support set is the second support set or the sixth support set, and the obstacle type prediction support set is the fourth support set or the fifth support set; combining the public support set, the obstacle detection support set and the obstacle type prediction support set to obtain the processing result of the s-th processing stage.

6. The method according to claim 1, characterized in that The point cloud rendering is performed on the point cloud data to obtain a point cloud rendering image, including: obtaining the point cloud data to be rendered; performing feature encoding based on the point cloud data to obtain point cloud coding tensor sets of multiple scales; using a point cloud coding tensor set with the smallest scale among the multiple point cloud coding tensor sets as a first point cloud coding tensor set, obtaining a first rendering result of a first spatial object based on the first point cloud coding tensor set, and obtaining a second rendering result of a second spatial object based on point cloud coding tensor sets other than the first point cloud coding tensor set; the first spatial object includes a target object; if the first rendering result and the second rendering result meet a preset rendering result, using the point cloud data, the first rendering result and the second rendering result as components of the point cloud rendering data to obtain the point cloud rendering data.

7. The method according to claim 6, characterized in that The feature encoding based on the point cloud data to obtain a set of point cloud coded tensors of multiple scales includes: performing multiple rounds of feature encoding based on the point cloud data to obtain a first set of coded tensors of multiple scales; performing feature encoding based on the first set of coded tensors of multiple scales to obtain a set of point cloud coded tensors of multiple scales; wherein each round of feature encoding includes: performing feature encoding based on first processed data to obtain a first transitional set of coded tensors; wherein the first processed data is data of a data slice of a set percentage in the input data corresponding to the current round of feature encoding; the input data corresponding to the first round of feature encoding is the point cloud data; the first transitional set of coded tensors is encoded. The first coding tensor set of the plurality of scales includes an initial coding tensor set obtained by multiple rounds of feature encoding in multiple rounds of feature encoding; and some data slices in the first processed data based on the two rounds of feature encoding have different spatial positions in the corresponding input data.

8. The method according to claim 7, characterized in that The method of performing feature encoding based on the first coding tensor set of multiple scales to obtain point cloud coding tensor sets of multiple scales includes: performing feature expansion based on the first coding tensor set of multiple scales to obtain second coding tensor sets of multiple scales; wherein the second coding tensor set of the largest scale is the first coding tensor set of the largest scale; second coding tensor sets of other scales are obtained by performing feature expansion on adjacent large-scale second coding tensor sets and merging first coding tensor sets of the same scale; performing feature compression based on the second coding tensor set of multiple scales to obtain point cloud coding tensor sets of multiple scales; wherein the point cloud coding tensor set of the smallest scale is the second coding tensor set of the smallest scale; point cloud coding tensor sets of other scales are obtained by performing feature compression on adjacent small-scale point cloud coding tensor sets and merging second coding tensor sets of the same scale; the method of performing feature expansion based on the first coding tensor set of multiple scales to obtain second coding tensor sets of multiple scales, includes : Perform feature expansion on the first set of coded tensors to be processed to obtain an extended set of coded tensors; wherein the first set of coded tensors to be processed based on which the first round of feature expansion is the first set of coded tensors obtained by the last round of feature compression; merge the extended set of coded tensors with the first set of coded tensors of the same scale to obtain a second set of coded tensors; determine the second set of coded tensors as the first set of coded tensors to be processed corresponding to the next round of feature expansion; perform multiple rounds of feature compression based on multiple second sets of coded tensors to obtain point cloud sets of coded tensors of multiple scales, including: perform feature compression on the second set of coded tensors to be processed to obtain a compressed set of coded tensors; wherein the second set of coded tensors obtained by the last round of feature expansion is the second set of coded tensors to be processed of the first round of feature compression; merge the compressed set of coded tensors with the second set of coded tensors of the same scale to obtain a point cloud set of coded tensors; determine the point cloud set of coded tensors as the second set of coded tensors to be processed corresponding to the next round of feature compression.

9. A computer system comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, wherein: When the processor executes the program, the steps in the method according to any one of claims 1 to 8 are implemented.

Images