Component identifying and positioning method and device, electronic equipment and readable storage medium

Abstract

The invention relates to the technical field of robots, in particular to a component identifying and positioning method and device, electronic equipment and a readable storage medium. The method comprises the following steps: acquiring multi-source sensing data of a target component; obtaining an instance segmentation mask of the target component through a pre-trained instance segmentation model based on the red, green and blue image data; on the basis of the instance segmentation mask, obtaining two-dimensional position data of an intersection point of center lines of the target component; generating a first coordinate estimation of the intersection point based on the two-dimensional position data and the depth map data, and generating a second coordinate estimation of the intersection point based on the two-dimensional position data and the laser radar point cloud data; and based on the first coordinate estimation and the second coordinate estimation, obtaining three-dimensional position data of the intersection point, and obtaining a positioning result of the target component. By adopting the method, more accurate and more reliable component position information can be provided.

Description

Technical Field

[0001] This application relates to the field of robotics, and in particular to a method, apparatus, electronic device, and readable storage medium for identifying and locating components. Background Technology

[0002] In industrial settings (e.g., electrical control cabinets, 3C assembly), the sorting and assembly of components is a fundamental yet demanding task, and its automation faces significant challenges.

[0003] Traditional technologies typically rely on fixed monocular or binocular vision systems to acquire images of the work area, followed by image processing algorithms to identify and locate components. However, traditional technologies suffer from significant errors in estimating component positions under complex conditions, and the results are highly uncertain, making it difficult to meet the high-precision positioning requirements of automated sorting and assembly operations. Therefore, there is an urgent need for a component identification and positioning method that can provide more accurate and reliable component location information. Summary of the Invention

[0004] Therefore, it is necessary to provide a component identification and positioning method, apparatus, electronic device, and readable storage medium that can provide more accurate and reliable component location information in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a method for identifying and locating electronic components, including:

[0006] Acquire multi-source sensing data of the target component. The multi-source sensing data is processed by spatiotemporal synchronization. The multi-source sensing data includes red-green-blue image data, depth map data, and lidar point cloud data.

[0007] Based on the red-green-blue image data, an instance segmentation mask of the target component is obtained through a pre-trained instance segmentation model; wherein, the instance segmentation model includes an instance segmentation network and a real-time segmentation network, the instance segmentation network is used to generate the instance mask of the component, and the real-time segmentation network is used to perform feature extraction and instance segmentation of the component and to provide feature enhancement for the instance segmentation network.

[0008] Based on the instance segmentation mask, obtain the two-dimensional position data of the intersection point of the center lines of the target component;

[0009] Based on the two-dimensional location data and the depth map data, a first coordinate estimate of the intersection point is generated, and based on the two-dimensional location data and the lidar point cloud data, a second coordinate estimate of the intersection point is generated.

[0010] Based on the first coordinate estimation and the second coordinate estimation, the three-dimensional position data of the intersection point is obtained, and the positioning result of the target component is obtained.

[0011] In one embodiment, generating a first coordinate estimate of the intersection point based on the two-dimensional location data and the depth map data, and generating a second coordinate estimate of the intersection point based on the two-dimensional location data and the lidar point cloud data, includes:

[0012] Based on the intrinsic parameters of the visual sensor that collects the red, green and blue image data, the depth values ​​of the corresponding positions in the two-dimensional position data and the depth map data are back-projected to obtain the first coordinate estimate of the intersection point;

[0013] Based on the two-dimensional location data, candidate point cloud regions are determined from the lidar point cloud data;

[0014] Cluster analysis is performed on the point cloud within the candidate point cloud region, and a second coordinate estimate of the intersection point is generated based on the results of the cluster analysis.

[0015] In one embodiment, after obtaining the three-dimensional position data of the intersection point based on the first coordinate estimation and the second coordinate estimation, the process includes:

[0016] Based on the three-dimensional location data, a semantic topology graph is constructed. The foundation layer of the semantic topology graph is a topology structure layer, which uses intersections as graph nodes and the connection relationships between components between intersections as graph edges. The attribute layer of the semantic topology graph is a task parameter layer, which assigns task attribute parameters to nodes and / or edges in the semantic topology graph. These task attribute parameters include at least one of task priority and task timeliness, and are used to form a semantic weighted task graph.

[0017] In one embodiment, the method for spatiotemporal synchronization processing of the multi-source sensing data includes:

[0018] Based on hardware trigger signals and software timestamps, the multi-source sensing data is time-aligned so that the difference between the software timestamps of any two types of sensing data in the multi-source sensing data is no greater than a preset difference.

[0019] The time-aligned multi-source sensing data is divided into multiple data frames according to the time series, and the sensing data in each data frame is unified to the same three-dimensional coordinate system.

[0020] In one embodiment, after acquiring the multi-source sensing data of the target component, the process includes:

[0021] The multi-source sensing data is input into a pre-trained cross-modal converter model, which enables the cross-modal converter model to perform semantic completion of occluded regions in the red-green-blue image data, geometric completion of occluded regions in the depth map data, or noise suppression of reflective regions in the red-green-blue image data.

[0022] In one embodiment, after obtaining the three-dimensional position data of the intersection point based on the first coordinate estimation and the second coordinate estimation, the process includes:

[0023] Based on the three-dimensional position data, the clamping component is controlled to move towards the intersection point of the target component, and the distance between the clamping component and the intersection point is determined based on the vision sensor.

[0024] If the rate of change of the distance is less than a preset rate of change, obtain the coordinates of the first center point of the intersection and the coordinates of the second center point of the end of the clamping component;

[0025] Calculate the spatial offset between the coordinates of the first center point and the coordinates of the second center point. If the spatial offset is greater than a preset offset, generate a pose correction amount based on the magnitude and direction of the spatial offset. The pose correction amount is used to convert into joint motion commands for the robot to control the gripping component to move in the direction that reduces the spatial offset until the spatial offset stabilizes within a preset error range.

[0026] In one embodiment, controlling the movement of the clamping component toward the intersection point of the target component based on the three-dimensional position data includes:

[0027] Based on the robot's hand-eye alignment, the three-dimensional position data is converted to the robot's base coordinate system;

[0028] Based on a pre-built accessibility assessment model, accessibility judgment results of the three-dimensional position data are generated. The accessibility assessment model is established based on the kinematic model of the robotic arm.

[0029] If the accessibility determination result indicates that the intersection point is within the range of motion of the robotic arm, the clamping component is controlled to move toward the intersection point of the target component based on the three-dimensional position data.

[0030] Secondly, this application also provides a component identification and positioning device, comprising:

[0031] The sensing data acquisition module is used to acquire multi-source sensing data of the target component. The multi-source sensing data is processed by spatiotemporal synchronization and includes red-green-blue image data, depth map data and lidar point cloud data.

[0032] The instance segmentation module is used to obtain the instance segmentation mask of the target component based on the red-green-blue image data through a pre-trained instance segmentation model; wherein, the instance segmentation model includes an instance segmentation network and a real-time segmentation network, the instance segmentation network is used to generate the instance mask of the component, and the real-time segmentation network is used to perform feature extraction and instance segmentation of the component and provide feature enhancement for the instance segmentation network.

[0033] A two-dimensional position acquisition module is used to acquire two-dimensional position data of the intersection of the center lines of the target component based on the instance segmentation mask;

[0034] The coordinate estimation module is used to generate a first coordinate estimate of the intersection point based on the two-dimensional position data and the depth map data, and to generate a second coordinate estimate of the intersection point based on the two-dimensional position data and the lidar point cloud data.

[0035] The three-dimensional position acquisition module is used to acquire the three-dimensional position data of the intersection point based on the first coordinate estimation and the second coordinate estimation, and obtain the positioning result of the target component.

[0036] Thirdly, this application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the above-mentioned embodiments.

[0037] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the preceding claims.

[0038] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the above claims.

[0039] The aforementioned component identification and localization method, device, electronic equipment, and readable storage medium achieve complementary and enhanced perception information under complex working conditions by processing multi-source perception data including RGB images, depth maps, and LiDAR point clouds in a spatiotemporal synchronous manner, overcoming the limitations of traditional single vision sensors. By employing a dual-model approach consisting of an instance segmentation network and a real-time segmentation network for collaborative segmentation, the system improves the fine-grained segmentation capability for overlapping and occluded component edges while maintaining processing efficiency, solving the problems of unstable recognition and unclear instance separation in traditional image algorithms under severe occlusion. Furthermore, by implementing dual-path coordinate estimation and fusion, redundant and mutually verified 3D localization is achieved, leveraging the advantages of different modal data to improve the accuracy and reliability of solving the 3D spatial coordinates of intersection points. Therefore, it can achieve highly robust and high-precision intelligent identification and localization of components in complex industrial environments. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating a component identification and positioning method in one embodiment;

[0042] Figure 2 This is a flowchart illustrating S04 in one embodiment;

[0043] Figure 3 This is a flowchart illustrating the component identification and positioning method in another embodiment;

[0044] Figure 4 This is a flowchart illustrating a method for spatiotemporal synchronization processing of multi-source sensing data in one embodiment;

[0045] Figure 5 This is a flowchart illustrating the component identification and positioning method in another embodiment;

[0046] Figure 6 This is a flowchart illustrating the component identification and positioning method in yet another embodiment;

[0047] Figure 7 This is a flowchart illustrating S081 in one embodiment;

[0048] Figure 8 This is a structural block diagram of a component identification and positioning device in one embodiment. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0051] In one embodiment, such as Figure 1 As shown, a component identification and positioning method is provided. This embodiment illustrates the application of this method to the processor of an electronic device. The electronic device can be a robot (e.g., an embodied intelligent industrial robot). In this embodiment, the method includes the following steps:

[0052] S01, acquire multi-source sensing data of the target component. The multi-source sensing data is processed in a time-space synchronization manner and includes red, green and blue image data, depth map data and lidar point cloud data.

[0053] The target components can be electronic components to be sorted or assembled for electrical control cabinet assembly, including but not limited to modular components such as PLCs, relays, fuse terminals, and circuit breakers. There can be multiple target components, which are distributed in overlapping and staggered patterns at storage stations or material preparation areas in the transit warehouse, with spatial layouts exhibiting localized shading and interference from metal surface reflections. Multi-source sensing data can be heterogeneous data collected by sensors based on different physical principles, including but not limited to: red, green, and blue image data for extracting color and texture features of components; depth map data recording the distance information between components and sensors in pixel array form; lidar point cloud data obtaining the three-dimensional spatial coordinates of components through laser ranging; and IMU data for capturing sensor motion states and assisting in correcting dynamic acquisition deviations.

[0054] For example, the robot can acquire perception data of target components from multiple angles and directions using equipped RGB cameras, depth cameras, LiDAR, and IMU units, ensuring coverage of key features of overlapping areas and occluded surfaces. The spatiotemporal synchronization processing of multi-source perception data can include: calibrating the acquisition time of each modality using a temporal interpolation algorithm to ensure that the acquisition time difference between any two modalities does not exceed a preset time threshold; first, performing intramodal spatial alignment of similar perception data; then, extracting common features of the target components in each modality (e.g., contours, key geometric points, etc.) for cross-modal feature matching; constructing coordinate transformation relationships between modalities based on the matching results; and unifying all multi-source perception data to a preset three-dimensional reference coordinate system. In one possible implementation, an industrial robot can collect data in stages around the target component placement area using its multi-sensor components at sorting stations in transit warehouses or material preparation areas during non-production periods. First, it collects global multi-source data from a fixed perspective. Then, it performs close-range supplementary data collection on local areas with severe overlap and obvious reflection. During the data collection process, the IMU provides real-time feedback on the motion status and dynamically adjusts the collection speed and angle to ensure that the collection range of multi-source data fully covers the key feature surfaces of each modular component. Finally, through spatiotemporal synchronization processing, different modal data can accurately correspond to the same component or the same spatial location.

[0055] S02, based on red, green and blue image data, obtain the instance segmentation mask of the target component through a pre-trained instance segmentation model; wherein, the instance segmentation model includes an instance segmentation network and a real-time segmentation network, the instance segmentation network is used to generate the instance mask of the component, and the real-time segmentation network is used to extract features of the component and segment the instance, and provide feature enhancement for the instance segmentation network.

[0056] For example, red, green, and blue image data can be input into a pre-trained instance segmentation model, enabling the instance segmentation model to extract features from the red, green, and blue image data through a real-time segmentation network. During the extraction process, an attention mechanism is introduced (which allows the instance segmentation model to focus on the edge, contour, and texture features of the components). Simultaneously, based on the extracted features, preliminary instance segmentation is completed, generating preliminary segmentation results. The real-time segmentation network outputs the extracted enhanced features and preliminary segmentation results to the instance segmentation network. The instance segmentation network receives enhanced features and optimizes the blurred regions, overlapping regions, and occluded regions in the preliminary segmentation results, correcting segmentation deviations and outputting an instance segmentation mask (which can be used to determine the contour range of a single target component, achieving instance separation of different target components).

[0057] The training method for the pre-trained instance segmentation model may include: constructing a training dataset containing images of components to be sorted or assembled for electrical control cabinet assembly. These images cover target components under different lighting conditions, varying degrees of occlusion, and different specifications, and include scenarios such as overlapping, shadowing, and metal surface reflection of components during pre-warehouse sorting, pre-outbound sorting, and pre-assembly. The training dataset is then subjected to occlusion-aware data augmentation processing (e.g., randomly simulating regular or irregular occlusions in the training images). The instance segmentation model is trained using a hybrid loss function, which includes cross-entropy loss and Dice loss, with weight coefficients adjusting the proportion of each loss. The weight coefficients range from [0,1]. The hybrid loss function... It can be represented as:

[0058] (1)

[0059] in, Cross-entropy loss; For Dice's loss; is the weighting coefficient, with a value range of [0,1].

[0060] S03, based on the instance segmentation mask, obtain the two-dimensional position data of the intersection of the center lines of the target component.

[0061] For example, the centerline of the target component can be extracted based on the instance segmentation mask, and two-dimensional position data can be obtained based on the intersection of the centerlines. Specifically, a morphological thinning algorithm can be applied to the instance segmentation mask of each component to extract the pixel-level centerline; a random sampling consensus algorithm can be used to fit the extracted centerline pixels with straight lines or arcs to remove noise interference; for the fitted centerlines of each target component, the intersection of the centerlines can be detected by Hough transform; the two-dimensional coordinates of the intersection points are recorded as the two-dimensional position data of the intersection of the centerlines of the target components.

[0062] This approach allows for individual processing of the instance segmentation mask for each target component, selecting the centerline type based on the component's structural characteristics. The centerline type can include at least one of geometric center lines, functional interface lines, or assembly datum lines. Geometric center lines can be determined based on the contour geometry of the component instance segmentation mask; functional interface lines can be determined based on the mask area corresponding to the component's interface structure; and assembly datum lines can be determined based on the mask area corresponding to the component's assembly positioning structure. In one possible implementation, for stacked components in pre-inbound and pre-outbound sorting scenarios, the intersection points of the centerlines between multiple components can be used as the objects for collecting two-dimensional position data. For modular components in pre-assembly scenarios, the intersection points of the functional interface lines or assembly datum lines of a single modular component with the corresponding lines of other modular components can be used as the objects for collecting two-dimensional position data.

[0063] It is understandable that after extracting the center line of multiple components in a stacked state through their respective instance segmentation masks, the intersection of the center lines corresponds to the structural stability area or assembly positioning reference area of ​​the component. This is suitable for the transfer and clamping of modular components in pre-assembly scenarios and the positioning and clamping operations before docking and assembly. Therefore, by confirming the two-dimensional position data of this intersection, the functional position of the clamping component can be clearly defined.

[0064] S04. Based on two-dimensional location data and depth map data, generate the first coordinate estimate of the intersection point, and based on two-dimensional location data and lidar point cloud data, generate the second coordinate estimate of the intersection point.

[0065] For example, the method for generating the first coordinate estimate of the intersection point based on two-dimensional location data and depth map data may include: delineating a local region centered on the pixel coordinates corresponding to the two-dimensional location data, extracting the effective depth values ​​of the depth map within this region and performing statistical fusion, and combining the intrinsic parameters of the visual sensor from the acquired red, green, and blue image data to generate three-dimensional coordinates through a spatial coordinate transformation algorithm. The method for generating the second coordinate estimate of the intersection point based on two-dimensional location data and LiDAR point cloud data may include: mapping the two-dimensional location data to the retrieval range of the LiDAR point cloud according to pre-calibrated cross-coordinate system transformation rules, filtering point clouds within this range that conform to the geometric constraints of the target component, and calculating the spatial center coordinates of the filtered point clouds.

[0066] S05. Based on the first coordinate estimation and the second coordinate estimation, obtain the three-dimensional position data of the intersection point to obtain the positioning result of the target component.

[0067] For example, the first coordinate estimate and the second coordinate estimate can be fused to obtain the three-dimensional position data of the intersection point. In one possible implementation, a Kalman filter algorithm can be used to fuse the first coordinate estimate and the second coordinate estimate, specifically including: initializing the state equation and observation equation of the Kalman filter, using the first coordinate estimate and the second coordinate estimate as two observation inputs respectively; calculating the prior state estimate value at the current moment through the state prediction equation, and correcting the prior estimate value by combining it with the observation equation to obtain the posterior state estimate value; iteratively executing the prediction and correction process until the fluctuation of the posterior state estimate value meets the preset stability condition, and using the posterior state estimate value as the three-dimensional position data of the intersection point. In this process, the spatial distance between the first coordinate estimate and the second coordinate estimate can be calculated as the deviation value; if the deviation value does not exceed the preset threshold, the two coordinate estimates (i.e., the first coordinate estimate and the second coordinate estimate) are directly fused; if the deviation value exceeds the preset threshold, a preset scene three-dimensional reconstruction model is called, and the geometric consistency between the two coordinate estimates and the corresponding region of the model is calculated respectively. The coordinate estimate with better geometric consistency is selected, and after correction by combining the effective dimensional information of the other coordinate estimate, the three-dimensional position data of the intersection point is obtained.

[0068] Furthermore, the robot can perform clamping and assembly operations based on the positioning results of the target components. Specifically, this includes: converting the three-dimensional position data to the robot's base coordinate system, planning the motion path from the robot's current pose to the clamping position of the target component, controlling the robot's clamping parts to move along the path and clamp the target component, and completing the sorting or assembly of components according to the preset assembly process.

[0069] The aforementioned component identification and localization method achieves complementary and enhanced perception information under complex working conditions by processing multi-source sensing data, including RGB images, depth maps, and LiDAR point clouds, through spatiotemporal synchronous processing, overcoming the limitations of traditional single vision sensors. By employing a dual-model approach consisting of an instance segmentation network and a real-time segmentation network for collaborative segmentation, it improves the ability to finely segment overlapping and occluded component edges while maintaining processing efficiency, solving the problems of unstable recognition and unclear instance separation in traditional image algorithms under severe occlusion. Furthermore, by implementing dual-path coordinate estimation and fusion, redundant and mutually verified 3D localization is achieved, leveraging the advantages of different modal data to improve the accuracy and reliability of solving the 3D spatial coordinates of intersection points. Therefore, it can achieve highly robust and high-precision intelligent identification and localization of components in complex industrial environments.

[0070] In one exemplary embodiment, such as Figure 2 As shown, the above S04 may include:

[0071] S041, based on the intrinsic parameters of the visual sensor that collects red, green and blue image data, back-project the depth values ​​of the corresponding positions in the two-dimensional position data and the depth map data to obtain the first coordinate estimate of the intersection point.

[0072] S042, Based on two-dimensional location data, candidate point cloud regions are determined from lidar point cloud data.

[0073] S043, perform cluster analysis on the point cloud within the candidate point cloud region, and generate the second coordinate estimate of the intersection point based on the results of the cluster analysis.

[0074] For example, back projection calculations can be performed based on a first formula, which includes:

[0075] , (2)

[0076] in, For 2D image pixel coordinates, For the corresponding three-dimensional spatial coordinates, and For the camera's focal length, and The primary point coordinates. That is, pixel coordinates can be determined based on the camera intrinsic parameter matrix. And its corresponding depth value Z is converted into three-dimensional spatial coordinates. .

[0077] Furthermore, a predetermined cubic region can be delineated within the point cloud data as a candidate point cloud region, centered on the image coordinates corresponding to the two-dimensional location data. Euclidean distance clustering can be used to segment the point cloud within the candidate region, and the centroid coordinates of the largest point cloud cluster can be calculated as the second coordinate estimate. For example, after obtaining the first and second coordinate estimates, a Kalman filter algorithm can be used to fuse the two coordinate estimates. When the difference in Euclidean distance between the first and second coordinate estimates exceeds a preset threshold, the scene model generated based on multi-view 3D reconstruction is activated, and the point with the highest geometric consistency with the two estimated coordinates is retrieved on the model surface as the final 3D localization result.

[0078] In this embodiment, by utilizing the intrinsic parameters of the visual sensor, the two-dimensional pixel coordinates in the image are precisely geometrically back-projected with the depth values ​​of the corresponding positions in the depth map to directly obtain the first coordinate estimate based on visual geometry. Simultaneously, based on the same two-dimensional position, candidate regions are delineated from the LiDAR point cloud and cluster analysis is performed. The statistical features of the main point cloud clusters are extracted to generate the second coordinate estimate. This method leverages the characteristics of the depth map, which provides direct depth information for corresponding pixels and is well aligned with the image, as well as the high accuracy of the LiDAR point cloud itself, which is unaffected by illumination and color. Ultimately, a more accurate and stable three-dimensional positioning result than that obtained by a single sensor is obtained.

[0079] In one exemplary embodiment, such as Figure 3 As shown, the above-mentioned component identification and positioning may further include:

[0080] S06. Based on 3D location data, construct a semantic topology graph. The basic layer of the semantic topology graph is the topology structure layer, which uses intersections as graph nodes and the component connections between intersections as graph edges. The attribute layer of the semantic topology graph is the task parameter layer, which is used to assign task attribute parameters to the nodes and / or edges in the semantic topology graph. The task attribute parameters include at least one of task priority and task timeliness. The task attribute parameters are used to form a semantic weighted task graph.

[0081] The component connection relationships include at least one of physical connection relationships and logical association relationships. Physical connection relationships refer to the physical connections between different components formed through mechanical or conductive structures, including but not limited to wire connections, snap-fit ​​connections, and slot mating. Physical connection relationships can be determined based on the structural characteristics of the target component. Logical association relationships refer to component association relationships formed based on production task requirements, including but not limited to task associations formed by categorizing by tunnel boring machine / TBM project number and process associations formed by the sequence of preset assembly steps. Logical association relationships can be determined based on production task parameters. Operation priority can be determined based on the production task scenario. For example, in a sorting scenario, priority is set based on the urgency of the tunnel boring machine / TBM project number; in an assembly scenario, priority is set based on the preset assembly step sequence of components (components with earlier assembly steps correspond to higher priority). Operation timeliness refers to time constraint parameters that match the production cycle time. Specifically, this includes: for collaborative operation scenarios with the main production line, a task completion threshold that matches the main production line cycle time (used to reduce interference with the main production line cycle time); for material preparation scenarios during non-production periods (such as after get off work hours), a preset material preparation completion time limit.

[0082] For example, graph nodes can be initialized based on the 3D positions of all located intersections, and graph edges can be initialized based on the connectivity of component centerlines. A graph neural network model is used to process the initial topology graph, aggregating the neighborhood features of each node through a message passing mechanism to update its node representation. Using the geometric prior constraints typically satisfied by component layouts, topological consistency checks and corrections are performed on the updated node positions and edge connections. For intersection nodes that were not identified and located during instance segmentation due to occlusion, supplementary nodes are generated and inserted into the graph using graph reasoning techniques based on the established local topology and geometric prior constraints to complete the topology. Then, based on the improved topology, nodes and edges are assigned operation priority and timeliness parameters obtained from industry standards, forming a semantically weighted task graph that can directly guide sorting and assembly operations. Topological consistency correction using geometric prior constraints can include: calculating the grid spacing based on detected intersections; if the local grid spacing variance exceeds a threshold, performing Laplace smoothing on the node positions according to the equal-spacing prior; or detecting component orientation and aligning edges that do not meet the parallel conditions according to the mainstream orientation.

[0083] In this embodiment, by elevating the discrete three-dimensional positions of intersections into a structured semantic topology graph and integrating geometric priors and operational specifications, it is possible to transform the geometric perception results of the physical world into structured and semantic task instructions that can guide robot actions, providing data support for the planning and decision-making of automated and intelligent sorting and assembly operations.

[0084] In one exemplary embodiment, such as Figure 4 As shown, methods for spatiotemporal synchronization processing of multi-source sensing data can include:

[0085] A1, based on hardware trigger signals and software timestamps, performs time alignment on multi-source sensing data, ensuring that the difference between the software timestamps of any two types of sensing data in the multi-source sensing data is no greater than a preset difference.

[0086] A2 divides the time-aligned multi-source sensing data into multiple data frames according to the time series, and unifies the sensing data in each data frame to the same three-dimensional coordinate system.

[0087] For example, the time synchronization accuracy of multimodal data can be expressed as:

[0088] (3)

[0089] in, and These are the timestamps for the i-th and j-th modal data, respectively. This is the threshold for time synchronization accuracy.

[0090] Furthermore, feature point pairs can be extracted from the RGB image and depth map of the same data frame. and And calculate its feature point matching error:

[0091] (4)

[0092] in, and Let be two feature points in three-dimensional space, and d be the Euclidean distance between the two points. When the matching error exceeds a preset threshold, the extrinsic parameter matrices of different sensors can be optimized using the bundle adjustment algorithm, and the spatial attitude of each sensor data can be iteratively adjusted until the matching error meets the accuracy requirements, thereby achieving accurate registration of multimodal data in three-dimensional space.

[0093] Optionally, the constructed topology layer can also be optimized, including using prior geometric constraints of component layout to correct topology consistency, and / or using graph reasoning techniques to complete missed intersection nodes due to occlusion.

[0094] In this embodiment, time alignment is achieved by combining hardware triggering with software timestamps, and unified spatial coordinates are achieved based on feature point matching, which enables consistency of multi-source heterogeneous sensing data in the spatiotemporal dimension.

[0095] In one exemplary embodiment, such as Figure 5 As shown, the above-mentioned component identification and positioning may further include:

[0096] S07, input the multi-source sensing data into the pre-trained cross-modal converter model so that the cross-modal converter model can perform semantic completion of occluded areas in red-green-blue image data, geometric completion of occluded areas in depth map data, or noise suppression of reflective areas in red-green-blue image data.

[0097] For example, time-synchronized RGB images, depth maps, and LiDAR point cloud data can be jointly input into a cross-modal converter model. This model, through its internal multi-head attention mechanism, calculates the similarity and association weights of features between different modalities, achieving cross-modal feature interaction and fusion. Based on the enhanced features after fusion, it can predict and fill in the missing texture and semantic information in the RGB image due to occlusion, completing semantic completion; simultaneously, using complete semantic context information, it infers and repairs the geometric structure and depth values ​​of the corresponding occluded areas in the depth map, completing geometric completion. For overexposed or noisy areas in the RGB image caused by metallic reflection, the model can suppress high-response features from reflective areas and enhance effective features from normally lit areas, achieving noise suppression and visual feature enhancement.

[0098] In this embodiment, high-quality and complete multimodal data is obtained after model processing. Combined with multi-view registration and 3D reconstruction techniques based on feature point matching and bundle adjustment, the spatial consistency of the data in the temporal dimension can be further optimized.

[0099] In one exemplary embodiment, such as Figure 6 As shown, the above-mentioned component identification and positioning may further include:

[0100] S081, based on three-dimensional position data, controls the movement of the clamping component toward the intersection point of the target component, and determines the distance between the clamping component and the intersection point based on a vision sensor.

[0101] S082, when the rate of change of distance is less than the preset rate of change, obtain the coordinates of the first center point of the intersection and the coordinates of the second center point of the end of the clamping component.

[0102] S083, calculate the spatial offset between the coordinates of the first center point and the coordinates of the second center point, and if the spatial offset is greater than the preset offset, generate a pose correction amount based on the magnitude and direction of the spatial offset. The pose correction amount is used to convert into joint motion commands for the robot to control the gripping component to move in the direction of reducing the spatial offset until the spatial offset stabilizes within the preset error range.

[0103] For example, the control gain can be dynamically adjusted based on the magnitude of the spatial offset. In one possible implementation, the clamping component can be controlled to move towards the target intersection point based on three-dimensional position data, and the three-dimensional spatial distance between the end of the clamping component and the intersection point can be continuously determined by a vision sensor; the rate of change of this distance per unit time is calculated. When the rate of change is less than a preset positive threshold, the system is determined to enter the alignment preparation state; in this state, the three-dimensional coordinates of the first center point of the intersection point and the three-dimensional coordinates of the second center point of the end of the clamping component are acquired in real time; the spatial offset δ between the coordinates of the first center point and the coordinates of the second center point is calculated; when the magnitude of the spatial offset is greater than the preset offset threshold ε, a pose correction amount is generated based on the direction vector and magnitude of the spatial offset, and its calculation relationship can be expressed as:

[0104] (5)

[0105] in, is the proportional gain coefficient, and u is the unit direction vector from the end of the clamping component to the intersection point.

[0106] Furthermore, the pose correction is converted into angle adjustment commands in joint space using the robot's inverse kinematics solver, driving the gripping component to move in the direction that reduces the spatial offset. During this process, the control gain is dynamically adjusted according to the magnitude of the offset, specifically: when... At that time, high gain was used. To achieve rapid approximation; when When switching to low gain To avoid overshoot and oscillation, the occlusion area of ​​the intersection point in the image is monitored synchronously. If the occlusion area exceeds 30%, a multi-view fusion strategy is activated, and a lateral camera is used for assisted positioning to ensure the reliability of the feedback signal. This closed-loop control process continues until the magnitude of the spatial offset is stably less than the threshold for N consecutive control cycles. If so, then the alignment is considered successful.

[0107] Furthermore, such as Figure 7 As shown, the above S081 may include:

[0108] B1, based on the robot's hand-eye calibration relationship, converts the three-dimensional position data to the robot's base coordinate system.

[0109] B2 generates accessibility judgment results for 3D position data based on a pre-built accessibility assessment model, which is established based on the kinematic model of the robotic arm.

[0110] B3, if the accessibility judgment result indicates that the intersection point is within the movement range of the robotic arm, based on the three-dimensional position data, control the gripping component to move towards the intersection point of the target component.

[0111] For example, based on a pre-calibrated hand-eye transformation matrix, the 3D position data of the intersection point in the camera coordinate system can be transformed to the robot's base coordinate system through homogeneous coordinate transformation. A pre-built reachability assessment model is then used to evaluate the transformed 3D position data. This model is established based on the kinematic parameters of the robotic arm (e.g., link length, joint angle limits, etc.) and the workspace envelope. Its output is a Boolean value, indicating whether the target point is within the robotic arm's non-singular movement range. If the reachability assessment result is true, a joint trajectory from the robot's current pose to the target pose is generated based on the 3D position data and combined with a path planning algorithm. The gripping component is then controlled to move along this trajectory towards the target intersection point. If the assessment result is false, the intersection point is marked as unreachable and reported to the task scheduling system for anomaly handling or task reassignment.

[0112] In this embodiment, by transforming the abstract gripping action into a quantifiable approach and alignment process based on a vision state machine, and introducing a dynamic negative feedback control mechanism, refined, autonomous, and high-precision control of the sorting and assembly processes can be achieved. Furthermore, by accurately transforming the visual perception coordinates to the robot's base coordinate system and introducing a kinematic model-based accessibility assessment, feasibility prediction and error prevention can be achieved before task execution.

[0113] In one possible implementation, scene data can be simultaneously acquired using multiple sensors (including RGB cameras, depth cameras, LiDAR, and IMU) to achieve robust perception of complex environments (e.g., overlapping components, reflections, occlusions). The texture information of the RGB images, the geometric information of the depth maps, the 3D structural information of the LiDAR, and the motion information of the IMU are then spatiotemporally synchronized and fused. Specifically, a Transformer-based cross-modal model can be used to perform semantic completion and noise suppression on occluded and reflective areas in the RGB images. Simultaneously, multi-view registration and bundle adjustment techniques are employed to ensure that all perceived data remain consistent in time and space. Then, a real-time segmentation network (ensuring efficiency) and a high-precision instance segmentation network (ensuring accuracy) work together to process the RGB image and accurately extract the instance mask of each component. Morphological refinement and fitting are performed on the segmentation mask to extract the component centerlines, and the intersection of the centerlines is found in the 2D image using Hough transform. The 2D intersections are back-projected to 3D space using the camera intrinsic parameters (K) through the depth map to obtain the first coordinate estimate. The corresponding region is searched in the LiDAR point cloud, and the second coordinate estimate is obtained through cluster analysis. The two sets of data are fused to output the precise 3D position data of the intersections. A topology graph is constructed using the located intersections as nodes and the component connections as edges. Furthermore, operation parameters such as task priority and timeliness are incorporated to form a semantically weighted task graph. Feasibility constraint analysis is then performed based on the robotic arm kinematics model to determine whether the gripping point is reachable. Finally, a visually discernible process map is established for the sorting and assembly process; the movement of the clamping component towards the target is controlled, and the rate of change of the distance between the end and the intersection point is calculated in real time. When the rate of change decreases and tends to stabilize, it is considered that it can be approached; the spatial offset between the component intersection point and the end of the clamping component is calculated. Once the spatial offset exceeds the threshold, the negative feedback control loop is immediately activated to generate the pose correction amount, drive the clamping component to dynamically fine adjust until the offset stabilizes within the error range, and the alignment is completed.

[0114] In summary, the aforementioned component identification and localization method achieves complementary and enhanced perception information under complex working conditions by processing multi-source sensing data, including RGB images, depth maps, and LiDAR point clouds, in a spatiotemporally synchronized manner, overcoming the limitations of traditional single-vision sensors. By employing a dual-model approach consisting of an instance segmentation network and a real-time segmentation network for collaborative segmentation, it improves the ability to finely segment overlapping and occluded component edges while maintaining processing efficiency, solving the problems of unstable recognition and unclear instance separation in traditional image algorithms under severe occlusion. Furthermore, by implementing dual-path coordinate estimation and fusion, it achieves redundant and mutually verified 3D localization, leveraging the advantages of different modal data to improve the accuracy and reliability of solving the 3D spatial coordinates of intersection points. Therefore, it can achieve highly robust and high-precision intelligent identification and localization of components in complex industrial environments.

[0115] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0116] Based on the same inventive concept, this application also provides a component identification and positioning device for implementing the component identification and positioning method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more component identification and positioning device embodiments provided below can be found in the limitations of the component identification and positioning method described above, and will not be repeated here.

[0117] In one exemplary embodiment, such as Figure 8 As shown, a component identification and positioning device 10 is provided, including: a sensing data acquisition module 11, an instance segmentation module 12, a two-dimensional position acquisition module 13, a coordinate estimation module 14, and a three-dimensional position acquisition module 15, wherein:

[0118] The sensing data acquisition module 11 is used to acquire multi-source sensing data of the target component. The multi-source sensing data is processed by spatiotemporal synchronization and includes red-green-blue image data, depth map data and lidar point cloud data.

[0119] The instance segmentation module 12 is used to obtain the instance segmentation mask of the target component based on red-green-blue image data through a pre-trained instance segmentation model. The instance segmentation model includes an instance segmentation network and a real-time segmentation network. The instance segmentation network is used to generate the instance mask of the component, and the real-time segmentation network is used to extract features of the component and segment the instance, and to provide feature enhancement for the instance segmentation network.

[0120] The two-dimensional position acquisition module 13 is used to acquire the two-dimensional position data of the intersection of the center lines of the target component based on the instance segmentation mask;

[0121] The coordinate estimation module 14 is used to generate a first coordinate estimate of the intersection point based on two-dimensional location data and depth map data, and to generate a second coordinate estimate of the intersection point based on two-dimensional location data and lidar point cloud data.

[0122] The three-dimensional position acquisition module 15 is used to acquire the three-dimensional position data of the intersection point based on the first coordinate estimation and the second coordinate estimation, so as to obtain the positioning result of the target component.

[0123] In one embodiment, the coordinate estimation module 14 is further configured to:

[0124] Based on the intrinsic parameters of the visual sensor that collects red, green and blue image data, the depth values ​​of corresponding positions in the two-dimensional position data and the depth map data are back-projected to obtain the first coordinate estimate of the intersection point.

[0125] Candidate point cloud regions are determined from lidar point cloud data based on two-dimensional location data.

[0126] Cluster analysis is performed on the point clouds within the candidate point cloud region, and the second coordinate estimate of the intersection point is generated based on the results of the cluster analysis.

[0127] In one embodiment, the component identification and positioning device 10 further includes:

[0128] The semantic topology graph construction module is used to construct a semantic topology graph based on 3D location data. The basic layer of the semantic topology graph is the topology structure layer, which uses intersections as graph nodes and the component connections between intersections as graph edges. The attribute layer of the semantic topology graph is the task parameter layer, which is used to assign task attribute parameters to the nodes and / or edges in the semantic topology graph. The task attribute parameters include at least one of task priority and task timeliness. The task attribute parameters are used to form a semantic weighted task graph.

[0129] In one embodiment, the component identification and positioning device 10 further includes a sensing data processing module, used for:

[0130] Based on hardware trigger signals and software timestamps, time alignment is performed on multi-source sensing data to ensure that the difference between the software timestamps of any two sensing data in the multi-source sensing data is no greater than a preset difference.

[0131] The time-aligned multi-source sensing data is divided into multiple data frames according to the time series, and the sensing data in each data frame is unified to the same three-dimensional coordinate system.

[0132] In one embodiment, the aforementioned sensing data processing module is further configured to:

[0133] Multi-source sensing data is input into a pre-trained cross-modal converter model, enabling the cross-modal converter model to perform semantic completion of occluded regions in red-green-blue image data, geometric completion of occluded regions in depth map data, or noise suppression of reflective regions in red-green-blue image data.

[0134] In one embodiment, the component identification and positioning device 10 further includes a motion control module, used for:

[0135] Based on three-dimensional position data, the clamping component is controlled to move towards the intersection point of the target component, and the distance between the clamping component and the intersection point is determined based on the vision sensor.

[0136] If the rate of change of distance is less than the preset rate of change, obtain the coordinates of the first center point of the intersection and the coordinates of the second center point of the end of the clamping component.

[0137] Calculate the spatial offset between the coordinates of the first center point and the coordinates of the second center point. If the spatial offset is greater than a preset offset, generate a pose correction amount based on the magnitude and direction of the spatial offset. The pose correction amount is used to convert into joint motion commands for the robot to control the gripping component to move in the direction that reduces the spatial offset until the spatial offset stabilizes within the preset error range.

[0138] In one embodiment, the motion control module is further configured to:

[0139] Based on the robot's hand-eye alignment relationship, the 3D position data is converted to the robot's base coordinate system;

[0140] Based on a pre-built accessibility assessment model, accessibility judgment results of 3D position data are generated. The accessibility assessment model is established based on the kinematic model of the robotic arm.

[0141] If the accessibility assessment result indicates that the intersection point is within the movement range of the robotic arm, the gripping component is controlled to move towards the intersection point of the target component based on the three-dimensional position data.

[0142] Each module in the aforementioned component identification and positioning device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the electronic device in hardware form or independent of it, or stored in the memory of the electronic device in software form, so that the processor can call and execute the corresponding operations of each module.

[0143] In one exemplary embodiment, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0144] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0145] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0146] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0147] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0148] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for identifying and locating electronic components, characterized in that, The method includes: Acquire multi-source sensing data of the target component. The multi-source sensing data is processed by spatiotemporal synchronization. The multi-source sensing data includes red-green-blue image data, depth map data, and lidar point cloud data. Based on the red-green-blue image data, an instance segmentation mask of the target component is obtained through a pre-trained instance segmentation model; wherein, the instance segmentation model includes an instance segmentation network and a real-time segmentation network, the instance segmentation network is used to generate the instance mask of the component, and the real-time segmentation network is used to perform feature extraction and instance segmentation of the component and to provide feature enhancement for the instance segmentation network. Based on the instance segmentation mask, obtain the two-dimensional position data of the intersection point of the center lines of the target component; Based on the two-dimensional location data and the depth map data, a first coordinate estimate of the intersection point is generated, and based on the two-dimensional location data and the lidar point cloud data, a second coordinate estimate of the intersection point is generated. Based on the first coordinate estimation and the second coordinate estimation, the three-dimensional position data of the intersection point is obtained, and the positioning result of the target component is obtained.

2. The method according to claim 1, characterized in that, The step of generating a first coordinate estimate of the intersection point based on the two-dimensional location data and the depth map data, and generating a second coordinate estimate of the intersection point based on the two-dimensional location data and the lidar point cloud data, includes: Based on the intrinsic parameters of the visual sensor that collects the red, green and blue image data, the depth values ​​of the corresponding positions in the two-dimensional position data and the depth map data are back-projected to obtain the first coordinate estimate of the intersection point; Based on the two-dimensional location data, candidate point cloud regions are determined from the lidar point cloud data; Cluster analysis is performed on the point cloud within the candidate point cloud region, and a second coordinate estimate of the intersection point is generated based on the results of the cluster analysis.

3. The method according to claim 1, characterized in that, After obtaining the three-dimensional position data of the intersection point based on the first coordinate estimation and the second coordinate estimation, the process includes: Based on the three-dimensional location data, a semantic topology graph is constructed. The foundation layer of the semantic topology graph is a topology structure layer, which uses intersections as graph nodes and the connection relationships between components between intersections as graph edges. The attribute layer of the semantic topology graph is a task parameter layer, which assigns task attribute parameters to nodes and / or edges in the semantic topology graph. These task attribute parameters include at least one of task priority and task timeliness, and are used to form a semantic weighted task graph.

4. The method according to claim 1, characterized in that, The methods for spatiotemporal synchronization processing of the multi-source sensing data include: Based on hardware trigger signals and software timestamps, the multi-source sensing data is time-aligned so that the difference between the software timestamps of any two types of sensing data in the multi-source sensing data is no greater than a preset difference. The time-aligned multi-source sensing data is divided into multiple data frames according to the time series, and the sensing data in each data frame is unified to the same three-dimensional coordinate system.

5. The method according to claim 1, characterized in that, After acquiring the multi-source sensing data of the target component, the following is included: The multi-source sensing data is input into a pre-trained cross-modal converter model, which enables the cross-modal converter model to perform semantic completion of occluded regions in the red-green-blue image data, geometric completion of occluded regions in the depth map data, or noise suppression of reflective regions in the red-green-blue image data.

6. The method according to claim 1, characterized in that, After obtaining the three-dimensional position data of the intersection point based on the first coordinate estimation and the second coordinate estimation, the process includes: Based on the three-dimensional position data, the clamping component is controlled to move towards the intersection point of the target component, and the distance between the clamping component and the intersection point is determined based on the vision sensor. If the rate of change of the distance is less than a preset rate of change, obtain the coordinates of the first center point of the intersection and the coordinates of the second center point of the end of the clamping component; Calculate the spatial offset between the coordinates of the first center point and the coordinates of the second center point. If the spatial offset is greater than a preset offset, generate a pose correction amount based on the magnitude and direction of the spatial offset. The pose correction amount is used to convert into joint motion commands for the robot to control the gripping component to move in the direction that reduces the spatial offset until the spatial offset stabilizes within a preset error range.

7. The method according to claim 6, characterized in that, The step of controlling the clamping component to move towards the intersection point of the target component based on the three-dimensional position data includes: Based on the robot's hand-eye alignment, the three-dimensional position data is converted to the robot's base coordinate system; Based on a pre-built accessibility assessment model, accessibility judgment results of the three-dimensional position data are generated. The accessibility assessment model is established based on the kinematic model of the robotic arm. If the accessibility determination result indicates that the intersection point is within the range of motion of the robotic arm, the clamping component is controlled to move toward the intersection point of the target component based on the three-dimensional position data.

8. A component identification and positioning device, characterized in that, The device includes: The sensing data acquisition module is used to acquire multi-source sensing data of the target component. The multi-source sensing data is processed by spatiotemporal synchronization and includes red-green-blue image data, depth map data and lidar point cloud data. The instance segmentation module is used to obtain the instance segmentation mask of the target component based on the red-green-blue image data through a pre-trained instance segmentation model; wherein, the instance segmentation model includes an instance segmentation network and a real-time segmentation network, the instance segmentation network is used to generate the instance mask of the component, and the real-time segmentation network is used to perform feature extraction and instance segmentation of the component and provide feature enhancement for the instance segmentation network. A two-dimensional position acquisition module is used to acquire two-dimensional position data of the intersection of the center lines of the target component based on the instance segmentation mask; The coordinate estimation module is used to generate a first coordinate estimate of the intersection point based on the two-dimensional position data and the depth map data, and to generate a second coordinate estimate of the intersection point based on the two-dimensional position data and the lidar point cloud data. The three-dimensional position acquisition module is used to acquire the three-dimensional position data of the intersection point based on the first coordinate estimation and the second coordinate estimation, and obtain the positioning result of the target component.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images