A decision method and system for mechanical arm action

Abstract

The application discloses a decision method and system for mechanical arm action. The method comprises the following steps: obtaining a task path of a mechanical arm, dividing the task path into a plurality of alternately arranged fixed execution sections and key areas; controlling the mechanical arm to execute according to the task path, when the mechanical arm enters a key area, continuously obtaining N image frames corresponding to the key area, performing sensing processing on each image frame to obtain a sensing result corresponding to each image frame; and based on the sensing result, determining a confidence degree corresponding to each image frame, and determining a confidence degree analysis result according to the obtained confidence degree; presetting a plurality of action modes and corresponding action mapping data relationships, determining a target action mode corresponding to the confidence degree analysis result based on the action mapping data relationships; and controlling the mechanical arm to execute a corresponding action according to the target action mode. The application can reduce unnecessary continuous sensing and real-time calculation, reduce the calculation overhead and improve the task execution efficiency, thereby adapting to the timing requirements of an industrial assembly line.

Description

Technical Field

[0001] This application relates to the field of industrial robot control technology, and in particular to a decision-making method and system for robotic arm movements. Background Technology

[0002] With the development of flexible manufacturing and intelligent automation technologies, robotic arms are widely used in industrial scenarios such as random material sorting and precision assembly. These operations typically require robotic arms to maintain high motion efficiency while also adapting to changes in material position, posture, and the status of upstream and downstream processes, placing higher demands on the real-time performance, adaptability, and deployment cost of robotic arm control methods.

[0003] Existing robotic arm control methods mainly include pre-programmed reproduction and online perception and real-time planning based on sensors such as vision. The former pre-generates a complete motion trajectory and executes according to the trajectory; the latter continuously acquires environmental information during the robotic arm's execution and performs positioning, trajectory generation, or trajectory correction based on the perception results.

[0004] However, pre-programmed methods struggle to adapt to changes in the position, posture, or environmental state of the work object, while continuous online sensing and real-time planning increase computational latency and power requirements, impacting the robotic arm's operating cycle time. Therefore, existing methods struggle to balance execution efficiency, environmental adaptability, and deployment costs. Summary of the Invention

[0005] This application aims to address the problems in existing robotic arm control processes, such as high computational overhead, low execution efficiency, inability to meet pipeline performance requirements, and difficulty in adaptively determining trajectory execution strategies based on the reliability of perception results when uncertain states exist during operation.

[0006] To solve the above-mentioned technical problems, this application provides the following technical solution: Firstly, this application provides a method for making decisions on robotic arm movements, including: Obtain the task path of the robotic arm and divide the task path into several alternating fixed execution segments and key areas; The robotic arm is controlled to execute according to the task path. When the robotic arm enters the key area, N image frames corresponding to the key area are continuously acquired. The image frames are processed to obtain the perception results corresponding to each image frame. The perception results include target position information, target preparation status information and target operation status information. Based on the perception results corresponding to each of the image frames, the confidence level corresponding to each of the image frames is determined, and the confidence level analysis result is determined based on the obtained confidence level. Several motion patterns and corresponding motion mapping data relationships are preset. Based on the motion mapping data relationships, a target motion pattern corresponding to the confidence analysis result is determined. The robotic arm is then controlled to perform the corresponding motion according to the target motion pattern. The motion mapping data relationships are used to indicate the correspondence between the confidence analysis result and the motion pattern.

[0007] As one feasible approach, dividing the task path into fixed execution segments and key regions includes: The path segments that do not require adjustment of the robotic arm's movements based on the perception results are defined as the fixed execution segments; The path segments for determining the robotic arm's movements based on the perception results are defined as the critical regions.

[0008] As one feasible approach, determining the confidence level corresponding to each of the image frames based on the perception results corresponding to each of the image frames includes: The confidence level of the corresponding image frame is determined based on the target detection score, the positional deviation between the target detection location and the expected location, and the target state judgment score.

[0009] As an implementable approach, the confidence level satisfies: , Where C is the confidence level calculation result, S det The target detection score is denoted by ΔP, where ΔP is the deviation between the detected target location and the desired location, T is the maximum tolerance deviation, and S is the target detection score. state α is the target state judgment score; β is the target detection score weight, β is the position deviation weight, and γ is the target state judgment score weight, and α+β+γ=1.

[0010] As an implementable approach, the various action modes include a fixed trajectory execution mode, a trajectory fine-tuning mode, a local trajectory replanning mode, and an exception handling mode.

[0011] As an implementable method, determining the confidence analysis result based on the obtained confidence level includes: In a sliding window consisting of N consecutively acquired image frames, determine whether there exist M consecutive image frames whose confidence scores are all greater than a preset confidence threshold, where M≤N; The confidence analysis results are determined based on the M consecutive image frames.

[0012] As one feasible approach, determining the confidence analysis result based on the M consecutive image frames includes: Obtain the confidence scores corresponding to the M image frames respectively; Calculate the average confidence level corresponding to each of the M image frames, and determine the average confidence level as the confidence level analysis result.

[0013] Secondly, this application provides a decision-making system for robotic arm movements, including: The task configuration module is used to obtain the task path of the robotic arm and divide the task path into several alternating fixed execution segments and key areas. The perception module is used to control the robotic arm to execute according to the task path. When the robotic arm enters the key area, it continuously acquires N image frames corresponding to the key area, performs perception processing on each image frame, and obtains the perception result corresponding to each image frame. The perception result includes target position information, target preparation status information, and target operation status information. The confidence analysis module is used to determine the confidence level of each image frame based on the perception results corresponding to each image frame, and to determine the confidence analysis result based on the obtained confidence level. The decision module is used to preset several action modes and corresponding action mapping data relationships, determine the target action mode corresponding to the confidence analysis results based on the action mapping data relationships, and control the robotic arm to perform the corresponding action according to the target action mode; the action mapping data relationships are used to indicate the correspondence between several confidence analysis results and action modes.

[0014] This application, by adopting the above technical solution, has significant technical effects: By dividing the robotic arm's task path into fixed execution segments and key areas, the robotic arm operates according to a predetermined task path in the fixed execution segments, and only triggers the perception model for perception processing when entering the key areas. This reduces unnecessary continuous perception and real-time calculations, thereby reducing the computational overhead in the robotic arm control process, improving task execution efficiency, and adapting to the timing requirements of industrial production lines. At the same time, by performing confidence analysis on multiple image frames continuously acquired within the key areas, and determining the confidence analysis results based on multiple image frames that continuously meet the preset confidence conditions, the impact of single-frame perception errors or occasional interferences on the decision-making results can be reduced, improving the stability and reliability of the judgment of the current operation status. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating a decision-making method for robotic arm movements provided in this application; Figure 2 A schematic diagram of the structure of a decision system for robotic arm movements provided in this application. Detailed Implementation

[0017] The present application will be further described in detail below with reference to the embodiments. The following embodiments are explanations of the present application, but the present application is not limited to the following embodiments.

[0018] Currently, robotic arms are widely used in industrial automation scenarios such as material handling, random material sorting, workpiece assembly, and multi-process collaborative operations. In these applications, robotic arms typically need to complete multiple consecutive work steps according to a pre-planned motion trajectory. For work steps where the material position and the status of upstream and downstream processes are relatively stable, rapid execution can be achieved through a pre-set motion trajectory. However, for steps such as handling, assembly, or material handover between upstream and downstream processes, the position, posture, preparation status, and working status of the materials may change. If the robotic arm still executes exactly according to the preset trajectory, situations such as handling position deviation, unmet assembly conditions, or mismatch between the current process and the status of upstream and downstream processes may occur. On the other hand, if the robotic arm continuously performs visual perception and adjusts the motion trajectory in real time throughout the entire task execution process, it will increase the operational burden of the perception model and trajectory calculation, thereby affecting the overall operating efficiency of the robotic arm.

[0019] To address the challenge of balancing the efficiency and environmental adaptability of robotic arms, this application provides a decision-making method for robotic arm movements. This method aims to reduce unnecessary perception calculations while improving the reliability of robotic arm decisions when faced with changes in material position, changes in process preparation status, or abnormal operating conditions.

[0020] like Figure 1 As shown, the decision-making method for robotic arm movements provided in this application may include the following steps.

[0021] S101. Obtain the task path of the robotic arm and divide the task path into several alternating fixed execution segments and key areas.

[0022] In this step, the task path can be the sequential motion path that the robotic arm traverses when performing a complete production task, along with the corresponding operational steps at each path position. The task path can be pre-established before the task begins, based on the robotic arm's workspace, target workstation location, material conveying direction, and the execution sequence between processes. The task path can include not only the spatial locations that the robotic arm's end effector needs to traverse, but also the corresponding task stages such as grasping, moving, placing, assembling, inspecting, and waiting for upstream materials. Therefore, the task path is not divided mechanically based solely on path length or spatial location, but rather by considering whether the robotic arm needs to determine subsequent actions based on current sensing results when traversing the corresponding path, thus dividing the task path into fixed execution segments and critical areas.

[0023] In one implementation, the task path is divided into fixed execution segments and critical regions, including: dividing the path segments that do not require adjustment of the robotic arm's movements based on the perception results into fixed execution segments; and dividing the path segments that require determination of the robotic arm's movements based on the perception results into critical regions.

[0024] It should be noted that a fixed execution segment can be a path segment in which the working environment, the moving object, and the moving target are relatively determined, and the robotic arm can complete the corresponding action according to the pre-generated motion trajectory.

[0025] For example, once the robotic arm has completed material grasping and the target placement station is fixed, the intermediate path from the grasping position to the target station can be considered a fixed execution segment. Within this segment, the robotic arm primarily operates according to a pre-planned trajectory and corresponding motion parameters such as speed and acceleration, eliminating the need to continuously call upon the visual perception model to assess the external environment, thereby reducing the computational load during task execution.

[0026] It should be noted that the critical area corresponds to the region where uncertainties exist during the robotic arm's task execution, and the path for subsequent actions needs to be determined based on real-time sensing results. Uncertainties can arise from changes in the target position or from changes in the state between upstream and downstream processes.

[0027] For example, before the grasping process, it is necessary to confirm whether the material to be grasped has reached the preset position; before the assembly process, it is necessary to confirm whether the component to be assembled is ready; or when assembling components such as nuts and bolts, it is necessary to confirm whether the target material already has the corresponding component and whether the current state of the material is normal. Locations with upstream and downstream logistics interactions, and those requiring re-detection due to the uncertainty of upstream and downstream states, are designated as key points, and the spatial range required for continuous shooting is further defined as key areas.

[0028] In the above steps, by dividing the fixed execution segment and key area, the trigger position of the perception model can be determined in advance before the task starts, so that the perception calculation is focused on the path area that really affects the task result, without having to continuously perform perception on the complete task path, reducing the operational burden of the perception model and trajectory calculation, and improving the overall efficiency of the robotic arm operation.

[0029] S102. Control the robotic arm to execute according to the task path. When the robotic arm enters the critical area, continuously acquire N image frames corresponding to the critical area, perform perception processing on each image frame, and obtain the perception result corresponding to each image frame. Among them, the perception result includes target position information, target preparation status information, and target operation status information.

[0030] In this step, after configuring the task path, fixed execution segments, and key areas, the robotic arm can begin executing the task according to the task path. When the robotic arm is in a fixed execution segment, it runs according to a pre-planned and stored motion trajectory without continuously calling the visual perception model related to the current task. The system can determine whether the robotic arm has entered a pre-configured key area based on the current end-effector position, the current task stage, or the progress of the current path execution. When the robotic arm enters a key area, the perception model corresponding to that key area is triggered, thus limiting the perception operation to the task stage that requires status confirmation.

[0031] It should be noted that different perception models can be configured for different key areas. The perception model can be pre-trained and configured according to the operational status that needs to be confirmed in that key area.

[0032] For example, one key area is mainly used to determine the spatial location of materials, and the corresponding perception model can output the location of the target material; another key area is mainly used to determine whether the upstream process has been completed, and the perception model can output the status result of whether the material is ready to be completed; at the assembly station, the perception model can be used to determine whether the material has missing parts, abnormal posture, or other states that do not meet the current operation requirements.

[0033] Understandably, in order to meet the requirements of lightweight deployment, the perception model can adopt a visual model that has been pruned, parameter compressed, or computationally simplified, so that it can be quickly invoked in key areas without occupying high computing resources for a long time.

[0034] Furthermore, since a single image is easily affected by short-term occlusion, changes in lighting, vibrations from the robotic arm, or instantaneous changes in the target's posture, this application does not acquire only one image frame after the robotic arm enters the critical area, but rather acquires N image frames continuously within a preset system time range. Adjacent image frames can be acquired according to a preset acquisition cycle. Those skilled in the art can configure the acquisition cycle based on the robotic arm's operating speed, the imaging speed of the vision device, and the spatial range of the critical area, rather than limiting it to a fixed value. The perception model processes each image frame separately, thus each image frame can generate a corresponding perception result.

[0035] It should be noted that the perception results include target location information, target readiness status information, and target operation status information. Target location information can include the spatial position of the target material in the robotic arm coordinate system; target readiness status information can be used to indicate whether the target material or upstream process has met the execution conditions of the current task; target operation status information can be used to indicate whether the current structure, posture, or component status of the material meets the preset operation requirements.

[0036] For example, in a component assembly task, continuous images can be used to determine whether the component to be assembled has been sent to the designated area by the upstream equipment, whether the actual position deviates from the expected position, and whether the structure to be confirmed on the target material is in a normal state.

[0037] S103. Based on the perception results corresponding to each image frame, determine the confidence level corresponding to each image frame, and determine the confidence level analysis result based on the obtained confidence level.

[0038] It should be noted that this step is used to convert the multidimensional perception results output by the perception model for each image frame into a confidence level that can characterize the reliability of the current robotic arm's operating conditions, and further use the changes in confidence level of multiple consecutive image frames to determine whether the current operating state has sufficient stability.

[0039] Understandably, unlike directly controlling the robotic arm based solely on the detection results of a single image frame, this application first determines the confidence level for each image frame, and then forms the final confidence level analysis result based on whether multiple consecutive image frames stably meet the preset conditions, thereby reducing the impact of single-frame false detections on the robotic arm's action decisions.

[0040] In one implementation, determining the confidence level for each image frame based on the perception results corresponding to each image frame includes: Based on the target detection score, the positional deviation between the detected target location and the expected location, and the target state judgment score, the confidence level of the corresponding image frame is determined, and the confidence level satisfies: , Where C is the confidence level, Sdet The target detection score is denoted by ΔP, where ΔP is the deviation between the detected target location and the desired location, T is the maximum tolerance deviation, and S is the target detection score. state α is the target state judgment score; β is the target detection score weight, β is the position deviation weight, and γ is the target state judgment score weight, and α+β+γ=1.

[0041] It should be noted that by imposing an upper limit constraint on the ratio of position deviation to the maximum tolerance deviation, the confidence component corresponding to the position deviation is kept within a preset value range. When the position deviation is less than the maximum tolerance deviation, the smaller the position deviation, the higher the corresponding confidence component. When the position deviation reaches or exceeds the maximum tolerance deviation, the confidence component corresponding to the position deviation takes the lowest value, thereby avoiding the position deviation from being too large and causing the component to generate a value that exceeds the preset range.

[0042] For example, the object detection score S det It is obtained by performing target detection on the image frame by the perception model corresponding to the current key region, and is used to characterize the confidence level of the perception model in that the detected target belongs to the preset target category.

[0043] For example, the deviation ΔP between the detected target position and the desired position is obtained as follows: The detected target position in the robotic arm coordinate system is determined based on the target position information output by the perception model, and the desired position is obtained from the preset task parameters of the current task path or the current key area; then, the spatial distance between the detected target position and the desired position is calculated, and this spatial distance is determined as the position deviation. As one implementation method, Euclidean distance can be used as the deviation between the detected target position and the desired position.

[0044] For example, the maximum tolerance deviation T can be preset during the task configuration phase. Those skilled in the art can determine it based on at least one of the following: the operational accuracy requirements of the current process, the target material size, and the sensing accuracy; furthermore, the higher the process accuracy requirement, the smaller the corresponding maximum tolerance deviation can be set. Different maximum tolerance deviations can be configured for different critical areas.

[0045] For example, a mapping relationship between target states and corresponding target state scores can be pre-established, where different target states correspond to preset state scores. Target states may include target preparation state, target alignment state, component presence state, and other states related to the current operation. A mapping table between each target state and its corresponding state score can be pre-established. When the perception model identifies multiple target states corresponding to the current target, it retrieves the corresponding state scores from the mapping table and accumulates the state scores to obtain the target state judgment score. To ensure that the target state judgment score and the target detection score are within a unified numerical range, the accumulated state scores are normalized, and the normalized result is used as the target state judgment score S. state .

[0046] For example, weighting coefficients can be set separately for different key areas during the task configuration phase and stored in the corresponding task configuration parameters. When the robotic arm enters the corresponding key area, the weighting coefficient for that key area is read. Those skilled in the art can determine the weighting coefficients based on the target detection reliability, position accuracy, and the importance of the target state to the robotic arm's motion decisions in the current key area.

[0047] Furthermore, weighting coefficients can be set for target detection scores, positional deviations, and status judgment scores for different key areas. Different key areas can have different weights determined based on the task objective they correspond to. For example, the pre-grabbing positioning area can have a higher weight for positional deviation factors, while the key area used to confirm whether materials are ready can have a higher weight for status judgment factors. These weighting parameters can be stored in the task configuration data and read when the robotic arm enters the corresponding key area.

[0048] In one implementation, determining the confidence analysis result based on the obtained confidence level includes: In a sliding window consisting of N consecutively acquired image frames, determine whether there are M consecutive image frames whose confidence scores are all greater than a preset confidence threshold, where M≤N; and determine the confidence analysis result based on the M consecutive image frames.

[0049] Specifically, N image frames can be arranged in chronological order to form a sliding window for state stability analysis. The system sequentially calculates the confidence level for each image frame in the window and checks whether there are M consecutive image frames whose confidence levels all exceed a preset confidence threshold. If only a few scattered image frames have confidence levels exceeding the threshold, while there are image frames in between that do not reach the threshold, it can be considered that the current perception state has not yet reached the condition of continuous stability, and it will not be immediately used as the basis for the robot arm's subsequent action decisions.

[0050] Understandably, using consecutive M frames instead of single frames for judgment can prevent the robotic arm from immediately changing its motion mode when an image frame has an abnormal confidence level due to momentary occlusion, local reflection, slight material movement, or visual noise. For example, if the detection confidence level is low at a certain moment, but the preceding and following images show that the target state is normal, performing local trajectory replanning based solely on that abnormal image may lead to unnecessary motion adjustments. Continuous frame judgment can filter out occasional anomalies by utilizing the consistency of the current state over time.

[0051] In another implementation, if no M consecutive image frames meeting the conditions are found in the current sliding window, the system can continue to acquire subsequent image frames and remove earlier image frames as new image frames enter, thereby forming a new sliding window to continue the judgment. Considering that the completion time of upstream processes may be uncertain, a fixed upper limit for status confirmation time may not be mandatory, but abnormal states that continuously fail to meet the conditions can be recorded and output as an operation log for subsequent equipment maintenance or production status traceability.

[0052] In one implementation, determining the confidence analysis result based on the M consecutive image frames includes: Obtain the confidence scores corresponding to M image frames; calculate the average confidence score of the M image frames, and use the average score as the confidence analysis result.

[0053] Specifically, after identifying M consecutive image frames that meet pre-set confidence conditions within a sliding window consisting of N image frames, the confidence values ​​corresponding to these M image frames can be obtained sequentially over time. These values ​​are then summed and divided by the number of image frames, M, to obtain a confidence analysis result that comprehensively represents the perceived reliability within the current consecutive time period. This confidence analysis result no longer corresponds to the image detection state at a single moment, but rather reflects the overall reliability exhibited by multiple consecutive image frames that meet stability requirements.

[0054] Furthermore, only the M image frames that have met the aforementioned continuity requirement are included in the average calculation, rather than directly averaging all N image frames in the sliding window. This is because, in the initial stage of the window, there may be situations where the robotic arm has not yet fully entered the effective observation position, the target is still occluded, or the upstream equipment is still performing preparatory actions; in such cases, the aforementioned image frames cannot accurately represent the subsequent stable state.

[0055] S104. Several motion modes and corresponding motion mapping data relationships are preset. The target motion mode corresponding to the confidence analysis result is determined based on the motion mapping data relationship. The robotic arm is controlled to perform the corresponding motion according to the target motion mode. The motion mapping data relationship is used to indicate the correspondence between the confidence analysis result and the motion mode.

[0056] In this step, after obtaining the confidence analysis results, the robotic arm action is not divided into "execution" or "non-execution" states based solely on a single threshold. Instead, based on the reliability of the current perception results corresponding to different confidence ranges and the degree of possible positional deviations or state anomalies, a target action mode that matches the current situation is selected from multiple preset action modes.

[0057] Understandably, by making decisions in a hierarchical manner, the robotic arm can only add visual feedback or trajectory planning calculations when it is truly necessary, thus avoiding over-processing of minor deviations and also avoiding continuing to execute the original task path when the perception results are obviously abnormal.

[0058] In one implementation, the multiple preset action modes include at least a fixed trajectory execution mode, a trajectory fine-tuning mode, a local trajectory replanning mode, and an anomaly handling mode. The preset confidence range is divided into a first confidence interval, a second confidence interval, a third confidence interval, and a fourth confidence interval, from high to low confidence. The boundaries of different intervals can be configured based on the robot arm's task accuracy, robot arm type, target material size, perception model performance, and actual production requirements, without being limited to fixed values. The threshold values ​​for each interval can be saved as task configuration parameters and adjusted for different tasks.

[0059] For example, the first confidence interval is the interval where the confidence analysis result is greater than or equal to 0.85; the second confidence interval is the interval where the confidence analysis result is greater than or equal to 0.6 and less than 0.85; the third confidence interval is the interval where the confidence analysis result is greater than or equal to 0.3 and less than 0.6; and the fourth confidence interval is the interval where the confidence analysis result is less than 0.3 and greater than 0.

[0060] Since the confidence analysis result is in the first confidence interval, it indicates that the current perception result is highly consistent with the expected operation state. Therefore, the fixed trajectory execution mode can be determined as the target action mode. The robotic arm does not need to call an additional trajectory correction process and continues to run according to the original task path.

[0061] Since the confidence analysis result falls within the second confidence interval, it indicates that the current target state is generally usable, but the actual target position or posture may have a small deviation. Therefore, the trajectory fine-tuning mode is determined as the target motion mode. The robotic arm can use the change in current position obtained from visual perception to perform local dynamic compensation for subsequent motion trajectories, enabling the robotic arm end effector to gradually track the actual target position without regenerating the complete motion trajectory.

[0062] Since the confidence analysis result falls within the third confidence interval, it indicates that the current deviation is no longer suitable for elimination through local fine-tuning alone. Therefore, a local trajectory replanning mode is determined. As a specific implementation, the local motion path between the current position and the target position can be redefined within the robot arm's Cartesian workspace based on the current pose of the end effector and the new target pose obtained through visual perception. When the new local trajectory connects with the original task path, the trajectory can be smoothed through chord length parameterization and curvature constraints, making the changes in motion direction and curvature at the connection point between the old and new paths more gradual, thereby reducing robot arm jitter caused by trajectory switching.

[0063] If the confidence analysis result falls within the fourth confidence interval, it can be considered that the current sensing result is abnormal or the current task does not meet the conditions for continued normal execution, thus entering an anomaly handling mode. Anomaly handling may include pausing the robotic arm, outputting alarm information, or executing a recovery procedure at least one of the following:

[0064] In one implementation, the decision-making method for the robotic arm's movements also includes a phase switch after the current critical area is completed.

[0065] After completing the action corresponding to the current key area according to the target action mode, the control robot continues to run according to the task path; in response to the robot entering the next key area, multiple image frames corresponding to the next key area are acquired again and confidence analysis and action mode determination are performed until the task corresponding to the task path is completed.

[0066] Specifically, a complete robotic arm task can include multiple fixed execution segments and multiple key areas, which can be alternated according to the actual production process. For example, the robotic arm first moves along a fixed execution segment to the material grasping area, where it uses visual perception to confirm the material's position and readiness status. After grasping the material, it moves along a pre-planned fixed execution segment to the assembly area. Upon entering the assembly area, it triggers the perception model corresponding to the assembly task again to determine the position of the component to be assembled and its current assembly status. Thus, the robotic arm does not need to maintain high-frequency perception continuously in the fixed execution segments, but instead executes perception and motion decisions matching the current task stage in different key areas.

[0067] Furthermore, after completing the action in the current critical area, if the determined target action mode is a fixed trajectory execution mode or a trajectory fine-tuning mode, it can directly connect to the subsequent fixed execution segment; if local trajectory replanning is performed, it can reconnect to the subsequent preset position in the original task path after completing the local replanning trajectory; if it enters the anomaly handling mode and completes recovery, it can re-execute the perception confirmation steps corresponding to the current critical area. In all the above situations, the task control system can record information such as the current task stage, completed critical areas, action mode, and abnormal status, enabling the robotic arm to continuously execute according to predetermined logic between multiple task stages.

[0068] The decision-making method for robotic arm movements provided in this application enables the robotic arm to run continuously along a preset task path in a fixed execution segment, and to determine subsequent actions based on real-time perception results in key areas related to the connection with upstream and downstream processes. This reduces unnecessary perception processing while improving the robotic arm's adaptability to the cycle time of industrial production lines and the timing requirements of upstream and downstream processes.

[0069] This application also provides a decision-making system for robotic arm movements, such as... Figure 2 As shown, it includes: The task configuration module 201 is used to obtain the task path of the robotic arm and divide the task path into several alternating fixed execution segments and key areas. The perception module 202 is used to control the robotic arm to execute according to the task path. When the robotic arm enters the key area, it continuously acquires N image frames corresponding to the key area, performs perception processing on each image frame, and obtains the perception result corresponding to each image frame. The perception result includes target position information, target preparation status information, and target operation status information. The confidence analysis module 203 is used to determine the confidence level of each image frame based on the perception results corresponding to each image frame, and to determine the confidence analysis result based on the obtained confidence level. The decision module 204 is used to preset several action modes and corresponding action mapping data relationships, determine the target action mode corresponding to the confidence analysis results based on the action mapping data relationships, and control the robotic arm to perform the corresponding action according to the target action mode; the action mapping data relationship is used to indicate the correspondence between several confidence analysis results and action modes.

[0070] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the robotic arm task triggering method based on spatiotemporal constraint visual state determination provided in any of the above embodiments.

[0071] As the system implementation is basically similar to the method implementation, it is described in a relatively simple way. For relevant details, please refer to the description of the method implementation.

[0072] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0073] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] This application is described with reference to flowchart illustrations and / or block diagrams of the method, terminal device (system), and computer program product according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0075] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0076] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0077] It should be noted that: The phrase "an embodiment" or "an embodiment" used in this specification means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of this application. Therefore, the phrase "an embodiment" or "an embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment.

[0078] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0079] Furthermore, it should be noted that the shapes and names of the parts and components described in the specific embodiments described in this specification may differ. All equivalent or simple variations made to the structure, features, and principles described in this application are included within the scope of protection of this application. Those skilled in the art to which this application pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, as long as they do not deviate from the structure of this application or exceed the scope defined by the claims, all of which should fall within the scope of protection of this application.

Claims

1. A decision-making method for robotic arm movements, characterized in that, include: Obtain the task path of the robotic arm and divide the task path into several alternating fixed execution segments and key areas; The robotic arm is controlled to execute according to the task path. When the robotic arm enters the key area, N image frames corresponding to the key area are continuously acquired. The image frames are processed to obtain the perception results corresponding to each image frame. The perception results include target position information, target preparation status information and target operation status information. Based on the perception results corresponding to each of the image frames, the confidence level corresponding to each of the image frames is determined, and the confidence level analysis result is determined based on the obtained confidence level. Several motion patterns and corresponding motion mapping data relationships are preset. Based on the motion mapping data relationships, a target motion pattern corresponding to the confidence analysis result is determined. The robotic arm is then controlled to perform the corresponding motion according to the target motion pattern. The motion mapping data relationships are used to indicate the correspondence between the confidence analysis result and the motion pattern.

2. The decision-making method for robotic arm movements according to claim 1, characterized in that, The process of dividing the task path into fixed execution segments and key regions includes: The path segments that do not require adjustment of the robotic arm's movements based on the perception results are defined as the fixed execution segments; The path segments for determining the robotic arm's movements based on the perception results are defined as the critical regions.

3. The decision-making method for robotic arm movements according to claim 1, characterized in that, The step of determining the confidence level corresponding to each image frame based on the perception results corresponding to each image frame includes: The confidence level of the corresponding image frame is determined based on the target detection score, the positional deviation between the target detection location and the expected location, and the target state judgment score.

4. The decision-making method for robotic arm movements according to claim 3, characterized in that, The confidence level satisfies: ， Where C is the confidence level calculation result, S det The target detection score is denoted by ΔP, where ΔP is the deviation between the detected target location and the desired location, T is the maximum tolerance deviation, and S is the target detection score. state α is the target state judgment score; β is the target detection score weight, β is the position deviation weight, and γ is the target state judgment score weight, and α+β+γ=1.

5. The decision-making method for robotic arm movements according to claim 1, characterized in that, The various action modes include fixed trajectory execution mode, trajectory fine-tuning mode, local trajectory replanning mode, and exception handling mode.

6. The decision-making method for robotic arm movements according to claim 1, characterized in that, The process of determining the confidence analysis result based on the obtained confidence level includes: In a sliding window consisting of N consecutively acquired image frames, determine whether there exist M consecutive image frames whose confidence scores are all greater than a preset confidence threshold, where M≤N; The confidence analysis results are determined based on the M consecutive image frames.

7. The decision-making method for robotic arm movements according to claim 6, characterized in that, The step of determining the confidence analysis result based on the M consecutive image frames includes: Obtain the confidence scores corresponding to the M image frames respectively; Calculate the average confidence level corresponding to each of the M image frames, and determine the average confidence level as the confidence level analysis result.

8. A decision-making system for robotic arm movements, characterized in that, include: The task configuration module is used to obtain the task path of the robotic arm and divide the task path into several alternating fixed execution segments and key areas. The perception module is used to control the robotic arm to execute according to the task path. When the robotic arm enters the key area, it continuously acquires N image frames corresponding to the key area, performs perception processing on each image frame, and obtains the perception result corresponding to each image frame. The perception result includes target position information, target preparation status information, and target operation status information. The confidence analysis module is used to determine the confidence level of each image frame based on the perception results corresponding to each image frame, and to determine the confidence analysis result based on the obtained confidence level. The decision module is used to preset several action modes and corresponding action mapping data relationships, determine the target action mode corresponding to the confidence analysis results based on the action mapping data relationships, and control the robotic arm to perform the corresponding action according to the target action mode; the action mapping data relationships are used to indicate the correspondence between several confidence analysis results and action modes.

Images